What a Confidence Interval Actually Is

In October 2024, Gallup interviewed 1,023 U.S. adults and reported Kamala Harris's job approval at 45%, with a sampling error of ±4 percentage points at the 95% confidence level. The next day's coverage turned that into a sentence everyone understands: there is a 95% chance her true approval lies between 41% and 49%.

I used to say that sentence too, and it rolls off the tongue. It has one flaw: it is false. The error is not a small slip; it reads the whole meaning of a confidence interval in reverse. Once the 41%-to-49% band is computed, it is fixed. It either contains the true value or it does not, and no "95% probability" attaches to it. The 95% is not a statement about this interval at all.

I am not here to scold anyone. "I just want to know the probability that the true value is in this range" is a completely reasonable wish — it is, in fact, the question everyone silently asks when they see a poll. The trouble is that the tool in your hand does not compute that. To see where the gap opens, build the interval by hand.

Build the interval first

Suppose you are estimating a mean. The population mean is $\mu$, unknown; you have $n$ observations, with sample mean $\bar X$ and sample standard deviation $s$. One quantity ties the unknown $\mu$ to the computable $\bar X$:

$$T = \frac{\bar X - \mu}{s / \sqrt{n}}$$

This $T$ has a useful property. Whatever $\mu$ actually equals, as long as the data are roughly normal, $T$ follows a $t$ distribution with $n-1$ degrees of freedom. Its shape does not depend on the unknown $\mu$. A quantity that carries an unknown parameter inside yet has a fixed distribution is called a pivot, and it is what the whole construction turns on.

Because the distribution of $T$ is fixed, I can write one probability statement on it that holds firmly. Let $t^* = t_{0.975,\,n-1}$, the critical value that brackets the middle 95% of the $t$ distribution:

$$P\!\left( -t^* \le \frac{\bar X - \mu}{s / \sqrt{n}} \le t^* \right) = 0.95$$

Nothing has gone wrong yet. Now a step of pure algebra: unwrap the inequality so that $\mu$ stands alone in the middle. Multiply through by $s/\sqrt n$, rearrange, flip signs, and

$$P\!\left( \bar X - t^* \frac{s}{\sqrt n} \;\le\; \mu \;\le\; \bar X + t^* \frac{s}{\sqrt n} \right) = 0.95$$

That $\bar X \pm t^*\, s/\sqrt n$ is the confidence interval. The poll's ±4-point band is the proportion version of the same object: score each "approve" as a 1 and each "disapprove" as a 0, and 45% is the mean of those 0s and 1s, with standard error $\sqrt{p(1-p)/n}$. Plug in $p=0.45$ and $n=1023$, and $1.96$ standard errors come to about 3 points; Gallup reports ±4 because the actual margin of error also absorbs weighting and the survey design, running a little above the textbook value. The skeleton is identical.

Now look at the subject of that probability statement. When $P(\dots) = 0.95$ is written down, $\mu$ is a fixed constant; the thing that varies is $\bar X$, still unobserved, still a random variable. The 95% says this: before sampling, while $\bar X$ still bounces around, the event "$\bar X$ lands close enough that the computed interval happens to cover $\mu$" has probability 95%. The probability rides on the moving $\bar X$.

Then you draw the sample, $\bar X = 45\%$, $s$ is fixed, and the interval comes out at 41% to 49%. That step swaps $\bar X$ from a random variable to a number in hand. $\bar X$ no longer moves, $\mu$ never moved, and the two endpoints are now nailed down. The 95% described "how often the interval covers while $\bar X$ bounces"; nothing bounces anymore, so it has nowhere left to stand. The segment 41% to 49% either contains $\mu$ or it does not — a fact that is zero or one, not a probability. The misreading is born at the plug-in step: a sentence about the long-run behaviour of a procedure gets read as a sentence about the one interval on the table.

That 95% is a coverage rate

Draw the sentence and it becomes plain. The population mean $\mu$ is a fixed line; each sample gives one interval. Draw a sample, draw a line, stack it:

Stack enough of them and about 95% cover the line while about 5% miss. The 95% is a property of the whole stack — the long-run fraction that this procedure covers — which is the coverage rate. It describes the machine that manufactures intervals, not any single interval the machine spits out. Your 41%-to-49% is one line in the stack; it carries no probability of its own. What carries the 95% is the procedure.

While we are here, clear away a cruder misreading: this interval is not the spread of the data. It is the uncertainty of the mean, and it narrows as $n$ grows; the spread of the data itself is $s$, which does not shrink because you collected more observations. Reading a confidence interval as "most individuals fall in here" gets the very object wrong.

Slide the level to 90% or 99% and watch the interval tighten and widen. Slide $n$. However you slide them, the percentage always names "the fraction of this stack that covers", never "how likely the line you picked is the right one".

The man who built it vetoed that reading on the spot

This is not hindsight. In the 1937 paper that founded confidence intervals, Jerzy Neyman blocked the tempting reading on the same page he proposed the construction. He first states that applying the rules permanently yields statements that are right in about $100a\%$ of cases. Then he asks and answers his own question: can we say that for this particular computed interval the probability of the true value falling inside equals the confidence coefficient? The answer is obviously in the negative. The parameter is an unknown constant, and no probability statement about its value may be made.

From the moment it was born, the confidence interval was never built to answer "the probability that this segment covers the truth". It answers a question about a procedure, and Neyman could not have been clearer about that.

Cracks inside the frequentist framework

Maybe you now want to say: fine, record the 95% as "the long-run fraction that covers", never pin it on a single interval, and the problem is gone. Even under that discipline, the frequentist framework has two cracks of its own, visible without ever calling on Bayes.

The first comes from an innocent-looking quantity: the ratio of two means. Such estimates are everywhere — a half-maximal dose, a ratio of two costs, a regression slope. Its 95% confidence set is given by Fieller's method, and what that lawful procedure returns can be an interval, the complement of an interval, or the entire real line; and to keep 95% coverage it must be infinite with positive probability.

Slide the denominator's signal-to-noise ratio down — the denominator mean shrinks relative to its error and grows more likely to cross zero. The confidence set is a finite interval, then it hollows out into the complement of an interval with two infinite tails, then it swells into the entire real line. The entire real line means "every value is compatible", carrying zero information. Nothing was miscalculated; this is the price the well-behaved procedure must pay, when the denominator might be zero, to keep 95% coverage. A lawful 95% procedure hands you an answer that says nothing with positive probability.

The second crack cuts harder, straight at "how much this interval can be trusted". Morey and colleagues told it in 2016 with a submarine. To be upfront, these authors are Bayesians whose paper concludes that for inference you should stop using confidence intervals altogether; I borrow their demonstration, not that verdict.

The story: a 10-metre submarine sits on the seabed, its rescue hatch at the exact centre of the hull, position unknown. The hatch releases bubbles, which rise uniformly along the hull within 5 metres on either side of it. You get one dive to attempt a rescue. The first two bubbles surface at $x_1$ and $x_2$. A frequentist builds a 50% confidence interval from them: the segment between the two bubbles, $\bar x \pm |x_1 - x_2|/2$. It is a genuine 50% procedure — the two bubbles straddle the hatch with probability exactly one half.

Drag the two bubbles. When they sit close together the interval is narrow, and the statistician calls it precise — yet two bubbles that near each other say almost nothing about where the hatch is, so the narrowness is unearned. Now pull them more than 5 metres apart: since a bubble can land at most 5 metres from the hatch, two bubbles more than 5 metres apart must straddle it, and the 50% interval is then guaranteed to contain the hatch. One lawful 50% procedure, and the segment in hand can be certainly right or almost certainly wrong, depending entirely on the gap between the bubbles.

Morey sorts the usual misreadings into three named fallacies. The precision fallacy: taking a narrow interval to mean a precise estimate — the narrowest interval in the submarine is the least informative. The likelihood fallacy: taking values inside the interval to be more probable than values outside — when the bubbles are nearly 10 metres apart, the interval is almost all impossible positions. The fundamental confidence fallacy: reading "the random procedure covers 50% in the long run" as "this observed interval has a 50% probability of containing the truth" — the very error the paper makes about the poll.

Behind all three is one thing, which Fisher named back in 1959: relevant subsets. Your observed data often fall in a subset where the interval's probability of covering the truth differs from the overall average, and the gap between the bubbles marks exactly such a subset. Look again at the coverage figure: sort the two thousand intervals by length, and the narrowest thousand cover only 92% while the widest thousand cover 98%. All of them are "95% intervals", yet their coverage sits below and above 95%. Which half your interval resembles is something its length already gave away.

Even the experts fail it

If by now this feels like a collection of contrived corner cases that daily research never meets, a 2014 survey by Hoekstra and colleagues is uncomfortable reading. They gave respondents a fictional vignette: a professor reports a 95% confidence interval of 0.1 to 0.4 for a mean, followed by six statements about that interval to judge true or false.

All six statements are false. And the 442 undergraduates and 120 researchers (PhD students and faculty) both endorsed three or more of them on average; statistical training had no bearing on the score, and the researchers barely beat first-years who had never taken an inference course. This was not a fluke — as early as 1986, Oakes put six misreadings of a significant p value to 70 academic psychologists, who endorsed about 2.5 on average, with only around one in ten fully correct.

Try it. All six statements are below; check your answers afterward:

The six err in the same place. A confidence interval does not directly indicate a property of the parameter; it indicates a property of the procedure, as is typical for a frequentist technique. Every misreading tries to squeeze, out of a procedural statement, one sentence about this parameter, this time. It cannot be squeezed out.

Said right, said wrong

Gathering the article's distinctions into one table: for the same confidence interval, which phrasings hold up and which are the misreadings you meet most often. Each misreading carries the fix that replaces it.

Name the wish

The circle closes back on the reasonable wish we started with: I just want the probability that the true value is in this range. That question has an answer; it simply belongs to another framework. What you want is a credible interval: given a model and a prior, the parameter has a 95% posterior probability of lying in this segment — this time, this segment, genuinely a probability. It looks like a confidence interval and answers precisely the question you meant to ask. The poll's band, redrawn with a prior under Bayesian arithmetic, would become the thing the newspaper already assumed it was. (How the prior enters, and how the credible interval moves with it, is left to the sequel.)

The same data, two intervals side by side. Numerically they nearly coincide, but they read as different things: the left says the procedure that made it covers the truth 95% of the time in the long run; the right actually says the parameter has a 95% posterior probability of being inside.

How to close? Not by announcing "so you should become a Bayesian". The frequentist tool is not broken — it answers a well-defined question exactly: what is this procedure's long-run coverage rate. What is broken is treating two frameworks' questions as one, using a frequentist interval to answer a Bayesian's question.

Some have lately wanted to change even the tool's name. In 2019, a Nature comment co-signed by more than 800 scientists stated plainly that one should never conclude "no difference" merely because a p value exceeds a threshold or a confidence interval includes zero — the grown-up form, in the scientific literature, of the poll's misreading. Within the same group, Greenland goes further and proposes to rename it the compatibility interval: the values inside are those more compatible with the data, under this model and its assumptions, than the values outside. The name shifts attention from whether a boundary is crossed to how compatible each value across the whole interval is.

Back to Gallup's 45%, ±4 points. The band is honest; it marks how much sampling wobble sits under the estimate. What gets broken is the sentence right after it, the one that sounds self-evident: "a 95% chance it is inside". The gap is not a matter of a little precision. It is a sentence about a procedure, mistaken for a sentence about this one time. The tool answered correctly. It just never answered the question you thought you asked.

Sources

1. 1.Gallup (2024). At 45%, Harris' Approval Rating Is Higher Than Biden's Gallup News. Telephone interviews conducted October 1-12, 2024 with a random sample of 1,023 U.S. adults; Kamala Harris job approval at 45%. The methods note states a sampling error of ±4 percentage points at the 95% confidence level. The 41%-to-49% band quoted at the top of this article is one 95% confidence interval.Read it
2. 2.Pew Research Center (2016). Understanding the margin of error in election polls Pew Research Center, Decoded / Short Reads. Uses a worked example of a candidate at 48% with a ±3 percentage point margin of error (95% confidence) to explain that the margin of sampling error describes how close a result falls to the true population value, and warns that one candidate leading by more than the individual-candidate margin is not enough to call the lead statistically real.Read it
3. 3.Jerzy Neyman (1937). Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability Philosophical Transactions of the Royal Society of London, Series A, vol. 231, pp.333-380. p.349 (the frequency reading of a confidence interval: in the long run the statements will be right in 100a per cent of cases; Neyman then asks whether one may say of a single computed interval that the parameter lies in it with probability a, and answers that one may not).Read it
4. 4.George Casella and Roger L. Berger (2002). Statistical Inference (2nd edition) Duxbury / Thomson Learning, Section 9.5.3 (Miscellanea). The 1-alpha confidence set for the ratio of two normal means (Fieller) depends on the signal-to-noise ratio in the denominator and can be an interval, the complement of an interval, or the entire real line; and to keep 1-alpha coverage it must be infinite with positive probability. This is a pathology internal to the frequentist framework, pointed out without any appeal to Bayes: a lawful confidence procedure that with positive probability returns the whole real line, an answer carrying no information.
5. 5.Richard D. Morey, Rink Hoekstra, Jeffrey N. Rouder, Michael D. Lee and Eric-Jan Wagenmakers (2016). The fallacy of placing confidence in confidence intervals Psychonomic Bulletin & Review, 23(1), pp.103-123. The sunken-submarine thought experiment (the rescue hatch sits at an unknown position, with bubbles surfacing uniformly within 5 metres on either side of it) is used to dismantle three named fallacies: the precision fallacy (a narrow interval means a precise estimate), the likelihood fallacy (values inside the interval are more probable), and the fundamental confidence fallacy (reading the X% long-run coverage of the procedure as an X% probability for the single observed interval). When the two bubbles are more than 5 metres apart, the 50% procedure's interval is certain to contain the hatch; when they are very close, an interval of the same width almost never does. One lawful procedure, and the interval in hand can be certainly right or almost certainly wrong (relevant subsets, Fisher 1959). Figure 1 draws 200 random 50% intervals from N=2 samples; 107 (53.5%) contain the true value, and sorting them by length shows the short ones almost never cover while the long ones almost always do.Read it
6. 6.Rink Hoekstra, Richard D. Morey, Jeffrey N. Rouder and Eric-Jan Wagenmakers (2014). Robust misinterpretation of confidence intervals Psychonomic Bulletin & Review, 21(5), pp.1157-1164. Respondents read one fictional vignette: a professor reports a 95% confidence interval of 0.1 to 0.4 for a mean, followed by six statements about that interval to be judged true or false. All six are false. The sample was 442 first-year students with no training in inferential statistics, 34 master's students, and 120 researchers (PhD students and faculty). Every group endorsed three or more of the false statements on average; statistical training was unrelated to performance, and the researchers barely outperformed the students.Read it ↑1↑2
7. 7.Michael Oakes (1986). Statistical Inference: A Commentary for the Social and Behavioural Sciences Wiley, Chichester. An earlier result of the same kind: 70 academic psychologists judged six statements about a significant p value, endorsing about 2.5 on average, with only around 11% fully correct. It is the direct forerunner of the confidence-interval quiz in Hoekstra et al. 2014.Read it
8. 8.Valentin Amrhein, Sander Greenland and Blake McShane (2019). Scientists rise up against statistical significance (Retire statistical significance) Nature, 567, pp.305-307. A comment led by three authors and co-signed by more than 800 scientists. It argues that significance is misused dichotomously and states plainly that one should never conclude 'no difference' merely because a p value exceeds a threshold or a confidence interval includes zero. This is the grown-up version, in the scientific literature, of the opening poll's misreading. The authors state they are not calling for a ban on p values.Read it
9. 9.Zad Rafi and Sander Greenland (2020). Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise arXiv:1909.08579 [stat.ME], v7 (1 Oct 2020); also in BMC Medical Research Methodology, 20:244. Renames the confidence interval as a 'compatibility interval': the parameter values inside are those more compatible with the data, under the background assumptions, than the values outside; the abbreviation stays 'CI'. The move shifts attention from whether a boundary is crossed to how compatible each value across the whole interval is, which is where this article's closing lands.Read it