Machine Learning ML-01.07
Statistics: sampling, CLT and confidence intervals
Understand why sample averages vary, why they form a bell curve, and how to put an honest error bar on any metric with a confidence interval or the bootstrap.
The big idea
Your new model scores 87.3 percent accuracy on a test set of 500 examples. The old one scored 86.1 percent. Is the new one better? Maybe. If you had drawn a different 500 test examples, both numbers would have come out a bit different. Statistics is the toolkit for answering "how different?", so you stop shipping changes that were really just luck.
The central insight is beautifully simple: averages are much more stable than individual values, they become more stable in a precise and predictable way as the sample grows, and, whatever the original data looks like, the average of a large sample follows a bell curve. That last fact, the central limit theorem, is why error bars work almost everywhere.
Picture estimating the average height in a city by measuring people at a bus stop. One person tells you very little. Ten people give a rough idea. A hundred give a good estimate, and if you repeated the experiment at many bus stops, your averages would cluster tightly around the truth.
How it works
Step 1: Population and sample
The populationEvery item you care about, such as all customers or all future emails. Usually too big or impossible to measure completely. is everything you care about: all your users, all future emails. You almost never see it all. You see a sampleThe subset of the population you actually measure. Its statistics estimate the population's., and compute statistics like the sample mean to estimate the population's true mean . A test set is a sample, and test accuracy is an estimate.
Step 2: The sampling distribution
Imagine drawing many samples of size and computing the mean of each. Those means form their own distribution, the sampling distributionThe distribution of a statistic, such as the mean, across many hypothetical samples of the same size.. Its spread has a name, the standard errorThe standard deviation of a statistic's sampling distribution: how much the estimate would wobble across repeated samples. For a mean it is sigma over the square root of n.:
The square root matters: to halve your uncertainty you need four times as much data.
Step 3: The central limit theorem
The central limit theoremFor large enough samples, the mean of independent draws is approximately normally distributed, whatever the shape of the original distribution. (CLT) says: for reasonably large , the sampling distribution of the mean is approximately normal, with mean and standard deviation , even when the population itself is skewed or lumpy.
import numpy as np
rng = np.random.default_rng(1)
population = rng.exponential(scale=2.0, size=1_000_000) # skewed, mean 2, sd 2
for n in (1, 5, 30):
means = rng.choice(population, size=(10_000, n)).mean(axis=1)
print(f"n={n:2d}: mean of sample means {means.mean():.3f}, spread {means.std():.3f}, theory 2/sqrt(n) = {2 / np.sqrt(n):.3f}")
# A 95% confidence interval from ONE sample of 40, two ways
sample = rng.choice(population, size=40)
se = sample.std(ddof=1) / np.sqrt(len(sample))
print(f"sample mean {sample.mean():.2f}, normal-approx 95% CI [{sample.mean() - 1.96 * se:.2f}, {sample.mean() + 1.96 * se:.2f}]")
boot = rng.choice(sample, size=(5000, len(sample))).mean(axis=1)
print(f"bootstrap 95% CI [{np.percentile(boot, 2.5):.2f}, {np.percentile(boot, 97.5):.2f}]")Output
n= 1: mean of sample means 2.020, spread 2.023, theory 2/sqrt(n) = 2.000
n= 5: mean of sample means 1.989, spread 0.883, theory 2/sqrt(n) = 0.894
n=30: mean of sample means 1.996, spread 0.365, theory 2/sqrt(n) = 0.365
sample mean 1.84, normal-approx 95% CI [1.37, 2.31]
bootstrap 95% CI [1.41, 2.30] The spread of the sample means matches almost exactly at every size, and by their histogram is a clean bell even though the population is anything but.
Step 4: Confidence intervals
Because the mean of a large sample is roughly normal, about 95 percent of sample means land within 1.96 standard errors of the truth. Flip that around and you get a 95 percent confidence intervalA range computed from a sample by a procedure that captures the true value in a stated share of repeated samples, such as 95 percent.:
where is the sample's standard deviation. The precise meaning is about the procedure: if you repeated the sampling many times, 95 percent of the intervals built this way would contain the true value. For model accuracy on 500 test examples at 87 percent, the standard error is , so the interval is roughly 84 to 90 percent. That makes the 86.1 versus 87.3 comparison look much less exciting.
Step 5: The bootstrap, when formulas get hard
For a median, an F1 score or any complicated metric, there may be no tidy formula. The bootstrapEstimating uncertainty by resampling your own data with replacement many times and recomputing the statistic each time. fakes repeated sampling: draw a new sample of the same size from your data, with replacement, compute the statistic, and repeat a few thousand times. The middle 95 percent of those values is a bootstrap confidence interval. In the code above it agrees closely with the formula, and it works just as well for metrics with no formula at all.
Try it
Try this:
- With the waiting-times population and , draw 500 samples. The histogram is just the skewed population.
- Set , then , drawing 500 each time. Watch the bell emerge and get narrower, matching the gold curve.
- Try "two humps". Even a population with no middle at all produces sample means that pile up in the middle.
Going deeper
- When the CLT is slow: with very skewed or heavy-tailed data (incomes, latencies), you need larger samples before the normal approximation is trustworthy. The bootstrap or a log transform can help.
- Independence matters: the formulas assume independent draws. Ten clicks from the same user are not ten independent observations; use one value per user or a clustered method.
- Error bars on model comparisons: compare two models on the same test set with paired methods, such as bootstrapping the difference per example, which is tighter than comparing two separate intervals.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Normal-approximation interval | Means and proportions with decent sample sizes; quick to compute | Small samples, heavy tails, or complex metrics |
| Bootstrap interval | Any statistic, including medians, F1 and AUC | Very small samples, or when compute per resample is expensive |
| Report the point estimate only | Never, for decisions | Always: it hides whether a difference is real |
Common mistakes
- Saying "there is a 95 percent chance the true value is in this interval". The 95 percent describes the procedure across repeated samples.
- Forgetting the square root: doubling the test set only shrinks the error bar by about 29 percent.
- Treating correlated observations (same user, same session) as independent, which makes intervals far too narrow.
- Declaring a winner from two point estimates whose intervals overlap heavily.
In the wild
Experimentation platforms at large tech companies show confidence intervals next to every metric in an A/B test, precisely so teams do not ship noise. ML papers increasingly report results over several random seeds with error bars, because a single run can mislead. Evaluating LLM applications on small eval sets of a few hundred examples is another place where the square-root law bites: small sets give wide intervals (see Building an eval set ).
Check yourself
-
A sample of 100 has standard deviation 20. What is the standard error of the mean?
Show answer
B. 2 SE = s / sqrt(n) = 20 / 10 = 2.
-
You want to cut the width of a confidence interval in half. How much more data do you need?
Show answer
B. Four times as much Width scales with 1 / sqrt(n). Halving it means sqrt(n) must double, so n must quadruple.
-
What does the central limit theorem say?
Show answer
B. Means of large samples are approximately normal, whatever the population shape The CLT is about the distribution of sample means, not the raw data. The data can be skewed; the averages still form a bell.
-
You need a confidence interval for your model's F1 score on a test set. What is the most practical approach?
Show answer
B. Bootstrap: resample the test set many times and recompute F1 F1 has no simple standard error formula; resampling test examples with replacement and recomputing F1 gives an honest interval.
Recap
- A sample estimates a population, and every estimate wobbles from sample to sample.
- The standard error of a mean is sigma over sqrt(n): four times the data halves the uncertainty.
- The central limit theorem makes sample means approximately normal, whatever the population looks like.
- A 95 percent confidence interval is roughly the estimate plus or minus 1.96 standard errors.
- The bootstrap resamples your own data to get intervals for any metric.
Next up
- From notebook to production learning path