Machine Learning ML-01.08
Hypothesis testing and p-values
Decide whether an observed difference is real or just chance, read a p-value correctly, and avoid the multiple-testing trap.
The big idea
You redesign the checkout page and the average time to pay drops from 10.1 to 9.0 seconds in a test with 50 users each. Did the redesign help, or did you just get a faster group of users this time? A hypothesis test is a disciplined way to answer that question: assume nothing changed, then ask how surprising your data would be in that boring world. If it would be very surprising, you have evidence that something did change.
It works like a courtroom. The defendant (your new feature) is presumed to have no effect. The data is the evidence. You only convict, declaring a real effect, if the evidence would be very unlikely under innocence. And just like a court, the procedure can make two kinds of mistake: convicting an innocent feature, or acquitting a guilty one.
How it works
Step 1: Two hypotheses
- The null hypothesisThe default claim that there is no effect or no difference, which a test tries to find evidence against. : there is no real difference. The new page is as fast as the old.
- The alternative hypothesisThe claim that there is a real effect, accepted only if the data is surprising enough under the null. : there is a difference.
You never prove . You either find enough evidence against it, or you do not.
Step 2: The p-value
Compute a test statistic that measures how far your result is from "no difference", in units of standard error. Then ask: if were true, how often would a result at least this extreme happen by chance? That probability is the p-valueThe probability, assuming the null hypothesis is true, of seeing data at least as extreme as what was observed..
A small p-value means your data would be unusual in a world with no effect. Before looking at the data you pick a threshold, the significance levelThe false positive rate you accept in advance, often 0.05: reject the null when p is below it. (commonly 0.05), and reject when .
What a p-value is not:
- It is not the probability that the null hypothesis is true.
- It is not the probability your result is a fluke in the everyday sense.
- It says nothing about how big or important the effect is. A tiny, useless effect can have a minuscule p-value with enough data.
Step 3: Two kinds of error
- A type I errorRejecting the null when it is actually true: a false positive. Its rate is alpha. is a false positive: declaring an effect that is not there. Its rate is .
- A type II errorFailing to reject the null when there really is an effect: a missed effect. Its rate is beta. is a miss: failing to detect a real effect. Its rate is .
- PowerThe probability a test detects a real effect of a given size: 1 minus beta. is the chance you detect an effect of a given size. Power grows with sample size and effect size. Tests with low power mostly find nothing, and the "significant" results they do find tend to exaggerate the effect.
Step 4: Common tests
- A t-test compares means of two groups (time to pay, old versus new page).
- A chi-square test compares counts in categories (how many converted versus not, per page).
- Both are available in SciPy:
import numpy as np
from scipy import stats
rng = np.random.default_rng(3)
a = rng.normal(loc=10.0, scale=2.0, size=50) # old checkout page: seconds to pay
b = rng.normal(loc=9.0, scale=2.0, size=50) # new page, truly 1 second faster
t, p = stats.ttest_ind(a, b)
print(f"t-test: t = {t:.2f}, p = {p:.4f}")
# Chi-square: did the new page change how many people convert?
table = np.array([[120, 880], # old: converted, did not
[150, 850]]) # new
chi2, p2, dof, _ = stats.chi2_contingency(table)
print(f"chi-square: chi2 = {chi2:.2f}, p = {p2:.4f}")
# Multiple testing: 20 tests where nothing is going on
false_alarms = 0
for i in range(20):
x, y = rng.normal(size=30), rng.normal(size=30)
false_alarms += stats.ttest_ind(x, y).pvalue < 0.05
print("tests with p < 0.05 when no effect exists:", false_alarms, "of 20")
print("Bonferroni threshold for 20 tests:", 0.05 / 20)Output
t-test: t = 3.14, p = 0.0023
chi-square: chi2 = 3.60, p = 0.0577
tests with p < 0.05 when no effect exists: 3 of 20
Bonferroni threshold for 20 tests: 0.0025 The t-test finds the 1-second difference with . The conversion change, 12 percent to 15 percent, gives : just above 0.05, so not significant at that threshold, even though it might be a real improvement. More data would tell.
Step 5: The multiple-testing trap
Run 20 tests where nothing is going on, each at , and you expect about one false positive. The simulation above found 3 out of 20 by pure chance. Testing many metrics, many segments or many model variants and reporting the winners guarantees false discoveries. Fixes: decide the main metric in advance, or correct the threshold. The Bonferroni correctionDivide the significance level by the number of tests, so the chance of any false positive across all of them stays near alpha. divides by the number of tests: 0.05 / 20 = 0.0025.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Strict alpha (0.01 or Bonferroni) | False positives are costly, such as shipping a risky change or a medical claim | Real but small effects get missed unless you collect much more data |
| Standard alpha 0.05 | A reasonable default for single, pre-registered comparisons | Many tests are run and only the winners are reported |
| Confidence interval on the effect | You need to know how big the effect is, not just whether it exists | Rarely; report it alongside the p-value |
Common mistakes
- Reading as "a 3 percent chance the null is true".
- Treating as "no effect". It means "not enough evidence at this threshold".
- Peeking at an A/B test every day and stopping as soon as , which inflates false positives (see Offline vs online evaluation and A/B tests ).
- Ignoring effect size: with millions of users a 0.01 percent change can be "significant" and worthless.
- Testing dozens of metrics and reporting only the significant one.
In the wild
A/B testing platforms at large product companies run hypothesis tests on every experiment and guard against peeking with sequential testing methods or fixed durations. Model comparisons in research increasingly use significance tests or confidence intervals across random seeds. The replication crisis in some sciences was driven partly by multiple testing and underpowered studies, the same traps that catch ML teams comparing many model variants on one small test set.
Check yourself
-
A test gives p = 0.01. What does that mean?
Show answer
B. If there were no real effect, data this extreme would happen about 1 percent of the time A p-value is computed assuming the null is true. It measures how surprising the data is in that world, not the probability of either hypothesis.
-
You test 50 product metrics at alpha = 0.05 and none of them truly changed. About how many will look significant?
Show answer
B. About 2 or 3 Each has a 5 percent false positive rate: 50 x 0.05 = 2.5 expected false alarms. That is the multiple-testing trap.
-
Which change increases a test's power to detect a real effect?
Show answer
B. Collecting a larger sample More data shrinks the standard error so the null and alternative distributions overlap less. A stricter alpha actually reduces power.
-
A conversion test gives p = 0.07. What is the most honest conclusion?
Show answer
C. The evidence is not strong enough at alpha 0.05; the confidence interval shows how big the effect could be Not significant is not the same as no effect. Report the interval and consider collecting more data, without moving the threshold after the fact.
Recap
- A hypothesis test assumes no effect and asks how surprising the data would be under that assumption.
- The p-value is that surprise level; it is not the probability that the null is true.
- Type I errors are false positives (rate alpha); type II errors are misses (rate beta); power is 1 minus beta.
- Use t-tests for means and chi-square tests for counts, and always report effect size with an interval.
- Many tests produce false positives by chance: pre-register the metric or correct the threshold.
Next up
- From notebook to production learning path