Machine Learning ML-04.06
Probability calibration
When a model says 80 percent, is it right 80 percent of the time? Read reliability diagrams and fix scores with Platt scaling or isotonic regression.
The big idea
A weather forecaster says "70 percent chance of rain" on 100 different days. If it rains on about 70 of them, the forecaster is calibratedA model is calibrated when, among all cases where it predicts probability p, the outcome happens about p of the time.: the numbers mean what they say. If it rains on 95 of them, the forecaster was too timid; on 40, too bold. Notice that the forecaster could still be excellent at telling rainy days from dry days, with probabilities that are simply on the wrong scale.
The same split exists in classifiers. Ranking quality (does the model put positives above negatives?) is measured by AUC. Calibration (do the numbers match reality?) is a separate property, and many strong models fail it. It matters whenever a probability is used as a number: pricing insurance from a claim probability, computing expected revenue, setting a threshold from costs, or combining a model's output with other probabilities.
How it works
Step 1: The reliability diagram
To check calibration, group test predictions into bins by predicted probability (0 to 0.2, 0.2 to 0.4, and so on). In each bin, compare the average prediction with the share of cases that were actually positive. Plot one point per bin: a reliability diagramA plot of the observed positive rate against the mean predicted probability for bins of predictions. A calibrated model lies on the diagonal.. A perfectly calibrated model lies on the diagonal.
- Points below the diagonal on the right and above it on the left: the model is overconfidentPredicting probabilities that are more extreme (closer to 0 or 1) than the observed frequencies justify., its probabilities too close to 0 and 1.
- The opposite S-shape: underconfident, probabilities squeezed toward the middle.
- Raw naive Bayes: overconfident Predictions near 0.02 come true 28 percent of the time; predictions near 0.99 only 88 percent. The curve is much flatter than the diagonal.
- Platt scaling: better, not perfect A fitted sigmoid pulls the extremes in. The ends now match, but the middle bins still sit well below the diagonal: one S-shaped curve cannot undo this distortion.
- Isotonic regression: on the diagonal A flexible monotone mapping fitted on held-out data brings every bin close to the diagonal: 0.47 predicted, 0.49 observed.
Step 2: Why models are miscalibrated
- Naive Bayes multiplies correlated evidence as if it were independent, so its probabilities rush toward 0 and 1 ( Naive Bayes ).
- SVMs output distances from the boundary, not probabilities at all.
- Boosted trees and deep neural networks trained to drive the loss very low often become overconfident on new data.
- Random forests average many trees' votes and rarely reach 0 or 1, so they are often underconfident at the extremes.
- Resampling and class weights ( Imbalanced data ) deliberately shift probabilities toward the rare class.
Logistic regression trained on representative data is usually close to calibrated, because cross-entropy rewards honest probabilities directly.
Step 3: Fix it with a calibration map
Calibration fits a second, tiny model that maps the raw score s to a corrected probability. It must be fitted on data the main model did not train on, a held-out calibration set or cross-validation folds, otherwise it learns the model's overconfidence on its own training data.
Two standard maps:
- Platt scalingCalibrating scores by fitting a logistic regression with one input, the model's raw score, on held-out data. fits a one-feature logistic regression:
p = sigmoid(a x s + b). Just two parameters, so it needs little data and cannot overfit much, but it can only apply an S-shaped correction. - Isotonic regressionFitting a non-decreasing step function from raw scores to observed outcomes. Flexible, but needs more data than Platt scaling. fits any non-decreasing step function. It can fix any monotone distortion, but with few calibration examples it overfits into jagged steps. A rough rule: prefer it when you have more than about a thousand calibration examples.
Both maps are monotone, so they keep the order of predictions. AUC does not change (0.815 before, 0.815 and 0.816 after); only the numbers do.
Step 4: Measure calibration
- The Brier scoreThe mean squared difference between predicted probabilities and the 0/1 outcomes. Lower is better; it rewards both calibration and discrimination. is the mean squared error between probability and outcome. In the code, it falls from 0.212 (raw) to 0.173 (Platt) and 0.161 (isotonic).
- Log loss also rewards calibration, and punishes confident mistakes harder.
- Expected calibration error (ECE) averages the gap between the reliability curve and the diagonal, weighted by bin size. It is easy to explain but depends on the binning.
Going deeper
Code: diagnose and fix
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.metrics import brier_score_loss, roc_auc_score
X, y = make_classification(n_samples=20000, n_features=30, n_informative=4, n_redundant=24,
random_state=0)
# train / calibration / test
Xtr, Xrest, ytr, yrest = train_test_split(X, y, test_size=0.5, random_state=0)
Xcal, Xte, ycal, yte = train_test_split(Xrest, yrest, test_size=0.5, random_state=0)
def show(name, p):
frac, mean = calibration_curve(yte, p, n_bins=5)
print(f"{name:<22} Brier {brier_score_loss(yte, p):.3f} AUC {roc_auc_score(yte, p):.3f}")
print(" predicted:", " ".join(f"{m:.2f}" for m in mean))
print(" actual :", " ".join(f"{f:.2f}" for f in frac))
nb = GaussianNB().fit(Xtr, ytr)
show("naive Bayes (raw)", nb.predict_proba(Xte)[:, 1])
# Fit the calibrator on held-out data the model never trained on
from sklearn.frozen import FrozenEstimator
for method in ["sigmoid", "isotonic"]:
cal = CalibratedClassifierCV(FrozenEstimator(nb), method=method).fit(Xcal, ycal)
show(f"NB + {'Platt' if method == 'sigmoid' else 'isotonic'}", cal.predict_proba(Xte)[:, 1])Output
naive Bayes (raw) Brier 0.212 AUC 0.815
predicted: 0.02 0.28 0.50 0.70 0.99
actual : 0.28 0.38 0.41 0.52 0.88
NB + Platt Brier 0.173 AUC 0.815
predicted: 0.26 0.49 0.71 0.85
actual : 0.28 0.40 0.51 0.89
NB + isotonic Brier 0.161 AUC 0.816
predicted: 0.12 0.28 0.47 0.68 0.98
actual : 0.11 0.29 0.49 0.74 0.97 The data is split three ways: training for the model, a calibration set for the map, and a test set for the verdict. FrozenEstimator tells scikit-learn not to refit the model. The Platt rows have only four bins because no calibrated prediction fell in the lowest one: the sigmoid squeezed everything above 0.2.
Calibration is not fairness, and not stability
A model can be calibrated overall and miscalibrated for a subgroup ( Fairness and bias ), so check reliability per segment that matters. And calibration drifts: if the base rate changes after launch, a calibrated model's probabilities become wrong even though its ranking is fine. Monitor the observed rate against the mean prediction over time ( Model serving patterns ).
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| No calibration | You only use the ranking or a threshold tuned on validation data | Probabilities feed prices, expected values or other models |
| Platt scaling | Small calibration sets; S-shaped distortions such as SVM scores | Distortions that are not sigmoid-shaped |
| Isotonic regression | Thousands of calibration examples; any monotone distortion | Small calibration sets; it overfits into steps |
| Temperature scaling | Neural network softmax outputs; one parameter for all classes | Distortions that differ by class |
Common mistakes
- Fitting the calibrator on the training data, which teaches it the model's overconfidence.
- Judging calibration by AUC, which is blind to it.
- Calibrating once and never checking again after the base rate shifts.
- Reading a reliability diagram with a handful of cases per bin; tiny bins are noise.
- Forgetting that class weights and resampling decalibrate a model on purpose.
In the wild
Any system that multiplies a probability by money needs calibration. Ad auctions estimate the expected value of showing an ad as click probability times the bid, so an overconfident click model systematically overpays. Insurance and credit pricing turn claim or default probabilities directly into premiums and interest rates. Weather forecast verification has long used reliability diagrams to check probability-of-precipitation forecasts, the same check you just ran on a classifier.
Check yourself
-
Among all cases where a model predicted 0.9, only 60 percent were positive. The model is...
Show answer
B. Overconfident at the high end It claims 90 percent but reality is 60 percent: its high probabilities are too extreme. Its ranking could still be excellent.
-
After isotonic calibration, what happens to ROC AUC?
Show answer
B. It stays (almost) the same, because the map is monotone and keeps the order A monotone map preserves the ranking of predictions, and AUC depends only on the ranking. Ties created by the step function can move it very slightly.
-
You have 300 labelled examples for calibration. Which method is the safer choice?
Show answer
B. Platt scaling Platt scaling has two parameters and is stable on small sets. Isotonic regression is flexible but overfits with few examples.
-
Why must the calibrator be fitted on data the model did not train on?
Show answer
B. On training data the model looks more confident and correct than it will be on new data, so the map would learn the wrong correction Models are overconfident precisely on the data they fitted. A calibrator fitted there learns to keep that overconfidence.
Recap
- Calibration asks whether predicted probabilities match observed frequencies; it is separate from ranking quality.
- Reliability diagrams compare mean prediction with observed rate per bin; the diagonal is perfect.
- Naive Bayes, SVMs, boosted trees, deep networks and resampled models are often miscalibrated.
- Platt scaling (two parameters) suits small calibration sets; isotonic regression fixes any monotone distortion with more data.
- Fit calibrators on held-out data, measure with Brier score or log loss, and re-check after launch.
Next up
- From notebook to production learning path