Machine Learning ML-04.05
Classification metrics
The confusion matrix and everything built on it: accuracy, precision, recall, F1, ROC and PR curves, and how to pick the metric that matches the cost.
The big idea
A churn model flags customers who are about to leave so a retention team can call them. Ask "how good is it?" and you get very different answers depending on what you count. It is right 91 percent of the time. Nine in ten customers it flags really are leaving. And it misses six out of seven customers who leave. All three statements describe the same model in the code below.
Every classification metric is a different way of summarising four counts: what the model flagged, what it did not, and whether each call was right. Choosing a metric means choosing which mistakes you care about. A cancer screen that misses tumours is a disaster even if it rarely raises false alarms; a spam filter that hides an important email is worse than one that lets an extra spam through. The right metric is the one that matches what each kind of error costs.
How it works
Step 1: The confusion matrix
Cross the true label with the predicted label and you get four cells, the confusion matrixA table counting true positives, false positives, false negatives and true negatives for a classifier at one threshold.:
- True positive (TP): flagged, and really positive.
- False positive (FP): flagged, but actually negative. A false alarm.
- False negative (FN): not flagged, but actually positive. A miss.
- True negative (TN): not flagged, and really negative.
Step 2: Accuracy, precision, recall and F1
Each metric is a ratio of cells. Using the numbers in Figure 1:
- AccuracyThe share of all predictions that are correct: (TP + TN) / all cases. = (TP + TN) / all = (45 + 2683) / 3000 = 0.909. Looks great, but always predicting "stay" would score 2688 / 3000 = 0.896.
- Precision = TP / (TP + FP) = 45 / 50 = 0.900. When the model flags someone, it is usually right.
- Recall = TP / (TP + FN) = 45 / 312 = 0.144. It finds only one churner in seven.
- F1 scoreThe harmonic mean of precision and recall, 2PR / (P + R). It is high only when both are high. = 2 x 0.900 x 0.144 / (0.900 + 0.144) = 0.249.
F1 uses the harmonic mean, which is dragged toward the smaller value, so a model cannot hide terrible recall behind great precision. Here it exposes the problem that accuracy hid.
Step 3: The threshold moves everything
All four cells depend on the threshold. Lower it and the model flags more people: recall rises, precision usually falls. From the code below:
| threshold | precision | recall | FPR | F1 |
|---|---|---|---|---|
| 0.7 | 1.000 | 0.026 | 0.000 | 0.050 |
| 0.5 | 0.900 | 0.144 | 0.002 | 0.249 |
| 0.3 | 0.691 | 0.308 | 0.016 | 0.426 |
| 0.2 | 0.459 | 0.471 | 0.064 | 0.465 |
| 0.1 | 0.200 | 0.670 | 0.311 | 0.308 |
No row is "correct". If a retention call is cheap and a lost customer is expensive, 0.2 or even 0.1 may be right. The model is the same; the decision rule is a business choice.
Step 4: The ROC curve and AUC
Instead of one threshold, sweep all of them. The ROC curveReceiver operating characteristic curve: true positive rate (recall) plotted against false positive rate as the threshold sweeps from high to low. plots recall (true positive rate) against the false positive rateThe share of real negatives that the model wrongly flags: FP / (FP + TN)., FP / (FP + TN). A threshold of infinity flags nobody (bottom-left); a threshold of minus infinity flags everybody (top-right). A random model traces the diagonal.
The area under the curve, ROC AUCArea under the ROC curve: the probability that the model scores a random positive above a random negative. 0.5 is random, 1.0 is perfect ranking., summarises ranking quality in one number that does not depend on any threshold: it is the probability that a randomly chosen positive gets a higher score than a randomly chosen negative.
Step 5: Precision-recall curves for rare classes
ROC has a blind spot. The false positive rate divides by all the negatives, so when negatives vastly outnumber positives, even thousands of false alarms barely move it. The precision-recall curvePrecision plotted against recall as the threshold sweeps. A random model's precision equals the positive rate, so improvements on rare classes are visible. plots precision against recall instead, and precision feels every false alarm. A random model's PR curve is flat at the positive rate (0.104 here), so the baseline is honest. Its area, average precision, is 0.469 for the churn model. Use PR curves when positives are rare and you care about the flagged list ( Imbalanced data ).
Step 6: Choose the metric from the cost
- Missing a positive is very costly (disease, fraud, safety): prioritise recall, then make precision acceptable.
- False alarms are costly (blocking legitimate payments, spam folder for real mail): prioritise precision.
- You need one number to compare models and both errors matter: F1, or better, a cost-weighted metric.
- You compare rankings independent of threshold: ROC AUC for balanced data, PR AUC for rare positives.
Best of all, write down the cost of each cell (a missed churner loses a year of revenue, a call costs ten minutes) and pick the threshold that minimises total cost on validation data.
Try it
Try this:
- Drag the threshold from left to right. Watch the gold dots travel down both curves at the same time and the confusion matrix move cases from FP to TN and from TP to FN.
- Set positives to 5 percent. The ROC curve does not change at all, but the PR curve drops sharply. Which picture better reflects how annoying the false alarms are?
- Raise model skill to 3.5. Both curves bend toward the ideal corner; now find a threshold with precision and recall both above 0.9.
Going deeper
Code: every metric from one model
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (confusion_matrix, accuracy_score, precision_score, recall_score,
f1_score, roc_auc_score, average_precision_score)
# 10 percent positives, e.g. customers who will churn
X, y = make_classification(n_samples=10000, n_features=8, n_informative=4, weights=[0.9],
class_sep=0.8, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
p = LogisticRegression().fit(Xtr, ytr).predict_proba(Xte)[:, 1]
print(f"test cases {len(yte)}, positives {yte.sum()}")
pred = (p >= 0.5).astype(int)
tn, fp, fn, tp = confusion_matrix(yte, pred).ravel()
print(f"\nthreshold 0.5: TP {tp} FP {fp} FN {fn} TN {tn}")
print(f"accuracy (TP+TN)/all = {accuracy_score(yte, pred):.3f}")
print(f"precision TP/(TP+FP) = {precision_score(yte, pred):.3f}")
print(f"recall TP/(TP+FN) = {recall_score(yte, pred):.3f}")
print(f"F1 = {f1_score(yte, pred):.3f}")
print("\nthreshold precision recall FPR F1")
for t in [0.1, 0.2, 0.3, 0.5, 0.7]:
pr = (p >= t).astype(int)
tn, fp, fn, tp = confusion_matrix(yte, pr).ravel()
print(f" {t:.1f} {tp / (tp + fp):.3f} {tp / (tp + fn):.3f} {fp / (fp + tn):.3f} {f1_score(yte, pr):.3f}")
print(f"\nROC AUC {roc_auc_score(yte, p):.3f} PR AUC (average precision) {average_precision_score(yte, p):.3f}")
print(f"a random ranking would score ROC AUC 0.5 and PR AUC about {yte.mean():.3f}")Output
test cases 3000, positives 312
threshold 0.5: TP 45 FP 5 FN 267 TN 2683
accuracy (TP+TN)/all = 0.909
precision TP/(TP+FP) = 0.900
recall TP/(TP+FN) = 0.144
F1 = 0.249
threshold precision recall FPR F1
0.1 0.200 0.670 0.311 0.308
0.2 0.459 0.471 0.064 0.465
0.3 0.691 0.308 0.016 0.426
0.5 0.900 0.144 0.002 0.249
0.7 1.000 0.026 0.000 0.050
ROC AUC 0.727 PR AUC (average precision) 0.469
a random ranking would score ROC AUC 0.5 and PR AUC about 0.104 Note how little accuracy tells you: 0.909, against 0.896 for a model that predicts "stay" for everyone. The threshold table and both AUCs give a far more useful picture.
Averages for more than two classes
With several classes, precision and recall are computed per class and then averaged. Macro and micro averaging give different answers when classes are imbalanced; see Multi-class and multi-label classification .
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Accuracy | Balanced classes and equal error costs | Rare positives; it rewards predicting the majority |
| Precision / recall at a threshold | A deployed decision rule with known costs | Comparing models before the threshold is chosen |
| F1 | One number when both errors matter about equally | Errors have very different costs |
| ROC AUC | Comparing ranking quality on roughly balanced data | Rare positives; it looks good while precision is poor |
| PR AUC | Rare positives and a focus on the flagged list | Explaining to stakeholders; its baseline varies with the positive rate |
Common mistakes
- Reporting accuracy alone on imbalanced data.
- Comparing two models at different thresholds and calling one "more precise".
- Choosing the threshold on the test set; choose it on validation data and report test once.
- Celebrating a high ROC AUC on a 1-in-1,000 problem where precision is tiny.
- Mixing up false positive rate (FP / all negatives) with the share of flags that are false (1 - precision).
In the wild
Medical tests are described by sensitivity and specificity, which are recall and 1 minus the false positive rate under other names. Search and recommendation teams track precision at the top of the ranked list, because users only see the first few results. Fraud and abuse teams choose thresholds from the cost of a missed case against the cost of manual review, and monitor precision and recall separately after launch because both drift as attackers adapt.
Check yourself
-
A model has TP = 80, FP = 20, FN = 120, TN = 780. What is its recall?
Show answer
B. 0.40 Recall = TP / (TP + FN) = 80 / 200 = 0.40. Precision would be 80 / 100 = 0.80, which is the tempting wrong answer.
-
For a cancer screening test that is followed by a cheap second test, which metric should the first test prioritise?
Show answer
B. Recall A missed cancer is very costly, while false alarms are filtered by the follow-up test. Maximise recall, then keep precision tolerable.
-
Positives are 0.5 percent of cases. Model A has ROC AUC 0.95. What else must you check?
Show answer
B. Precision or PR AUC, since thousands of false alarms barely move the false positive rate With so many negatives, a small false positive rate can still mean false alarms outnumber true positives many times over. Precision shows that; ROC does not.
-
Lowering the threshold from 0.5 to 0.2 usually does what?
Show answer
B. Raises recall and lowers precision More cases are flagged: more positives are caught (recall up), and more negatives slip in (precision usually down). AUC is threshold-free, so it does not change.
Recap
- The confusion matrix (TP, FP, FN, TN) at one threshold is the source of every classification metric.
- Precision asks how many flags are right; recall asks how many positives were caught; F1 balances them.
- Changing the threshold trades precision against recall; it is a business decision made on validation data.
- ROC AUC measures ranking quality across thresholds; PR curves are more honest when positives are rare.
- Pick metrics from the cost of each error, not from habit.
Next up
- From notebook to production learning path