Skip to content
learn
  1. Learn
  2. Machine Learning
  3. Classification

Machine Learning ML-04.05

Classification metrics

The confusion matrix and everything built on it: accuracy, precision, recall, F1, ROC and PR curves, and how to pick the metric that matches the cost.

Beginner 16 min Last reviewed

The big idea

A churn model flags customers who are about to leave so a retention team can call them. Ask "how good is it?" and you get very different answers depending on what you count. It is right 91 percent of the time. Nine in ten customers it flags really are leaving. And it misses six out of seven customers who leave. All three statements describe the same model in the code below.

Every classification metric is a different way of summarising four counts: what the model flagged, what it did not, and whether each call was right. Choosing a metric means choosing which mistakes you care about. A cancer screen that misses tumours is a disaster even if it rarely raises false alarms; a spam filter that hides an important email is worse than one that lets an extra spam through. The right metric is the one that matches what each kind of error costs.

How it works

Step 1: The confusion matrix

Cross the true label with the predicted label and you get four cells, the confusion matrixA table counting true positives, false positives, false negatives and true negatives for a classifier at one threshold.:

  • True positive (TP): flagged, and really positive.
  • False positive (FP): flagged, but actually negative. A false alarm.
  • False negative (FN): not flagged, but actually positive. A miss.
  • True negative (TN): not flagged, and really negative.
A confusion matrix at threshold 0.5 A two by two grid. Rows are the actual class, columns the predicted class. True positives 45 (green), false negatives 267 (red, glowing), false positives 5 (red), true negatives 2683 (green). Labels around the grid show predicted positive and predicted negative across the top, actual positive and actual negative down the side. TP = 45 caught churners FN = 267 missed churners FP = 5 false alarms TN = 2,683 correctly left alone predicted churn predicted stay actually churned (312) actually stayed (2,688) Figure 1. The churn model from the code below at threshold 0.5, on 3,000 test customers. It flags only 50 people, and 45 of them are right, but 267 churners slip through unflagged. The glowing cell is the problem accuracy hides.

Step 2: Accuracy, precision, recall and F1

Each metric is a ratio of cells. Using the numbers in Figure 1:

  • AccuracyThe share of all predictions that are correct: (TP + TN) / all cases. = (TP + TN) / all = (45 + 2683) / 3000 = 0.909. Looks great, but always predicting "stay" would score 2688 / 3000 = 0.896.
  • Precision = TP / (TP + FP) = 45 / 50 = 0.900. When the model flags someone, it is usually right.
  • Recall = TP / (TP + FN) = 45 / 312 = 0.144. It finds only one churner in seven.
  • F1 scoreThe harmonic mean of precision and recall, 2PR / (P + R). It is high only when both are high. = 2 x 0.900 x 0.144 / (0.900 + 0.144) = 0.249.

F1 uses the harmonic mean, which is dragged toward the smaller value, so a model cannot hide terrible recall behind great precision. Here it exposes the problem that accuracy hid.

Step 3: The threshold moves everything

All four cells depend on the threshold. Lower it and the model flags more people: recall rises, precision usually falls. From the code below:

thresholdprecisionrecallFPRF1
0.71.0000.0260.0000.050
0.50.9000.1440.0020.249
0.30.6910.3080.0160.426
0.20.4590.4710.0640.465
0.10.2000.6700.3110.308

No row is "correct". If a retention call is cheap and a lost customer is expensive, 0.2 or even 0.1 may be right. The model is the same; the decision rule is a business choice.

Step 4: The ROC curve and AUC

Instead of one threshold, sweep all of them. The ROC curveReceiver operating characteristic curve: true positive rate (recall) plotted against false positive rate as the threshold sweeps from high to low. plots recall (true positive rate) against the false positive rateThe share of real negatives that the model wrongly flags: FP / (FP + TN)., FP / (FP + TN). A threshold of infinity flags nobody (bottom-left); a threshold of minus infinity flags everybody (top-right). A random model traces the diagonal.

ROC curve of the churn model A square plot with false positive rate on the x axis and true positive rate on the y axis. A dashed diagonal marks a random model. The model's glowing curve rises steeply from the origin through gold points for thresholds 0.5, 0.3, 0.2 and 0.1, reaching true positive rate 0.67 at false positive rate 0.31, then runs to the top-right corner. The area under it is shaded and labelled AUC 0.727. 00.250.50.75100.250.50.751false positive ratetrue positive rate (recall) t = 0.5 t = 0.3 t = 0.2 t = 0.1 random: AUC 0.5 model: AUC 0.727 Figure 2. Each gold point is one threshold from the table. The curve shows every trade-off the model offers at once; its area, AUC 0.727, is the chance that a random churner gets a higher score than a random loyal customer.

The area under the curve, ROC AUCArea under the ROC curve: the probability that the model scores a random positive above a random negative. 0.5 is random, 1.0 is perfect ranking., summarises ranking quality in one number that does not depend on any threshold: it is the probability that a randomly chosen positive gets a higher score than a randomly chosen negative.

Step 5: Precision-recall curves for rare classes

ROC has a blind spot. The false positive rate divides by all the negatives, so when negatives vastly outnumber positives, even thousands of false alarms barely move it. The precision-recall curvePrecision plotted against recall as the threshold sweeps. A random model's precision equals the positive rate, so improvements on rare classes are visible. plots precision against recall instead, and precision feels every false alarm. A random model's PR curve is flat at the positive rate (0.104 here), so the baseline is honest. Its area, average precision, is 0.469 for the churn model. Use PR curves when positives are rare and you care about the flagged list ( Imbalanced data ).

Precision-recall points of the churn model Precision on the y axis against recall on the x axis. Five gold points for thresholds 0.7, 0.5, 0.3, 0.2 and 0.1, joined by a glowing pink line falling from precision 1.0 at recall 0.026 to precision 0.2 at recall 0.67. A dashed horizontal line at precision 0.104 marks a random model. 00.250.50.75100.250.50.751recallprecision t = 0.7 t = 0.5 t = 0.3 t = 0.2 t = 0.1 random: precision 0.104 Figure 3. The same five thresholds as precision-recall points. Every step down in threshold buys recall with precision. The random baseline sits at the positive rate, 0.104, not at 0.5.

Step 6: Choose the metric from the cost

  • Missing a positive is very costly (disease, fraud, safety): prioritise recall, then make precision acceptable.
  • False alarms are costly (blocking legitimate payments, spam folder for real mail): prioritise precision.
  • You need one number to compare models and both errors matter: F1, or better, a cost-weighted metric.
  • You compare rankings independent of threshold: ROC AUC for balanced data, PR AUC for rare positives.

Best of all, write down the cost of each cell (a missed churner loses a year of revenue, a call costs ten minutes) and pick the threshold that minimises total cost on validation data.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Drag the threshold from left to right. Watch the gold dots travel down both curves at the same time and the confusion matrix move cases from FP to TN and from TP to FN.
  2. Set positives to 5 percent. The ROC curve does not change at all, but the PR curve drops sharply. Which picture better reflects how annoying the false alarms are?
  3. Raise model skill to 3.5. Both curves bend toward the ideal corner; now find a threshold with precision and recall both above 0.9.

Going deeper

Code: every metric from one model

metrics.pyruns with python3 + numpy + scikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (confusion_matrix, accuracy_score, precision_score, recall_score,
                             f1_score, roc_auc_score, average_precision_score)

# 10 percent positives, e.g. customers who will churn
X, y = make_classification(n_samples=10000, n_features=8, n_informative=4, weights=[0.9],
                           class_sep=0.8, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
p = LogisticRegression().fit(Xtr, ytr).predict_proba(Xte)[:, 1]
print(f"test cases {len(yte)}, positives {yte.sum()}")

pred = (p >= 0.5).astype(int)
tn, fp, fn, tp = confusion_matrix(yte, pred).ravel()
print(f"\nthreshold 0.5:  TP {tp}  FP {fp}  FN {fn}  TN {tn}")
print(f"accuracy  (TP+TN)/all = {accuracy_score(yte, pred):.3f}")
print(f"precision TP/(TP+FP)  = {precision_score(yte, pred):.3f}")
print(f"recall    TP/(TP+FN)  = {recall_score(yte, pred):.3f}")
print(f"F1                    = {f1_score(yte, pred):.3f}")

print("\nthreshold  precision  recall  FPR    F1")
for t in [0.1, 0.2, 0.3, 0.5, 0.7]:
    pr = (p >= t).astype(int)
    tn, fp, fn, tp = confusion_matrix(yte, pr).ravel()
    print(f"   {t:.1f}      {tp / (tp + fp):.3f}    {tp / (tp + fn):.3f}  {fp / (fp + tn):.3f}  {f1_score(yte, pr):.3f}")

print(f"\nROC AUC {roc_auc_score(yte, p):.3f}   PR AUC (average precision) {average_precision_score(yte, p):.3f}")
print(f"a random ranking would score ROC AUC 0.5 and PR AUC about {yte.mean():.3f}")

Output

test cases 3000, positives 312

threshold 0.5:  TP 45  FP 5  FN 267  TN 2683
accuracy  (TP+TN)/all = 0.909
precision TP/(TP+FP)  = 0.900
recall    TP/(TP+FN)  = 0.144
F1                    = 0.249

threshold  precision  recall  FPR    F1
   0.1      0.200    0.670  0.311  0.308
   0.2      0.459    0.471  0.064  0.465
   0.3      0.691    0.308  0.016  0.426
   0.5      0.900    0.144  0.002  0.249
   0.7      1.000    0.026  0.000  0.050

ROC AUC 0.727   PR AUC (average precision) 0.469
a random ranking would score ROC AUC 0.5 and PR AUC about 0.104

Note how little accuracy tells you: 0.909, against 0.896 for a model that predicts "stay" for everyone. The threshold table and both AUCs give a far more useful picture.

Averages for more than two classes

With several classes, precision and recall are computed per class and then averaged. Macro and micro averaging give different answers when classes are imbalanced; see Multi-class and multi-label classification .

Trade-offs

OptionWins whenLoses when
Accuracy Balanced classes and equal error costs Rare positives; it rewards predicting the majority
Precision / recall at a threshold A deployed decision rule with known costs Comparing models before the threshold is chosen
F1 One number when both errors matter about equally Errors have very different costs
ROC AUC Comparing ranking quality on roughly balanced data Rare positives; it looks good while precision is poor
PR AUC Rare positives and a focus on the flagged list Explaining to stakeholders; its baseline varies with the positive rate

Common mistakes

  • Reporting accuracy alone on imbalanced data.
  • Comparing two models at different thresholds and calling one "more precise".
  • Choosing the threshold on the test set; choose it on validation data and report test once.
  • Celebrating a high ROC AUC on a 1-in-1,000 problem where precision is tiny.
  • Mixing up false positive rate (FP / all negatives) with the share of flags that are false (1 - precision).

In the wild

Medical tests are described by sensitivity and specificity, which are recall and 1 minus the false positive rate under other names. Search and recommendation teams track precision at the top of the ranked list, because users only see the first few results. Fraud and abuse teams choose thresholds from the cost of a missed case against the cost of manual review, and monitor precision and recall separately after launch because both drift as attackers adapt.

Check yourself

  1. A model has TP = 80, FP = 20, FN = 120, TN = 780. What is its recall?

    Show answer

    B. 0.40 Recall = TP / (TP + FN) = 80 / 200 = 0.40. Precision would be 80 / 100 = 0.80, which is the tempting wrong answer.

  2. For a cancer screening test that is followed by a cheap second test, which metric should the first test prioritise?

    Show answer

    B. Recall A missed cancer is very costly, while false alarms are filtered by the follow-up test. Maximise recall, then keep precision tolerable.

  3. Positives are 0.5 percent of cases. Model A has ROC AUC 0.95. What else must you check?

    Show answer

    B. Precision or PR AUC, since thousands of false alarms barely move the false positive rate With so many negatives, a small false positive rate can still mean false alarms outnumber true positives many times over. Precision shows that; ROC does not.

  4. Lowering the threshold from 0.5 to 0.2 usually does what?

    Show answer

    B. Raises recall and lowers precision More cases are flagged: more positives are caught (recall up), and more negatives slip in (precision usually down). AUC is threshold-free, so it does not change.

Recap

  • The confusion matrix (TP, FP, FN, TN) at one threshold is the source of every classification metric.
  • Precision asks how many flags are right; recall asks how many positives were caught; F1 balances them.
  • Changing the threshold trades precision against recall; it is a business decision made on validation data.
  • ROC AUC measures ranking quality across thresholds; PR curves are more honest when positives are rare.
  • Pick metrics from the cost of each error, not from habit.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up