Machine Learning ML-02.09
Imbalanced data
Why accuracy lies when one class is rare, and how class weights, resampling, SMOTE, threshold moving and the right metrics fix it.
The big idea
One card transaction in a thousand is fraud. A model that says "not fraud" to every transaction is 99.9 percent accurate, ships in an afternoon and catches nothing. That is the trap of imbalanced dataA classification dataset where one class is much rarer than the other, such as fraud, disease or equipment failure.: when one class is rare, the number everyone reaches for first, accuracy, rewards ignoring it.
Think of an airport scanner. Almost every bag is harmless, so a scanner that never alarms is "right" almost every time. Nobody would call it good, because the rare bag it misses is the one that matters. With imbalanced data you have to decide how much a missed rare case costs compared with a false alarm, measure that, and push the model toward it.
How it works
Step 1: Why accuracy lies
Accuracy counts every correct answer equally. When 99 percent of cases are negative, the negatives decide the score. A useful model and a useless one can have nearly the same accuracy. Instead, look at the rare class directly:
- RecallOf all the truly positive cases, the share the model flagged. Also called sensitivity or true positive rate.: of the real fraud cases, what share did you catch?
- PrecisionOf all the cases the model flagged as positive, the share that really were positive.: of the cases you flagged, what share were really fraud?
- PR-AUC (average precision): precision and recall across every threshold, summarised in one number. Unlike accuracy, it starts at the base rate (0.01 for 1 percent positives) for a useless model, so improvements are visible. These metrics are covered fully in Classification metrics .
Step 2: Where the threshold sits
A classifier usually outputs a probability, and you flag a case when it passes a threshold, 0.5 by default. With a rare class, the probability that a case is positive stays low for all but the most extreme scores, because the base rate drags it down, exactly as in Bayes' theorem. The 0.5 line ends up deep in the rare class's territory and most positives fall below it. Nothing is wrong with the model; the threshold is wrong for your costs.
Step 3: Fix the decision, not just the data
- Threshold movingChoosing the decision threshold from the costs of each type of error, instead of using the default 0.5.: keep the model and lower the threshold until the trade-off between catches and false alarms fits your costs. If a missed fraud costs 500 euros and reviewing a false alarm costs 5, you can afford many false alarms per catch. Pick the threshold on validation data.
- Class weights: make each rare example count more in the loss, for example 99 times more when it is 1 percent of the data. The model now pays as much attention to the rare class as to the common one. In scikit-learn,
class_weight="balanced"does this.
Step 4: Resampling
- Undersampling: throw away common-class examples until the classes are closer. Fast and simple, but you discard data.
- Oversampling: repeat rare-class examples. Keeps all data, but the model can memorise the repeated rows.
- SMOTESynthetic Minority Over-sampling Technique: create new rare-class examples by interpolating between a rare example and one of its nearest rare neighbours.: create new synthetic rare-class examples by picking a rare example, picking one of its nearest rare neighbours, and placing a new point somewhere on the line between them. It adds variety instead of exact copies, but it can invent unrealistic points where classes overlap.
Two rules: resample the training data only, never validation or test, which must reflect reality. And remember that resampling and class weights shift the model's probabilities upward, so its scores are no longer calibrated probabilities ( Probability calibration ).
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, precision_score, recall_score, average_precision_score
# 1 percent positives, like fraud. Positives are shifted in 2 of 6 features.
rng = np.random.default_rng(0)
y = (rng.random(20000) < 0.01).astype(int)
X = rng.normal(size=(20000, 6))
X[y == 1, :2] += 1.8
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
def report(name, model, threshold=0.5, Xfit=Xtr, yfit=ytr):
model.fit(Xfit, yfit)
p = model.predict_proba(Xte)[:, 1]
pred = (p >= threshold).astype(int)
print(f"{name:<26} acc {accuracy_score(yte, pred):.3f} "
f"precision {precision_score(yte, pred, zero_division=0):.2f} "
f"recall {recall_score(yte, pred):.2f} PR-AUC {average_precision_score(yte, p):.2f}")
print(f"positives in test set: {yte.sum()} of {len(yte)}\n")
report("always 'no'", DummyClassifier(strategy="most_frequent"))
report("logistic, threshold 0.5", LogisticRegression())
report("logistic, threshold 0.1", LogisticRegression(), threshold=0.1)
report("class_weight='balanced'", LogisticRegression(class_weight="balanced"))
# Random oversampling: repeat rare rows (training data only!) until classes match
pos = np.flatnonzero(ytr == 1)
extra = rng.choice(pos, (ytr == 0).sum() - len(pos))
report("random oversampling", LogisticRegression(),
Xfit=np.vstack([Xtr, Xtr[extra]]), yfit=np.concatenate([ytr, ytr[extra]]))Output
positives in test set: 59 of 6000
always 'no' acc 0.990 precision 0.00 recall 0.00 PR-AUC 0.01
logistic, threshold 0.5 acc 0.991 precision 0.59 recall 0.22 PR-AUC 0.43
logistic, threshold 0.1 acc 0.982 precision 0.29 recall 0.59 PR-AUC 0.43
class_weight='balanced' acc 0.895 precision 0.08 recall 0.92 PR-AUC 0.40
random oversampling acc 0.895 precision 0.08 recall 0.92 PR-AUC 0.40 Read the output row by row. "Always no" gets 0.990 accuracy and catches nothing. Plain logistic regression at 0.5 has the best accuracy, 0.991, but recall of only 0.22. Lowering the threshold to 0.1 raises recall to 0.59. Class weights and oversampling push recall to 0.92 but precision drops to 0.08: most flags are false alarms. PR-AUC barely changes across the logistic models, because they rank cases in nearly the same order. Most of the difference is where each one puts its threshold.
Try it
Try this:
- Keep the threshold at 0.50 and drag the rare class share from 50 percent down to 0.1 percent. Watch accuracy (gold) climb while recall (purple) collapses. At 0.1 percent, how many of the 10 rare cases does the model catch?
- At 1 percent, lower the threshold until recall passes 70 percent. How many false alarms per 10,000 cases does that cost? Would that be acceptable for fraud review? For cancer screening?
- Raise the model skill. Which helps recall at a 1 percent share more: a better model, or a better threshold?
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Threshold moving | You have a decent model and know the cost of each error type | Costs are unknown or change often; revisit the threshold regularly |
| Class weights | A simple, cheap fix inside training, supported by most libraries | You need calibrated probabilities without recalibrating |
| Undersampling | Huge data where the common class is redundant and training is slow | Data is limited; you throw away information |
| Oversampling and SMOTE | Few rare examples and a model that benefits from balance | Classes overlap heavily, or resampling leaks into the validation data |
| Collect more rare examples | Rare cases can be found or labelled at reasonable cost | Rare events are truly rare and slow to accumulate |
Common mistakes
- Reporting accuracy on imbalanced data, or comparing it with the always-majority baseline only in your head.
- Oversampling before splitting, so copies of the same rare rows land in both training and test.
- Resampling the validation or test set, which makes reported precision meaningless.
- Leaving the threshold at 0.5 without asking what each error costs.
- Treating probabilities from a reweighted or resampled model as calibrated risks.
In the wild
Fraud detection, disease screening, equipment failure prediction and content moderation are all imbalanced problems, and teams in these areas typically report precision, recall and PR-AUC rather than accuracy. Fraud teams set thresholds from the cost of a missed fraud against the cost of manual review or a blocked legitimate customer, and often use several thresholds: approve, send to review, block. Screening programmes in medicine accept many false positives at the first stage because a follow-up test can filter them out, while a missed case is far more costly.
Check yourself
-
A defect detector reports 99.5 percent accuracy. Defects are 0.5 percent of items. What do you check first?
Show answer
B. Recall on defects, since always predicting 'no defect' also scores 99.5 percent The always-majority baseline matches that accuracy exactly. Only recall and precision on the rare class show whether it detects anything.
-
Your model outputs probabilities. With 1 percent positives, almost nothing passes 0.5. What is the cheapest fix?
Show answer
B. Lower the decision threshold, chosen on validation data from the cost of each error Base rates keep probabilities low. Moving the threshold uses the same model and lets you trade false alarms for catches.
-
When should you apply SMOTE or oversampling?
Show answer
B. On the training data only, inside each cross-validation fold Resampling before splitting puts copies or near-copies of test rows in training, and resampled evaluation sets misrepresent reality.
-
Class weights raise recall from 0.22 to 0.92 but precision falls from 0.59 to 0.08. What does that mean?
Show answer
B. It now flags far more cases, catching most fraud but with about 11 false alarms per real one Precision 0.08 means 8 in 100 flags are real. Whether that is acceptable depends on the cost of reviewing a false alarm compared with missing a fraud.
Recap
- With a rare class, accuracy rewards ignoring it; always compare against the always-majority baseline.
- Measure the rare class with recall, precision and PR-AUC.
- The default 0.5 threshold is usually wrong for rare classes; choose it from the cost of each error.
- Class weights, undersampling, oversampling and SMOTE rebalance training, never evaluation data.
- Rebalancing shifts predicted probabilities, so recalibrate if you need them as real risks.
Next up
- From notebook to production learning path