Machine Learning ML-02.08
Data leakage
How information from the future or the test set sneaks into training, why it makes offline scores lie, and how to detect and prevent it.
The big idea
Your churn model scores 100 percent on the test set. The team is thrilled. In production it is barely better than guessing. What happened is almost always the same thing: the model was given information during training and testing that it will not have when it makes real predictions. That is data leakageWhen information that would not be available at prediction time ends up in training data or features, making offline results far better than real performance..
It is like a student who scores perfectly on practice exams because the answer key was stapled to the back. The scores are real, but they measure the wrong thing, and the first exam without the answer key shows it. Leakage is dangerous precisely because it does not look like a bug. It looks like success.
How it works
Step 1: Target leakage
A feature is computed from the label, or from something that happens because of the label. Examples:
- "days since cancellation call" in a churn model: only churners have one.
- "account in collections" when predicting loan default: collections happen after default.
- "treatment given" when predicting a diagnosis: the treatment reflects the diagnosis already made.
The feature is real data and in your table, so nothing looks wrong. The test is a single question for every feature: at the exact moment the model must make its prediction, would this value exist, and would it have this value?
Step 2: Temporal leakage
The model trains on data from after the period it is tested on. A random split of time-ordered data does this: the model learns from March to "predict" February. Window features that include the current event or later ones do it too, such as "total spend this month" used to predict a purchase made in the middle of that month. Split by time ( Train, validation and test splits ) and compute every feature as of the prediction timestamp, often called a point-in-time correctJoining feature values exactly as they were at the time of each training example, never using later values. join.
Step 3: Train-test contamination
The same or nearly the same example appears on both sides of the split: duplicate rows, the same customer with several records, the same photo resized, the same document with a different title. The model is partly tested on things it memorised. Remove duplicates before splitting and use group splits for repeated entities.
Step 4: Preprocessing leakage
Any step that learns from data, such as scaling, imputing, encoding or feature selection, fitted on the full dataset before splitting, has seen the test rows. Usually this inflates results only slightly. With feature selection on many columns it can be dramatic:
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score, train_test_split
rng = np.random.default_rng(0)
# 1. Preprocessing leakage: 100 rows, 5,000 columns of pure noise, random labels.
X = rng.normal(size=(100, 5000))
y = rng.integers(0, 2, 100)
X_sel = SelectKBest(f_classif, k=20).fit_transform(X, y) # selected using ALL rows
leaky = cross_val_score(LogisticRegression(), X_sel, y, cv=5).mean()
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression())
honest = cross_val_score(pipe, X, y, cv=5).mean() # selected inside each fold
print(f"noise data, select then CV: {leaky:.2f} select inside CV: {honest:.2f}")
# 2. Target leakage: churn data with a column written AFTER the customer left.
n = 2000
usage = rng.normal(size=n)
tenure = rng.normal(size=n)
churn = (rng.random(n) < 1 / (1 + np.exp(1.0 + 2.5 * usage + 1.5 * tenure))).astype(int)
days_since_cancel_call = np.where(churn == 1, rng.integers(1, 30, n), 0) # the leak
X2 = np.column_stack([usage, tenure, days_since_cancel_call])
names = ["usage", "tenure", "days_since_cancel_call"]
Xtr, Xte, ytr, yte = train_test_split(X2, churn, test_size=0.3, random_state=0)
for cols in ([0, 1, 2], [0, 1]):
rf = RandomForestClassifier(min_samples_leaf=10, random_state=0).fit(Xtr[:, cols], ytr)
imp = dict(zip([names[c] for c in cols], rf.feature_importances_.round(2).tolist()))
print(f"\ntest accuracy {rf.score(Xte[:, cols], yte):.3f} with {len(cols)} features")
print("importances:", imp)
print(f"\nalways predicting 'stays' would score {1 - yte.mean():.3f}")Output
noise data, select then CV: 0.85 select inside CV: 0.48
test accuracy 1.000 with 3 features
importances: {'usage': 0.19, 'tenure': 0.1, 'days_since_cancel_call': 0.71}
test accuracy 0.807 with 2 features
importances: {'usage': 0.67, 'tenure': 0.33}
always predicting 'stays' would score 0.615 The first line uses data that is pure noise: 5,000 random columns and random labels. Choosing the 20 columns most correlated with the label using all rows, then cross-validating, reports 85 percent accuracy on a problem where nothing can be learned. Doing the selection inside each training fold, with a pipeline, gives the honest answer: 48 percent, a coin flip. The selection step had found columns that happened to match the test rows' labels by chance.
Step 5: Detecting leakage
- Too good to be true. Results far above published benchmarks, above the agreement between human experts, or near-perfect on a hard problem. In the churn example, the leaky model scores 1.000 against 0.807 without the leaking column.
- One feature dominates importance. The leaky column takes 0.71 of the importance. A single feature carrying most of the model is a reason to check when and how it is recorded.
- Offline and online disagree. A big drop from validation to production, or to a later time period, points to leakage or drift.
- Ablation. Remove the suspicious feature and retrain. If the score collapses to something plausible, you probably found the leak.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Strict point-in-time features | Any prediction about the future, especially with event logs | Your data store only keeps current values, so building history takes real work |
| Pipelines fitted inside each fold | Any preprocessing that learns from data | Rarely; it costs a little extra compute |
| Group and time splits | Repeated entities or time-ordered data | Few groups or a short history, so splits are lumpy |
| Fresh holdout from a later period | You need confidence before launch | Data accumulates slowly, so you must wait for it |
Common mistakes
- Including a column without knowing when and by what process it is filled in.
- Fitting a scaler, imputer, encoder or selector on all data before splitting.
- Computing window aggregates that include the current event or later ones.
- Deduplicating after splitting instead of before.
- Accepting a big jump in validation score without asking what changed and why.
In the wild
Leakage is a recurring theme in Kaggle competitions, where participants have repeatedly found identifiers, file orderings or timestamps that encoded the answer, and organisers now design data releases to prevent it. In healthcare research, models have been reported to learn from hospital-specific artefacts, such as which scanner or which ward produced an image, instead of from the disease. Feature platforms in industry offer point-in-time joins precisely so that training examples see feature values as they were at the time, not as they are today.
Check yourself
-
A hospital readmission model uses 'number of follow-up appointments booked'. Appointments are booked at discharge for patients flagged as high risk. What kind of problem is this?
Show answer
B. Target leakage: the feature reflects a decision already informed by the outcome risk If the feature is created by the same judgement you want to predict, or after it, the model learns that judgement, not the patient's risk. Check exactly when it is recorded.
-
You standardise all features using the mean and standard deviation of the whole dataset, then split. What is the issue?
Show answer
B. Test-set statistics influenced training, a form of preprocessing leakage The scaler saw the test rows. Fit it on the training data only, ideally inside a pipeline so cross-validation does it per fold.
-
On pure-noise data, selecting features on all rows and then cross-validating gives 85 percent accuracy. Why?
Show answer
B. The selection used the test folds' labels, so it picked columns that match them by chance With thousands of random columns, some match any set of labels by chance. Selecting them with the test labels in view bakes that luck into the score.
-
Which result is the strongest hint of leakage?
Show answer
B. One feature with 70 percent of the importance and a near-perfect score on a known-hard problem A dominant feature plus an implausibly good score is the classic leakage signature. Check when that feature becomes known.
Recap
- Leakage is information in training that will not be available at prediction time.
- Target leakage uses features caused by the outcome; temporal leakage uses the future.
- Contamination puts the same examples on both sides; preprocessing leakage fits steps on test data.
- Ask of every feature: would this value exist, with this value, at the moment of prediction?
- Suspiciously good scores and one dominant feature are the warning signs; fit everything inside the training folds.
Next up
- From notebook to production learning path