Machine Learning ML-02.07
Train, validation and test splits
Why you need three data splits, how to choose random, stratified, group or time-based splits, and how a test set gets used up.
The big idea
A model that has memorised its training data can score 100 percent on it and still be useless on anything new. The only honest way to know how a model will do on tomorrow's data is to test it on data it has never seen. So before training, you hold some data back.
School exams work the same way. Students learn from the textbook exercises (training data). Practice tests tell them which study methods work and where to focus (validation data). The final exam uses fresh questions nobody has seen (test data), and it is only meaningful if it is taken once. A student who sees the final exam questions in advance gets a great score that says nothing about what they know.
How it works
- Random split. Shuffle the rows, then cut: about 70 percent to train on, 15 percent to tune choices, and 15 percent locked away for one final measurement.
- Time-based split. When the model will predict the future, the split must respect time. Each fold trains on the past and validates on the block that follows, moving forward. The newest data is the test set.
Step 1: Why three sets
- The training setThe portion of data the model learns its parameters from. is what the model learns from.
- The validation setData held out from training and used to compare models and tune choices such as hyperparameters and features. is for decisions: which model, which features, which settings. You look at it many times.
- The test setData held out until the very end and used once to estimate how the final model will perform on new data. is for one final, honest estimate of performance after every decision is made.
Why not just two? Because every decision you make by looking at the validation score fits your choices a little to that particular set. After fifty experiments, the best validation score is partly luck. The untouched test set tells you how much. Typical splits are 70/15/15 or 80/10/10; with millions of rows, a few percent is plenty for validation and test. With little data, cross-validationRotating which part of the data is held out, training several times, and averaging the scores, so every row is used for both training and validation. rotates the validation role across the training data instead ( Cross-validation ).
Step 2: Random and stratified splits
A random split shuffles rows and cuts. It is fine when rows are independent and the classes are reasonably balanced.
With a rare class it gets shaky. If 5 percent of 400 rows are positive, a random 20 percent test set should hold 4 positives, but it might hold 2 or 6. A stratified splitA split that keeps each class's share the same in every part, so a rare class is not over- or under-represented by chance. keeps the class proportions identical in every part. The code below shows the difference: random splits give 2 to 6 positives, stratified always gives 4.
Step 3: Group splits
Rows are often not independent. One user has fifty sessions, one patient has twelve scans, one house appears in three listings. With a random split, the same user lands on both sides, and the model gets credit for recognising that person rather than for generalising to new people. A group splitA split that keeps all rows belonging to one group, such as a user or patient, on the same side, so the model is tested on unseen groups. keeps every row of a user, patient or house on one side. Ask: "in production, will the model see new users, or only new rows from known users?" Split to match the answer.
Step 4: Time-based splits
If the model predicts the future, such as next week's demand or whether a transaction is fraud, train on the past and test on the later period. A random split lets the model learn from data that comes after the rows it is tested on, which it can never do in real life. Figure 1 shows forward folds: train on months 1 to 4 and validate on 5, then train on 1 to 5 and validate on 6, and so on, with the newest period as the test set. This also tells you how fast performance decays as the world drifts.
import numpy as np
from sklearn.model_selection import train_test_split, GroupShuffleSplit, TimeSeriesSplit
rng = np.random.default_rng(0)
y = (rng.random(400) < 0.05).astype(int) # 5 percent positives (20 of 400)
for strat in (None, y):
counts = [train_test_split(y, test_size=0.2, stratify=strat, random_state=s)[1].sum().item()
for s in range(8)]
print("stratified" if strat is not None else "random ", "positives in test:", counts)
users = rng.integers(0, 80, size=400) # 80 users, ~5 rows each
tr, te = train_test_split(np.arange(400), test_size=0.2, random_state=0)
print("\nrandom split: test users also seen in train:",
f"{np.isin(users[te], users[tr]).mean():.0%}")
tr, te = next(GroupShuffleSplit(test_size=0.2, random_state=0).split(users, groups=users))
print("group split: test users also seen in train:",
f"{np.isin(users[te], users[tr]).mean():.0%}")
print("\ntime-series folds (row indices, oldest first):")
for tr, va in TimeSeriesSplit(n_splits=4).split(np.arange(20)):
print(f" train {tr[0]:>2}-{tr[-1]:>2} validate {va[0]:>2}-{va[-1]:>2}")
# Using up the test set: 200 "models" that guess at random, all scored on one test set
test_y = rng.integers(0, 2, 300)
fresh_y = rng.integers(0, 2, 300)
guesses = rng.integers(0, 2, (200, 300))
best = (guesses == test_y).mean(axis=1).argmax()
print(f"\nbest of 200 coin-flip models on the test set: {(guesses[best] == test_y).mean():.2f}")
print(f"the same model on fresh data: {(guesses[best] == fresh_y).mean():.2f}")Output
random positives in test: [3, 2, 4, 4, 6, 4, 3, 3]
stratified positives in test: [4, 4, 4, 4, 4, 4, 4, 4]
random split: test users also seen in train: 100%
group split: test users also seen in train: 0%
time-series folds (row indices, oldest first):
train 0- 3 validate 4- 7
train 0- 7 validate 8-11
train 0-11 validate 12-15
train 0-15 validate 16-19
best of 200 coin-flip models on the test set: 0.57
the same model on fresh data: 0.48 Step 5: How the test set gets "used up"
The last two lines of the output are the warning. Two hundred "models" that guess completely at random were each scored on the same test set. The best one scored 0.57, which looks like a real if modest model. On fresh data it scores 0.48, a coin flip. Nothing was learned; choosing the best of many by their test scores manufactured the result.
Every time you look at test results and change something in response, you leak a little information from the test set into your choices. Do it often enough and the test set becomes just another validation set. Rules of thumb: decide your metric and split before you start, do all tuning on validation data, look at the test set once at the end, and if you must iterate after that, collect a fresh test set.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Random split | Independent rows, balanced classes, no time dependence | Repeated users or patients, rare classes, or forecasting |
| Stratified split | Classification with a rare class or a small dataset | Rarely; it is the safe default for classification |
| Group split | Many rows per user, patient, device or item, and production will see new ones | Very few groups, so each split is lumpy |
| Time-based split | Anything that predicts the future or data that drifts over time | Rarely; the test score is lower but honest |
Common mistakes
- Tuning hyperparameters against the test set, then reporting the test score as if it were unbiased.
- Random splits on time-ordered data, which let the model peek at the future.
- Splitting rows when the same user, patient or image appears many times.
- Fitting preprocessing (scalers, imputers, encoders) before the split, which leaks test statistics into training (see Data leakage ).
- Re-shuffling to a "luckier" split when results disappoint.
In the wild
Benchmark datasets in research ship with fixed train, validation and test splits so results are comparable, and some competitions keep the test labels hidden on a server so they cannot be tuned against. Forecasting teams evaluate with rolling-origin backtests, the time-based folds from Figure 1 ( Backtesting and forecast evaluation ). Medical imaging studies split by patient, because several scans of one patient on both sides would inflate results.
Check yourself
-
What is the validation set for?
Show answer
B. Comparing models and tuning choices like features and hyperparameters Validation guides decisions. The test set is kept separate so that the final estimate is not biased by those decisions.
-
You predict next month's sales from three years of history. How should you split?
Show answer
C. Train on earlier months, validate and test on later ones In production you only have the past. A time-based split reproduces that; a random split lets future information leak into training.
-
A dataset has 20,000 chest X-rays from 3,000 patients. What split should you use?
Show answer
B. Group by patient Images of the same patient are highly similar. Splitting by patient checks that the model works on new patients, which is the real use.
-
You tried 40 model variants and picked the one with the best test accuracy, 91 percent. What is the problem?
Show answer
B. The test set was used for selection, so 91 percent is optimistic Picking the best of many on the test set fits your choice to that set's noise. Select on validation and use the test once.
Recap
- Train to learn, validate to decide, test once for an honest final estimate.
- Stratify to keep class balance, especially with rare classes or small datasets.
- Group by user, patient or item so the same entity never sits on both sides.
- Split by time for anything that predicts the future.
- Every look at the test set that changes a decision uses it up a little.
Next up
- From notebook to production learning path