Machine Learning ML-05.02
Cross-validation
Rotate the validation fold for a reliable score: k-fold, stratified, group and time-series splits, and nested cross-validation for honest tuning.
The big idea
You hold out 20 percent of your data, train on the rest, and get 96 percent accuracy. Is the model a 96 percent model? Maybe. Shuffle differently and the same model might score 99 percent, or 95. On 569 tumour samples in the code below, twenty different random 80/20 splits give accuracies anywhere from 0.947 to 0.991. A single split is one roll of the dice, and on small datasets the dice are loud.
Cross-validationEstimating model performance by splitting the data into k parts, training k times on all but one part and validating on the remaining one, then averaging. rolls the dice several times in a structured way. Split the data into k equal parts, called foldsOne of the k equal parts the data is split into for cross-validation.. Train on k - 1 folds and validate on the one left out, then rotate until every fold has been the validation set exactly once. Average the k scores. It is like judging a chef on five different dishes instead of one: the average is fairer, and the spread tells you how consistent they are.
How it works
Step 1: k-fold
With k = 5, each model trains on 80 percent of the data and you get five scores. Report the mean and the standard deviation: "0.979 plus or minus 0.014" is far more honest than "0.982". Common choices are k = 5 or 10: more folds means more training data per model and less pessimistic estimates, but k times the compute.
Once cross-validation has told you how good the approach is, retrain the final model on all the training data. The five fold models were only measuring instruments.
Step 2: Stratified folds for classification
If positives are 10 percent of the data, a random fold might get 4 percent or 16 percent, and the scores swing for no reason. Stratified k-foldCross-validation where each fold keeps the same class proportions as the full dataset. builds folds that keep the class proportions of the whole dataset. Use it by default for classification, and always for imbalanced data ( Imbalanced data ).
Step 3: Group folds when rows are related
Suppose you have 10 scans from each of 100 patients. If scans from the same patient land in both training and validation folds, the model can recognise the patient instead of learning the disease. Validation looks superb and real performance on new patients is not. Group k-foldCross-validation that keeps all rows sharing a group id (such as a patient, user or store) in the same fold. keeps every patient's rows together in one fold.
The code below builds exactly this trap with labels that are pure noise. Plain k-fold reports 0.985 accuracy; group k-fold reports 0.509, the coin flip it really is. Whenever a single entity (user, patient, device, store, document) contributes several rows, ask: "will the model see this entity at prediction time?" If not, group by it.
Step 4: Time-series folds
For data ordered in time, random folds let the model train on the future and validate on the past, which never happens in production. Time-series cross-validationCross-validation for ordered data where each round trains on an initial stretch of time and validates on the period right after it. uses an expanding window: train on months 0 to 3 and validate on 4 and 5, then train on 0 to 5 and validate on 6 and 7, and so on.
This mirrors deployment: you always forecast forward. It also reveals whether performance decays as the gap between training and prediction grows. Backtesting and forecast evaluation covers backtesting for forecasts in depth.
Step 5: Nested cross-validation for tuning
If you try 50 hyperparameter settings and pick the one with the best cross-validated score, that best score is optimistic: you selected it partly for its luck. Nested cross-validationCross-validation with two loops: an inner loop that tunes hyperparameters and an outer loop that estimates the performance of the whole tuning procedure. wraps the whole tuning procedure in an outer loop. The inner loop picks hyperparameters using only the outer training folds; the outer fold, never seen by the tuning, scores the result. In the code, nested CV gives 0.974 plus or minus 0.012, a little below the untuned 0.979: that small gap is the optimism you avoid. With plenty of data, a single held-out test set does the same job more cheaply ( Hyperparameter tuning ).
Going deeper
Code: every flavour of cross-validation
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import (KFold, StratifiedKFold, GroupKFold, TimeSeriesSplit,
cross_val_score, train_test_split, GridSearchCV)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
X, y = load_breast_cancer(return_X_y=True) # 569 tumours, built into scikit-learn
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
# One split is a noisy estimate: try 20 different random 80/20 splits
single = [model.fit(Xa, ya).score(Xb, yb) for Xa, Xb, ya, yb in
(train_test_split(X, y, test_size=0.2, random_state=s) for s in range(20))]
print(f"20 single splits: min {min(single):.3f} max {max(single):.3f}")
scores = cross_val_score(model, X, y, cv=StratifiedKFold(5, shuffle=True, random_state=0))
print("5-fold scores:", scores.round(3), f"mean {scores.mean():.3f} +/- {scores.std():.3f}")
# Grouped data: 100 patients, 10 scans each. The label depends on the patient, not the scan.
rng = np.random.default_rng(0)
patient = np.repeat(np.arange(100), 10)
pid_signature = rng.normal(size=(100, 5))[patient] + 0.1 * rng.normal(size=(1000, 5))
label = rng.integers(0, 2, 100)[patient] # random: there is nothing real to learn
rf = RandomForestClassifier(n_estimators=100, random_state=0)
leaky = cross_val_score(rf, pid_signature, label, cv=KFold(5, shuffle=True, random_state=0))
honest = cross_val_score(rf, pid_signature, label, cv=GroupKFold(5), groups=patient)
print(f"\nKFold (same patient in train and test): {leaky.mean():.3f}")
print(f"GroupKFold (patients kept together): {honest.mean():.3f}")
# Time series: always train on the past, validate on the future
print("\nTimeSeriesSplit on 12 months:")
for tr, va in TimeSeriesSplit(n_splits=4).split(np.arange(12)):
print(f" train months {tr.min()}-{tr.max()} validate {va.min()}-{va.max()}")
# Nested CV: tune C inside, estimate outside
inner = GridSearchCV(model, {"logisticregression__C": [0.01, 0.1, 1, 10]}, cv=5)
outer = cross_val_score(inner, X, y, cv=StratifiedKFold(5, shuffle=True, random_state=1))
print(f"\nnested CV accuracy {outer.mean():.3f} +/- {outer.std():.3f}")Output
20 single splits: min 0.947 max 0.991
5-fold scores: [0.956 0.974 0.982 1. 0.982] mean 0.979 +/- 0.014
KFold (same patient in train and test): 0.985
GroupKFold (patients kept together): 0.509
TimeSeriesSplit on 12 months:
train months 0-3 validate 4-5
train months 0-5 validate 6-7
train months 0-7 validate 8-9
train months 0-9 validate 10-11
nested CV accuracy 0.974 +/- 0.012 Two details matter in practice. First, the scaler sits inside the pipeline, so it is refitted on each training fold; scaling all data before cross-validation leaks validation statistics into training ( Data leakage ). Second, the group example uses features that identify the patient and labels that are random: there is nothing to learn, and only group folds reveal it.
What cross-validation does not fix
Cross-validation estimates performance on data drawn the same way as your dataset. It cannot warn you that next year's customers differ from last year's, or that your labels are biased. It also multiplies training cost by k, which is why deep learning projects with large datasets usually rely on a single fixed validation set instead.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Single hold-out split | Large datasets or expensive models (deep learning) | Small datasets: the estimate is noisy |
| Stratified k-fold | Most classification on small to medium data | Rows are grouped or ordered in time |
| Group k-fold | Several rows per user, patient or store | Few groups: folds become unbalanced |
| Time-series split | Forecasting and any time-ordered prediction | Data with no time dependence (it wastes early data) |
| Nested CV | Small data where you need an unbiased estimate after tuning | Compute budget: k_outer x k_inner x settings fits |
Common mistakes
- Preprocessing (scaling, imputation, feature selection) on all data before cross-validation.
- Shuffled k-fold on time series, letting the model peek at the future.
- Ignoring groups, so the same user appears in training and validation folds.
- Reporting the best of many cross-validated configurations as the expected performance.
- Treating the fold models as the final model instead of retraining on all training data.
In the wild
Kaggle competitors rely on carefully designed cross-validation that mimics how the hidden test set was split, because public leaderboard scores on a small sample are noisy and easy to overfit. In industry, group-aware and time-aware validation are standard wherever data comes from repeat users or evolves over time, such as recommendations, credit and demand forecasting. Medical imaging studies split by patient for exactly the reason in Step 3.
Check yourself
-
Why is 5-fold cross-validation more reliable than one 80/20 split on 500 rows?
Show answer
B. It averages five estimates, each on different validation data, reducing the luck of one split One split can be lucky or unlucky. Averaging over folds that together cover every row gives a steadier estimate and a measure of its spread.
-
A churn dataset has one row per customer per month. Which split should you use?
Show answer
B. Group k-fold by customer, or a time-based split Rows from the same customer are related. Group by customer so the model is judged on unseen customers, and respect time if you predict future months.
-
You standardize all features, then run 10-fold cross-validation. What is wrong?
Show answer
B. The scaler saw the validation folds, leaking their statistics into training Every preprocessing step that learns from data must be fitted inside each training fold, for example with a pipeline.
-
You tried 200 settings and the best had CV accuracy 0.91. What does nested CV estimate?
Show answer
B. The performance of the whole procedure 'tune, then pick the best', usually a bit lower Selecting the best of 200 rewards luck. The outer loop scores the tuning procedure on data it never saw.
Recap
- Cross-validation trains k times, each time validating on a different fold, and averages the scores.
- Report the mean and the spread; a single split on small data is a noisy estimate.
- Stratify for classification, group by entity when rows are related, and respect time order for temporal data.
- Fit all preprocessing inside each training fold, ideally with a pipeline.
- Nested cross-validation gives an honest estimate after hyperparameter tuning.
Next up
- From notebook to production learning path