Machine Learning ML-05.01
Bias, variance, underfitting and overfitting
Why models fail in two opposite ways, how to tell which one from training and validation error, and the right remedy for each.
The big idea
A model can fail in two opposite ways. It can be too simple to capture the pattern, like fitting a straight line to a curve: it is wrong on the training data and wrong on new data. That is underfittingWhen a model is too simple to capture the real pattern, so it performs poorly even on its training data. Caused by high bias.. Or it can be so flexible that it memorises the noise in its training data, like a student who memorises past exam answers without understanding them: perfect on the training data, poor on anything new. That is overfittingWhen a model fits noise in the training data, performing well on it but poorly on new data. Caused by high variance..
Think of a dartboard. BiasSystematic error: how far the model's average prediction, over many possible training sets, is from the truth. is how far your throws land from the bullseye on average: a consistent aim error. VarianceHow much the model's predictions change when it is trained on a different sample of data. is how scattered the throws are around their own centre. You want both small, but the two usually pull against each other.
How it works
Step 1: Capacity drives the trade-off
A model's capacityHow flexible a model is: how many different patterns it can represent. More parameters, deeper trees or higher polynomial degree mean more capacity. is how complex a pattern it can represent: polynomial degree, tree depth, number of neural network weights. As capacity grows:
- Bias falls. A flexible model can bend to follow the real pattern.
- Variance rises. A flexible model also bends to follow the noise, so it changes a lot from one training sample to the next.
Expected error on new data splits into three parts:
The noise is the randomness in the data that no model can predict. The other two are yours to trade. In the code below, fitting 500 different training sets of 30 points: a straight line has bias squared 0.068 and variance 0.013; a cubic has bias squared 0.001 and variance 0.013, the best total; a degree-8 polynomial has bias squared 0.000 but variance 0.231.
Step 2: Diagnose with training and validation error
You cannot see bias and variance directly on a real problem, because you have one dataset and do not know the truth. But you can see their symptoms by comparing training error with validation error:
- Both high, close together: underfitting (high bias). The model cannot even fit the data it has seen.
- Training low, validation much higher: overfitting (high variance). The gap is the memorised noise.
- Both low, close together: the sweet spot.
Step 3: Remedies for each
They need opposite medicine, which is why the diagnosis matters.
For underfitting (high bias):
- Use a more flexible model or add capacity (higher degree, deeper trees, more layers).
- Add better features, including interactions ( Feature engineering ).
- Reduce regularization.
- Train longer, if training stopped early.
For overfitting (high variance):
- Get more training data. The single most reliable fix.
- Add regularization ( Regularization: ridge, lasso, elastic net ), dropout or early stopping.
- Reduce capacity or remove noisy features.
- Average many models: ensembles such as random forests cut variance ( Boosting: AdaBoost to gradient boosting ).
More data helps variance but does nothing for bias: a straight line fitted to a million points from a curve is still a straight line.
Try it
Try this:
- Degree 0 or 1: the twenty fits agree with each other (low variance) but their gold average misses the green curve (high bias).
- Degree 9 with 12 points: the average is close to the truth, but individual fits fly all over the place. That spread is variance.
- Keep degree 9 and raise the points per set to 120. Variance collapses while bias stays low: more data is the cure for variance.
Going deeper
Code: the U-curve and the decomposition
import numpy as np
rng = np.random.default_rng(0)
truth = lambda x: np.sin(2.6 * x) + 0.3 * x
noise_sd = 0.3
def sample(n):
x = rng.uniform(-1, 1, n)
return x, truth(x) + rng.normal(0, noise_sd, n)
# Train vs validation error as model capacity grows (one training set of 20 points)
xtr, ytr = sample(20)
xva, yva = sample(1000)
print("degree train MSE valid MSE")
for d in [0, 1, 3, 5, 9, 15]:
c = np.polynomial.chebyshev.chebfit(xtr, ytr, d)
tr = np.mean((np.polynomial.chebyshev.chebval(xtr, c) - ytr) ** 2)
va = np.mean((np.polynomial.chebyshev.chebval(xva, c) - yva) ** 2)
print(f"{d:>6} {tr:9.3f} {va:9.3f}")
# Bias and variance by brute force: 500 training sets of 30 points each
grid = np.linspace(-0.9, 0.9, 50)
print("\ndegree bias^2 variance noise expected test MSE")
for d in [1, 3, 8]:
preds = np.array([np.polynomial.chebyshev.chebval(grid, np.polynomial.chebyshev.chebfit(*sample(30), d))
for _ in range(500)])
bias2 = np.mean((preds.mean(axis=0) - truth(grid)) ** 2)
var = np.mean(preds.var(axis=0))
print(f"{d:>6} {bias2:6.3f} {var:8.3f} {noise_sd**2:5.3f} {bias2 + var + noise_sd**2:8.3f}")Output
degree train MSE valid MSE
0 0.744 0.979
1 0.057 0.229
3 0.025 0.123
5 0.023 0.129
9 0.020 0.151
15 0.012 493.813
degree bias^2 variance noise expected test MSE
1 0.068 0.013 0.090 0.171
3 0.001 0.013 0.090 0.103
8 0.000 0.231 0.090 0.321 The first table is what you would see on a real project: one training set, one validation set. The second is what you can only do in a simulation, because the true curve is known: train on 500 fresh samples and measure bias and variance directly. Note that degree 15 on 20 points has a tiny training error (0.012) and a catastrophic validation error (493.8): with almost as many parameters as points, it threads every training point and swings wildly between them.
Beyond the U-curve
Very large neural networks sometimes break the simple picture: past the point where they can fit the training data perfectly, validation error can fall again as capacity grows further, a phenomenon called double descent. Regularization, the training procedure and the amount of data all shape where that happens. For classic models and most tabular work, the U-curve remains the right mental model, and the diagnosis in Step 2 always applies. Learning curves and baselines adds learning curves, the other diagnostic tool.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| More capacity | Training and validation error are both high (underfitting) | There is already a big gap between them |
| More data | A large train-validation gap (overfitting) and data is obtainable | The model underfits; data alone will not fix bias |
| Regularization | Overfitting when more data is not available | The model already underfits |
| Better features | Underfitting because the signal is not in the inputs | Adding many noisy features to an overfit model |
Common mistakes
- Looking only at training error and declaring success.
- Collecting more data for a model that underfits; it will not help.
- Tuning capacity on the test set, which makes the test set part of training ( Train, validation and test splits ).
- Treating a small validation set's noise as a real difference between models.
- Assuming a bigger model is always better; it is only better if you have the data to feed it.
In the wild
The train-versus-validation diagnosis is the first thing practitioners check when a model disappoints, and it shapes most day-to-day decisions: whether to spend a week labelling more data, engineering features, or tuning regularization. Training dashboards for deep learning plot training and validation loss together for exactly this reason, and early stopping, which halts training when validation loss starts to rise, is a direct application of the U-curve.
Check yourself
-
Training error 2 percent, validation error 19 percent. What is the problem and the first fix?
Show answer
B. Overfitting; get more data or add regularization A large gap between training and validation error is the signature of high variance. More data, regularization or less capacity reduce it.
-
Training error 18 percent, validation error 19 percent, and humans manage 2 percent. What should you do?
Show answer
B. Increase capacity or add better features Both errors are high and close: the model underfits. More data will not fix bias; a more expressive model or better features will.
-
As you add capacity to a model, what usually happens?
Show answer
B. Bias falls and variance rises Flexibility lets the model follow the true pattern (less bias) but also the noise (more variance).
-
Which part of the expected error can no model remove?
Show answer
C. Irreducible noise Noise is randomness in the labels themselves. Even the true function makes that error on new data.
Recap
- Underfitting is high bias: poor on training and validation data alike.
- Overfitting is high variance: good on training data, much worse on validation data.
- Expected test error = bias squared + variance + irreducible noise; capacity trades the first two.
- Diagnose with the gap between training and validation error before choosing a remedy.
- More data and regularization cure variance; more capacity and better features cure bias.
Next up
- From notebook to production learning path