Skip to content
learn
  1. Learn
  2. Machine Learning
  3. Model Evaluation and Selection

Machine Learning ML-05.01

Bias, variance, underfitting and overfitting

Why models fail in two opposite ways, how to tell which one from training and validation error, and the right remedy for each.

Beginner 14 min Last reviewed

The big idea

A model can fail in two opposite ways. It can be too simple to capture the pattern, like fitting a straight line to a curve: it is wrong on the training data and wrong on new data. That is underfittingWhen a model is too simple to capture the real pattern, so it performs poorly even on its training data. Caused by high bias.. Or it can be so flexible that it memorises the noise in its training data, like a student who memorises past exam answers without understanding them: perfect on the training data, poor on anything new. That is overfittingWhen a model fits noise in the training data, performing well on it but poorly on new data. Caused by high variance..

Think of a dartboard. BiasSystematic error: how far the model's average prediction, over many possible training sets, is from the truth. is how far your throws land from the bullseye on average: a consistent aim error. VarianceHow much the model's predictions change when it is trained on a different sample of data. is how scattered the throws are around their own centre. You want both small, but the two usually pull against each other.

Bias and variance as darts Four dartboards, each with concentric rings and a green bullseye. Gold darts: on the first board they cluster tightly on the bullseye (low bias, low variance). On the second they are spread widely around the bullseye (low bias, high variance). On the third they cluster tightly but up and to the right of the bullseye (high bias, low variance). On the fourth they are spread widely and also off-centre (high bias, high variance). low bias, low variance low bias, high variance high bias, low variance high bias, high variance Figure 1. Each dart is one model trained on a different sample of data. Bias is the distance from the darts' centre to the bullseye; variance is how scattered the darts are. Underfit models live on the third board, overfit models on the second.

How it works

Step 1: Capacity drives the trade-off

A model's capacityHow flexible a model is: how many different patterns it can represent. More parameters, deeper trees or higher polynomial degree mean more capacity. is how complex a pattern it can represent: polynomial degree, tree depth, number of neural network weights. As capacity grows:

  • Bias falls. A flexible model can bend to follow the real pattern.
  • Variance rises. A flexible model also bends to follow the noise, so it changes a lot from one training sample to the next.

Expected error on new data splits into three parts:

The noise is the randomness in the data that no model can predict. The other two are yours to trade. In the code below, fitting 500 different training sets of 30 points: a straight line has bias squared 0.068 and variance 0.013; a cubic has bias squared 0.001 and variance 0.013, the best total; a degree-8 polynomial has bias squared 0.000 but variance 0.231.

Step 2: Diagnose with training and validation error

You cannot see bias and variance directly on a real problem, because you have one dataset and do not know the truth. But you can see their symptoms by comparing training error with validation error:

  • Both high, close together: underfitting (high bias). The model cannot even fit the data it has seen.
  • Training low, validation much higher: overfitting (high variance). The gap is the memorised noise.
  • Both low, close together: the sweet spot.
Training and validation error against model capacity Mean squared error against polynomial degree for a fit to 20 points. Training error falls steadily: 0.744, 0.057, 0.025, 0.023, 0.020 for degrees 0, 1, 3, 5 and 9. Validation error falls from 0.979 to a minimum of 0.123 at degree 3, then rises to 0.129 and 0.151, and explodes to 493.8 at degree 15. The left region is shaded as underfit and the right region as overfit. 01234567891000.250.50.751model capacity (polynomial degree)mean squared error underfit overfit sweet spot training error validation error 493.8 at degree 15 Figure 2. Numbers from the code below. Training error only goes down as capacity grows; validation error falls, bottoms out at degree 3, then climbs. The widening gap between the curves is the signature of overfitting.

Step 3: Remedies for each

They need opposite medicine, which is why the diagnosis matters.

For underfitting (high bias):

  • Use a more flexible model or add capacity (higher degree, deeper trees, more layers).
  • Add better features, including interactions ( Feature engineering ).
  • Reduce regularization.
  • Train longer, if training stopped early.

For overfitting (high variance):

More data helps variance but does nothing for bias: a straight line fitted to a million points from a curve is still a straight line.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Degree 0 or 1: the twenty fits agree with each other (low variance) but their gold average misses the green curve (high bias).
  2. Degree 9 with 12 points: the average is close to the truth, but individual fits fly all over the place. That spread is variance.
  3. Keep degree 9 and raise the points per set to 120. Variance collapses while bias stays low: more data is the cure for variance.

Going deeper

Code: the U-curve and the decomposition

bias-variance.pyruns with python3 + numpy
import numpy as np

rng = np.random.default_rng(0)
truth = lambda x: np.sin(2.6 * x) + 0.3 * x
noise_sd = 0.3

def sample(n):
    x = rng.uniform(-1, 1, n)
    return x, truth(x) + rng.normal(0, noise_sd, n)

# Train vs validation error as model capacity grows (one training set of 20 points)
xtr, ytr = sample(20)
xva, yva = sample(1000)
print("degree  train MSE  valid MSE")
for d in [0, 1, 3, 5, 9, 15]:
    c = np.polynomial.chebyshev.chebfit(xtr, ytr, d)
    tr = np.mean((np.polynomial.chebyshev.chebval(xtr, c) - ytr) ** 2)
    va = np.mean((np.polynomial.chebyshev.chebval(xva, c) - yva) ** 2)
    print(f"{d:>6}  {tr:9.3f}  {va:9.3f}")

# Bias and variance by brute force: 500 training sets of 30 points each
grid = np.linspace(-0.9, 0.9, 50)
print("\ndegree  bias^2  variance  noise  expected test MSE")
for d in [1, 3, 8]:
    preds = np.array([np.polynomial.chebyshev.chebval(grid, np.polynomial.chebyshev.chebfit(*sample(30), d))
                      for _ in range(500)])
    bias2 = np.mean((preds.mean(axis=0) - truth(grid)) ** 2)
    var = np.mean(preds.var(axis=0))
    print(f"{d:>6}  {bias2:6.3f}  {var:8.3f}  {noise_sd**2:5.3f}  {bias2 + var + noise_sd**2:8.3f}")

Output

degree  train MSE  valid MSE
     0      0.744      0.979
     1      0.057      0.229
     3      0.025      0.123
     5      0.023      0.129
     9      0.020      0.151
    15      0.012    493.813

degree  bias^2  variance  noise  expected test MSE
     1   0.068     0.013  0.090     0.171
     3   0.001     0.013  0.090     0.103
     8   0.000     0.231  0.090     0.321

The first table is what you would see on a real project: one training set, one validation set. The second is what you can only do in a simulation, because the true curve is known: train on 500 fresh samples and measure bias and variance directly. Note that degree 15 on 20 points has a tiny training error (0.012) and a catastrophic validation error (493.8): with almost as many parameters as points, it threads every training point and swings wildly between them.

Beyond the U-curve

Very large neural networks sometimes break the simple picture: past the point where they can fit the training data perfectly, validation error can fall again as capacity grows further, a phenomenon called double descent. Regularization, the training procedure and the amount of data all shape where that happens. For classic models and most tabular work, the U-curve remains the right mental model, and the diagnosis in Step 2 always applies. Learning curves and baselines adds learning curves, the other diagnostic tool.

Trade-offs

OptionWins whenLoses when
More capacity Training and validation error are both high (underfitting) There is already a big gap between them
More data A large train-validation gap (overfitting) and data is obtainable The model underfits; data alone will not fix bias
Regularization Overfitting when more data is not available The model already underfits
Better features Underfitting because the signal is not in the inputs Adding many noisy features to an overfit model

Common mistakes

  • Looking only at training error and declaring success.
  • Collecting more data for a model that underfits; it will not help.
  • Tuning capacity on the test set, which makes the test set part of training ( Train, validation and test splits ).
  • Treating a small validation set's noise as a real difference between models.
  • Assuming a bigger model is always better; it is only better if you have the data to feed it.

In the wild

The train-versus-validation diagnosis is the first thing practitioners check when a model disappoints, and it shapes most day-to-day decisions: whether to spend a week labelling more data, engineering features, or tuning regularization. Training dashboards for deep learning plot training and validation loss together for exactly this reason, and early stopping, which halts training when validation loss starts to rise, is a direct application of the U-curve.

Check yourself

  1. Training error 2 percent, validation error 19 percent. What is the problem and the first fix?

    Show answer

    B. Overfitting; get more data or add regularization A large gap between training and validation error is the signature of high variance. More data, regularization or less capacity reduce it.

  2. Training error 18 percent, validation error 19 percent, and humans manage 2 percent. What should you do?

    Show answer

    B. Increase capacity or add better features Both errors are high and close: the model underfits. More data will not fix bias; a more expressive model or better features will.

  3. As you add capacity to a model, what usually happens?

    Show answer

    B. Bias falls and variance rises Flexibility lets the model follow the true pattern (less bias) but also the noise (more variance).

  4. Which part of the expected error can no model remove?

    Show answer

    C. Irreducible noise Noise is randomness in the labels themselves. Even the true function makes that error on new data.

Recap

  • Underfitting is high bias: poor on training and validation data alike.
  • Overfitting is high variance: good on training data, much worse on validation data.
  • Expected test error = bias squared + variance + irreducible noise; capacity trades the first two.
  • Diagnose with the gap between training and validation error before choosing a remedy.
  • More data and regularization cure variance; more capacity and better features cure bias.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up