Learning curves and baselines

The big idea

A team spends three weeks on a neural network and reports a mean absolute error of 2.0. Is that good? Without a reference point, nobody knows. If predicting last week's average already gives 2.1, three weeks bought almost nothing. If it gives 4.0, the model halved the error.

A baselineA deliberately simple model, such as predicting the mean, the majority class or a one-feature rule, used as the reference that any real model must beat. is that reference point: the simplest thing that could possibly work. And once you have a model, a learning curveA plot of training and validation performance against the number of training examples. tells you what to do next. It answers a question that saves months: will more data help, or do I need a better model? Like a runner's training log, it shows not just where you are but whether more of the same will get you further.

How it works

Step 1: Always start with a dumb baseline

Before any real model, measure:

• Regression: predict the training mean (or median) for everyone.
• Classification: predict the most frequent class, or random guesses in class proportions.
• A one-feature rule: the single most obvious signal, such as "churn if no login in 30 days" or "price = 3,000 per square metre".
• The current system: if a heuristic or an older model is in production, that is the baseline that matters.

Baselines catch bugs (a model worse than the mean is broken, or leaking, if suspiciously good), set expectations, and sometimes win outright: a simple rule that is 95 percent as good may be the right thing to ship.

Step 2: Draw the learning curve

Train the same model on growing subsets of the training data, say 100, 400, 1,600 and 3,200 examples, and record training and validation error at each size (cross-validated, so the points are not noise). Plot both against training set size, usually on a log scale.

Step 3: Read the shape

• Both curves flat and close together, far above an acceptable error: high bias. The model has learned all it can. More data is wasted; use a more flexible model, add features, or reduce regularization.
• A big gap, validation still falling: high variance. Extrapolate the slope: more data will probably help. Regularization or a simpler model also narrows the gap.
• Both curves flat, close together, near the noise floor: you are done with this data. Only better features or less noisy labels can help.

The noise floor is the error even a perfect model would make. Here the labels have noise with standard deviation 1.0, which means a mean absolute error of about 0.80 is unavoidable. Human performance or the agreement between two human labellers is a common proxy for it.

Step 4: Use them together

Baselines tell you whether you are making progress at all. Learning curves tell you where the next unit of effort should go: collecting and labelling data (often expensive), engineering features, or changing the model. The forest in Figure 1 beats both models in Figure 2 because it combines the flexibility of trees with the variance reduction of averaging ( Boosting: AdaBoost to gradient boosting ).

Going deeper

Code: baselines and two learning curves

import numpy as np
from sklearn.datasets import make_friedman1
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import learning_curve, cross_val_score, KFold

# A nonlinear target with noise, 10 features (5 useful)
X, y = make_friedman1(n_samples=4000, n_features=10, noise=1.0, random_state=0)
cv = KFold(5, shuffle=True, random_state=0)
mae = lambda m, Xs=X: -cross_val_score(m, Xs, y, cv=cv, scoring="neg_mean_absolute_error").mean()

print("baselines first (cross-validated MAE, lower is better)")
print(f"  predict the mean          {mae(DummyRegressor()):.2f}")
print(f"  linear on one feature     {mae(LinearRegression(), X[:, [3]]):.2f}")
print(f"  linear on all features    {mae(LinearRegression()):.2f}")
print(f"  random forest             {mae(RandomForestRegressor(n_estimators=100, random_state=0, n_jobs=-1)):.2f}")

sizes = [100, 400, 1600, 3200]
for name, model in [("linear regression (high bias)", LinearRegression()),
                    ("deep decision tree (high variance)", DecisionTreeRegressor(random_state=0))]:
    n, tr, va = learning_curve(model, X, y, train_sizes=sizes, cv=cv,
                               scoring="neg_mean_absolute_error")
    print(f"\n{name}")
    print("  train size   train MAE   valid MAE")
    for k in range(len(n)):
        print(f"  {n[k]:>10}   {abs(tr[k].mean()):9.2f}   {-va[k].mean():9.2f}")

baselines first (cross-validated MAE, lower is better)
  predict the mean          4.08
  linear on one feature     3.40
  linear on all features    2.02
  random forest             1.32

linear regression (high bias)
  train size   train MAE   valid MAE
         100        2.22        2.23
         400        2.06        2.03
        1600        1.99        2.03
        3200        2.01        2.02

deep decision tree (high variance)
  train size   train MAE   valid MAE
         100        0.00        3.41
         400        0.00        2.70
        1600        0.00        2.21
        3200        0.00        1.98

learning_curve does the bookkeeping: for each training size it runs cross-validation and returns training and validation scores. scikit-learn reports errors as negative scores ("higher is better"), hence the minus signs in the code.

Learning curves over training time

Deep learning practitioners also plot loss against training steps or epochs, which is a different curve with a similar reading: training loss falling while validation loss rises means overfitting has begun (the point where early stopping kicks in). Do not confuse the two: one varies the amount of data, the other the amount of training on fixed data.

Trade-offs

Common mistakes

• Skipping the baseline and celebrating an accuracy that the majority class already achieves.
• Collecting more data for a model whose learning curves are already flat.
• Plotting learning curves from single splits, so noise looks like a trend.
• Comparing against a baseline evaluated on different data or a different metric.
• Forgetting the noise floor and chasing improvements that are impossible.

In the wild

Experienced teams treat "beat the baseline" as the first milestone of any ML project, and production launches are usually judged against the current heuristic or model rather than against zero. Labelling budgets are commonly justified with learning curves: if validation error is still falling steeply with data, buying more labels is the cheapest improvement available; if it has flattened, the money is better spent on features or modelling.

Check yourself

Recap