Backtesting and forecast evaluation

The big idea

Your team has built a demand forecast. Every Monday it will predict the next 14 days, planners will order stock from it, and next Monday it runs again with one more week of data. The question is not "how well does the model fit the past?" but "how good were its Monday forecasts, had we used it all last year?"

That is a backtestEvaluating a forecasting method by replaying history: train on data up to a past date, forecast the following period, compare with what happened, then move the date forward and repeat.: you replay history as if you were living through it. Stand at some past date, use only what was known then, forecast, then look at what actually happened. Step forward, repeat. It is like testing a trading strategy on last year's prices, but only letting it see each day's newspaper on that day.

Two more things make the evaluation honest. Errors should be reported per forecast horizonHow far ahead a forecast is made, for example 1 day or 14 days after the last observed data point., because predicting tomorrow and predicting two weeks out are different problems. And a forecast should come with a range, not just a number, because the planner ordering stock needs to know how wrong it might be.

How it works

Step 1: Rolling-origin evaluation

Ordinary k-fold cross-validation ( Cross-validation ) shuffles rows into folds. On a time series that puts the future into the training set and tests on the past, a leak ( Data leakage ). The forecasting version keeps time in order.

Pick a forecast originThe last time point whose data a forecast may use; the forecast starts right after it., a date in the past. Train on everything before it, forecast the next h steps, and record the errors. Then move the origin forward and do it again. The forecasting textbook by Hyndman and Athanasopoulos calls this "evaluation on a rolling forecasting origin" because the origin "rolls forward in time", and computes accuracy by averaging over all the test sets.

Choices to make:

• Expanding or sliding window. Expanding uses all history and matches most production setups. Sliding keeps a fixed length, so old regimes stop pulling on the model.
• Step between origins. Moving the origin by one day gives the most folds but costs a refit each time. Weekly origins that match the real retraining schedule are a common compromise.
• A gap. If production data arrives two days late, leave a two-day gap between the end of training and the forecast window. scikit-learn's TimeSeriesSplit has gap and max_train_size parameters for exactly this; its successive training sets are supersets of the earlier ones.
• Refit or not. Refitting the model at every origin is the most faithful. If that is too slow, refit occasionally and only update the data in between, as production would.

Step 2: Metrics, per horizon

Each fold gives an error for every horizon step. Arrange them as a table, folds by horizons, and average down the columns. The result is one number per horizon: how wrong you are, on average, 1 day out, 7 days out, 14 days out. Errors usually grow with the horizon, and the curve tells you how far ahead the forecast is worth trusting.

The common point-forecast metrics, with their catch:

• MAE, the mean absolute error, is in the units of the data and easy to explain. A forecast that minimizes MAE aims at the median.
• RMSE punishes large misses more. Minimizing it aims at the mean.
• MAPE, the mean absolute percentage error, is unit-free, but it is undefined when the actual value is 0, explodes near 0, and treats over- and under-forecasts differently. Avoid it for intermittent demand.
• MASE (mean absolute scaled errorMean absolute scaled error: the forecast's MAE divided by the in-sample MAE of a naive or seasonal naive forecast, so values below 1 beat that baseline.) divides the forecast's errors by the in-sample MAE of a naive forecast (for seasonal data, the seasonal naive forecast). Below 1 means you beat the baseline; above 1 means you did not. Because it is unit-free, you can average it across series of very different sizes.

Here $m$ is the season length (7 for daily data with a weekly pattern) and the denominator is computed on the training data. Worked example: if the seasonal naive forecast misses by 5.6 units a day on average in training and your model's backtest MAE is 4.2, then MASE = 4.2 / 5.6 = 0.75: a quarter less error than the baseline.

Step 3: Prediction intervals

A prediction intervalA range that should contain the future value with a stated probability, such as 80 percent, given the information at the forecast origin. says "with 80 percent probability, demand will be between 85 and 115". Two properties matter:

• Coverage. Over many forecasts, about 80 percent of actual values should land inside an 80 percent interval. The backtest is how you check: count how often they did.
• Width. Among intervals with the right coverage, narrower is more useful. An interval of 0 to infinity always covers and tells you nothing.

Where intervals come from:

• The model. ARIMA and exponential smoothing give intervals from their error model; Prophet from its trend and noise model ( Prophet and modern forecasting libraries ). They assume the model is right, so they are often too narrow.
• Quantile models. Train one model per quantile, for example the 10th and 90th percentiles, with the pinball lossAn asymmetric loss for quantile forecasts: under-forecasts cost p times the miss and over-forecasts cost (1 - p) times the miss (up to a constant factor), so minimizing it gives the p-th quantile.. Gradient boosting libraries support this loss directly.
• Backtest errors. Take the errors your backtest collected at each horizon and use their 10th and 90th percentiles as the interval around new forecasts. This makes no assumption about the model at all, as long as the future behaves like the backtested past.

To score whole intervals in one number, the Winkler scoreAn interval score: the interval's width plus a penalty of 2/alpha times the distance by which the actual value falls outside it. is the width of the interval plus a penalty of $2/\alpha$ times how far the actual falls outside it, where $\alpha =0.2$ for an 80 percent interval. Lower is better, so it rewards narrow intervals and punishes misses.

Going deeper

Code: a backtest with per-horizon errors and intervals

import warnings; warnings.filterwarnings("ignore")
import numpy as np
from statsmodels.tsa.holtwinters import ExponentialSmoothing

# Two years of synthetic daily demand: level that wanders, weekly shape, noise.
rng = np.random.default_rng(3)
T, H, m = 730, 14, 7
level = 100 + np.cumsum(rng.normal(0.05, 0.8, T))
y = level + 12 * np.sin(2 * np.pi * np.arange(T) / m) + rng.normal(0, 4, T)

def snaive(train):   # repeat the last observed week
    return np.resize(train[-m:], H)
def holt_winters(train):
    fit = ExponentialSmoothing(train, trend="add", seasonal="add", seasonal_periods=m).fit()
    return fit.forecast(H)

origins = range(365, T - H + 1, 7)         # expanding window: train on y[:o], test y[o:o+H]
err = {"seasonal naive": [], "Holt-Winters": []}
for o in origins:
    for name, f in (("seasonal naive", snaive), ("Holt-Winters", holt_winters)):
        err[name].append(y[o:o + H] - f(y[:o]))
scale = np.mean(np.abs(y[m:365] - y[:365 - m]))   # in-sample seasonal naive MAE (for MASE)

print(f"{len(origins)} forecast origins, horizon 1..{H} days")
print(f"{'':16s} {'h=1':>6s} {'h=7':>6s} {'h=14':>6s}   MASE")
for name, e in err.items():
    mae_h = np.mean(np.abs(e), axis=0)          # one MAE per horizon step
    print(f"{name:16s} {mae_h[0]:6.2f} {mae_h[6]:6.2f} {mae_h[13]:6.2f}   {mae_h.mean() / scale:.2f}")

e, sn = np.array(err["Holt-Winters"]), np.array(err["seasonal naive"])
wins = np.mean(np.abs(e), axis=1) < np.mean(np.abs(sn), axis=1)   # who wins each single split?
print(f"Holt-Winters beats seasonal naive at {wins.sum()} of {len(wins)} origins")

# 80% intervals from backtest errors: first half of origins calibrate, second half test.
half = len(e) // 2
lo, hi = np.quantile(e[:half], [0.1, 0.9], axis=0)        # error quantiles for each horizon
lo1, hi1 = np.full(H, lo[0]), np.full(H, hi[0])           # naive: reuse the h=1 width everywhere
for label, a, b in (("per-horizon widths", lo, hi), ("h=1 width for all h", lo1, hi1)):
    cover = np.mean((e[half:] >= a) & (e[half:] <= b))
    print(f"80% interval, {label:19s}: width h=1 {b[0] - a[0]:5.1f}, h=14 {b[-1] - a[-1]:5.1f}, coverage {cover:.0%}")

51 forecast origins, horizon 1..14 days
                    h=1    h=7   h=14   MASE
seasonal naive     4.66   4.63   5.46   1.03
Holt-Winters       3.29   4.21   4.71   0.78
Holt-Winters beats seasonal naive at 47 of 51 origins
80% interval, per-horizon widths : width h=1   9.6, h=14  14.5, coverage 79%
80% interval, h=1 width for all h: width h=1   9.6, h=14   9.6, coverage 71%

Two years of synthetic daily demand, with a wandering level and a weekly pattern. Starting after one year, every 7 days the code fits Holt-Winters on all data so far and forecasts 14 days, 51 origins in all, alongside a seasonal naive forecast.

• Error grows with the horizon. Holt-Winters has an MAE of 3.29 one day ahead and 4.71 at 14 days. Its MASE of 0.78 means 22 percent less error than the in-sample seasonal naive; the seasonal naive forecast itself scores 1.03.
• One split is an anecdote. Holt-Winters wins at 47 of 51 origins. Had you happened to test on one of the other 4, a single split would have told you the opposite.
• Intervals must widen. 80 percent intervals built from the first half of the backtest errors, one width per horizon, cover 79 percent of the later actuals: on target. Reusing the 1-day width at every horizon gives only 71 percent, because the 14-day errors are larger.

Pitfalls

• Tuning on the backtest. If you try 50 configurations and pick the best backtest score, that score is optimistic. Keep the last stretch of time as a final holdout you touch once.
• Features from the future. Rolling means that include the target day, or prices that were revised later, leak into every fold ( ML and deep learning for forecasting ).
• Too few folds. Five origins all in the same quiet season say nothing about the holiday peak. Cover at least one full seasonal cycle.
• Averaging scale-dependent errors across series. An MAE averaged over stores is dominated by the biggest store. Use MASE or weight the series by what matters to the business.

Trade-offs

Common mistakes

• Shuffling rows into k-fold cross-validation on a time series and celebrating the leaked accuracy.
• Reporting one MAE averaged over all horizons when the business only acts on the 7-day forecast.
• Using MAPE on a product that often sells 0 units.
• Shipping intervals without checking their coverage in a backtest, then finding that the "95 percent" band misses one day in four.
• Leaving out the seasonal naive baseline, so nobody can tell whether a MASE of 0.95 is impressive or a hard-won 5 percent.

Where you will see this

Every serious forecasting library builds backtesting in: scikit-learn's TimeSeriesSplit, the cross-validation utilities in statsforecast, sktime and Darts, and Prophet's own cross_validation function, which uses rolling cutoffs and reports metrics by horizon. Forecasting competitions score entries on held-out future periods; the M4 competition ranked them on a blend of MASE and a percentage error, each relative to a naive benchmark. And the overview from Uber in ML and deep learning for forecasting stresses the same two ideas: chronological backtests with sliding or expanding windows, and prediction intervals reported alongside every point forecast.

Check yourself

Recap