ML and deep learning for forecasting

The big idea

Classical models fit one series at a time and see only that series' past. A grocery chain with 1,000 stores and 5,000 products has 5 million series, many of them short, sparse or brand new, and sales that depend on prices, promotions and weather. Fitting 5 million ARIMA models, each blind to promotions and to what similar products did, wastes most of the information.

The machine learning approach turns forecasting into ordinary supervised learning. Each row says: "here is what we knew at the time (recent sales, day of week, whether a promotion is planned); here is what actually happened". Then one model learns from the rows of all series together. Think of a weather forecaster who has studied thousands of cities: faced with a new town, they already know how fronts behave and only need its recent readings.

How it works

Step 1: Turn a series into a table

Lag featuresA feature built from an earlier value of the series, such as sales 7 days before the target date. are past values of the series: sales 7, 14 and 28 days before the target day. Add rolling statisticsA summary such as a mean or standard deviation over a past window of the series, computed only from data available at forecast time. (the mean of the last four weeks), calendar features (weekday, month, holiday flags) and known-in-advance inputs such as planned promotions or prices. The target is the value on the day you forecast. Every time step of every series becomes one training row, and any regressor can learn from it; gradient-boosted trees ( Boosting: AdaBoost to gradient boosting ) are the usual first choice.

The one rule that matters most: every feature must be available at the moment the forecast is made. If you forecast up to 7 days ahead, a lag of 1 day is a leak ( Data leakage ), because for the 7th day it will not exist yet. Use lags of at least the horizon, or forecast recursively.

• Recursive: predict one step ahead, feed the prediction back in as a lag, repeat. One model, but errors compound.
• Direct: train a separate model (or a horizon feature) for each step ahead, using only lags that will exist. No compounding, more models.

Step 2: One global model for many series

A global modelA single forecasting model trained on rows from many related time series at once, as opposed to one local model per series. is trained on all series at once, unlike the local, one-model-per-series approach of ARIMA and exponential smoothing. It wins in three ways:

• Pooled data. A promotion effect seen 10,000 times across products is learned far better than from the 3 promotions in one product's history.
• Cold start. A product launched last month gets sensible forecasts from patterns learned on similar products.
• External features such as price, promotion, weather and product attributes go in as ordinary columns.

Two details make it work. Scale each series (divide by its mean, or model log sales) so the model learns shapes rather than sizes. Add static features such as store, category or region so it can still tell series apart.

Retail forecasting competitions made the case publicly. The M5 "Accuracy" competition asked for forecasts of 42,840 hierarchical unit-sales series from Walmart, and its winner used multiple direct and recursive LightGBM models.

Step 3: Deep forecasting models

Deep models learn the features themselves and, like global GBMs, train across many series.

• DeepARAn autoregressive recurrent network trained on many related series that outputs a probability distribution for each future step (Salinas et al., 2017). (Salinas et al., 2017) trains an autoregressive recurrent network ( LSTM and GRU ) on a large number of related series and produces probabilistic forecasts: a distribution for each future step, not just a number. The authors report accuracy improvements of around 15 percent over state-of-the-art methods on several real-world data sets.
• N-BEATS (Oreshkin et al., 2019) is a very deep stack of fully connected layers with backward and forward residual links. Its first configuration uses no time-series-specific components, and it improved accuracy by 11 percent over a statistical benchmark and by 3 percent over the previous winner of the M4 competition.
• The Temporal Fusion Transformer (Lim et al., 2019) targets multi-horizon forecasting with a mix of inputs: static covariates, known future inputs and series only observed in the past. It combines recurrent layers for local processing with interpretable self-attention for long-term dependencies ( Seq2seq and attention ), plus components that select relevant features.
• Hybrids blend both worlds. The winning M4 entry, described in Uber's forecasting overview, combined Holt-Winters-style smoothing formulas with a stack of dilated LSTMs.

Step 4: Foundation models for time series

The newest idea borrows from language models: pretrain one large model on a huge corpus of diverse series, then forecast a new series zero-shotApplying a pretrained model to a new task or data set without any training on it., with no training on it at all. TimesFM pretrains a patched-decoder attention model and reports zero-shot accuracy that comes close to supervised models trained on each data set. Chronos tokenizes values by scaling and quantization into a fixed vocabulary, trains standard language-model architectures on them, and reports zero-shot performance comparable to, and occasionally better than, models trained on the target data sets.

Going deeper

Code: one boosted model for 200 series

import numpy as np
from sklearn.ensemble import HistGradientBoostingRegressor   # LightGBM-style boosted trees

# 200 related synthetic series (think store x product daily sales): own level, shared weekly shape,
# slow drift, and promotions known in advance that lift sales by 40 percent.
rng = np.random.default_rng(0)
N, T, test_days = 200, 365, 28
level = rng.lognormal(3, 0.8, N)[:, None]
dow = np.arange(T) % 7
weekly = np.array([-0.2, -0.15, -0.1, 0.0, 0.1, 0.3, 0.05])[dow]
drift = 1 + np.cumsum(rng.normal(0, 0.004, (N, T)), axis=1)
promo = (rng.random((N, T)) < 0.06).astype(float)
y = level * (1 + weekly) * drift * (1 + 0.4 * promo) * rng.lognormal(0, 0.1, (N, T))

scale = y[:, : T - test_days].mean(axis=1, keepdims=True)   # per-series scale from training data only
z = y / scale

def table(days, use_promo=True):   # one row per (series, day); every lag is at least 7 days old
    X, tgt, s = [], [], []
    for t in days:
        lags = [z[:, t - k] for k in (7, 14, 21, 28)]
        feats = lags + [z[:, t - 34:t - 6].mean(axis=1), np.full(N, dow[t])] + ([promo[:, t]] if use_promo else [])
        X.append(np.column_stack(feats)); tgt.append(z[:, t]); s.append(scale[:, 0])
    return np.vstack(X), np.concatenate(tgt), np.concatenate(s)

train_days, test_days_idx = range(35, T - test_days), range(T - test_days, T)
actual = y[:, T - test_days:].T.ravel()
mae = lambda pred: np.mean(np.abs(pred - actual))
print(f"{N} series x {T} days; forecasting the last {test_days} days, each from data at least 7 days old")
print(f"  seasonal naive (same weekday last week): MAE {mae(y[:, T - test_days - 7:T - 7].T.ravel()):.2f}")
for use_promo in (False, True):
    Xtr, ytr, _ = table(train_days, use_promo)
    Xte, _, s = table(test_days_idx, use_promo)
    model = HistGradientBoostingRegressor(max_iter=300, learning_rate=0.05, random_state=0).fit(Xtr, ytr)
    label = "global GBM, lags + weekday" + (" + promo" if use_promo else "")
    print(f"  {label:39s}: MAE {mae(model.predict(Xte) * s):.2f}   (one model, {len(ytr):,} training rows)")
print(f"mean daily sales across series: {actual.mean():.1f}")

200 series x 365 days; forecasting the last 28 days, each from data at least 7 days old
  seasonal naive (same weekday last week): MAE 3.98
  global GBM, lags + weekday             : MAE 2.83   (one model, 60,400 training rows)
  global GBM, lags + weekday + promo     : MAE 2.36   (one model, 60,400 training rows)
mean daily sales across series: 27.9

The data are 200 synthetic daily series, like store-product sales, each with its own level, a shared weekly shape, slow drift and promotions that lift sales by 40 percent. Every series is divided by its training mean, then windowed into rows with lags of 7, 14, 21 and 28 days, a rolling mean, the weekday and, in the second model, the promotion flag. All lags are at least 7 days old, so each forecast uses only data that would exist up to a week ahead.

• Seasonal naive (same weekday last week): MAE 3.98, on series that average 27.9 units a day.
• One global GBM on lags and weekday: 2.83. It averages several weeks of history instead of trusting one noisy week.
• Adding the promotion flag: 2.36. The model learned the lift from thousands of promotion days pooled across all 200 series. A per-series model would see about 20.

Pitfalls that are specific to forecasting

• Leakage through features. Rolling means that include the target day, "future" prices that were revised after the fact, or target encodings computed on the whole history all inflate test scores.
• Random cross-validation. Shuffled folds put the future in the training set. Use time-based backtests, the subject of the next lesson.
• Trees do not extrapolate. A tree predicts within the range of targets it saw. If the series trends beyond it, model differences or scaled values, or detrend first.
• Point forecasts are not enough. Inventory decisions need quantiles. Train with a quantile loss, or use a probabilistic model such as DeepAR.

Trade-offs

Common mistakes

• Using lag-1 features for a 7-day-ahead forecast, then discovering in production that yesterday's value is not there yet.
• Training a global model on unscaled series, so the largest stores dominate the loss.
• Comparing a tuned deep model against an untuned ARIMA and no seasonal naive baseline.
• Evaluating recursive forecasts one step ahead only, hiding how errors compound over the horizon.
• Forgetting that a forecast becomes a decision: optimizing MAE when the business cost of running out is ten times the cost of overstock.

In the wild

Check yourself

Recap