Stacking and blending

The big idea

A hospital gets a second opinion from a radiologist, a lab test and a family doctor. Simply averaging them is crude: the radiologist may be excellent on tumours and weak on fractures. A senior physician who has watched all three for years learns how much to trust each one, and when. StackingAn ensemble in which a meta-model is trained to combine the predictions of several different base models, using their predictions as its input features. trains that senior physician: a meta-modelThe second-level model in stacking. Its inputs are the base models' predictions and its output is the final prediction. whose inputs are the predictions of several different base modelsThe first-level models in a stack, usually of different types (linear, trees, neighbours) so their errors differ..

The catch is subtle. If the meta-model learns from predictions the base models made on their own training data, it learns to trust whoever memorised best. Its training data must be predictions on data each base model never saw.

How it works

Step 1: Pick diverse base models

Stacking helps when base models make different mistakes, exactly like bagging. Five gradient boosting models with different seeds add little. A linear model, a k-nearest-neighbours model, a random forest and a boosted model see the data in different ways, so a combiner has something to choose between.

Step 2: Produce out-of-fold predictions

For each base model, run k-fold cross-validation ( Cross-validation ) and keep every held-out prediction. After k rounds every training row has exactly one out-of-fold predictionA prediction for a training row made by a model that was trained without that row, as in k-fold cross-validation.. Put the columns side by side: that table, plus the true labels, trains the meta-model.

Step 3: Fit the meta-model, then refit the bases

Use a simple meta-model, usually logistic or linear regression, sometimes constrained to non-negative weights. A flexible meta-model on a handful of correlated columns overfits easily. Finally, refit each base model on all the training data; at prediction time, the base models predict and the meta-model combines their predictions.

Step 4: Or blend with a holdout

BlendingA simpler form of stacking: base models train on one part of the data and the meta-model trains on their predictions for a separate holdout part. skips the folds: train bases on, say, 80 percent of the data and fit the meta-model on their predictions for the other 20 percent. It is simpler and leak-proof by construction, but the meta-model learns from fewer rows and the bases see less data.

Going deeper

Code: four models, an average, a leaky stack and an honest one

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier, StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import cross_val_predict, train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=2000, n_features=20, n_informative=10, flip_y=0.08, random_state=3)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=0)
base = {
    "logistic": make_pipeline(StandardScaler(), LogisticRegression()),
    "kNN (k=15)": make_pipeline(StandardScaler(), KNeighborsClassifier(15)),
    "random forest": RandomForestClassifier(100, random_state=0),
    "boosting": HistGradientBoostingClassifier(random_state=0),
}
oof, insample, test = {}, {}, {}
for name, m in base.items():
    oof[name] = cross_val_predict(m, Xtr, ytr, cv=5, method="predict_proba")[:, 1]
    m.fit(Xtr, ytr)
    insample[name] = m.predict_proba(Xtr)[:, 1]
    test[name] = m.predict_proba(Xte)[:, 1]
    print(f"{name:<14} test AUC {roc_auc_score(yte, test[name]):.4f}")

T = np.column_stack(list(test.values()))
print(f"\nsimple average  test AUC {roc_auc_score(yte, T.mean(1)):.4f}")
for label, Z in [("leaky stack (in-sample preds)", insample), ("stack on out-of-fold preds  ", oof)]:
    meta = LogisticRegression().fit(np.column_stack(list(Z.values())), ytr)
    w = ", ".join(f"{c:+.2f}" for c in meta.coef_[0])
    print(f"{label}: test AUC {roc_auc_score(yte, meta.predict_proba(T)[:, 1]):.4f}, weights [{w}]")

# scikit-learn does the out-of-fold bookkeeping for you
s = StackingClassifier(list(base.items()), final_estimator=LogisticRegression(), cv=5).fit(Xtr, ytr)
print(f"\nStackingClassifier test AUC {roc_auc_score(yte, s.predict_proba(Xte)[:, 1]):.4f}")

logistic       test AUC 0.9049
kNN (k=15)     test AUC 0.9386
random forest  test AUC 0.9430
boosting       test AUC 0.9421

simple average  test AUC 0.9443
leaky stack (in-sample preds): test AUC 0.9454, weights [+0.75, +1.21, +3.93, +5.64]
stack on out-of-fold preds  : test AUC 0.9472, weights [+0.40, +2.68, +2.36, +2.39]

StackingClassifier test AUC 0.9472

Look at the meta-model weights. Trained on in-sample predictions, it puts +5.64 on boosting and +3.93 on the forest, the models that fit their own training data almost perfectly, and only +0.75 on logistic regression. Trained on out-of-fold predictions, the weights are more balanced (+0.40, +2.68, +2.36, +2.39), because it sees how each model actually performs on unseen rows. Here the leak cost only a little; with more flexible base models and a more flexible meta-model it can cost a lot. StackingClassifier gives the same 0.9472 as the hand-written version, because it does the same out-of-fold bookkeeping.

Practical variations

• Pass-through: give the meta-model the original features as well (passthrough=True), so it can learn that one base model is better for some segments. More power, more overfitting risk.
• Multi-level stacks: stack the stacks. Common in competitions, rarely justified in production.
• Time series: out-of-fold predictions must respect time order, using forward-chaining folds ( Backtesting and forecast evaluation ), or the meta-model learns from the future.

Trade-offs

Common mistakes

• Training the meta-model on base predictions for rows the bases trained on.
• Using a deep, flexible meta-model on four correlated columns.
• Stacking near-identical models and expecting a gain.
• Reporting the stack's score on the same folds used to build the out-of-fold predictions and to tune the meta-model.
• Shipping a five-model stack for a 0.2 point gain without counting serving cost and maintenance.

In the wild

Stacking and blending are staples of data science competitions, where winning solutions often combine dozens of models and a fraction of a point decides the ranking. In production, teams more often ship a single strong model or a simple average, because every extra model must be retrained, served, monitored and debugged. Stacking shows up where scoring happens offline in batches and accuracy is worth the cost, such as risk scoring or demand forecasting pipelines.

Check yourself

Recap