Bagging and random forests

The big idea

Ask one person to guess the number of jelly beans in a jar and the answer may be wildly off. Ask a hundred people and average their guesses, and the average is usually much closer than most individuals. The errors point in different directions and cancel. This only works if the guesses are somewhat independent: a hundred people copying the same loud neighbour are no better than one.

A deep decision tree is like one confident, erratic guesser. It has low bias (it can fit almost anything) but high variance: change a few training rows and you get a different tree. BaggingBootstrap aggregating: train many copies of a model on different bootstrap samples of the data and average (or vote on) their predictions. trains many trees on slightly different versions of the data and averages them. A random forestBagged decision trees where each split also considers only a random subset of the features, which makes the trees less alike and their average more accurate. adds one more trick to make the trees disagree more, so their errors cancel better.

How it works

Step 1: Make many training sets from one

A bootstrap sampleA sample of n rows drawn with replacement from a dataset of n rows; some rows appear several times and about 37 percent do not appear at all. draws n rows from your n training rows with replacement. Some rows are picked twice or three times, others not at all. The chance a given row is never picked is (1 - 1/n)^n, which approaches 1/e, about 0.368. So each sample contains about 63 percent of the distinct rows (the code below measures 63.4 percent), and each tree trains on a different 63 percent.

Step 2: Grow a deep tree on each sample, then average

Each tree is grown deep, with little or no pruning: low bias, high variance. For classification the forest takes a majority vote or, better, averages the predicted probabilities. For regression it averages the predicted numbers. Training is embarrassingly parallelWork that can be split into independent pieces with no communication between them, so it scales almost perfectly across cores or machines., since no tree depends on another.

Step 3: Why averaging helps, and where it stops helping

If each tree's prediction has variance sigma^2 and any two trees have correlation rho, the average of B trees has variance

The second term shrinks as you add trees. The first does not: correlated trees make the same mistakes, and no amount of voting removes a shared mistake. With rho = 0.66 the ensemble variance can never fall below 66 percent of one tree's; with rho = 0.3 it can fall to 30 percent.

Step 4: Random forests decorrelate the trees

If one feature is very strong, every bagged tree splits on it at the root and the trees look alike. A random forest forbids this: at each split, it considers only a random subset of features, typically the square root of the number of features for classification (max_features="sqrt") and a third of them for regression. Each tree is a little weaker, but the trees are less correlated, and Figure 2 shows why that usually wins.

In the code, 50 trees using all 20 features at every split have an average pairwise correlation of 0.66; with 4 random features per split it falls to 0.58.

Step 5: Out-of-bag error, a free validation set

Each row was left out of about 37 percent of the trees. Predict each row using only those trees and you get an out-of-bag (OOB)An accuracy or error estimate for a bagged ensemble computed by predicting each training row only with the trees whose bootstrap sample did not include it. estimate of test performance, without a separate validation split. It is close to a cross-validation estimate at no extra training cost.

Going deeper

Code: tree vs bagging vs forest, OOB and correlation

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(n_samples=1200, n_features=20, n_informative=8,
                           n_redundant=6, flip_y=0.05, random_state=0)

# A bootstrap sample leaves out about 1/e = 36.8 percent of rows
rng = np.random.default_rng(0)
idx = rng.integers(0, len(X), len(X))
print(f"unique rows in one bootstrap sample: {len(np.unique(idx)) / len(X):.1%}")

models = {
    "one deep tree": DecisionTreeClassifier(random_state=0),
    "bagging, 100 trees": BaggingClassifier(DecisionTreeClassifier(), n_estimators=100, random_state=0),
    "random forest, 100": RandomForestClassifier(n_estimators=100, random_state=0),
}
print("\nmodel                 5-fold accuracy")
for name, m in models.items():
    s = cross_val_score(m, X, y, cv=5)
    print(f"{name:<20}  {s.mean():.3f} +/- {s.std():.3f}")

# Out-of-bag accuracy is a free validation estimate
print("\ntrees  OOB accuracy")
for n in [25, 100, 300]:
    rf = RandomForestClassifier(n_estimators=n, oob_score=True, random_state=0).fit(X, y)
    print(f"{n:>5}  {rf.oob_score_:.3f}")

print()
# Feature subsampling makes trees less alike: correlation of their predictions on held-out rows
Xtr, Xte, ytr = X[:900], X[900:], y[:900]
for mf, label in [(None, "all 20 features per split"), ("sqrt", "sqrt(20) = 4 per split")]:
    rf = RandomForestClassifier(n_estimators=50, max_features=mf, random_state=0).fit(Xtr, ytr)
    P = np.array([t.predict_proba(Xte)[:, 1] for t in rf.estimators_])
    c = np.corrcoef(P)[np.triu_indices(50, 1)].mean()
    print(f"{label}: mean correlation between trees {c:.2f}")

unique rows in one bootstrap sample: 63.4%

model                 5-fold accuracy
one deep tree         0.859 +/- 0.018
bagging, 100 trees    0.908 +/- 0.020
random forest, 100    0.913 +/- 0.017

trees  OOB accuracy
   25  0.901
  100  0.916
  300  0.918

all 20 features per split: mean correlation between trees 0.66
sqrt(20) = 4 per split: mean correlation between trees 0.58

One deep tree scores 0.859 in 5-fold cross-validation. Bagging 100 trees scores 0.908 and a random forest 0.913: the big jump comes from averaging, and the extra step from feature subsampling is small on this dataset, inside the noise. OOB accuracy climbs from 0.901 with 25 trees to 0.916 with 100 and 0.918 with 300, the flattening Figure 2 predicts.

Tuning (there is not much)

• n_estimators: more is never worse for accuracy, only slower. Stop when OOB error flattens; a few hundred is typical.
• max_features: the main knob. Smaller means less correlated, weaker trees.
• min_samples_leaf and max_depth: usually left loose; raising min_samples_leaf smooths probabilities and shrinks the model.
• Forests do not overfit by adding trees, but they can still overfit noisy data through very deep trees, which is why the leaf size sometimes matters.

Limits

A forest's prediction is an average of leaf values, so a regression forest can never predict beyond the range of training targets: it cannot extrapolate a trend. Forests are also large (hundreds of deep trees) and slower to predict than one tree, and you lose the readable flowchart. Feature importance ( Feature importance and SHAP ) partly restores interpretability.

Trade-offs

Common mistakes

• Bagging a low-variance model such as linear regression and expecting a big gain; bagging helps unstable models.
• Treating OOB accuracy as valid when rows are not independent (repeated customers, time series); use grouped or time-based validation instead.
• Pruning the individual trees hard: forests want deep, low-bias trees and let averaging handle variance.
• Expecting a regression forest to predict prices higher than any seen in training.
• Tuning n_estimators by grid search; just use enough trees.

In the wild

Random forests are a standard first strong model for tabular problems in industry and science: fraud and credit scoring, churn prediction, ecology and genomics studies, and quality control. Their appeal is robustness: they work reasonably well with default settings, handle mixed feature types and give a built-in validation estimate. Many teams use a forest as the baseline that a tuned gradient boosting model must beat.

Check yourself

Recap