Regularization for deep nets

The big idea

A neural network with a few hundred hidden units can memorize a small dataset perfectly, noise and all. In the code, a 200-unit network reaches 100 percent accuracy on 60 noisy training points but only 87.3 percent on new data. It learned the training set, not the pattern.

RegularizationAny technique that discourages a model from fitting noise in the training data, so it generalizes better to new data, usually at a small cost in training accuracy. is everything you do to stop that. Think of a student preparing for an exam. Memorizing last year's answers works perfectly on last year's paper and fails on this one. A good teacher makes the student solve problems with some notes randomly missing (dropout), discourages over-complicated answers (weight decay), gives many variations of each problem (augmentation), and stops the cramming before it turns into rote memory (early stopping).

These four are the everyday toolkit for deep networks. They are cheap, they combine well, and most real training recipes use several at once.

How it works

Step 1: Dropout

DropoutA regularizer that sets each hidden unit's output to zero with probability p on every training step, so the network cannot rely on any single unit. All units are used at test time. switches off each hidden unit with probability p (often 0.1 to 0.5) on every training step. A unit cannot count on any particular partner being present, so the network cannot build fragile, co-adapted combinations that only fit the training points. Each step trains a different thinned sub-network; at test time, using all units is like averaging an ensemble of them.

To keep the expected size of each layer's output the same, modern implementations use inverted dropoutDropout that divides the kept activations by (1 - p) during training, so nothing needs rescaling at test time.: during training, the surviving activations are divided by (1 - p). At test time, dropout simply does nothing. This is one of the reasons model.train() and model.eval() exist.

Step 2: Weight decay

Weight decayShrinking every weight slightly towards zero at each step, a form of L2 regularization that discourages large weights. adds a pull towards zero on every weight at every step: w := w - lr x wd x w, the same as an L2 penalty with plain SGD (with Adam, use the decoupled AdamW form from Optimizers: SGD, momentum, RMSProp, Adam, AdamW ). Large weights let a network draw sharp, wiggly boundaries around individual points; keeping weights small keeps the function smoother. Typical values range from 1e-5 to 0.1 depending on the optimizer and model; it needs tuning, because too much makes the model underfit.

Step 3: Data augmentation

The most effective regularizer is more data. Data augmentationCreating modified copies of training examples (flips, crops, noise, paraphrases) that keep the label, so the model sees more variety and learns invariances. creates it from what you have by applying changes that keep the label: flip, crop, rotate or recolour an image; add noise to audio; shift a time series. The model learns that these changes do not matter, which is exactly the kind of knowledge it cannot get from a small dataset. The key is that the augmentation must preserve the label: flipping a photo of a cat is fine, flipping a "6" into a "9" is not.

Step 4: Early stopping

Training loss keeps falling, but validation loss usually falls, flattens and then rises as the network starts memorizing. Early stoppingMonitoring validation performance during training and keeping the model from the point where it was best, instead of the final one. watches a validation set during training and keeps the checkpoint where validation performance was best, then stops after it has not improved for a while (the patience). It costs nothing extra to compute and is almost always worth having.

Going deeper

Code: four regularizers on noisy two moons

import numpy as np
from sklearn.datasets import make_moons

# Small, noisy training set; a validation set for choosing; a large test set
Xtr, ytr = make_moons(60, noise=0.3, random_state=1)
Xte, yte = make_moons(2000, noise=0.3, random_state=2)
Xva, yva = make_moons(300, noise=0.3, random_state=3)

def train(wd=0.0, drop=0.0, augment=False, steps=8000, lr=1.0, seed=0):
    rng = np.random.default_rng(seed)
    W1, b1 = rng.normal(0, np.sqrt(2 / 2), (2, 200)), np.zeros(200)
    W2, b2 = rng.normal(0, np.sqrt(1 / 200), (200, 1)), np.zeros(1)
    hist = []
    for t in range(steps):
        X, y = Xtr, ytr
        if augment:  # data augmentation: a jittered copy of each point, fresh every step
            X = np.vstack([Xtr, Xtr + rng.normal(0, 0.15, Xtr.shape)]); y = np.r_[ytr, ytr]
        h = np.maximum(0, X @ W1 + b1)
        mask = (rng.random(h.shape) > drop) / (1 - drop)          # inverted dropout
        hd = h * mask
        p = 1 / (1 + np.exp(-(hd @ W2 + b2).ravel()))
        g = (p - y)[:, None] / len(y)                              # cross-entropy gradient
        gh = (g @ W2.T) * mask * (h > 0)
        W2 -= lr * (hd.T @ g + wd * W2); b2 -= lr * g.sum(0)       # weight decay: + wd * W
        W1 -= lr * (X.T @ gh + wd * W1); b1 -= lr * gh.sum(0)
        if t % 200 == 199:  # evaluation: no dropout
            acc = lambda A, b: np.mean(((np.maximum(0, A @ W1 + b1) @ W2 + b2).ravel() > 0) == b)
            hist.append((t + 1, acc(Xtr, ytr), acc(Xte, yte), acc(Xva, yva)))
    return hist

for name, kw in [("no regularization", {}), ("weight decay 0.001", {"wd": 0.001}),
                 ("dropout 0.3", {"drop": 0.3}), ("augmentation", {"augment": True})]:
    t, tr, te, _ = train(**kw)[-1]
    print(f"{name:<19} train acc {tr:.3f}   test acc {te:.3f}")

# Early stopping: keep the checkpoint with the best VALIDATION accuracy, then report test
hist = train()
best = max(hist, key=lambda r: r[3])
print(f"\nearly stopping: best validation acc at step {best[0]} -> test acc {best[2]:.3f}"
      f" (train acc {best[1]:.3f}); full run ends at test acc {hist[-1][2]:.3f}")

no regularization   train acc 1.000   test acc 0.873
weight decay 0.001  train acc 0.950   test acc 0.886
dropout 0.3         train acc 0.967   test acc 0.872
augmentation        train acc 0.983   test acc 0.889

early stopping: best validation acc at step 4600 -> test acc 0.885 (train acc 0.983); full run ends at test acc 0.873

The network has 200 hidden ReLU units and 60 noisy training points, so it can memorize. Without regularization: 100 percent train, 87.3 percent test. Weight decay 0.001 trades training accuracy for generalization (95.0 percent train, 88.6 percent test). Augmentation with jittered copies does best here, 88.9 percent test. Early stopping chosen on a separate validation set gives 88.5 percent test. Dropout 0.3 barely changes the result on this tiny two-input network; dropout pays off mostly in large, wide layers with many co-adapting units. The gains are a point or two because the data itself is noisy: no model can be right about the points that sit in the other class's region.

Note that early stopping picks the step by validation accuracy, not test accuracy. If you choose anything by looking at the test set, the test score is no longer an honest estimate.

More regularizers you will meet

• Label smoothing: train on targets like 0.9 and 0.1 instead of 1 and 0, so the network does not push its outputs to extreme confidence.
• Stochastic depth: randomly skip whole residual blocks during training, a dropout for layers.
• Mixup and CutMix: blend two images and their labels into one training example.
• Normalization ( Weight initialization and normalization ) and small batch sizes also have a mild regularizing effect.

Regularization at scale

Very large models trained on huge datasets for about one pass often use little or no dropout, because they see each example only once and cannot memorize it; weight decay remains standard. Small datasets and fine-tuning are where the full toolkit matters most.

Trade-offs

Common mistakes

• Leaving the model in training mode at evaluation, so dropout randomly removes units from predictions.
• Choosing the early-stopping point, or any hyperparameter, using the test set.
• Augmenting in ways that change the label, such as flipping digits or text images.
• Adding every regularizer at full strength at once and then wondering why the model underfits.
• Applying augmentation to the validation and test sets, which should reflect real inputs.

In the wild

Image classifiers rely heavily on augmentation (random crops, flips, colour changes) plus weight decay. Transformer models use dropout in attention and feed-forward layers when data is limited, weight decay through AdamW, and sometimes label smoothing. Speech systems add noise and masking to spectrograms. Early stopping on a validation metric is part of nearly every training script, from small tabular networks to fine-tuning large models.

Check yourself

Recap