Vanishing and exploding gradients, residual connections

The big idea

Backpropagation multiplies one local derivative per layer on the way back. In a 30-layer network, the gradient reaching the first layer is a product of 30 factors. Products are unforgiving: 0.5^30 is about one in a billion, and 1.5^30 is about 190,000. If the typical factor is a little below 1, early layers get essentially no gradient and stop learning: vanishing gradientsWhen gradients shrink exponentially as they are propagated back through many layers, so early layers learn extremely slowly or not at all.. A little above 1, and updates become huge and training blows up: exploding gradientsWhen gradients grow exponentially as they are propagated back through many layers, causing huge, unstable updates and often NaN losses..

Think of a bucket brigade passing water down a line of 30 people, each spilling a little. By the end, almost nothing arrives. Now add a pipe that runs alongside the whole line. Each person can still add water, but the pipe delivers the full flow end to end no matter what anyone does.

That pipe is a residual connectionA connection that adds a block's input directly to its output, y = x + F(x), so the signal and its gradient can bypass the block. Also called a skip connection.. Instead of each block replacing its input, it adds to it. Gradients flow back along the identity path undiminished, which is what made networks with dozens and then hundreds of layers trainable.

How it works

Step 1: Gradient magnitude through depth

For a plain stack h_(l+1) = f(W_l h_l), the gradient at layer l is the gradient at layer l + 1 times W_l transposed times the activation's slope. Its size changes by roughly the same factor at every layer, so after L layers it has been scaled by that factor to the power L. Two things set the factor: the weight scale ( Weight initialization and normalization ) and the activation's slope. Sigmoid's slope is at most 0.25, so a sigmoid stack loses at least a factor of 4 per layer on that alone.

In the code, a 30-layer sigmoid network with sensible weights delivers a gradient norm of 0.26 at the last layer but 1.7 x 10^-19 at the first. With tanh it falls more gently, from 1.07 to 0.114. With weights three times too large, tanh's gradient grows instead, from 1.37 to 4,600.

Step 2: Fixing the factor, partly

Many earlier lessons attack the factor: ReLU-family activations have slope 1 for active units ( Activation functions ); He and Xavier initialization set the weight scale ( Weight initialization and normalization ); normalization layers keep activations in range during training. These help a lot, but they balance the product on a knife edge. Through 100 layers, a factor of 0.95 still shrinks the gradient to 0.6 percent, and a factor of 1.05 grows it 130 times.

Step 3: Gradient clipping for explosions

Gradient clippingRescaling the gradient before the update when its norm exceeds a threshold, keeping its direction but capping its size, to prevent exploding updates. is a safety valve for explosions. Compute the norm of the whole gradient vector; if it exceeds a threshold c, scale it down to norm c:

The direction is kept; only the step size is capped. In the code, a gradient of (300, -400), norm 500, becomes (0.6, -0.8) with c = 1. Clipping by the global norm (over all parameters together) with c around 1 is standard in training transformers and recurrent networks, which are especially prone to sudden spikes. It does nothing for vanishing gradients.

Step 4: Residual connections

A residual blockA block that outputs its input plus a learned transformation of it, y = x + F(x), so it only has to learn a correction to the identity. outputs y = x + F(x). Its local derivative is the identity plus the derivative of F:

That "I" is the point. Even if dF/dx is tiny, the gradient passes through each block at least unchanged along the identity path. Across L blocks the gradient is no longer a pure product of small factors; it includes a direct route from the loss to every layer. In the code, the same 30-layer sigmoid stack with residual connections delivers a gradient of 1.27 at the first layer instead of 1.7 x 10^-19.

There is a second benefit. Each block only has to learn a correction to its input rather than a whole new representation. A block that is not useful can learn F close to 0 and become the identity, so adding layers rarely makes a residual network worse. Plain deep networks, by contrast, often train worse than shallower ones: the degradation problem that residual networks were introduced to solve.

Step 5: Keeping residual sums in check

Adding block after block can make the signal grow (the residual tanh stack in the code grows the gradient to 2.57 even with small branch weights). Real architectures control this with normalization inside each block (pre-norm: x + F(norm(x)), as in transformers) and by initializing each block's last layer small or at zero, so the network starts close to the identity and grows its corrections gradually.

Going deeper

Code: gradient norms through 30 layers

import numpy as np

def grad_norms(depth=30, width=64, scale=1.0, act="sigmoid", residual=False, seed=0):
    """Forward a batch through `depth` layers, backprop a unit gradient, return |grad| per layer."""
    rng = np.random.default_rng(seed)
    Ws = [rng.normal(0, scale / np.sqrt(width), (width, width)) for _ in range(depth)]
    f = (lambda z: 1 / (1 + np.exp(-z))) if act == "sigmoid" else np.tanh
    df = (lambda a: a * (1 - a)) if act == "sigmoid" else (lambda a: 1 - a ** 2)
    h, acts = rng.normal(size=(32, width)), []
    for W in Ws:
        a = f(h @ W)
        acts.append(a)
        h = h + a if residual else a              # residual: the layer adds to its input
    g, norms = np.ones_like(h) / np.sqrt(h.size), []
    for W, a in zip(reversed(Ws), reversed(acts)):
        gl = (g * df(a)) @ W.T                     # gradient through the layer's branch
        g = g + gl if residual else gl             # residual: plus the identity path
        norms.append(np.linalg.norm(g))
    return norms[::-1]                             # index 0 = first layer

print("gradient norm reaching layer:        30       20       10        1")
for name, kw in [("sigmoid, plain", dict(act="sigmoid")),
                 ("tanh, plain", dict(act="tanh")),
                 ("tanh, plain, weights x3", dict(act="tanh", scale=3.0)),
                 ("sigmoid, residual", dict(act="sigmoid", residual=True)),
                 ("tanh, residual, weights x0.3", dict(act="tanh", residual=True, scale=0.3))]:
    n = grad_norms(**kw)
    print(f"{name:<28}" + "".join(f"{n[i]:9.3g}" for i in (29, 19, 9, 0)))

# Exploding: a product of 30 factors of 1.5, and gradient clipping by global norm
print(f"\n1.5 ** 30 = {1.5 ** 30:,.0f}   0.5 ** 30 = {0.5 ** 30:.2e}")
g = np.array([300.0, -400.0])                      # an exploding gradient, norm 500
clipped = g * min(1.0, 1.0 / np.linalg.norm(g))    # clip to max norm 1.0
print("gradient", g, "-> clipped", np.round(clipped, 2), " (same direction, norm 1)")

gradient norm reaching layer:        30       20       10        1
sigmoid, plain                   0.26 1.19e-07  5.1e-14 1.67e-19
tanh, plain                      1.07    0.781    0.381    0.114
tanh, plain, weights x3          1.37     32.7      568  4.6e+03
sigmoid, residual                   1     1.03     1.09     1.27
tanh, residual, weights x0.3     1.05     1.31     1.86     2.57

1.5 ** 30 = 191,751   0.5 ** 30 = 9.31e-10
gradient [ 300. -400.] -> clipped [ 0.6 -0.8]  (same direction, norm 1)

Each row pushes a batch through 30 layers of width 64, backpropagates a unit gradient from the top, and prints the gradient norm reaching layers 30, 20, 10 and 1. Compare the first and fourth rows: the only difference is h = h + a instead of h = a, and the first-layer gradient goes from 1.7 x 10^-19 to 1.27.

Gradients in recurrent networks

A recurrent network applies the same weights at every time step, so backpropagating through 100 steps of a sequence is a 100-layer product with the same matrix each time, which makes vanishing and exploding even more severe. LSTMs and GRUs add gated, additive memory paths, the same idea as a residual connection, and gradient clipping became standard in that setting. Attention-based transformers sidestep long chains by connecting every position directly ( Seq2seq and attention ).

Diagnosing it in practice

Log the gradient norm per layer (or per block) during training. Early-layer norms many orders of magnitude below later ones point to vanishing; sudden spikes in the global norm, followed by loss spikes or NaNs, point to explosions. The debugging checklist in Debugging neural network training starts from these signals.

Trade-offs

Common mistakes

• Stacking many sigmoid or tanh layers without residuals or normalization and blaming the optimizer.
• Clipping each parameter's gradient separately when you meant to clip the global norm, which changes the gradient's direction.
• Treating a NaN loss as bad luck instead of checking for exploding gradients and the learning rate.
• Adding residual connections but no normalization or careful initialization, so activations grow with depth.

In the wild

Residual connections are everywhere: residual networks for images, every transformer block (around both attention and the feed-forward layer), and most modern speech and vision architectures. Large language model training typically clips the global gradient norm at about 1.0 and monitors gradient norms continuously, because a single spike can derail a very expensive run. The combination of residual paths, normalization and careful initialization is what allows networks with hundreds of layers to train at all.

Check yourself

Recap