Backpropagation

The big idea

Gradient descent needs one number per weight: if I nudge this weight up a little, does the loss go up or down, and how fast? A modern network has millions or billions of weights. Nudging each one separately and re-running the network would take millions of forward passes per training step.

BackpropagationThe algorithm that computes the gradient of the loss with respect to every parameter of a network in one backward sweep, by applying the chain rule from the output back to the inputs. gets all of them in one backward sweep that costs about as much as the forward pass. The trick is bookkeeping. Every operation in the network only needs to know its own small, local derivative. Multiply those local derivatives along the path from the loss back to a weight, and you have that weight's gradient: the chain rule, applied systematically.

Think of a company that missed its target. The CEO tells each division head how much of the miss is on them. Each head splits their share among their managers in proportion to each manager's influence, and so on down. Nobody needs to understand the whole company; each person just passes blame back to the people who fed them, weighted by how much they relied on each. When the message reaches the bottom, everyone knows exactly how their own decisions moved the final number.

How it works

Step 1: The network as a computational graph

Write the forward pass as a computational graphA graph whose nodes are simple operations (add, multiply, activation) and whose edges carry values. Frameworks record it during the forward pass so they can run it backwards.: each node is one simple operation, each edge carries a number. Take the 2-2-1 network from Forward pass by hand with input (1, 2), target t = 1 and cross-entropy loss. Follow the path through hidden neuron 2:

Step 2: Local gradients

Each node knows how its output changes when its input changes, using only its own inputs:

• A sum z = w x + b: dz/dw = x, dz/dx = w, dz/db = 1.
• ReLU: slope 1 if z > 0, else 0.
• Sigmoid: dy/dz = y (1 - y).
• Cross-entropy with target 1, L = -log y: dL/dy = -1 / y.

These are local gradientsThe derivative of one node's output with respect to its own input, computed from values available at that node alone.. None of them needs to know anything about the rest of the network.

Step 3: The chain rule, backwards

The chain ruleThe calculus rule that the derivative of a composition is the product of the derivatives of its parts: dL/dx = dL/dy times dy/dx. says that if L depends on y and y depends on z, then dL/dz = dL/dy x dy/dz. Start at the loss with dL/dL = 1 and walk backwards, multiplying by one local gradient per step:

Positive: increasing w22 would increase the loss, so gradient descent will decrease it. Check the sign by intuition: the target is 1 but y is only 0.242; h2 pushes the output down through the negative weight v2; so making h2 smaller (by lowering w22) helps.

A useful shortcut: sigmoid followed by cross-entropy simplifies to dL/dz_out = y - t = 0.242 - 1 = -0.758, the same number as -4.127 x 0.184. Libraries use this fused form because it is faster and numerically safer.

Step 4: Reuse is what makes it fast

Notice that dL/dz_out = -0.758 is needed for every parameter upstream: v1, v2, c, and everything in the first layer. Backprop computes it once and passes it back. Each node receives the gradient of the loss with respect to its output (the upstream gradientDuring backpropagation, the gradient of the loss with respect to a node's output, passed back from the nodes that consumed it.), multiplies by its local gradient and sends the result to its inputs. When a value feeds several nodes, the gradients coming back from each are added.

That is why the whole backward pass costs roughly the same as the forward pass (in practice about two times), whatever the number of weights. The naive alternative, nudging each weight and re-running, costs one forward pass per weight.

Step 5: Gradient flow and blocked paths

Gradients can be stopped or shrunk on the way back. Hidden neuron 1 had z = -0.2, so ReLU's local gradient is 0: no gradient reaches w11, w12 or b1 for this input, even though dL/dh1 = -0.909 arrived at it. Long chains of local gradients smaller than 1 shrink the signal (vanishing gradients); larger than 1 grow it (exploding gradients). This is why the choice of activation ( Activation functions ) and the residual connections of Vanishing and exploding gradients, residual connections matter so much.

Step 6: The full update

Collect all nine gradients and take one gradient descent step with learning rate 0.5. In the code the output rises from 0.242 to 0.604 and the loss falls from 1.417 to 0.504. One forward pass, one backward pass, one update: that is a training step, repeated millions of times in real training.

Try it

Try this:

1. Press Next through all six stages with the defaults. Which first-layer weights get a gradient of 0, and which stage explains why?
2. At stage 6, press Next to apply the update. Run the six stages again. How much did the loss drop per update over three or four updates?
3. Reset and set the target to 0. Which way do the gradients point now? Before stepping, predict the sign of dL/dz_out.

Going deeper

Code: backprop by hand, checked numerically

import numpy as np

# The 2-2-1 network from the forward-pass lesson, with target t = 1 and cross-entropy loss
x, t = np.array([1.0, 2.0]), 1.0
W1, b1 = np.array([[0.5, -0.4], [0.3, 0.8]]), np.array([0.1, -0.2])
w2, b2 = np.array([1.2, -0.7]), 0.05

def forward(W1, b1, w2, b2):
    z1 = W1 @ x + b1
    h = np.maximum(0, z1)
    z2 = w2 @ h + b2
    y = 1 / (1 + np.exp(-z2))
    loss = -(t * np.log(y) + (1 - t) * np.log(1 - y))
    return z1, h, z2, y, loss

z1, h, z2, y, loss = forward(W1, b1, w2, b2)
print(f"forward:  z1 {z1}  h {h}  z2 {z2:.3f}  y {y:.4f}  loss {loss:.4f}")

# Backward pass: chain rule, from the loss back to every parameter
dz2 = y - t                       # sigmoid + cross-entropy simplify to this
dw2, db2 = dz2 * h, dz2           # z2 = w2 . h + b2
dh = dz2 * w2                     # how the loss changes with each hidden output
dz1 = dh * (z1 > 0)               # ReLU passes the gradient only where z1 > 0
dW1, db1 = np.outer(dz1, x), dz1  # z1 = W1 x + b1
print(f"dL/dz2 {dz2:.4f}")
print("dL/dw2", np.round(dw2, 4) + 0.0, " dL/db2", round(dz2, 4))
print("dL/dh ", np.round(dh, 4), " dL/dz1", np.round(dz1, 4) + 0.0)
print("dL/dW1\n", np.round(dW1, 4) + 0.0, "\ndL/db1", np.round(db1, 4) + 0.0)

# Gradient check: nudge one weight and measure the loss change numerically
eps = 1e-6
W1p = W1.copy(); W1p[1, 1] += eps
print(f"\nnumerical dL/dW1[1,1] {(forward(W1p, b1, w2, b2)[4] - loss) / eps:.4f}"
      f"   backprop {dW1[1, 1]:.4f}")

# One gradient descent step with learning rate 0.5
lr = 0.5
new = forward(W1 - lr * dW1, b1 - lr * db1, w2 - lr * dw2, b2 - lr * dz2)
print(f"after one step: y {y:.4f} -> {new[3]:.4f}, loss {loss:.4f} -> {new[4]:.4f}")

forward:  z1 [-0.2  1.7]  h [0.  1.7]  z2 -1.140  y 0.2423  loss 1.4175
dL/dz2 -0.7577
dL/dw2 [ 0.     -1.2881]  dL/db2 -0.7577
dL/dh  [-0.9092  0.5304]  dL/dz1 [0.     0.5304]
dL/dW1
 [[0.     0.    ]
 [0.5304 1.0608]] 
dL/db1 [0.     0.5304]

numerical dL/dW1[1,1] 1.0608   backprop 1.0608
after one step: y 0.2423 -> 0.6041, loss 1.4175 -> 0.5039

The gradient check at the end is a standard debugging tool: nudge one weight by a tiny epsilon, measure how much the loss moves, and divide. The numerical estimate (1.0608) matches backprop (1.0608). If you ever implement a layer's backward pass yourself, check it this way before trusting it.

Matrix form

For a layer z = W x + b with upstream gradient g = dL/dz, the three gradients are compact matrix expressions:

In the code, dL/dW1 is the outer product of dL/dz1 = (0, 0.530) and x = (1, 2): the first row is all zero (neuron 1 was inactive) and the second row is (0.530, 1.061). For a batch, the per-example gradients are summed or averaged, which turns these into matrix-matrix products, just like the forward pass.

Automatic differentiation

Nobody writes backward passes by hand for real networks. Automatic differentiationSoftware that records the operations of a computation and applies the chain rule automatically to produce exact gradients. Reverse mode, used for neural networks, is backpropagation. frameworks record the computational graph as your forward code runs, then apply each operation's known local gradient in reverse. That is exactly what loss.backward() does in PyTorch ( Autograd and your first PyTorch training loop ). The price is memory: every intermediate value from the forward pass must be kept until the backward pass uses it, which is why training needs far more memory than inference.

Trade-offs

Common mistakes

• Forgetting to add gradients when one value feeds several nodes (the gradients from each use must be summed).
• Dropping the ReLU mask, so gradient flows through units that were inactive.
• Mixing up shapes in the matrix form; dL/dW must have the same shape as W.
• Not resetting accumulated gradients between steps in frameworks that add into them by default.
• Trusting a hand-written backward pass without a numerical gradient check.

In the wild

Check yourself

Recap