Machine Learning ML-01.04
Derivatives and the chain rule
Read a derivative as a slope, combine partial derivatives into a gradient, and see how the chain rule lets backpropagation train networks.
The big idea
Training a model means nudging its numbers so its mistakes get smaller. To nudge in the right direction you need to know one thing: if I increase this number a tiny bit, does the error go up or down, and how fast? That question has a name. It is the derivativeHow fast a function's output changes as its input changes: the slope of the function at a point., and it is the single most important piece of calculus in machine learning.
You already have the intuition. Driving up a hill, the steepness under your wheels is the derivative of your height with respect to distance. Steep and positive: you are climbing fast. Zero: you are at the top, or at the bottom of a dip. Negative: downhill. A model in training is a driver trying to reach the lowest point of a valley, and derivatives are how it feels the slope.
How it works
Step 1: Derivative as slope
For a function , the derivative is the slope of the curve at : rise over run, for a run so tiny it is practically zero.
Worked example: . At , try a small step : . As shrinks it settles on 6. The general rule for powers gives , and .
A few rules cover most of what ML needs: the derivative of is ; of is ; of is ; and the derivative of a sum is the sum of the derivatives.
Step 2: Partial derivatives and the gradient
A model's loss depends on many numbers at once, not one. A partial derivativeThe derivative of a many-input function with respect to one input, holding all the others fixed. asks about one input at a time while freezing the rest. For at :
- Change only : .
- Change only : .
Stack the partials into a vector and you have the gradientThe vector of all partial derivatives of a function. It points in the direction of steepest increase., written . The gradient points in the direction where the function rises fastest. Training walks the opposite way, downhill, which is gradient descent ( Gradient descent, the engine of ML ).
Step 3: The chain rule, multiplying local slopes
Models are functions of functions: the loss depends on the prediction, which depends on a layer's output, which depends on the weights. The chain ruleThe derivative of a composition equals the product of the derivatives of its parts: dy/dx = dy/du times du/dx. says the slope of a chain is the product of the slopes of its links:
Worked example: . Let , so . Locally, and . At : , so .
This is the foundation of training neural networks. A network with many layers is a long chain; backpropagationThe algorithm that computes the gradient of the loss for every weight by applying the chain rule backward through the network. computes every weight's gradient by walking the chain backward and multiplying local slopes, reusing work at each step ( Backpropagation ).
def f(x): return x ** 2
def df(x): return 2 * x # derivative worked out by hand
def numeric_slope(f, x, h=1e-5):
return (f(x + h) - f(x - h)) / (2 * h) # tiny rise over tiny run
print(round(numeric_slope(f, 3.0), 6), df(3.0))
# Chain rule: y = (3x + 1)^2. Outer g(u) = u^2, inner u = 3x + 1.
x = 2.0
u = 3 * x + 1
dy_du = 2 * u # slope of the outer function
du_dx = 3 # slope of the inner function
print("chain rule:", dy_du * du_dx, " numeric:", round(numeric_slope(lambda t: (3 * t + 1) ** 2, x), 4))
# Partial derivatives of L(w, b) = (w * 2 + b - 5)^2 at w = 1, b = 0
L = lambda w, b: (w * 2 + b - 5) ** 2
print("dL/dw:", round(numeric_slope(lambda w: L(w, 0.0), 1.0), 4), " dL/db:", round(numeric_slope(lambda b: L(1.0, b), 0.0), 4))Output
6.0 6.0
chain rule: 42.0 numeric: 42.0
dL/dw: -12.0 dL/db: -6.0 The numeric_slope trick, rise over a tiny run on both sides, is called a finite difference. It is slow for millions of weights, but it is the standard way to check a hand-written or automatic gradient.
Try it
Try this:
- On , drag the point to . The slope is 0: you are at the bottom of the valley. That is where training wants to end up.
- Switch to and find both places where the slope is 0. One is a hilltop, one a valley. A slope of zero alone does not tell you which.
- On the sine curve, watch the slope change sign as you cross each peak. Imagine stepping downhill from different starting points: where would each one end up?
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Derivatives by hand | Small formulas, understanding, writing a custom layer | Large models: slow and error-prone |
| Finite differences (numeric) | Checking that a gradient implementation is right | Training: one or two extra function calls per weight, and rounding error |
| Automatic differentiation (autograd) | Training any real model: exact gradients for millions of weights in one backward pass | Very little; just keep track of what is recorded and free memory |
Common mistakes
- Treating the gradient as pointing downhill. It points uphill; training subtracts it.
- Forgetting one link in the chain rule, especially the inner derivative (the 3 in ).
- Using a finite-difference step that is too large (inaccurate) or too tiny (rounding noise). Around to with a centered difference is a good default.
- Concluding that a slope of zero means a minimum. It can be a maximum or a saddle point.
In the wild
Every deep learning framework is built around automatic differentiation: PyTorch's backward(), JAX's grad and TensorFlow's gradient tapes all apply the chain rule through the operations your model ran. When a team trains a model with billions of parameters, each training step is one forward pass to compute the loss and one backward pass of chained derivatives to get all of the gradients.
Check yourself
-
What is the derivative of f(x) = x^3 at x = 2?
Show answer
B. 12 The power rule gives f'(x) = 3x^2, and 3 x 2^2 = 12.
-
The gradient of the loss is [4, -2]. Which direction does gradient descent move the weights?
Show answer
B. [-4, 2] The gradient points uphill. Descent moves the opposite way, so the update is in the direction [-4, 2], scaled by the learning rate.
-
y = (2x + 5)^2. What is dy/dx at x = 0?
Show answer
B. 20 Chain rule: dy/du = 2u with u = 5 gives 10, and du/dx = 2, so 10 x 2 = 20.
-
Why does backpropagation use the chain rule?
Show answer
B. Because a network is a composition of functions, so each weight's gradient is a product of local slopes along the path Each layer feeds the next, so the loss is a long composition. The chain rule turns its derivative into a product of easy local slopes, computed backward.
Recap
- A derivative is the slope of a function at a point: how fast the output changes per unit of input.
- Partial derivatives take one input at a time; together they form the gradient, which points uphill.
- Training moves weights against the gradient to reduce the loss.
- The chain rule multiplies local slopes along a chain of functions.
- Backpropagation is the chain rule applied backward through a network.
Next up
- I know nothing yet learning path
- From notebook to production learning path