Skip to content
learn
  1. Learn
  2. Machine Learning
  3. Mathematics for ML (only what you need, visually)

Machine Learning ML-01.04

Derivatives and the chain rule

Read a derivative as a slope, combine partial derivatives into a gradient, and see how the chain rule lets backpropagation train networks.

Beginner 14 min Last reviewed

The big idea

Training a model means nudging its numbers so its mistakes get smaller. To nudge in the right direction you need to know one thing: if I increase this number a tiny bit, does the error go up or down, and how fast? That question has a name. It is the derivativeHow fast a function's output changes as its input changes: the slope of the function at a point., and it is the single most important piece of calculus in machine learning.

You already have the intuition. Driving up a hill, the steepness under your wheels is the derivative of your height with respect to distance. Steep and positive: you are climbing fast. Zero: you are at the top, or at the bottom of a dip. Negative: downhill. A model in training is a driver trying to reach the lowest point of a valley, and derivatives are how it feels the slope.

How it works

Step 1: Derivative as slope

For a function , the derivative is the slope of the curve at : rise over run, for a run so tiny it is practically zero.

Worked example: . At , try a small step : . As shrinks it settles on 6. The general rule for powers gives , and .

The derivative is the slope of the tangent line The curve f of x equals x squared, with a glowing tangent line touching it at x equals 3. The tangent rises 3 for every half step to the right, a slope of 6. -3-2-1012345-2024681012 f(x) = x^2 rise 3, run 0.5 slope 6 at x = 3 Figure 1. At x = 3 the tangent to x^2 rises 3 for every half step right: the derivative there is 6.

A few rules cover most of what ML needs: the derivative of is ; of is ; of is ; and the derivative of a sum is the sum of the derivatives.

Step 2: Partial derivatives and the gradient

A model's loss depends on many numbers at once, not one. A partial derivativeThe derivative of a many-input function with respect to one input, holding all the others fixed. asks about one input at a time while freezing the rest. For at :

  • Change only : .
  • Change only : .

Stack the partials into a vector and you have the gradientThe vector of all partial derivatives of a function. It points in the direction of steepest increase., written . The gradient points in the direction where the function rises fastest. Training walks the opposite way, downhill, which is gradient descent ( Gradient descent, the engine of ML ).

Step 3: The chain rule, multiplying local slopes

Models are functions of functions: the loss depends on the prediction, which depends on a layer's output, which depends on the weights. The chain ruleThe derivative of a composition equals the product of the derivatives of its parts: dy/dx = dy/du times du/dx. says the slope of a chain is the product of the slopes of its links:

Worked example: . Let , so . Locally, and . At : , so .

The chain rule as a chain of local slopes x feeds a box computing u equals 3x plus 1, which feeds a box computing y equals u squared. Forward, values flow left to right. Backward, the local slopes 3 and 2u are multiplied right to left to get dy over dx equals 42 at x equals 2. forward forward dy/du = 2u = 14 du/dx = 3, so 42 x = 2 u = 3x + 1 u = 7 y = u^2 y = 49 Figure 2. Forward: values flow right (x = 2, u = 7, y = 49). Backward: local slopes multiply on the way back (2u = 14, then times 3 = 42). That backward pass is backpropagation in miniature.

This is the foundation of training neural networks. A network with many layers is a long chain; backpropagationThe algorithm that computes the gradient of the loss for every weight by applying the chain rule backward through the network. computes every weight's gradient by walking the chain backward and multiplying local slopes, reusing work at each step ( Backpropagation ).

derivatives.pyruns with python3 + numpy
def f(x):  return x ** 2
def df(x): return 2 * x                      # derivative worked out by hand

def numeric_slope(f, x, h=1e-5):
    return (f(x + h) - f(x - h)) / (2 * h)   # tiny rise over tiny run

print(round(numeric_slope(f, 3.0), 6), df(3.0))

# Chain rule: y = (3x + 1)^2. Outer g(u) = u^2, inner u = 3x + 1.
x = 2.0
u = 3 * x + 1
dy_du = 2 * u                                # slope of the outer function
du_dx = 3                                    # slope of the inner function
print("chain rule:", dy_du * du_dx, " numeric:", round(numeric_slope(lambda t: (3 * t + 1) ** 2, x), 4))

# Partial derivatives of L(w, b) = (w * 2 + b - 5)^2 at w = 1, b = 0
L = lambda w, b: (w * 2 + b - 5) ** 2
print("dL/dw:", round(numeric_slope(lambda w: L(w, 0.0), 1.0), 4), " dL/db:", round(numeric_slope(lambda b: L(1.0, b), 0.0), 4))

Output

6.0 6.0
chain rule: 42.0  numeric: 42.0
dL/dw: -12.0  dL/db: -6.0

The numeric_slope trick, rise over a tiny run on both sides, is called a finite difference. It is slow for millions of weights, but it is the standard way to check a hand-written or automatic gradient.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. On , drag the point to . The slope is 0: you are at the bottom of the valley. That is where training wants to end up.
  2. Switch to and find both places where the slope is 0. One is a hilltop, one a valley. A slope of zero alone does not tell you which.
  3. On the sine curve, watch the slope change sign as you cross each peak. Imagine stepping downhill from different starting points: where would each one end up?

Trade-offs

OptionWins whenLoses when
Derivatives by hand Small formulas, understanding, writing a custom layer Large models: slow and error-prone
Finite differences (numeric) Checking that a gradient implementation is right Training: one or two extra function calls per weight, and rounding error
Automatic differentiation (autograd) Training any real model: exact gradients for millions of weights in one backward pass Very little; just keep track of what is recorded and free memory

Common mistakes

  • Treating the gradient as pointing downhill. It points uphill; training subtracts it.
  • Forgetting one link in the chain rule, especially the inner derivative (the 3 in ).
  • Using a finite-difference step that is too large (inaccurate) or too tiny (rounding noise). Around to with a centered difference is a good default.
  • Concluding that a slope of zero means a minimum. It can be a maximum or a saddle point.

In the wild

Every deep learning framework is built around automatic differentiation: PyTorch's backward(), JAX's grad and TensorFlow's gradient tapes all apply the chain rule through the operations your model ran. When a team trains a model with billions of parameters, each training step is one forward pass to compute the loss and one backward pass of chained derivatives to get all of the gradients.

Check yourself

  1. What is the derivative of f(x) = x^3 at x = 2?

    Show answer

    B. 12 The power rule gives f'(x) = 3x^2, and 3 x 2^2 = 12.

  2. The gradient of the loss is [4, -2]. Which direction does gradient descent move the weights?

    Show answer

    B. [-4, 2] The gradient points uphill. Descent moves the opposite way, so the update is in the direction [-4, 2], scaled by the learning rate.

  3. y = (2x + 5)^2. What is dy/dx at x = 0?

    Show answer

    B. 20 Chain rule: dy/du = 2u with u = 5 gives 10, and du/dx = 2, so 10 x 2 = 20.

  4. Why does backpropagation use the chain rule?

    Show answer

    B. Because a network is a composition of functions, so each weight's gradient is a product of local slopes along the path Each layer feeds the next, so the loss is a long composition. The chain rule turns its derivative into a product of easy local slopes, computed backward.

Recap

  • A derivative is the slope of a function at a point: how fast the output changes per unit of input.
  • Partial derivatives take one input at a time; together they form the gradient, which points uphill.
  • Training moves weights against the gradient to reduce the loss.
  • The chain rule multiplies local slopes along a chain of functions.
  • Backpropagation is the chain rule applied backward through a network.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up