Machine Learning ML-03.03
Gradient descent, the engine of ML
See how models learn by walking downhill on a loss surface, why the learning rate matters so much, and how batch, mini-batch and SGD differ.
The big idea
Imagine you are standing on a foggy hillside at night and you want to reach the lowest point in the valley. You cannot see the valley. All you can feel is the slope of the ground under your feet. A sensible plan: feel which way is downhill, take a step that way, and repeat. Big steps when the ground is steep, small steps as it flattens out.
That is gradient descent, and it is how almost every modern model learns, from a two-parameter line fit to a neural network with billions of weights. The "hillside" is the loss functionA function that measures how wrong the model's predictions are on the training data. Lower is better., your position is the current set of model weights, and "feeling the slope" is computing the gradientThe vector of partial derivatives of the loss with respect to each weight. It points in the direction of steepest increase..
How it works
Step 1: The loss is a landscape
Take the line model from Linear regression intuition : y_hat = w * x + b. For every possible pair (w, b) you can compute the mean squared error on your data. Plot that error as a height above the (w, b) plane and you get a bowl-shaped surface. The best line is the bottom of the bowl.
Seen from above, the bowl becomes contour lines, like a topographic map: each ellipse joins points with the same loss, and the minimum sits in the middle.
Step 2: The gradient points uphill, so step the other way
The gradient is a vector made of the partial derivatives, one per weight:
It points in the direction where the loss increases fastest. We want the opposite, so the update rule is:
The Greek letter eta is the learning rateA small positive number that scales the size of each update step.. That single line is the whole algorithm. Everything else in this lesson is about doing it well.
Step 3: A worked example with real numbers
Data: three points (1, 2), (2, 4), (3, 6). The true answer is w = 2, b = 0. Start at w = 0, b = 0, learning rate 0.1.
For MSE, the gradients are (derived in Solving linear regression: normal equation and gradient descent ):
With w = b = 0, every prediction is 0, so the errors are -2, -4, -6.
- dL/dw = (2/3) x (-2 x 1 - 4 x 2 - 6 x 3) = (2/3) x (-28) = -18.67
- dL/db = (2/3) x (-2 - 4 - 6) = -8
Update: w = 0 - 0.1 x (-18.67) = 1.87, b = 0 - 0.1 x (-8) = 0.8. One step and w jumped from 0 to 1.87, close to the true 2. That is the long first step in Figure 1. The next steps make smaller corrections as the errors shrink.
Step 4: The learning rate decides everything
- Too small (0.05). Twelve steps and the ball has only crept part of the way down. Training works but wastes time.
- Just right (0.3). Big steps where it is steep, small ones near the bottom. Six steps land almost exactly on the minimum.
- Too large (1.02). Each step jumps past the minimum to a higher point on the other side. The loss grows every step: training diverges.
- Too small: every step is tiny. Training takes forever and may stop early in a flat region.
- Just right: fast at first, then settles into the minimum.
- Too large: each step jumps past the minimum to the other side of the valley, which may be even higher. The loss bounces or explodes to infinity (you will see
NaNin your logs).
There is no universal good value. Common starting points are 0.1 or 0.01 for plain gradient descent on scaled features, and around 0.001 for Adam on neural networks (see Optimizers: SGD, momentum, RMSProp, Adam, AdamW ).
Try it
Try this:
- Leave the learning rate at 0.1 and press Play. Count how many steps it takes to reach the minimum.
- Raise the learning rate to 0.9 and run again. Watch the path zig-zag. Push it to 1.1 and watch it diverge.
- Switch the surface to "ravine" (a long thin valley). Plain gradient descent stalls along the valley floor. Now set momentum to 0.9 and compare.
Going deeper
Batch, mini-batch and stochastic
The exact gradient averages over every training example. With 10 million examples, that is expensive for one step. So in practice we estimate it:
- Batch gradient descent: use all examples per step. Accurate, smooth, slow per step.
- Stochastic gradient descentStochastic gradient descent: estimating the gradient from one example, or a small random batch, per step instead of the whole dataset. (SGD): use one example per step. Very noisy, but cheap and fast to start.
- Mini-batch: use a small batch, typically 32 to 1024 examples. The standard choice: noise is moderate, and batches map well onto GPU parallelism.
One full pass over the training set is an epochOne complete pass of the training algorithm over the whole training dataset.. Noise from mini-batches is not purely bad: it can help the optimizer escape shallow local minima and saddle points in neural networks.
Code: gradient descent from scratch
import numpy as np
X = np.array([1.0, 2.0, 3.0]); y = np.array([2.0, 4.0, 6.0])
w, b, lr = 0.0, 0.0, 0.1
for step in range(50):
y_hat = w * X + b
err = y_hat - y
dw = 2 * np.mean(err * X) # dL/dw
db = 2 * np.mean(err) # dL/db
w, b = w - lr * dw, b - lr * db
if step % 10 == 0:
print(f"step {step:2d} loss {np.mean(err**2):.4f} w {w:.3f} b {b:.3f}")
print("final:", round(w, 3), round(b, 3))
# Mini-batch SGD on a bigger version of the same problem: 10,000 noisy points
rng = np.random.default_rng(0)
Xb = rng.uniform(0, 3, 10_000); yb = 2 * Xb + rng.normal(0, 0.3, 10_000)
w, b = 0.0, 0.0
for epoch in range(3):
for idx in np.array_split(rng.permutation(10_000), 10_000 // 64): # batches of ~64
err = w * Xb[idx] + b - yb[idx]
w -= 0.05 * 2 * np.mean(err * Xb[idx]); b -= 0.05 * 2 * np.mean(err)
print(f"epoch {epoch + 1}: w {w:.3f} b {b:.3f}")Output
step 0 loss 18.6667 w 1.867 b 0.800
step 10 loss 0.0479 w 1.752 b 0.564
step 20 loss 0.0295 w 1.805 b 0.442
step 30 loss 0.0181 w 1.847 b 0.347
step 40 loss 0.0111 w 1.880 b 0.272
final: 1.904 0.218
epoch 1: w 1.981 b 0.037
epoch 2: w 1.986 b -0.001
epoch 3: w 1.997 b -0.012 The first part is the worked example. Notice that after 50 steps w is 1.904 and b is 0.218: close, but still creeping. That is the narrow valley of Figure 1. The second part runs mini-batch gradient descent with batches of about 64 on 10,000 noisy points: within one epoch, about 156 small cheap steps, it is already at w = 1.98.
Why feature scaling matters
If one feature ranges from 0 to 1 and another from 0 to 100,000, the loss bowl becomes an extremely stretched ellipse. A learning rate small enough for the steep direction is far too small for the flat one, so training crawls. Standardizing features (zero mean, unit variance, see Feature engineering ) makes the bowl rounder and gradient descent dramatically faster. Even in the tiny example, the valley is stretched because x is never centred on zero, which ties w and b together.
Convex and non-convex
For linear regression, the loss is convexA function shaped like a single bowl, with one global minimum and no other dips.: any downhill path ends at the global minimum. Neural network losses are non-convex, full of saddle points and many minima. Gradient descent still works remarkably well there, and momentumAn optimizer tweak that keeps a running average of past gradients, so steps build up speed along consistent directions and damp out zig-zags.-based optimizers help it move through flat and curved regions. The Distill article below shows this visually.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Batch GD | Small datasets; you want smooth, deterministic convergence | Each step touches all data; impractical for millions of rows |
| Mini-batch GD | Almost every real training job; uses GPU parallelism well | Adds batch size as another hyperparameter to tune |
| Pure SGD | Streaming or online learning, one example at a time | Very noisy updates; needs a decaying learning rate to settle |
| Closed-form solution | Linear regression with few features | Matrix inversion becomes too slow and memory-hungry at scale; does not exist for most models |
Common mistakes
- Not scaling features, then blaming the learning rate for slow training.
- Picking a learning rate once and never checking the loss curve. Always plot loss per step.
- Mixing up the sign: adding the gradient climbs uphill and the loss grows every step.
- Judging progress on training loss alone; watch validation loss too (see Bias, variance, underfitting and overfitting ).
- Treating a
NaNloss as a mystery. It is almost always a learning rate that is too high, or bad input data.
In the wild
Distill
Why momentum really works (opens in a new tab)An interactive article that studies gradient descent on simple quadratic bowls, where "pathological curvature", long narrow valleys, makes plain gradient descent either bounce between the walls or creep timidly along the floor. It shows how momentum, a short-term memory of past gradients, speeds progress along the valley and lets larger step sizes work, and explains why the improvement is so large: in the best case, momentum's convergence depends on the square root of the valley's stretch rather than the stretch itself. It is the clearest visual explanation of the ravine in Try it, step 3.
Check yourself
-
The gradient of the loss points in which direction?
Show answer
B. The direction of steepest increase in loss The gradient points uphill. That is why the update subtracts it: we move in the opposite direction to go downhill.
-
Your loss goes 2.1, 5.8, 31.0, then NaN. What is the most likely fix?
Show answer
B. Decrease the learning rate A loss that grows each step means updates overshoot the minimum. A smaller learning rate keeps steps inside the valley.
-
Why is mini-batch gradient descent the default for deep learning?
Show answer
B. It balances gradient accuracy with cheap steps that parallelize well on GPUs Mini-batches give a good-enough gradient estimate at a fraction of the cost, and batch operations map naturally onto GPU hardware.
-
Training is very slow, and one feature is in dollars (0 to 500,000) while another is a 0 to 1 ratio. What should you try first?
Show answer
B. Standardize the features Wildly different scales stretch the loss surface into a thin ellipse. Scaling makes it rounder so one learning rate works in every direction.
Recap
- Gradient descent repeats one rule: new weight = old weight minus learning rate times gradient.
- The gradient points uphill; subtracting it walks downhill on the loss surface.
- The learning rate is the most important knob: too small crawls, too large diverges.
- Mini-batches give cheap, slightly noisy gradient estimates and are the standard in practice.
- Scale your features: it turns a stretched valley into a round bowl that is easy to descend.
Next up
- I know nothing yet learning path
- From notebook to production learning path
- I know backend, teach me AI learning path