Machine Learning ML-01.09
Optimization basics
See training as optimization: objective functions, local and global minima, convex versus non-convex losses, and how constraints shape solutions.
The big idea
Every model you will train in this track is solving the same kind of problem: find the parameter values that make some number as small as possible. That number is the objective functionThe function training tries to minimize (or maximize), such as the average prediction error. Also called the loss or cost., usually called the loss. Training is optimizationFinding the inputs that make a function as small (or as large) as possible.: searching a landscape of possible parameter settings for its lowest point.
Picture the loss as a mountain landscape at night, with each location a possible set of weights and the height the error. Some landscapes are a single smooth bowl: walk downhill from anywhere and you reach the bottom. Others are rugged ranges with many valleys, ridges and flat plateaus. Which kind of landscape you are in decides how hard training is and whether you can trust that you found the best answer.
How it works
Step 1: Minima, local and global
A global minimumThe lowest point of the whole function: no other parameter setting gives a smaller loss. is the lowest point anywhere. A local minimumA point lower than all its immediate neighbours, but not necessarily the lowest overall. is lower than everything around it but possibly higher than a valley somewhere else. At both, the gradient is zero: the ground is flat. So are saddle pointsA point where the slope is zero but the surface goes up in some directions and down in others, like the middle of a horse saddle., which curve up in one direction and down in another.
import numpy as np
f = lambda x: x**4 - 3 * x**2 + x # non-convex: two valleys
df = lambda x: 4 * x**3 - 6 * x + 1
for start in (-2.0, 2.0):
x = start
for _ in range(200):
x -= 0.01 * df(x) # plain gradient descent
print(f"start {start:+.1f} -> x = {x:+.3f}, f(x) = {f(x):+.3f}")Output
start -2.0 -> x = -1.301, f(x) = -3.514
start +2.0 -> x = +1.131, f(x) = -1.070 Same algorithm, same learning rate, different starting points, different answers: the start at +2 settles into the shallower valley and stays there, because the ground slopes uphill in every direction around it.
Step 2: Convex versus non-convex
A function is convexShaped like a single bowl: the straight line between any two points on the curve never goes below it, so there is one global minimum and no other dips. if it is one bowl with no other dips. Formally, a straight line between any two points on its graph never dips below the graph. Convex problems are the friendly case: any local minimum is the global minimum, so walking downhill from anywhere finds the best answer.
- Convex: linear regression with squared error, logistic regression, linear SVMs. Training is reliable and the answer does not depend on where you start.
- Non-convex: neural networks. The landscape has huge numbers of minima and saddle points.
Surprisingly, non-convex training works well in practice. In very high dimensions, most points with zero gradient are saddle points rather than bad local minima, gradient methods with noise slip past them, and many of the minima networks find perform about equally well. Momentum-based optimizers help cross flat regions ( Optimizers: SGD, momentum, RMSProp, Adam, AdamW ).
Step 3: Constraints and regularization
Sometimes not every parameter setting is allowed or desirable: probabilities must sum to 1, weights should stay small. A constraintA restriction on which parameter values are allowed, such as all weights being non-negative. limits the search to a region. In ML, a softer version is common: add a penalty to the objective, such as , so that large weights cost something. This regularizationAdding a penalty to the loss that discourages complex solutions, such as large weights, to improve generalization. reshapes the landscape so the optimizer prefers simpler solutions, which usually generalize better ( Regularization: ridge, lasso, elastic net ).
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Convex model | You want guarantees, reproducibility and fast reliable training on modest data | The pattern is too complex for a linear boundary, such as images or language |
| Non-convex model (neural network) | Complex patterns with lots of data; the flexibility pays off | Results depend on initialization, random seeds and tuning |
| Multiple restarts or seeds | Non-convex problems where you need robust results | Compute is tight |
Common mistakes
- Assuming a zero gradient means a minimum. It can be a maximum or a saddle.
- Reporting a single training run of a neural network as if the result were exact. Different seeds land in different minima.
- Forgetting that regularization changes the objective, so "the best training loss" is no longer the goal by itself.
- Treating optimization and generalization as the same thing. The lowest training loss can be the most overfit model.
In the wild
Linear and logistic regression are solved as convex problems, which is why libraries return the same answer every time. Deep learning practitioners routinely train the same network with several random seeds and report the spread, because non-convex optimization is sensitive to the start. Hyperparameter search itself is an outer optimization problem over a noisy, non-convex landscape ( Hyperparameter tuning ).
Check yourself
-
What is special about a convex loss function?
Show answer
B. Any local minimum is the global minimum A convex function is a single bowl: walking downhill from anywhere reaches the same global minimum.
-
Gradient descent stops where the gradient is zero. Which of these could that point be?
Show answer
B. A minimum, a maximum or a saddle point A zero gradient only means the ground is flat. Checking the curvature, or simply the loss, tells you which kind of point it is.
-
Two training runs of the same neural network with different random seeds reach different final losses. Why?
Show answer
B. The loss is non-convex, so different starting points settle in different minima Neural network losses have many minima. Different initializations and mini-batch orders lead to different, often similarly good, solutions.
-
Adding lambda times the squared norm of the weights to the loss does what?
Show answer
B. Penalizes large weights so the optimizer prefers simpler solutions That is L2 regularization: it reshapes the objective so large weights cost extra, which usually improves generalization.
Recap
- Training is optimization: find the parameters that minimize the loss.
- A global minimum is the lowest point overall; local minima and saddle points also have zero gradient.
- Convex losses have one bowl, so downhill always finds the best answer; neural network losses are non-convex.
- Non-convex training still works well in practice, but results depend on the starting point.
- Constraints and regularization reshape the objective to prefer simpler solutions.
Next up
- From notebook to production learning path