Loss functions

The big idea

A network learns by nudging its weights to make one number smaller. That number is the lossA function that measures how wrong a model's prediction is for one example; averaged over the data it gives the number training tries to minimize.. It is the only thing the network is told about what "good" means, so choosing it is choosing what the model will care about.

Think of a teacher grading homework. One teacher takes off points in proportion to the square of each mistake, so one wild answer ruins the grade. Another takes off a fixed amount per unit of error, so a wild answer counts, but not catastrophically. A third only cares whether you were confidently right. Students adapt to whichever teacher they have, and models do too.

Every loss has a shape: how fast the penalty grows as the error grows. That shape decides how strongly each example pulls on the weights, which is the gradient. Four losses cover most of practice: mean squared error, cross-entropy, hinge and Huber.

How it works

Step 1: Mean squared error for regression

Mean squared errorMean squared error: the average of squared differences between predictions and targets. The standard loss for regression. (MSE) squares each residual. Small errors cost little; large errors cost a lot. The gradient is proportional to the error itself, so big mistakes pull hardest. Minimizing MSE makes the model predict the mean of y for each input, which is usually what you want for regression.

The flip side: in the code, three predictions with one error of 0.5 give MSE 0.08. Make that one error 10 and MSE jumps to 33.33, a factor of 400 for a factor of 20 in error. A few bad labels can dominate training.

Step 2: MAE and Huber for robustness

Mean absolute errorMean absolute error: the average absolute difference between predictions and targets. Less sensitive to outliers than MSE; minimizing it predicts the median. (MAE) grows linearly: the same example scores 3.33 instead of 33.33. It predicts the median and shrugs off outliers, but its gradient has the same size for tiny and huge errors, so it does not slow down near the answer.

Huber lossA loss that is quadratic for small errors (below a threshold delta) and linear for large ones, combining MSE's smoothness with MAE's robustness to outliers. takes the best of both: squared for errors below a threshold delta, linear above it.

In the code, four corrupted labels (each 40 too high) drag an ordinary least-squares line to slope 0.98 instead of the true 2.00. A Huber fit on the same data finds slope 2.01.

Step 3: Cross-entropy for classification

For a classifier that outputs a probability p for the true class:

Cross-entropyThe loss -log(p) of the probability a model assigned to the correct class. It punishes confident wrong predictions very heavily. Also called log loss. is 0.11 when p = 0.9, 0.69 at a coin flip (p = 0.5), 2.30 at p = 0.1 and 4.61 at p = 0.01. It grows without limit as the model becomes confidently wrong. Squared error on the same probabilities tops out at 1, so a confidently wrong classifier barely feels it, and combined with a sigmoid its gradient almost vanishes exactly when the model is most wrong.

Cross-entropy is also what maximum likelihood gives for a model that outputs probabilities: minimizing it makes the model's probabilities match the data's frequencies. Pair binary cross-entropy with a sigmoid output, and categorical cross-entropy with softmax ( Multi-class and multi-label classification ). Libraries combine them in one numerically stable function that takes raw scores (logits), so do not apply the sigmoid or softmax yourself first.

Step 4: Hinge loss for margins

For labels y = -1 or +1 and a raw score s, the hinge lossThe loss max(0, 1 - y * s) for labels -1 or +1 and raw score s. It is zero once an example is on the correct side with a margin of at least 1. Used by support vector machines. is max(0, 1 - y s). It is zero once the example is on the right side with margin at least 1, and grows linearly otherwise: scores of +2 and +1 cost nothing, +0.3 costs 0.70, -0.5 costs 1.50. It ignores examples that are already safely right and focuses on those near or across the boundary. It is the loss of the support vector machine ( Support Vector Machines and kernels ) and gives scores, not probabilities.

Step 5: Match the loss to the task

• Predicting a number: MSE by default; MAE or Huber when there are outliers or heavy tails.
• Predicting one of two classes: binary cross-entropy with a sigmoid output.
• Predicting one of many classes: categorical cross-entropy with softmax.
• Margin-based classifiers without probabilities: hinge.

Also remember that the training loss and the business metric can differ. You may train with cross-entropy and report recall at a fixed precision ( Classification metrics ). The loss must be smooth enough to give useful gradients; the metric is what you actually care about.

Going deeper

Code: the four losses side by side

import numpy as np
from sklearn.linear_model import HuberRegressor, LinearRegression

mse = lambda y, p: np.mean((y - p) ** 2)
mae = lambda y, p: np.mean(np.abs(y - p))
def huber(y, p, d=1.0):
    r = np.abs(y - p)
    return np.mean(np.where(r <= d, 0.5 * r ** 2, d * (r - 0.5 * d)))
bce = lambda y, p: -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
hinge = lambda y, s: np.mean(np.maximum(0, 1 - y * s))  # y in {-1, +1}, s = raw score

# Regression: one prediction off by 0.5, then off by 10
y = np.array([3.0, 5.0, 7.0])
for p in [np.array([3.5, 5.0, 7.0]), np.array([13.0, 5.0, 7.0])]:
    print(f"errors {y - p}:  MSE {mse(y, p):6.2f}   MAE {mae(y, p):5.2f}   Huber {huber(y, p):5.2f}")

# Classification: the true class is 1. How does the loss react to the predicted probability?
for p in [0.9, 0.6, 0.5, 0.1, 0.01]:
    print(f"p(true class) = {p:<4}  cross-entropy {bce(np.array([1]), np.array([p])):5.2f}   "
          f"squared error {mse(np.array([1]), np.array([p])):4.2f}")

# Hinge loss on raw scores: zero once the margin is at least 1
for s in [2.0, 1.0, 0.3, -0.5]:
    print(f"score {s:+.1f} for a +1 example: hinge {hinge(np.array([1]), np.array([s])):.2f}")

# Outliers: fit a line with squared loss vs Huber loss
rng = np.random.default_rng(0)
x = rng.uniform(0, 10, 50)
t = 2 * x + 1 + rng.normal(0, 1, 50)
t[:4] += 40  # four corrupted labels
X = x.reshape(-1, 1)
ls, hb = LinearRegression().fit(X, t), HuberRegressor().fit(X, t)
print(f"\ntrue line:        slope 2.00, intercept 1.00")
print(f"squared loss fit: slope {ls.coef_[0]:.2f}, intercept {ls.intercept_:.2f}")
print(f"Huber loss fit:   slope {hb.coef_[0]:.2f}, intercept {hb.intercept_:.2f}")

errors [-0.5  0.   0. ]:  MSE   0.08   MAE  0.17   Huber  0.04
errors [-10.   0.   0.]:  MSE  33.33   MAE  3.33   Huber  3.17
p(true class) = 0.9   cross-entropy  0.11   squared error 0.01
p(true class) = 0.6   cross-entropy  0.51   squared error 0.16
p(true class) = 0.5   cross-entropy  0.69   squared error 0.25
p(true class) = 0.1   cross-entropy  2.30   squared error 0.81
p(true class) = 0.01  cross-entropy  4.61   squared error 0.98
score +2.0 for a +1 example: hinge 0.00
score +1.0 for a +1 example: hinge 0.00
score +0.3 for a +1 example: hinge 0.70
score -0.5 for a +1 example: hinge 1.50

true line:        slope 2.00, intercept 1.00
squared loss fit: slope 0.98, intercept 9.63
Huber loss fit:   slope 2.01, intercept 1.10

Variations you will meet

Class weights multiply the loss of rare classes so they are not ignored. Focal lossA variant of cross-entropy that down-weights easy, well-classified examples so training focuses on hard ones; common in object detection. scales cross-entropy by (1 - p)^gamma, shrinking the loss of easy examples. Label smoothing replaces the hard target 1 with something like 0.9, which stops the model pushing its logits to extremes. Quantile (pinball) loss predicts a chosen percentile instead of the mean, useful for "90 percent of deliveries arrive by" estimates. Contrastive losses, which pull matching pairs together and push others apart, train embedding models.

Trade-offs

Common mistakes

• Using MSE on probabilities for classification instead of cross-entropy.
• Applying softmax yourself and then passing the result to a loss that expects logits, applying it twice.
• Training regression with MSE on data with a handful of huge, wrong labels.
• Comparing loss values across different loss functions as if they were on the same scale.
• Assuming a lower loss always means a better product metric.

In the wild

Cross-entropy trains nearly every classifier and every language model, which predicts the next token with categorical cross-entropy over its whole vocabulary. Ranking and recommendation systems often train with cross-entropy on clicks. Forecasting systems use quantile losses to give ranges instead of single numbers. Robust losses like Huber are standard in reinforcement learning value updates and in regression on noisy sensor data.

Check yourself

Recap