Machine Learning ML-03.06
Regression metrics
MAE, MSE, RMSE, MAPE and R squared: what each one measures, how outliers sway them, and how to pick the one that matches the real cost of errors.
The big idea
Two models predict food delivery times. One is always about 4 minutes off. The other is perfect on seven orders and 32 minutes late on the eighth. Which is better? It depends on what a mistake costs you. If customers mostly care about the typical wait, the two are equally good. If one very late order means a refund and an angry review, the second model is much worse. Every regression metric is a different answer to "how bad is this error?", and choosing one is a business decision, not a technical detail.
It is like grading a darts player. You could average how far each dart lands from the bullseye, or you could heavily punish the dart that hit the wall. Same throws, different scores, because you value different things.
How it works
The examples below use these 8 orders: true delivery times and two models' predictions.
import numpy as np
from sklearn.metrics import (mean_absolute_error, mean_squared_error, r2_score,
mean_absolute_percentage_error)
# Delivery times in minutes for 8 orders, and two models' predictions
y = np.array([30, 25, 40, 35, 50, 28, 45, 32.0])
steady = y + np.array([4, -4, 4, -4, 4, -4, 4, -4]) # always 4 minutes off
spiky = y + np.array([0, 0, 0, 0, 32, 0, 0, 0]) # perfect, except one 32-minute miss
def report(name, y, p):
mse = mean_squared_error(y, p)
print(f"{name:<8} MAE {mean_absolute_error(y, p):5.1f} MSE {mse:6.1f} RMSE {np.sqrt(mse):5.1f} "
f"MAPE {mean_absolute_percentage_error(y, p):6.1%} R2 {r2_score(y, p):5.2f}")
report("steady", y, steady)
report("spiky", y, spiky)
report("mean", y, np.full_like(y, y.mean())) # baseline: always predict the mean
# MAPE breaks near zero: daily sales of a slow product
sales = np.array([0.5, 1, 2, 10, 12.0])
pred = np.array([1.5, 2, 3, 11, 13.0]) # every prediction off by exactly 1
print(f"\nsales: MAE {mean_absolute_error(sales, pred):.1f}, "
f"MAPE {mean_absolute_percentage_error(sales, pred):.0%}")
print("per-row percentage errors:", (np.abs(pred - sales) / sales * 100).round(0).tolist())Output
steady MAE 4.0 MSE 16.0 RMSE 4.0 MAPE 11.8% R2 0.76
spiky MAE 4.0 MSE 128.0 RMSE 11.3 MAPE 8.0% R2 -0.93
mean MAE 7.0 MSE 66.2 RMSE 8.1 MAPE 20.3% R2 0.00
sales: MAE 1.0, MAPE 74%
per-row percentage errors: [200.0, 100.0, 50.0, 10.0, 8.0] Step 1: MAE, the typical miss
Mean absolute errorMean absolute error: the average of the absolute residuals. In the target's units and equally sensitive to every minute of error. averages the size of the errors:
It is in the same units as the target, so "MAE 4.0" means "on average about 4 minutes off". Every minute of error counts the same, wherever it occurs. Both models score exactly 4.0: MAE cannot tell them apart.
Step 2: MSE and RMSE, punishing big misses
MSE averages squared errors, as in Linear regression intuition . Squaring makes one 32-minute miss count as much as 64 four-minute misses (). The spiky model's MSE is 128 against the steady model's 16. RMSERoot mean squared error: the square root of MSE, back in the target's units. Always at least as large as MAE, and much larger when there are a few big errors. takes the square root to return to minutes: 11.3 versus 4.0. A useful habit: compare RMSE with MAE. When RMSE is far above MAE, a few large errors dominate.
Step 3: MAPE, errors as percentages
MAPEMean absolute percentage error: the average of absolute errors divided by the true values. Scale-free, but undefined at zero and explosive near it. divides each error by the true value:
Percentages are easy to explain and let you compare across products of very different sizes. But they break near zero. In the sales example every prediction is off by exactly 1 unit, yet MAPE is 74 percent: the product that sold 0.5 units contributes a 200 percent error. MAPE is undefined when a true value is 0, and it punishes over-prediction more than under-prediction on the same item, because under-prediction can never exceed 100 percent.
Step 4: R squared, compared with a baseline
R squaredThe share of the target's variance explained by the model: 1 minus the model's MSE divided by the MSE of always predicting the mean. 1 is perfect, 0 matches the mean, negative is worse than the mean. () compares your model with the dumbest sensible model, one that always predicts the mean:
is perfect, is no better than the mean, and it can be negative when the model is worse than the mean. The steady model explains 76 percent of the variance (). The spiky model scores -0.93: its single huge miss makes its squared error larger than simply predicting the average every time. has no units, which helps when comparing across problems, but it says nothing about whether the errors are acceptable in minutes or euros.
Step 5: Pick the metric that matches the cost
- Every minute late costs the same: MAE.
- Big misses are disproportionately bad (stockouts, missed deadlines, safety margins): RMSE.
- Errors matter relative to size, and no true values are near zero: MAPE, or the weighted variant that divides total absolute error by total actuals.
- Explaining how much of the variation the model captures: R squared, alongside an error in real units.
Train with a loss that matches the metric you report where you can. Minimising MSE gives predictions that aim at the mean; minimising MAE aims at the median, which is more robust to outliers in the targets.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| MAE | Costs grow linearly with error; you want a robust, easy-to-explain number | Rare large errors are much more costly than small ones |
| RMSE / MSE | Large errors are disproportionately bad; standard training loss | Outliers in the data dominate the score |
| MAPE | Comparing across items of very different scale, explaining to non-experts | True values at or near zero |
| R squared | Comparing against the mean baseline, unitless summaries | You need to know whether errors are acceptable in real units |
Common mistakes
- Reporting only R squared, with no sense of the error in real units.
- Using MAPE on data with zeros or tiny values, such as sales of slow-moving products.
- Comparing RMSE across datasets with different units or scales.
- Optimising one metric in training and judging on another without noticing they disagree.
- Ignoring the error distribution: always look at the largest errors, not just the average.
In the wild
Forecasting teams commonly report MAE or RMSE in units the business understands, such as units sold or minutes late, and often a scale-free measure for comparing many products at once ( Backtesting and forecast evaluation ). Many forecasting competitions and benchmarks use scaled error measures specifically because MAPE breaks down for intermittent demand with zeros. In pricing and finance, asymmetric costs (under-predicting is worse than over-predicting, or the reverse) lead teams to use quantile losses that penalise each direction differently.
Check yourself
-
Model A has MAE 5 and RMSE 6. Model B has MAE 5 and RMSE 15. What is different about B?
Show answer
B. B has a few very large errors Same average miss, but RMSE squares errors. A much higher RMSE means some errors are far larger than the typical one.
-
A model has R squared = -0.3. What does that mean?
Show answer
B. It does worse than always predicting the mean R squared compares the model's squared error with the mean baseline's. Negative means the model's error is larger than the baseline's.
-
Why is MAPE a poor choice for daily sales of a product that often sells 0 or 1 units?
Show answer
B. Dividing by tiny or zero true values makes errors explode or become undefined An error of 1 unit on a day with 0.5 sales is a 200 percent error. With zeros, MAPE is undefined.
-
A late delivery costs the same per minute regardless of how late it is. Which metric fits best?
Show answer
B. MAE MAE counts each minute equally, which matches a cost that grows linearly with lateness.
Recap
- MAE is the typical miss in real units and treats every unit of error equally.
- MSE and RMSE square errors, so a few big misses dominate; compare RMSE with MAE.
- MAPE gives percentages but breaks near zero and is lopsided.
- R squared compares with always predicting the mean and can be negative.
- Choose the metric that matches the real cost of errors, and report errors in real units too.
Next up
- From notebook to production learning path