Machine Learning ML-03.04
Polynomial regression and basis functions
Fit curves with a linear model by adding powers of x as features, and watch higher degrees slide from underfitting into overfitting.
The big idea
Fuel economy rises with speed up to a point and then falls. Sales climb in spring, peak in summer, drop in winter. A straight line cannot follow a curve like that. The surprising fix is not a new model: keep linear regression, but feed it extra features made from the original one, such as and . The model is still a weighted sum, but the sum of curved pieces can bend.
Think of building a shape from Lego bricks. A straight line is one long brick. Give yourself a few different curved bricks and you can build an arch, as long as you choose how many of each to use. Polynomial regression gives the model curved bricks and lets least squares choose the amounts. The catch: with too many bricks you can build a shape that passes through every training point exactly, noise and all.
How it works
Step 1: Turn one feature into several
Polynomial regressionLinear regression on powers of a feature (x, x squared, x cubed and so on), which lets a linear model fit curves. of degree predicts
The curve is not linear in , but it is linear in the weights, so everything from the last three lessons still works: MSE, the normal equation, gradient descent. You just build a new feature matrix whose columns are . With and degree 3, the row is .
More generally, any fixed transformation of the inputs is a basis functionA fixed function applied to the raw inputs to create new features, such as a power, a sine wave or a bump around a point. A linear model on basis functions can fit non-linear shapes.: powers, logs, sine waves for seasonal patterns, or bumps centred on chosen points. A linear model on good basis functions can fit very non-linear shapes while staying fast and convex.
Step 2: Degree controls flexibility
- Degree 0: a flat line at the average. Ignores completely.
- Degree 1: an ordinary straight line.
- Degree 3 or 4: smooth curves with a bend or two. Often enough.
- Degree 11 with 12 training points: a curve with 12 weights can pass through all 12 points exactly. Training error hits zero.
That last case sounds perfect and is the worst of all. The curve swings wildly between the points to hit each one, so on any new point it can be badly wrong.
Step 3: Training error versus validation error
UnderfittingWhen a model is too simple to capture the real pattern, so it does badly on training and new data alike. (left): the model is too simple, and both errors are high. OverfittingWhen a model fits the training data, including its noise, so closely that it does worse on new data. (right): training error keeps falling, but validation error climbs, because the model is memorising the noise in 12 points rather than learning the curve. The degree is a hyperparameterA setting chosen before training rather than learned from the data, such as polynomial degree or learning rate. Chosen using validation data.: you pick it by validation error, never by training error. This trade-off has a name, bias versus variance, covered in Bias, variance, underfitting and overfitting .
Step 4: Warning signs of overfitting
- Training error far below validation error.
- Huge weights that nearly cancel each other. In the code below, the largest weight goes from under 2 at degree 3 to over 4,000 at degree 15. Ridge and lasso regularization exist to stop exactly this.
- Wild swings at the edges of the data, where high powers of explode. Never trust a high-degree polynomial outside the range it was trained on.
import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(1)
x = rng.uniform(-1, 1, 60)
y = np.sin(2.6 * x) + 0.3 * x + rng.normal(0, 0.25, 60) # a curve plus noise
# Keep only 18 points for training, so high degrees have room to overfit
x_tr, x_va, y_tr, y_va = train_test_split(x.reshape(-1, 1), y, test_size=0.7, random_state=0)
print("PolynomialFeatures(3) turns x = 0.5 into:",
PolynomialFeatures(3).fit_transform([[0.5]])[0].tolist())
print()
for degree in (1, 3, 5, 9, 15):
model = make_pipeline(PolynomialFeatures(degree), StandardScaler(), LinearRegression())
model.fit(x_tr, y_tr)
mse_tr = np.mean((model.predict(x_tr) - y_tr) ** 2)
mse_va = np.mean((model.predict(x_va) - y_va) ** 2)
biggest = np.abs(model[-1].coef_).max()
print(f"degree {degree:>2}: train MSE {mse_tr:.3f} validation MSE {mse_va:8.3f} "
f"largest weight {biggest:10.1f}")Output
PolynomialFeatures(3) turns x = 0.5 into: [1.0, 0.5, 0.25, 0.125]
degree 1: train MSE 0.108 validation MSE 0.177 largest weight 0.9
degree 3: train MSE 0.021 validation MSE 0.050 largest weight 1.8
degree 5: train MSE 0.016 validation MSE 0.051 largest weight 1.8
degree 9: train MSE 0.008 validation MSE 0.129 largest weight 17.9
degree 15: train MSE 0.004 validation MSE 0.969 largest weight 4309.3 Try it
Try this:
- Start at degree 1 and step up one at a time. At which degree does the curve first follow the shape of the dashed truth line?
- Go to degree 11. Training error is 0: the curve hits every white point. What does it do between and beyond them? Look at the validation error.
- Switch to "training points only" and pick the degree you would choose without validation data. Then switch the validation points back on. Would you have picked well?
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Low degree (1 to 3) | Little data, noisy data, or you need safe behaviour near the edges | The true curve has several bends |
| High degree | Lots of data covering the whole range, with a genuinely wiggly pattern | Few points, extrapolation, or unscaled inputs |
| Polynomial plus regularization | You want flexibility but must keep weights in check (next lesson) | You need an exact, unpenalised fit |
| Other basis functions (splines, sines) | Seasonal patterns or local bends, with stable behaviour at the edges | You need a single simple formula |
Common mistakes
- Choosing the degree by training error, which always favours the highest degree.
- Extrapolating a polynomial beyond the training range.
- Forgetting to scale: of a feature in the thousands overflows and wrecks numerical precision.
- Expanding many features to high degree: with 20 features, degree 3 already creates 1,771 columns including all interactions.
- Reading the individual polynomial weights as meaningful; the shape of the curve is what matters.
In the wild
Polynomial and other basis-function features are standard in engineering and science for smooth curves such as calibration curves or dose-response relationships. Forecasting libraries model yearly and weekly seasonality with sine and cosine basis functions ( Prophet and modern forecasting libraries ). Splines, piecewise polynomials joined smoothly, are a staple of statistics because they bend where the data needs it without the wild edges of a single high-degree polynomial.
Check yourself
-
Why is polynomial regression still called linear regression?
Show answer
B. Because the prediction is linear in the weights, even though it is curved in x The features x, x^2, x^3 are fixed transformations. The model is still a weighted sum of them, so least squares and all its tools still apply.
-
As degree increases, what always happens to training error (on the same data)?
Show answer
A. It goes down or stays the same A higher-degree polynomial can always copy a lower-degree one by setting the extra weights to 0, so its best training fit is at least as good.
-
Degree 9 has training MSE 0.008 and validation MSE 0.129; degree 3 has 0.021 and 0.050. Which do you pick?
Show answer
B. Degree 3, lower validation error Validation error estimates performance on new data. The degree 9 model's lower training error is partly memorised noise.
-
A degree 12 model fitted on house sizes from 40 to 150 m2 predicts a negative price for a 200 m2 house. Why?
Show answer
B. High-degree polynomials swing wildly outside the training range High powers grow explosively beyond the data, and nothing in training constrained the curve there. Do not extrapolate polynomials.
Recap
- Adding powers of x as features lets a linear model fit curves; the model stays linear in its weights.
- Any fixed transformation of the inputs is a basis function; powers are just one choice.
- Higher degree always lowers training error, but validation error falls then rises.
- Underfitting means too simple; overfitting means memorising noise. Pick the degree on validation data.
- Huge weights and wild edges are signs of overfitting; never extrapolate a high-degree polynomial.
Next up
- From notebook to production learning path