Skip to content
learn
  1. Learn
  2. Machine Learning
  3. Regression

Machine Learning ML-03.04

Polynomial regression and basis functions

Fit curves with a linear model by adding powers of x as features, and watch higher degrees slide from underfitting into overfitting.

Intermediate 10 min Last reviewed

The big idea

Fuel economy rises with speed up to a point and then falls. Sales climb in spring, peak in summer, drop in winter. A straight line cannot follow a curve like that. The surprising fix is not a new model: keep linear regression, but feed it extra features made from the original one, such as and . The model is still a weighted sum, but the sum of curved pieces can bend.

Think of building a shape from Lego bricks. A straight line is one long brick. Give yourself a few different curved bricks and you can build an arch, as long as you choose how many of each to use. Polynomial regression gives the model curved bricks and lets least squares choose the amounts. The catch: with too many bricks you can build a shape that passes through every training point exactly, noise and all.

How it works

Step 1: Turn one feature into several

Polynomial regressionLinear regression on powers of a feature (x, x squared, x cubed and so on), which lets a linear model fit curves. of degree predicts

The curve is not linear in , but it is linear in the weights, so everything from the last three lessons still works: MSE, the normal equation, gradient descent. You just build a new feature matrix whose columns are . With and degree 3, the row is .

More generally, any fixed transformation of the inputs is a basis functionA fixed function applied to the raw inputs to create new features, such as a power, a sine wave or a bump around a point. A linear model on basis functions can fit non-linear shapes.: powers, logs, sine waves for seasonal patterns, or bumps centred on chosen points. A linear model on good basis functions can fit very non-linear shapes while staying fast and convex.

Step 2: Degree controls flexibility

  • Degree 0: a flat line at the average. Ignores completely.
  • Degree 1: an ordinary straight line.
  • Degree 3 or 4: smooth curves with a bend or two. Often enough.
  • Degree 11 with 12 training points: a curve with 12 weights can pass through all 12 points exactly. Training error hits zero.

That last case sounds perfect and is the worst of all. The curve swings wildly between the points to hit each one, so on any new point it can be badly wrong.

Step 3: Training error versus validation error

Training and validation error as the degree grows Error on a log scale against polynomial degree from 0 to 10, from the lab below. Training error, in purple, falls steadily as the degree rises. Validation error, in green, falls until degree 4, at 0.101, then rises sharply, reaching about 23 at degree 8. 012345678910polynomial degreeerror (MSE, log scale) 0.01 0.1 1 10 underfitting best on validation overfitting training error validation error Figure 1. Training error only goes down as the degree grows. Validation error falls, bottoms out at degree 4, then shoots up: past that point the extra flexibility is spent fitting noise.

UnderfittingWhen a model is too simple to capture the real pattern, so it does badly on training and new data alike. (left): the model is too simple, and both errors are high. OverfittingWhen a model fits the training data, including its noise, so closely that it does worse on new data. (right): training error keeps falling, but validation error climbs, because the model is memorising the noise in 12 points rather than learning the curve. The degree is a hyperparameterA setting chosen before training rather than learned from the data, such as polynomial degree or learning rate. Chosen using validation data.: you pick it by validation error, never by training error. This trade-off has a name, bias versus variance, covered in Bias, variance, underfitting and overfitting .

Step 4: Warning signs of overfitting

  • Training error far below validation error.
  • Huge weights that nearly cancel each other. In the code below, the largest weight goes from under 2 at degree 3 to over 4,000 at degree 15. Ridge and lasso regularization exist to stop exactly this.
  • Wild swings at the edges of the data, where high powers of explode. Never trust a high-degree polynomial outside the range it was trained on.
poly.pyruns with python3 + numpy + scikit-learn
import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(1)
x = rng.uniform(-1, 1, 60)
y = np.sin(2.6 * x) + 0.3 * x + rng.normal(0, 0.25, 60)   # a curve plus noise
# Keep only 18 points for training, so high degrees have room to overfit
x_tr, x_va, y_tr, y_va = train_test_split(x.reshape(-1, 1), y, test_size=0.7, random_state=0)

print("PolynomialFeatures(3) turns x = 0.5 into:",
      PolynomialFeatures(3).fit_transform([[0.5]])[0].tolist())
print()
for degree in (1, 3, 5, 9, 15):
    model = make_pipeline(PolynomialFeatures(degree), StandardScaler(), LinearRegression())
    model.fit(x_tr, y_tr)
    mse_tr = np.mean((model.predict(x_tr) - y_tr) ** 2)
    mse_va = np.mean((model.predict(x_va) - y_va) ** 2)
    biggest = np.abs(model[-1].coef_).max()
    print(f"degree {degree:>2}: train MSE {mse_tr:.3f}  validation MSE {mse_va:8.3f}  "
          f"largest weight {biggest:10.1f}")

Output

PolynomialFeatures(3) turns x = 0.5 into: [1.0, 0.5, 0.25, 0.125]

degree  1: train MSE 0.108  validation MSE    0.177  largest weight        0.9
degree  3: train MSE 0.021  validation MSE    0.050  largest weight        1.8
degree  5: train MSE 0.016  validation MSE    0.051  largest weight        1.8
degree  9: train MSE 0.008  validation MSE    0.129  largest weight       17.9
degree 15: train MSE 0.004  validation MSE    0.969  largest weight     4309.3

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Start at degree 1 and step up one at a time. At which degree does the curve first follow the shape of the dashed truth line?
  2. Go to degree 11. Training error is 0: the curve hits every white point. What does it do between and beyond them? Look at the validation error.
  3. Switch to "training points only" and pick the degree you would choose without validation data. Then switch the validation points back on. Would you have picked well?

Trade-offs

OptionWins whenLoses when
Low degree (1 to 3) Little data, noisy data, or you need safe behaviour near the edges The true curve has several bends
High degree Lots of data covering the whole range, with a genuinely wiggly pattern Few points, extrapolation, or unscaled inputs
Polynomial plus regularization You want flexibility but must keep weights in check (next lesson) You need an exact, unpenalised fit
Other basis functions (splines, sines) Seasonal patterns or local bends, with stable behaviour at the edges You need a single simple formula

Common mistakes

  • Choosing the degree by training error, which always favours the highest degree.
  • Extrapolating a polynomial beyond the training range.
  • Forgetting to scale: of a feature in the thousands overflows and wrecks numerical precision.
  • Expanding many features to high degree: with 20 features, degree 3 already creates 1,771 columns including all interactions.
  • Reading the individual polynomial weights as meaningful; the shape of the curve is what matters.

In the wild

Polynomial and other basis-function features are standard in engineering and science for smooth curves such as calibration curves or dose-response relationships. Forecasting libraries model yearly and weekly seasonality with sine and cosine basis functions ( Prophet and modern forecasting libraries ). Splines, piecewise polynomials joined smoothly, are a staple of statistics because they bend where the data needs it without the wild edges of a single high-degree polynomial.

Check yourself

  1. Why is polynomial regression still called linear regression?

    Show answer

    B. Because the prediction is linear in the weights, even though it is curved in x The features x, x^2, x^3 are fixed transformations. The model is still a weighted sum of them, so least squares and all its tools still apply.

  2. As degree increases, what always happens to training error (on the same data)?

    Show answer

    A. It goes down or stays the same A higher-degree polynomial can always copy a lower-degree one by setting the extra weights to 0, so its best training fit is at least as good.

  3. Degree 9 has training MSE 0.008 and validation MSE 0.129; degree 3 has 0.021 and 0.050. Which do you pick?

    Show answer

    B. Degree 3, lower validation error Validation error estimates performance on new data. The degree 9 model's lower training error is partly memorised noise.

  4. A degree 12 model fitted on house sizes from 40 to 150 m2 predicts a negative price for a 200 m2 house. Why?

    Show answer

    B. High-degree polynomials swing wildly outside the training range High powers grow explosively beyond the data, and nothing in training constrained the curve there. Do not extrapolate polynomials.

Recap

  • Adding powers of x as features lets a linear model fit curves; the model stays linear in its weights.
  • Any fixed transformation of the inputs is a basis function; powers are just one choice.
  • Higher degree always lowers training error, but validation error falls then rises.
  • Underfitting means too simple; overfitting means memorising noise. Pick the degree on validation data.
  • Huge weights and wild edges are signs of overfitting; never extrapolate a high-degree polynomial.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up