Skip to content
learn
  1. Learn
  2. Machine Learning
  3. Regression

Machine Learning ML-03.01

Linear regression intuition

Fit a straight line through data, measure its mistakes with residuals and mean squared error, and read what each coefficient means.

Beginner 14 min Last reviewed

The big idea

You have 14 recent house sales: the size of each house and the price it sold for. A friend asks what a 90 square metre flat should go for. You could squint at the numbers, or you could draw the straight line that runs through the middle of the points and read the answer off it. That line is a model, and finding the best one is linear regressionA model that predicts a number as a weighted sum of the features plus a constant: a straight line for one feature, a flat plane for two, and so on., the oldest and still one of the most useful models in machine learning.

Think of stretching a rubber band through a cloud of pins. Each pin pulls the band toward itself; the band settles where the pulls balance. Linear regression does the same thing with arithmetic: it finds the line where the total "pull" of the errors is as small as possible.

How it works

Best-fit line and residuals Fourteen house sales plotted as size against price. A glowing straight line runs through them: price equals 2.48 times size plus 19.7. Short pink vertical segments connect each point to the line; they are the residuals, the errors the line makes on each house. 406080100120140100150200250300350400450size (m2)price (k) price = 2.48 x size + 19.7 Figure 1. The glowing line is the least squares fit. Each pink segment is a residual: how far the true price is from the line's prediction. The best line makes the squared lengths of these segments as small as possible in total.

Step 1: The model is a line

With one feature, the model is

where is the feature (size), ("y hat") is the prediction (price), is the weightA number the model learns that multiplies a feature. In linear regression it is the change in prediction per unit of that feature. or slope, and is the biasThe constant added to every prediction: where the line crosses x = 0. Also called the intercept. or intercept. Training means choosing and . With many features it is the same idea, one weight per feature, which you met as a dot product in Vectors and vector spaces :

Two features give a flat plane through a 3D cloud; more give a flat "hyperplane" you cannot draw but can compute just the same.

Step 2: Residuals measure each mistake

For each training example, the residualThe difference between the true value and the model's prediction for one example: y minus y hat. is the true value minus the prediction, . Positive means the line guessed too low; negative means too high. The pink segments in Figure 1 are the residuals. A good line has small residuals everywhere, with no obvious pattern in them.

Step 3: Mean squared error adds them up

To compare lines you need one number. You cannot just add residuals, because positive and negative ones cancel. Squaring makes every error positive and punishes big misses much more than small ones:

Mean squared errorMean squared error: the average of the squared residuals. The standard loss for regression. (MSE) is the lossA function that measures how wrong a model is on the training data. Training searches for the parameters that make it smallest. linear regression minimises. Worked example with the first house: size 109, price 305. The line predicts , so the residual is and its squared error is 225. Do that for all 14 houses and average: the best line has an MSE of 1,079. Because MSE is in squared units (thousands squared), people often report its square root, the RMSE: about 33, meaning a typical miss of around 33 thousand.

The line with the smallest MSE is called the least squaresThe line, or plane, that makes the sum of squared residuals as small as possible. fit. For linear regression there is exactly one such line, and it can be computed directly ( Solving linear regression: normal equation and gradient descent ).

Step 4: Reading the coefficients

The fitted line is price = 2.48 x size + 19.7. The weight says: each extra square metre adds about 2.48 thousand to the predicted price. So 80 m2 predicts 218k and 90 m2 predicts 243k, about 25k more for 10 more square metres. That direct reading is why linear models are popular in finance, medicine and policy: anyone can check what the model does.

Two cautions:

  • The bias of 19.7 is the prediction for a 0 m2 house, which is meaningless here. The intercept often has no real-world meaning; it just positions the line.
  • With several features, each weight means "the change in prediction per unit of this feature with the other features held fixed". Add house age to the model and the size weight drops from 2.48 to 1.46, because in this data bigger houses also tend to be newer, and some of the price difference credited to size really belonged to age.
linreg.pyruns with python3 + numpy + scikit-learn
import numpy as np
from sklearn.linear_model import LinearRegression

# The 14 house sales from the lab: size (m2) and price (thousands)
size = np.array([109, 102, 112, 63, 106, 70, 116, 42, 67, 139, 125, 139, 62, 128])
price = np.array([305, 233, 338, 190, 214, 205, 318, 146, 145, 415, 305, 340, 186, 358])

model = LinearRegression().fit(size.reshape(-1, 1), price)
w, b = model.coef_[0], model.intercept_
print(f"price = {w:.2f} x size + {b:.1f}")

pred = model.predict(size.reshape(-1, 1))
residuals = price - pred
print("first 4 residuals:", residuals[:4].round(1))
print(f"MSE  = {np.mean(residuals ** 2):.0f}   (units: thousands squared)")
print(f"RMSE = {np.sqrt(np.mean(residuals ** 2)):.1f}   (back in thousands)")

# Interpreting the weight: every extra square metre adds about w thousand
print(f"predicted 80 m2: {model.predict([[80]])[0]:.0f}k, 90 m2: {model.predict([[90]])[0]:.0f}k")

# Two features: the same idea, a flat plane instead of a line
age = np.array([12, 30, 5, 40, 35, 22, 8, 50, 45, 3, 18, 25, 33, 10])
X2 = np.column_stack([size, age])
m2 = LinearRegression().fit(X2, price)
print("size and age weights:", m2.coef_.round(2), " bias:", round(m2.intercept_, 1))

Output

price = 2.48 x size + 19.7
first 4 residuals: [ 15.  -39.6  40.6  14.1]
MSE  = 1079   (units: thousands squared)
RMSE = 32.9   (back in thousands)
predicted 80 m2: 218k, 90 m2: 243k
size and age weights: [ 1.46 -2.73]  bias: 186.3

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Drag the handles to make the line as good as you can by eye. How close to the best MSE of 1,079 can you get?
  2. Switch to "squares". MSE is the average area of these squares. Notice how one far-away point makes a huge square: that is why squared error is sensitive to outliers.
  3. Press Solve, then drag one end a little. MSE rises whichever way you move it: the least squares line sits at the bottom of a bowl, which is exactly what gradient descent ( Gradient descent, the engine of ML ) walks down.

Trade-offs

OptionWins whenLoses when
Linear regression The relationship is roughly straight, you need speed and an explainable model The relationship curves or depends on combinations of features
Linear with engineered features You know the shape: logs, squares, interactions (polynomial features) The shape is complicated and unknown
Tree ensembles or neural nets Complex, non-linear relationships with plenty of data You need to explain each prediction simply, or data is small

Common mistakes

  • Reading a weight as a cause. A weight describes association in this data, given the other features, not what would happen if you changed one feature in the real world.
  • Comparing weights of features on different scales: a weight per square metre and a weight per bedroom are not comparable without scaling.
  • Extrapolating far outside the training data, such as predicting a 1,000 m2 mansion from flats.
  • Ignoring residual patterns. If residuals curve or fan out, a straight line is the wrong shape.
  • Letting a few outliers drag the line, because squared error weights them heavily.

In the wild

Linear regression is everywhere a number must be predicted and explained: house price estimates, demand forecasts, the effect of price changes in economics, and dose-response models in medicine. Many teams start every regression project with it as a baseline ( Learning curves and baselines ), and the same weighted-sum-plus-bias unit is the building block of every neural network layer ( The perceptron and the artificial neuron ).

Check yourself

  1. A model predicts salary = 1.8 x years_experience + 30 (in thousands). What does 1.8 mean?

    Show answer

    B. Each extra year of experience adds about 1.8k to the predicted salary The weight is the change in prediction per unit of the feature, here per year of experience.

  2. Why does MSE square the residuals instead of just adding them?

    Show answer

    B. So positive and negative errors do not cancel, and big errors are punished more Raw residuals can sum to zero for a terrible line. Squaring makes every error count and weights large misses heavily.

  3. A house sold for 300k and the model predicts 320k. What is the residual?

    Show answer

    B. -20 Residual = true minus predicted = 300 - 320 = -20. The model guessed too high.

  4. Adding 'age' to a size-only model changes the size weight from 2.48 to 1.46. Why?

    Show answer

    B. Size and age are related in this data, so part of the effect credited to size now goes to age Each weight is the effect holding the other features fixed. When features are correlated, adding one redistributes credit among them.

Recap

  • Linear regression predicts with a weighted sum of features plus a bias: a line, plane or hyperplane.
  • A residual is true minus predicted; MSE averages the squared residuals.
  • Training finds the weights with the smallest MSE: the least squares fit.
  • Each weight is the change in prediction per unit of its feature, holding other features fixed.
  • Linear models are fast and explainable, but squared error makes them sensitive to outliers.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up