Machine Learning ML-03.01
Linear regression intuition
Fit a straight line through data, measure its mistakes with residuals and mean squared error, and read what each coefficient means.
The big idea
You have 14 recent house sales: the size of each house and the price it sold for. A friend asks what a 90 square metre flat should go for. You could squint at the numbers, or you could draw the straight line that runs through the middle of the points and read the answer off it. That line is a model, and finding the best one is linear regressionA model that predicts a number as a weighted sum of the features plus a constant: a straight line for one feature, a flat plane for two, and so on., the oldest and still one of the most useful models in machine learning.
Think of stretching a rubber band through a cloud of pins. Each pin pulls the band toward itself; the band settles where the pulls balance. Linear regression does the same thing with arithmetic: it finds the line where the total "pull" of the errors is as small as possible.
How it works
Step 1: The model is a line
With one feature, the model is
where is the feature (size), ("y hat") is the prediction (price), is the weightA number the model learns that multiplies a feature. In linear regression it is the change in prediction per unit of that feature. or slope, and is the biasThe constant added to every prediction: where the line crosses x = 0. Also called the intercept. or intercept. Training means choosing and . With many features it is the same idea, one weight per feature, which you met as a dot product in Vectors and vector spaces :
Two features give a flat plane through a 3D cloud; more give a flat "hyperplane" you cannot draw but can compute just the same.
Step 2: Residuals measure each mistake
For each training example, the residualThe difference between the true value and the model's prediction for one example: y minus y hat. is the true value minus the prediction, . Positive means the line guessed too low; negative means too high. The pink segments in Figure 1 are the residuals. A good line has small residuals everywhere, with no obvious pattern in them.
Step 3: Mean squared error adds them up
To compare lines you need one number. You cannot just add residuals, because positive and negative ones cancel. Squaring makes every error positive and punishes big misses much more than small ones:
Mean squared errorMean squared error: the average of the squared residuals. The standard loss for regression. (MSE) is the lossA function that measures how wrong a model is on the training data. Training searches for the parameters that make it smallest. linear regression minimises. Worked example with the first house: size 109, price 305. The line predicts , so the residual is and its squared error is 225. Do that for all 14 houses and average: the best line has an MSE of 1,079. Because MSE is in squared units (thousands squared), people often report its square root, the RMSE: about 33, meaning a typical miss of around 33 thousand.
The line with the smallest MSE is called the least squaresThe line, or plane, that makes the sum of squared residuals as small as possible. fit. For linear regression there is exactly one such line, and it can be computed directly ( Solving linear regression: normal equation and gradient descent ).
Step 4: Reading the coefficients
The fitted line is price = 2.48 x size + 19.7. The weight says: each extra square metre adds about 2.48 thousand to the predicted price. So 80 m2 predicts 218k and 90 m2 predicts 243k, about 25k more for 10 more square metres. That direct reading is why linear models are popular in finance, medicine and policy: anyone can check what the model does.
Two cautions:
- The bias of 19.7 is the prediction for a 0 m2 house, which is meaningless here. The intercept often has no real-world meaning; it just positions the line.
- With several features, each weight means "the change in prediction per unit of this feature with the other features held fixed". Add house age to the model and the size weight drops from 2.48 to 1.46, because in this data bigger houses also tend to be newer, and some of the price difference credited to size really belonged to age.
import numpy as np
from sklearn.linear_model import LinearRegression
# The 14 house sales from the lab: size (m2) and price (thousands)
size = np.array([109, 102, 112, 63, 106, 70, 116, 42, 67, 139, 125, 139, 62, 128])
price = np.array([305, 233, 338, 190, 214, 205, 318, 146, 145, 415, 305, 340, 186, 358])
model = LinearRegression().fit(size.reshape(-1, 1), price)
w, b = model.coef_[0], model.intercept_
print(f"price = {w:.2f} x size + {b:.1f}")
pred = model.predict(size.reshape(-1, 1))
residuals = price - pred
print("first 4 residuals:", residuals[:4].round(1))
print(f"MSE = {np.mean(residuals ** 2):.0f} (units: thousands squared)")
print(f"RMSE = {np.sqrt(np.mean(residuals ** 2)):.1f} (back in thousands)")
# Interpreting the weight: every extra square metre adds about w thousand
print(f"predicted 80 m2: {model.predict([[80]])[0]:.0f}k, 90 m2: {model.predict([[90]])[0]:.0f}k")
# Two features: the same idea, a flat plane instead of a line
age = np.array([12, 30, 5, 40, 35, 22, 8, 50, 45, 3, 18, 25, 33, 10])
X2 = np.column_stack([size, age])
m2 = LinearRegression().fit(X2, price)
print("size and age weights:", m2.coef_.round(2), " bias:", round(m2.intercept_, 1))Output
price = 2.48 x size + 19.7
first 4 residuals: [ 15. -39.6 40.6 14.1]
MSE = 1079 (units: thousands squared)
RMSE = 32.9 (back in thousands)
predicted 80 m2: 218k, 90 m2: 243k
size and age weights: [ 1.46 -2.73] bias: 186.3 Try it
Try this:
- Drag the handles to make the line as good as you can by eye. How close to the best MSE of 1,079 can you get?
- Switch to "squares". MSE is the average area of these squares. Notice how one far-away point makes a huge square: that is why squared error is sensitive to outliers.
- Press Solve, then drag one end a little. MSE rises whichever way you move it: the least squares line sits at the bottom of a bowl, which is exactly what gradient descent ( Gradient descent, the engine of ML ) walks down.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Linear regression | The relationship is roughly straight, you need speed and an explainable model | The relationship curves or depends on combinations of features |
| Linear with engineered features | You know the shape: logs, squares, interactions (polynomial features) | The shape is complicated and unknown |
| Tree ensembles or neural nets | Complex, non-linear relationships with plenty of data | You need to explain each prediction simply, or data is small |
Common mistakes
- Reading a weight as a cause. A weight describes association in this data, given the other features, not what would happen if you changed one feature in the real world.
- Comparing weights of features on different scales: a weight per square metre and a weight per bedroom are not comparable without scaling.
- Extrapolating far outside the training data, such as predicting a 1,000 m2 mansion from flats.
- Ignoring residual patterns. If residuals curve or fan out, a straight line is the wrong shape.
- Letting a few outliers drag the line, because squared error weights them heavily.
In the wild
Linear regression is everywhere a number must be predicted and explained: house price estimates, demand forecasts, the effect of price changes in economics, and dose-response models in medicine. Many teams start every regression project with it as a baseline ( Learning curves and baselines ), and the same weighted-sum-plus-bias unit is the building block of every neural network layer ( The perceptron and the artificial neuron ).
Check yourself
-
A model predicts salary = 1.8 x years_experience + 30 (in thousands). What does 1.8 mean?
Show answer
B. Each extra year of experience adds about 1.8k to the predicted salary The weight is the change in prediction per unit of the feature, here per year of experience.
-
Why does MSE square the residuals instead of just adding them?
Show answer
B. So positive and negative errors do not cancel, and big errors are punished more Raw residuals can sum to zero for a terrible line. Squaring makes every error count and weights large misses heavily.
-
A house sold for 300k and the model predicts 320k. What is the residual?
Show answer
B. -20 Residual = true minus predicted = 300 - 320 = -20. The model guessed too high.
-
Adding 'age' to a size-only model changes the size weight from 2.48 to 1.46. Why?
Show answer
B. Size and age are related in this data, so part of the effect credited to size now goes to age Each weight is the effect holding the other features fixed. When features are correlated, adding one redistributes credit among them.
Recap
- Linear regression predicts with a weighted sum of features plus a bias: a line, plane or hyperplane.
- A residual is true minus predicted; MSE averages the squared residuals.
- Training finds the weights with the smallest MSE: the least squares fit.
- Each weight is the change in prediction per unit of its feature, holding other features fixed.
- Linear models are fast and explainable, but squared error makes them sensitive to outliers.
Next up
- I know nothing yet learning path
- From notebook to production learning path