Skip to content
learn
  1. Learn
  2. Machine Learning
  3. Regression

Machine Learning ML-03.05

Regularization: ridge, lasso, elastic net

Ridge, lasso and elastic net explained: L2 shrinks every weight, L1 sets some to exactly zero, elastic net mixes both, and how to pick the penalty.

Intermediate 14 min Last reviewed

The big idea

In the last lesson, the overfitting polynomial gave itself away with enormous weights: thousands, carefully cancelling each other to thread the curve through every noisy point. RegularizationAdding a penalty for model complexity, usually the size of the weights, to the training loss so the model prefers simpler solutions that generalise better. attacks that directly. Instead of minimising the error alone, minimise the error plus a price on how large the weights are. The model can still use big weights, but only if they buy a big reduction in error.

Think of packing for a flight with a strict weight allowance. Without a limit you would bring everything. With one, you keep only the items that really earn their weight. Ridge regression is a limit that makes everything a bit lighter; lasso is a limit that makes you leave whole items at home.

How it works

Step 1: Add a penalty to the loss

measures the size of the weights, and the regularization strengthThe strength of the regularization penalty, often written lambda or alpha. 0 means no penalty; larger values force smaller weights. (called alpha in scikit-learn) sets the exchange rate. At you get plain least squares; as grows, weights are pushed toward zero. The bias is usually not penalised: shifting predictions up or down is not complexity.

Step 2: Ridge (L2): shrink everything

Ridge regressionLinear regression with an L2 penalty, the sum of squared weights. It shrinks all weights smoothly toward zero but rarely makes any exactly zero. penalises the sum of squared weights, . Big weights become very expensive, small ones cost almost nothing, so ridge shrinks every weight a little and the largest ones most. Weights get small but almost never exactly zero. With correlated features, ridge shares the weight between them instead of picking one arbitrarily, and it fixes the instability of the normal equation when features are nearly duplicates: the solution becomes , which is always solvable.

Step 3: Lasso (L1): zero some out

LassoLinear regression with an L1 penalty, the sum of absolute weights. It drives some weights exactly to zero, performing feature selection. penalises the sum of absolute values, . The cost of a weight grows at the same rate however small it gets, so there is no discount for being "nearly zero". For features that help only a little, the cheapest choice is to switch them off entirely. Lasso therefore does feature selection, which you met in Feature selection : the model it returns uses only some of the features.

Step 4: Why L1 gives zeros: the geometric view

Loss contours meeting the L1 diamond and the L2 circle Two weights on the axes. Ellipses are contours of the training loss, centred on the unpenalised solution at w1 = 0.6, w2 = 1.4. A green circle shows all weights with the same L2 size, and a purple diamond all weights with the same L1 size. The pink ellipse first touches the circle at about (0.46, 0.89), where both weights are non-zero: the ridge solution. The gold ellipse first touches the diamond at its corner (0, 1), where w1 is exactly zero: the lasso solution. -1012-10123weight w1weight w2 no penalty ridge lasso: w1 = 0 L2 circle L1 diamond Figure 1. Regularization picks the point on the penalty shape with the lowest loss: where the expanding loss ellipses first touch it. The circle is smooth, so ridge lands at (0.46, 0.89) with both weights non-zero. The diamond has corners on the axes, so lasso lands on the corner (0, 1) and w1 becomes exactly 0.

An equivalent way to see regularization: keep the weights inside a fixed "budget" region and find the lowest-loss point in it. For L2 the budget region is a circle (a sphere in more dimensions); for L1 it is a diamond. The loss contours are ellipses around the unpenalised solution. Grow the ellipses until they first touch the region: that touch point is the answer.

The circle has no corners, so the touch point is usually somewhere with both weights non-zero. The diamond's corners stick out along the axes, and an ellipse coming from most directions hits a corner first. At a corner, one weight is exactly zero. In many dimensions the diamond has very many corners and edges, so lasso solutions typically have many zero weights.

Step 5: Elastic net, and choosing lambda

Elastic netRegularization that mixes L1 and L2 penalties, getting lasso's sparsity with ridge's stability on correlated features. uses both penalties at once, with a mixing ratio (l1_ratio). It keeps lasso's ability to zero out features but behaves better when features come in correlated groups, where pure lasso tends to keep one at random and drop the rest.

is a hyperparameter. Too small and you are back to overfitting; too large and every weight is crushed toward zero, so the model underfits. Choose it with validation data or cross-validation ( Cross-validation ), usually by trying values spread across several orders of magnitude, such as 0.001, 0.01, 0.1, 1, 10, 100. Always scale features first: the penalty treats all weights alike, so a feature measured in large units would need only a tiny weight and would escape the penalty unfairly.

regularize.pyruns with python3 + numpy + scikit-learn
import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet, LassoCV
from sklearn.model_selection import train_test_split

# 150 examples, 30 features, only the first 3 actually matter. Train on 45.
rng = np.random.default_rng(0)
X = rng.normal(size=(150, 30))
y = 3 * X[:, 0] - 2 * X[:, 1] + 1.5 * X[:, 2] + rng.normal(0, 1.5, 150)
X_tr, X_va, y_tr, y_va = train_test_split(X, y, test_size=0.7, random_state=0)

models = {
    "no penalty": LinearRegression(),
    "ridge (L2)": Ridge(alpha=1.0),
    "lasso (L1)": Lasso(alpha=0.2),
    "elastic net": ElasticNet(alpha=0.2, l1_ratio=0.5),
}
for name, m in models.items():
    m.fit(X_tr, y_tr)
    va = np.mean((m.predict(X_va) - y_va) ** 2)
    w = m.coef_
    print(f"{name:<12} val MSE {va:6.2f}  first 3 weights {w[:3].round(2)}  "
          f"zeros {np.sum(np.abs(w) < 1e-8):>2}/30  sum|w| {np.abs(w).sum():5.1f}")

# Ridge shrinks every weight smoothly as alpha grows
print("\nridge weight on feature 0 as alpha grows:",
      [round(float(Ridge(alpha=a).fit(X_tr, y_tr).coef_[0]), 2) for a in (0.01, 1, 10, 100, 1000)])

# Choose the penalty strength by cross-validation instead of guessing
cv = LassoCV(cv=5, random_state=0).fit(X_tr, y_tr)
print(f"LassoCV picked alpha = {cv.alpha_:.3f}, keeping features {np.flatnonzero(cv.coef_).tolist()}")

Output

no penalty   val MSE   5.97  first 3 weights [ 2.92 -2.1   1.15]  zeros  0/30  sum|w|  13.4
ridge (L2)   val MSE   5.47  first 3 weights [ 2.65 -1.78  1.01]  zeros  0/30  sum|w|  12.0
lasso (L1)   val MSE   3.44  first 3 weights [ 2.43 -1.5   0.95]  zeros 20/30  sum|w|   6.0
elastic net  val MSE   5.03  first 3 weights [ 2.14 -1.29  0.83]  zeros 12/30  sum|w|   6.7

ridge weight on feature 0 as alpha grows: [2.91, 2.65, 1.84, 0.68, 0.11]
LassoCV picked alpha = 0.065, keeping features [0, 1, 2, 3, 4, 7, 10, 13, 15, 19, 21, 22, 23, 24, 25, 26, 27, 28, 29]

Only 3 of the 30 features matter (true weights 3, -2 and 1.5), and there are just 45 training rows. Plain least squares spreads weight over all 30 features and has the worst validation error. Ridge shrinks every weight a little and helps slightly. Lasso zeroes 20 of the 30 weights and nearly halves the validation error. The ridge row of the output shows the smooth shrinking: feature 0's weight falls from 2.91 to 0.11 as alpha rises from 0.01 to 1,000. LassoCV, which picks alpha by cross-validation, chooses a weaker penalty and keeps 19 features: it optimises prediction, not sparsity, so expect it to keep a few extra.

Trade-offs

OptionWins whenLoses when
Ridge (L2) Many features that each matter a little, correlated features, unstable least squares You want a short list of features or exact zeros
Lasso (L1) You expect few features to matter and want automatic selection Groups of correlated features, where it picks one arbitrarily
Elastic net Sparse solutions with correlated features You want a single hyperparameter to tune
No penalty Few features and plenty of data Many features, little data, or near-duplicate features

Common mistakes

  • Regularizing without scaling features, so the penalty hits features unequally.
  • Choosing lambda by training error, which always prefers lambda = 0.
  • Penalising the bias term.
  • Reading lasso's dropped features as "irrelevant": with correlated features, a dropped one may be just as predictive as the one kept.
  • Forgetting that regularization biases the weights toward zero; for unbiased effect estimates you need other tools.

In the wild

Ridge and lasso are standard tools wherever there are many features and limited data, such as genomics, where thousands of gene measurements describe a few hundred patients. Weight decay, the L2 penalty applied during gradient descent, is a default part of training neural networks ( Regularization for deep nets ), and L1 and L2 penalties are built into gradient boosting libraries as well. Large-scale linear models for click prediction commonly use L1-style penalties to keep the number of active features, and so the memory needed to serve them, manageable.

Check yourself

  1. What does increasing the regularization strength lambda do?

    Show answer

    B. Pushes weights toward zero, trading training fit for simplicity Lambda is the price on weight size. A higher price means the model accepts more training error to keep weights small.

  2. Why does lasso set some weights exactly to zero while ridge does not?

    Show answer

    B. The L1 penalty's diamond shape has corners on the axes, where weights are zero The L1 penalty does not get cheaper near zero, and its constraint region has corners on the axes. The optimum often lands on a corner, where some weights are exactly zero.

  3. You have 500 features, most probably useless, and 300 rows. Which is a good first choice?

    Show answer

    B. Lasso or elastic net, with lambda chosen by cross-validation With more features than rows and many useless ones, an L1 penalty selects a small useful set and prevents overfitting.

  4. You fit ridge on raw features: income in dollars and age in years. What is the problem?

    Show answer

    B. The penalty treats weights equally, but income's weight is tiny because of its units, so it barely gets penalised Weights depend on units. Standardize features first so the penalty compares like with like.

Recap

  • Regularization adds a penalty on weight size to the loss; lambda sets how strong it is.
  • Ridge (L2) shrinks all weights smoothly and stabilises correlated features.
  • Lasso (L1) sets some weights exactly to zero, selecting features.
  • Geometrically, L1's diamond has corners on the axes, which is where the zeros come from.
  • Elastic net mixes both; scale features first and choose lambda on validation data.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up