Skip to content
learn
  1. Learn
  2. Machine Learning
  3. Data: the most important part

Machine Learning ML-02.06

Feature selection

Why more features can hurt, and how filter, wrapper and embedded methods pick the ones worth keeping, plus the curse of dimensionality.

Intermediate 10 min Last reviewed

The big idea

After a burst of feature engineering you have 400 columns. Surely more information can only help? It cannot. Every irrelevant feature is noise the model may latch onto by chance, a column to compute and store at serving time, and one more thing that can break when an upstream system changes. Feature selectionChoosing a subset of the available features to train on, dropping those that add noise, cost or risk without improving predictions. keeps the features that earn their place.

Packing for a trip is a good picture. Taking everything you own does not make the trip better: the bag gets heavy, you cannot find what you need, and the airline charges you. You pack by asking of each item, "will I actually use this?" Feature selection asks the same question of every column, with data instead of guesswork.

How it works

Three families of feature selection Three lanes. Filter: score each feature on its own with a statistic, then keep the top ones, before any model. Wrapper: repeatedly train a model on candidate subsets and keep the subset that scores best. Embedded: the model selects while it trains, through L1 regularisation or tree importances. cheap expensive all features 20 columns filter score each feature alone wrapper train on many subsets embedded select while training kept features the useful 4 Figure 1. Filter methods score features before modelling, wrapper methods search subsets by training many models, and embedded methods select during a single training run.

Step 1: Filter methods

Score each feature on its own, against the target, and keep the best. Fast, model-free, and a good first sweep over hundreds of columns.

  • Correlation with the target catches straight-line relationships and nothing else.
  • Mutual informationA measure of how much knowing one variable reduces uncertainty about another. Zero means independent; it detects non-linear relationships too. measures how much knowing the feature reduces uncertainty about the target, in the sense of Information theory for ML . It detects curved and non-monotonic relationships as well.

The weakness: features are judged one at a time. Two features that are useless alone but powerful together (an interaction) both get dropped, and two near-identical copies of the same feature both get kept.

Step 2: Wrapper methods

Train a model on candidate subsets and keep the subset that validates best. Recursive feature eliminationRecursive feature elimination: train a model, drop the least important feature or features, and repeat until the desired number remains. (RFE) trains, removes the weakest feature, and repeats. Wrappers account for how features work together, and they are tuned to the model you will actually use. They are also expensive: dozens or hundreds of training runs, and with so many subsets tried, a real risk of picking one that is lucky on the validation data.

Step 3: Embedded methods

Let the model select while it trains, in one run:

  • L1 regularisation (lasso) adds a penalty on the absolute size of the weights, which pushes the weights of unhelpful features to exactly zero. Whatever remains non-zero is selected. Covered in depth in Regularization: ridge, lasso, elastic net .
  • Tree importances: tree ensembles record how much each feature reduced error across all splits. Rank by that and drop the bottom. Beware that impurity-based importance favours features with many distinct values; permutation importance and SHAP ( Feature importance and SHAP ) are more reliable.

On a dataset where only the first 4 of 20 features carry signal, all four approaches find them:

select.pyruns with python3 + numpy + scikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.feature_selection import mutual_info_classif, RFE
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score

# 20 features; only the first 4 carry signal (shuffle=False keeps them first).
X, y = make_classification(n_samples=600, n_features=20, n_informative=4, n_redundant=0,
                           shuffle=False, random_state=1)
top4 = lambda scores: sorted(np.argsort(scores)[-4:].tolist())

print("filter   (mutual information):", top4(mutual_info_classif(X, y, random_state=0)))
rfe = RFE(LogisticRegression(max_iter=1000), n_features_to_select=4).fit(X, y)
print("wrapper  (RFE):               ", np.flatnonzero(rfe.support_).tolist())
l1 = LogisticRegression(l1_ratio=1, C=0.05, solver="liblinear").fit(X, y)  # L1 penalty (sklearn >= 1.8)
print("embedded (L1 non-zero):       ", np.flatnonzero(l1.coef_[0]).tolist())
rf = RandomForestClassifier(n_estimators=200, random_state=0).fit(X, y)
print("embedded (tree importance):   ", top4(rf.feature_importances_))

# Curse of dimensionality: add pure-noise features and watch k-NN degrade.
rng = np.random.default_rng(0)
knn = KNeighborsClassifier(n_neighbors=5)
for extra in (0, 16, 100, 500):
    Xn = np.hstack([X[:, :4], rng.normal(size=(len(X), extra))])
    acc = cross_val_score(knn, Xn, y, cv=5).mean()
    print(f"k-NN with 4 useful + {extra:>3} noise features: accuracy {acc:.2f}")

Output

filter   (mutual information): [0, 1, 2, 3]
wrapper  (RFE):                [0, 1, 2, 3]
embedded (L1 non-zero):        [0, 1, 2, 3]
embedded (tree importance):    [0, 1, 2, 3]
k-NN with 4 useful +   0 noise features: accuracy 0.87
k-NN with 4 useful +  16 noise features: accuracy 0.73
k-NN with 4 useful + 100 noise features: accuracy 0.61
k-NN with 4 useful + 500 noise features: accuracy 0.56

On real data they will disagree at the margins. That disagreement is useful: features every method keeps are safe bets, and features only one method likes deserve a closer look.

Step 4: The curse of dimensionality

The last lines of the output show why selection matters even when you have a decent model.

k-NN accuracy as noise features are added Bar chart of 5-fold cross-validated k-nearest-neighbour accuracy with the 4 useful features plus 0, 16, 100 and 500 pure-noise features: 0.87, 0.73, 0.61 and 0.56. accuracy (5-fold cross-validation) 4 useful + 0 noise 0.87 4 useful + 16 noise 0.73 4 useful + 100 noise 0.61 4 useful + 500 noise 0.56 Figure 2. The same 4 useful features, buried under more and more random columns. Accuracy falls from 0.87 toward a coin flip as noise dominates every distance.

The curse of dimensionalityThe collection of problems that appear as the number of features grows: data becomes sparse, distances between points become similar, and far more data is needed to learn. is the name for what happens as you add dimensions. The volume of the space grows exponentially, so a fixed number of examples becomes very sparse. Distances also stop being informative: with hundreds of random dimensions, the nearest and the farthest neighbour end up almost equally far away, so "nearest" means little. Distance-based methods like k-NN suffer first, but every model needs more data to separate signal from noise as the noise columns pile up.

Trade-offs

OptionWins whenLoses when
Filter (correlation, mutual information) Hundreds or thousands of features, a fast first pass Features matter only in combination, or many are near-duplicates
Wrapper (RFE, forward selection) A modest number of features and enough compute Many features, slow models, or a small validation set that rewards luck
Embedded (L1, tree importance) You want selection as a by-product of one training run Correlated features: L1 picks one arbitrarily, impurity importance is biased
Keep everything Strong regularisation, lots of data and features are cheap to serve Features are costly, fragile, or the model is distance-based

Common mistakes

  • Selecting features using the whole dataset, then evaluating on a test set that influenced the selection. Do selection inside cross-validation, on training folds only.
  • Dropping a feature because its correlation with the target is near zero, when its relationship is curved or works through an interaction.
  • Reading tree importances as causal effects, or trusting impurity importance for high-cardinality features.
  • Keeping a feature with suspiciously high importance without checking it for leakage ( Data leakage ).
  • Ignoring serving cost: a feature that adds 0.1 percent accuracy but needs a slow external call may not be worth it.

In the wild

Credit scoring teams often prefer a compact set of features because each one must be explained to regulators and customers. Large ad and recommendation systems add and remove features through controlled experiments, since each feature carries computation and maintenance cost at serving time. In genomics, with tens of thousands of genes per sample and only hundreds of samples, filter methods and L1-regularised models are the standard way to make learning possible at all.

Check yourself

  1. Two features are each useless on their own but predict the target perfectly together (their product). Which method is most likely to miss them?

    Show answer

    A. A filter method that scores features one at a time Filters judge each feature in isolation, so features that only matter jointly score near zero. Models that see features together can pick up the combination.

  2. Why does L1 regularisation perform feature selection?

    Show answer

    B. Its penalty pushes the weights of unhelpful features to exactly zero The absolute-value penalty has a corner at zero, so the optimum often sets weak features' weights exactly to zero, and those features drop out.

  3. You add 300 random columns to a k-NN model's 5 good features. What happens?

    Show answer

    C. Accuracy drops because distances become dominated by noise Each noise dimension adds random distance. With many of them, nearest neighbours are no longer the truly similar points.

  4. You ran feature selection on the full dataset, then measured test accuracy. What is wrong?

    Show answer

    B. The test set influenced which features were chosen, so the test score is optimistic Selection is part of training. If it saw the test rows, the test is no longer an unbiased estimate. Select inside each training fold.

Recap

  • Irrelevant features add noise, cost and fragility; more is not automatically better.
  • Filter methods score features one by one, quickly but blind to interactions.
  • Wrapper methods search subsets with a model, accurately but expensively.
  • Embedded methods such as L1 and tree importance select during a single training run.
  • The curse of dimensionality makes data sparse and distances meaningless as noise dimensions grow, so do selection inside cross-validation.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up