Feature importance and SHAP

The big idea

A forest of 100 trees predicts a house price of 376,600. The buyer, the bank and the regulator all ask the same question: why? No one can read 100 trees. You need tools that summarise what the model relies on.

There are two different questions. Global: which features matter to the model overall? Local: why did this one house get this one price? Feature importanceA score for how much a model relies on a feature, either overall (global) or for one prediction (local). methods answer the first. SHAPSHapley Additive exPlanations: attributes one prediction to features using Shapley values from game theory, so the contributions add up exactly to the prediction minus the average prediction. answers the second, and its values can be averaged to answer the first too.

Think of a team that wins a prize and has to split it fairly. Shapley values, from cooperative game theory, give each player their average added value over every possible order in which the team could have been assembled. SHAP treats features as the players and the prediction as the prize.

How it works

Step 1: Impurity importance is free but biased

Every tree split reduces impurity ( Decision trees ). Summing those reductions per feature, over all trees, gives impurity importanceFeature importance computed from the total impurity reduction of all splits on a feature during training; also called mean decrease in impurity., the feature_importances_ attribute. It costs nothing, but has two flaws:

• It is computed on training data, so it rewards splits that only fit noise.
• Features with many distinct values (continuous numbers, ids) offer many thresholds, so they get chosen for noise-fitting splits more often than binary features.

In the code below, a column of pure random numbers gets impurity importance 0.026, almost as much as house age (0.030), which genuinely affects price.

Step 2: Permutation importance measures what breaks

Permutation importanceThe drop in a model's score on held-out data when one feature's values are randomly shuffled, breaking its relation to the target. asks a cleaner question: if I shuffle this feature's column in the test set, destroying its information while keeping its distribution, how much worse does the model get?

Permutation importance works with any model and any metric. Its weak spot is correlated featuresStrong correlation between features. Importance methods split or misattribute credit among correlated features.: shuffle one of two near-duplicate columns and the model uses the other, so both look unimportant. Group correlated features and permute them together, or drop one and retrain to check.

Step 3: Shapley values split one prediction fairly

For one example, define the value of a subset S of features as the model's expected prediction when only the features in S are known (the others are filled in from background data). A feature's Shapley valueA feature's average marginal contribution to the prediction, weighted over all subsets of the other features, as defined in cooperative game theory. is its marginal contribution, averaged over all subsets of the other features:

The weights make every ordering of features equally likely. The result has a property no other attribution has in general: the values add up exactly. The base value (the average prediction) plus all the Shapley values equals the model's prediction.

Step 4: From single explanations to global pictures

Compute SHAP values for many rows and you get rich global views: the mean absolute SHAP value per feature (a more reliable importance ranking than impurity), and scatter plots of a feature's value against its SHAP value, which show the shape of its effect, like a partial dependence plot with interactions visible as vertical spread.

Going deeper

Code: three importance methods, and SHAP by brute force

from itertools import combinations
from math import factorial
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

# House prices: size and location matter, age a little, plus a pure-noise column
rng = np.random.default_rng(0)
n = 2000
size = rng.normal(100, 25, n)                 # square metres
location = rng.integers(0, 3, n)              # 0 suburb, 1 city, 2 centre
age = rng.uniform(0, 50, n)
noise_id = rng.random(n)                       # unique random number, no signal
price = 2.0 * size + 60 * location - 0.3 * age + rng.normal(0, 20, n)
X = np.column_stack([size, location, age, noise_id])
names = ["size", "location", "age", "noise_id"]
Xtr, Xte, ytr, yte = train_test_split(X, price, test_size=0.3, random_state=0)
rf = RandomForestRegressor(100, min_samples_leaf=1, random_state=0).fit(Xtr, ytr)

perm = permutation_importance(rf, Xte, yte, n_repeats=5, random_state=0)
print("feature    impurity  permutation (drop in R^2 on test)")
for i, nm in enumerate(names):
    print(f"{nm:<9}  {rf.feature_importances_[i]:.3f}     {perm.importances_mean[i]:.3f}")

# Exact Shapley values for one house: average marginal contribution over all feature subsets.
# "Feature absent" = replaced by values from 200 background rows (interventional SHAP).
bg = Xtr[:200]
def value(x, S):
    Z = bg.copy()
    Z[:, list(S)] = x[list(S)]
    return rf.predict(Z).mean()

x = Xte[(Xte[:, 1] == 2) & (Xte[:, 2] > 40)][0]       # an old house in the centre
p = len(names)
phi = np.zeros(p)
for j in range(p):
    others = [k for k in range(p) if k != j]
    for r in range(p):
        for S in combinations(others, r):
            w = factorial(r) * factorial(p - r - 1) / factorial(p)
            phi[j] += w * (value(x, S + (j,)) - value(x, S))
base = value(x, ())
print(f"\nhouse: size {x[0]:.0f}, location {x[1]:.0f}, age {x[2]:.0f}")
print(f"base value (average prediction) {base:.1f}")
for nm, v in zip(names, phi):
    print(f"  {nm:<9} {v:+7.1f}")
print(f"base + sum of SHAP = {base + phi.sum():.1f}; model prediction = {rf.predict(x[None])[0]:.1f}")

feature    impurity  permutation (drop in R^2 on test)
size       0.503     0.917
location   0.441     0.892
age        0.030     0.007
noise_id   0.026     0.001

house: size 135, location 2, age 45
base value (average prediction) 255.5
  size        +63.5
  location    +63.3
  age          -4.6
  noise_id     -1.1
base + sum of SHAP = 376.6; model prediction = 376.6

The code computes exact Shapley values by visiting all 2^4 = 16 subsets of the four features. That is impossible for 100 features, which is where the SHAP paper's contribution comes in. The shap library provides fast approximations for any model (KernelSHAP), and TreeSHAP, an algorithm that computes exact values for tree ensembles in polynomial time.

What explanations do not tell you

• Not causation. SHAP explains the model, not the world. If the model uses postcode as a proxy for income, SHAP will credit postcode.
• Background data matters. "Feature absent" is simulated by background samples; a different background gives a different base value and different attributions.
• Correlated features confuse every method. Filling in a feature from background data can create impossible combinations, like a 20 square metre flat with six bedrooms.

These limits matter for fairness reviews ( Fairness and bias ): an explanation can show what a model uses, but not whether using it is justified.

Trade-offs

Common mistakes

• Trusting feature_importances_ on a forest that includes ids or other high-cardinality columns.
• Computing permutation importance on training data, where an overfit model makes noise look important.
• Concluding that a feature with low importance is unrelated to the target, when a correlated twin took its credit.
• Reading SHAP values as causal effects ("raising size by 10 m2 will add X").
• Presenting a single SHAP explanation without saying what the base value and background data are.

In the wild

Check yourself

Recap