Explainability

The big idea

A loan model says no. The applicant asks why, the risk team asks whether the model is relying on something it should not, and the regulator asks how decisions are made in general. Those are three different questions, and one chart of feature importances answers none of them well.

Think of a doctor explaining a diagnosis. To a colleague: the lab values and how each moved the probability. To the patient: what it means and what they can do about it. To a hospital board: how the diagnostic rule performs across all patients. Same model of the disease, three explanations, each tailored to what the listener needs to decide.

ExplainabilityThe degree to which people can understand why a model produced its outputs, either overall or for a single case. in ML works the same way. Pick the question first (overall behaviour or one decision, understanding or action), then the method, then the words.

How it works

Step 1: Global or local

A global explanationAn explanation of a model's overall behaviour across all inputs, such as which features it relies on most or the average shape of a feature's effect. describes the model as a whole: which features matter most and how the prediction changes as a feature changes. Tools: coefficients of a linear model, permutation importance ( Feature importance and SHAP ), partial dependence plotsA plot of the model's average prediction as one feature is varied over its range, with the other features left at their observed values., or a small decision tree trained to mimic the big model (a global surrogate).

A local explanationAn explanation of a single prediction: why this input received this output. describes one prediction. Tools: SHAP, LIME and counterfactuals, the subject of this lesson. Local explanations can be averaged into global ones (mean absolute SHAP), but not the other way round: a feature that matters a lot on average can be irrelevant for a particular person.

There is also a choice before any of this: use an interpretable modelA model simple enough to read directly, such as a short decision tree, a sparse linear model or a scorecard. (a scorecard, a shallow tree, a sparse linear model) and you need no separate explanation, because the model is the explanation. Post-hoc methods like SHAP and LIME explain a black box from the outside, approximately.

Step 2: SHAP, a fair split against a baseline

SHAP ( Feature importance and SHAP ) splits the gap between this prediction and the average prediction among the features, using Shapley values. The parts add up exactly: base value plus contributions equals the prediction. In the lab below, the average applicant has P(repay) 0.51, this one 0.07; income accounts for -0.30 and debt for -0.14.

Notice what SHAP answers: "compared with a typical applicant, which features pushed this one down?" The comparison group matters. Explain against all applicants and low income looks bad; explain against other low-income applicants and it looks normal.

Step 3: LIME, a local straight line

LIMELocal Interpretable Model-agnostic Explanations: explains one prediction by sampling points near it, weighting them by closeness, and fitting a simple model such as a sparse linear model to the black box's outputs there. (Ribeiro, Singh and Guestrin, arXiv 1602.04938) treats the model as a black box. It samples points around the instance, asks the model for each, weights each sample by a proximity kernel ${\pi }_x(z)=\exp (-D(x,z{)}^2/{\sigma }^2)$ with width $\sigma$, and fits a sparse linear model to the answers. The coefficients are the explanation. The paper's key requirement is local fidelity: the explanation "must correspond to how the model behaves in the vicinity of the instance being predicted", which does not imply it describes the model anywhere else.

LIME and SHAP can disagree and both be right. In the code below, SHAP gives income -0.16 (this applicant's income is below average, which hurts) while LIME gives income +0.06 (locally, more income would help). One compares with a baseline, the other measures a slope.

LIME's weakness is stability. The result depends on the random samples and on the kernel width $\sigma$, which has no principled default; narrow kernels use few effective samples and give noisy coefficients. Always rerun with several seeds before trusting a LIME explanation.

The LIME paper also shows why local explanations matter. The authors deliberately trained a bad wolf-versus-husky classifier on 20 images in which every wolf photo had snow in the background. Shown only predictions, 10 of 27 graduate students with ML training trusted it and 12 mentioned snow as a possible feature. After seeing LIME explanations, which highlighted the snow, only 3 trusted it and 25 named snow.

Step 4: Counterfactuals, what would change the outcome

A counterfactual explanationAn explanation of the form: if your inputs had been these instead, the decision would have been different; usually the smallest such change. answers the question an applicant actually has: what would I need to change? Wachter, Mittelstadt and Russell (arXiv 1711.00399) describe it as "the smallest change to the world that can be made to obtain a desirable outcome", and note it needs no description of the model's internals.

Finding one is an optimisation: minimise distance from $x$ to $x^{\prime }$ subject to $f(x^{\prime })\ge$ threshold. Three practical rules:

• Scale the distance. Measure each change in standard deviations (or by real-world cost), otherwise the feature with the biggest units dominates.
• Only move actionable features. Income and debt can change; age and past late payments cannot. Restrict the search or the advice is useless.
• Offer a few options. "Income 76k, or debt ratio 0.07, or income 59k with debt ratio 0.41" lets people pick the path open to them.

A counterfactual is also a probe for unfairness: if changing only a protected attribute, or a proxy for one, flips the decision, you have found a problem ( Fairness and bias ).

Step 5: Explaining to non-experts

Match the explanation to the decision the listener has to make:

For the affected person, use the feature's plain name ("your debt is high relative to your income"), not a code or a number like "-0.14". Lead with the top two or three reasons, add one actionable change, and say how to contest the decision. Some decisions come with legal duties: in US credit, Regulation B (12 CFR 1002.9) requires a statement of "specific" principal reasons for an adverse action, and says that stating the applicant "failed to achieve a qualifying score on the creditor's credit scoring system" is insufficient.

Try it

1. With SHAP selected and the applicant at the start position (40k, 0.52), read the bars: base 0.51, income -0.30, debt -0.14, total 0.07. Now drag the applicant to about 90k income and a debt ratio of 0.30. SHAP credits income with about +0.37, because 90k is far above the average.
2. Keep the applicant at 90k and switch to LIME. Income's local coefficient is close to 0: at that income the model has saturated, so a little more or less income changes nothing. Global importance and local relevance are different things.
3. Move back to the start, choose LIME and set the kernel width to 0.2. Press New LIME sample a few times and watch the coefficients jump (income 0.12, then 0.08). At width 0.5 they barely move. Then switch to Counterfactual and compare "income only" (58.5k) with "debt only" (debt ratio 0.14). Drag the applicant to 90k with a debt ratio of 0.70 and choose "income only": no amount of income helps, only reducing debt does.

Going deeper

Code: SHAP, LIME and counterfactuals on one denied loan

A gradient boosting model on synthetic credit data with four features. The code computes exact SHAP values by enumerating all subsets against a 200-row background, runs a from-scratch LIME twice with different seeds, and searches a grid for counterfactuals that only change income and debt.

# Explain one denied loan three ways: SHAP (exact), LIME (from scratch), a counterfactual.
import itertools, math
import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.linear_model import Ridge

rng = np.random.default_rng(1)
names = ["income_k", "debt_ratio", "years_job", "late_pays"]
n = 4000
X = np.column_stack([rng.normal(55, 18, n).clip(15), rng.uniform(0, 0.8, n),
                     rng.exponential(5, n), rng.poisson(0.6, n)])
z = 0.06 * (X[:, 0] - 55) - 5 * (X[:, 1] - 0.4) + 0.15 * np.minimum(X[:, 2], 8) - 0.9 * X[:, 3]
y = rng.random(n) < 1 / (1 + np.exp(-z))                  # 1 = repaid
f = GradientBoostingClassifier(random_state=0).fit(X, y)
p = lambda A: f.predict_proba(A)[:, 1]

x = np.array([38.0, 0.55, 1.5, 1.0])                        # the applicant
bg = X[rng.choice(n, 200, replace=False)]                   # background data
print(f"P(repay) = {p(x[None])[0]:.2f}, cut-off 0.50 -> denied")

def v(S):                                                   # value of a feature subset
    Z = bg.copy(); Z[:, list(S)] = x[list(S)]; return p(Z).mean()
phi = [sum(math.factorial(len(S)) * math.factorial(3 - len(S)) / 24 * (v(S + (j,)) - v(S))
           for k in range(4) for S in itertools.combinations([i for i in range(4) if i != j], k))
       for j in range(4)]
print(f"SHAP  base {v(()):.2f}:", " ".join(f"{nm} {s:+.2f}" for nm, s in zip(names, phi)),
      f"| base + sum = {v(()) + sum(phi):.2f}")

sd = X.std(axis=0)
for seed in (0, 1):                                         # LIME: perturb, weight, fit
    r = np.random.default_rng(seed)
    Z = x + r.normal(0, 1, (300, 4)) * sd
    w = np.exp(-(((Z - x) / sd) ** 2).sum(1) / 0.75 ** 2)   # kernel width 0.75
    g = Ridge(alpha=1e-3).fit((Z - x) / sd, p(Z), sample_weight=w)
    print(f"LIME seed {seed}:", " ".join(f"{nm} {c:+.2f}" for nm, c in zip(names, g.coef_)))

def flips(inc, debt):                                       # only actionable features move
    c = x.copy(); c[0], c[1] = inc, debt; return p(c[None])[0] >= 0.5
incs, debts = np.arange(38, 120.1, 1), np.arange(0.55, -0.001, -0.01)
print("income only:", next((f"{i:.0f}k" for i in incs if flips(i, 0.55)), "never"))
print("debt only:  ", next((f"{d:.2f}" for d in debts if flips(38, d)), "never"))
cost, inc, debt = min(((i - 38) / sd[0] + (0.55 - d) / sd[1], i, d)
                      for i, d in itertools.product(incs, debts) if flips(i, d))
print(f"cheapest mix: income {inc:.0f}k and debt ratio {debt:.2f}")

P(repay) = 0.11, cut-off 0.50 -> denied
SHAP  base 0.55: income_k -0.16 debt_ratio -0.14 years_job -0.08 late_pays -0.07 | base + sum = 0.11
LIME seed 0: income_k +0.06 debt_ratio -0.08 years_job +0.06 late_pays -0.04
LIME seed 1: income_k +0.08 debt_ratio -0.07 years_job +0.06 late_pays -0.03
income only: 76k
debt only:   0.07
cheapest mix: income 59k and debt ratio 0.41

• SHAP splits the gap between the base value 0.55 and the prediction 0.11 across all four features, and the parts add up exactly.
• LIME gives the same signs for debt and late payments, but a positive sign for income and job years: below-average values hurt relative to the baseline, yet locally more of either would help. Its numbers shift between seeds (income +0.06 vs +0.08).
• Counterfactuals give three honest options: raise income to 76k, cut the debt ratio to 0.07, or a mix of 59k and 0.41. The mix is the cheapest in standard-deviation units, and probably the most realistic.

Limits worth knowing

• Explanations describe the model, not the world. A SHAP value for postcode says the model uses postcode, not that postcode causes default.
• Correlated features confuse every method: perturbation creates impossible inputs, and credit gets split arbitrarily between twins.
• Explanations can be gamed. Slack et al. (arXiv 1911.02508) built classifiers that stay biased on real inputs but detect LIME's and SHAP's perturbed samples and answer those innocuously, so the explanations look clean. Explanations are evidence, not proof.
• Faithfulness vs simplicity. The simpler the explanation, the less of the model it captures. For high-stakes decisions, consider an interpretable model that needs no approximation.

Trade-offs

Common mistakes

• Answering "why was I denied?" with a global importance chart.
• Reporting SHAP values without saying what the base value and background data are.
• Trusting a single LIME run; the coefficients move with the seed and kernel width.
• Suggesting counterfactuals that change immutable features such as age, or that are absurdly far away.
• Treating an explanation as proof that a model is fair or causal.

In the wild

Where you will see this: credit and insurance decisions, where reason codes for adverse actions are a legal requirement in some jurisdictions; fraud and risk teams, who use SHAP to review why transactions were flagged; and model debugging in general, where local explanations expose shortcuts like the snow in the LIME paper. The open-source shap and lime libraries implement the methods from this lesson, and scikit-learn includes permutation importance and partial dependence plots in sklearn.inspection.

Check yourself

Recap