Fairness and bias

The big idea

A model learns whatever pattern makes its loss small. If the data records a world where one group was treated worse, or measures one group less accurately, the model learns that too, and then applies it to millions of decisions at machine speed. Nobody has to write a biased line of code for a loan, hiring or medical model to treat groups of people differently.

Think of a referee who trained by watching old match footage in which one team always got the benefit of the doubt. The referee is consistent, fast and sincere, and still biased, because the footage was. Fixing it means deciding what a fair call is, checking the referee against that standard, and correcting either the footage, the training or the calls.

That is the shape of this lesson: find where biasSystematic differences in how a model treats, or errs on, different groups of people, introduced anywhere from data collection to deployment. enters, measure it with a fairness metric you chose on purpose, and mitigate it at the right stage. The hard part is the middle step: the common definitions of fairness contradict each other, so choosing one is a decision about values, not just engineering.

How it works

Step 1: Where bias comes from

Suresh and Guttag (arXiv 1901.10002) name seven sources of harm across the ML life cycle. They are worth memorising, because each has a different fix.

Two examples make them concrete. In the Gender Shades study (Buolamwini and Gebru, 2018), two facial analysis benchmarks were 79.6 and 86.2 percent lighter-skinned subjects (representation bias in the data and evaluation bias in the benchmark), and commercial gender classifiers misclassified darker-skinned women at error rates up to 34.7 percent, against at most 0.8 percent for lighter-skinned men. And Suresh and Guttag's measurement example: using "arrests" as the label for "crime" bakes in where police chose to patrol.

Step 2: Dropping the sensitive column does not work

The first instinct is fairness through unawarenessTrying to make a model fair by leaving the protected attribute, such as race or sex, out of its inputs. It fails when other features act as proxies.: never show the model the group. Hardt, Price and Srebro (arXiv 1610.02413) call this "ineffective due to the existence of redundant encodings": postcode, first name, school or shopping history can predict the group, so the model reconstructs it. Worse, if you delete the column you can no longer measure the gaps. Keep the attribute for auditing, even when the model does not use it.

Step 3: Three ways to measure fairness

Write $\hat{Y}$ for the decision (1 = approve), $Y$ for the true outcome and $A$ for the group. Each metric compares one rate across groups:

Hardt et al. define equalized odds as "$\hat{Y}$ and $A$ are independent conditional on $Y$": among people who would repay, approval does not depend on group, and the same among people who would not. The relaxed version, equal opportunityFairness criterion requiring only the true positive rate to be equal across groups: qualified people are approved at the same rate whatever their group., asks only for equal TPR.

Worked example. Group A: 1,000 applicants, 500 would repay. Group B: 1,000 applicants, 250 would repay. A model approves 420 of A's repayers and 80 of A's non-repayers, 210 of B's repayers and 120 of B's non-repayers.

• Approval rates: A $500/1000=0.50$, B $330/1000=0.33$. Demographic parity fails.
• TPR: A $420/500=0.84$, B $210/250=0.84$. FPR: A $80/500=0.16$, B $120/750=0.16$. Equalized odds holds.
• PPV: A $420/500=0.84$, B $210/330=0.64$. Predictive parity fails.

The same model is fair by one definition and unfair by two others.

Step 4: Why you cannot have them all

That was not bad luck. Chouldechova (arXiv 1610.07524) shows that for any classifier, the rates are tied together by the base rateThe share of a group that truly belongs to the positive class; also called prevalence. $p$:

If two groups have different base rates and you force equal PPV, the right-hand side differs between groups, so FPR and FNR cannot both be equal. Kleinberg, Mullainathan and Raghavan (arXiv 1609.05807) prove the general version for risk scores: apart from highly constrained special cases (a perfect predictor, or equal base rates), calibration within groups and balanced error rates cannot all hold at once. Chouldechova uses this to explain the COMPAS recidivism debate: a score free of predictive bias in the psychometric sense still produced, by ProPublica's figures, a false positive rate of 45 percent for Black defendants against 23 percent for White defendants, and a false negative rate of 28 against 48 percent.

So the real question is which mistake matters most for this decision. For a loan, a false negative denies credit to someone who would repay; equal opportunity protects them. For a screening test that triggers a cheap follow-up, equal approval rates may matter less than catching every case. Write the choice down, with the reason.

Step 5: Mitigation at three stages

• Pre-processing fixes the data: collect more examples of under-represented groups, relabel with better proxies, reweight examples so group and label are balanced, or remove correlation between features and the group.
• In-processing changes training: add a fairness constraint or penalty to the loss, as in the reductions approach of Agarwal et al., or train an adversary that tries to predict the group from the model's output.
• Post-processing changes decisions: keep the model, pick a separate threshold per group so the chosen rates match. Hardt et al. show how to derive the optimal equalized-odds or equal-opportunity predictor from any score this way.

Post-processing is the easiest to bolt on, but it needs the group at decision time, which some laws restrict. Pre-processing fixes root causes but rarely closes the gap alone. The Fairlearn documentation adds a warning worth keeping: its algorithms give "mathematical guarantees as to how close they can drive some unfairness metric to zero. However, this does not mean that the results are fair."

Try it

1. Start with the defaults (same threshold, same score quality, B's base rate 25 percent). TPR and FPR are equal, so equalized odds holds, but B's approval rate is 0.33 against 0.50 and B's precision is 0.64 against 0.84. This is the worked example above.
2. Press Equal approval rate. B's threshold drops to 0.40 and demographic parity holds, but B's TPR jumps to 0.95 and its FPR to 0.35, and B's precision falls to 0.48. Now set B's base rate to 50 percent and press Same threshold: every gap closes. The conflict exists only because base rates differ.
3. Set B's base rate back to 25 percent, lower B's score quality to 1.2 (noisier features for B, a form of measurement bias) and press Same threshold. B's TPR falls to 0.58: qualified people in B are now missed far more often. Equal TPR fixes that by lowering B's threshold to 0.2, at the price of a 0.42 FPR for B. Better data for group B would beat any threshold.

Going deeper

Code: auditing a model that never saw the group

A synthetic lending dataset: group B is 30 percent of applicants, has lower average income, and mostly lives in one area. The historical labels carry measurement bias: a quarter of group B's good payers were recorded as defaults. The model gets income, debt and area, but never the group.

# Fairness audit of a loan model that never sees the group column.
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)
n = 40_000
b = rng.random(n) < 0.3                                   # group B is 30% of applicants
area = (rng.random(n) < np.where(b, 0.8, 0.2)) * 1.0      # neighbourhood: a proxy for group
income = rng.normal(np.where(b, -0.4, 0.0), 1.0)
debt = rng.normal(0, 1, n)
repaid = rng.random(n) < 1 / (1 + np.exp(-(1.2 * income - 0.8 * debt + 0.4)))
# Historical labels: 25% of group B's good payers were recorded as defaults (measurement bias)
label = repaid & ~(b & (rng.random(n) < 0.25))

X = np.column_stack([income, debt, area])                 # no group column: "unaware"
tr, te = np.arange(n) < n // 2, np.arange(n) >= n // 2
model = LogisticRegression().fit(X[tr], label[tr])
print("weights income, debt, area:", model.coef_.round(2)[0])
proxy = LogisticRegression().fit(X[tr], b[tr])
print(f"group predicted from features, AUC = {roc_auc_score(b[te], proxy.predict_proba(X[te])[:, 1]):.2f}")

p = model.predict_proba(X[te])[:, 1]
y, g = repaid[te], b[te]                                  # audit against true outcomes

def report(name, approve):
    for grp, m in (("A", ~g), ("B", g)):
        sel, tpr = approve[m].mean(), approve[m & y].mean()
        fpr, ppv = approve[m & ~y].mean(), y[m & approve].mean()
        print(f"{name:13s} {grp}: approve {sel:.2f}  TPR {tpr:.2f}  FPR {fpr:.2f}  PPV {ppv:.2f}")
    print(f"{'':13s} accuracy {(approve == y).mean():.3f}")

report("one cut 0.5", p > 0.5)
# Post-processing: lower B's cut until its TPR matches A's (equal opportunity)
tpr_a = (p[~g & y] > 0.5).mean()
cuts = np.linspace(0.2, 0.5, 301)
t_b = cuts[np.argmin([abs((p[g & y] > c).mean() - tpr_a) for c in cuts])]
print(f"cut for B = {t_b:.3f}")
report("per-group cut", np.where(g, p > t_b, p > 0.5))

weights income, debt, area: [ 1.1  -0.73 -0.38]
group predicted from features, AUC = 0.84
one cut 0.5   A: approve 0.58  TPR 0.77  FPR 0.32  PPV 0.77
one cut 0.5   B: approve 0.38  TPR 0.59  FPR 0.17  PPV 0.77
              accuracy 0.726
cut for B = 0.360
per-group cut A: approve 0.58  TPR 0.77  FPR 0.32  PPV 0.77
per-group cut B: approve 0.55  TPR 0.77  FPR 0.33  PPV 0.69
              accuracy 0.730

Read the output in order:

• The area feature gets a negative weight even though true repayment does not depend on area. The model learned the biased labels through a proxy, and the features predict group membership with an AUC of 0.84. Unawareness did nothing.
• Against true outcomes, one cut-off approves 58 percent of A and 38 percent of B, and B's good payers are approved at a TPR of 0.59 against 0.77.
• Lowering B's cut-off to 0.36 equalizes TPR at 0.77, and FPR comes out equal too (0.32 vs 0.33). B's precision drops from 0.77 to 0.69, because B's base rate is lower: exactly the Chouldechova trade-off. Accuracy against the true outcomes rises slightly, from 0.726 to 0.730, because the post-processing partly undoes the label bias.

The last point is not general. When labels are biased, "accuracy" measured on those labels can reward the bias. Audit against the best ground truth you can get, even if it is a small, carefully labelled sample.

Practical checklist

• Decide the protected attributes and the fairness criterion with domain, legal and affected stakeholders before training.
• Report every metric per group, with confidence intervals; small groups give noisy rates.
• Check intersections (for example, women in one age band), not just one attribute at a time; Gender Shades found the largest gap at an intersection.
• Monitor after launch. Feedback loops, such as approved applicants being the only ones whose repayment you observe, change the data over time.

Trade-offs

Common mistakes

• Removing the protected attribute and declaring the model fair, while proxies carry the same information.
• Reporting one fairness metric without saying why it was chosen over the others.
• Measuring fairness against labels that are themselves biased, and trusting the result.
• Auditing only overall accuracy, or only one attribute at a time, and missing gaps at intersections.
• Treating fairness as a one-time check instead of monitoring it after deployment.

In the wild

Where you will see this: lenders, insurers and employers audit credit, pricing and screening models for group gaps, often under anti-discrimination law. The Gender Shades audit of commercial face analysis and the debate over the COMPAS recidivism score are the two most cited real cases, and both are described above from their primary papers. Open-source toolkits such as Fairlearn and IBM's AI Fairness 360 implement the metrics and the pre-, in- and post-processing methods from this lesson, and model cards (Mitchell et al., arXiv 1810.03993) propose that every released model report its performance per demographic and intersectional group.

Check yourself

Recap