Privacy-preserving ML

The big idea

A model is a compressed summary of its training data, and summaries can give away details. A hospital trains a readmission model on patient records and publishes an API. If an attacker can tell from the API's answers that a particular person's record was in the training set, they have learned that this person was a patient. No database was breached and no row was copied, yet private information left the building inside the model.

Think of a town that publishes its average salary every month. It looks harmless. But if you know the average before and after one family moved in, and the number of residents, you can work out that family's income exactly. The fix is not to stop publishing, it is to publish numbers that would look almost the same whether or not any single family lived there.

That is the promise of differential privacyA mathematical guarantee that the output of an analysis or training run is almost equally likely whether or not any single person's data was included; the strength is set by epsilon (and delta).: add carefully calibrated noise so that no individual's record noticeably changes what is released, including a trained model. Federated learningTraining a shared model across many devices or organisations that keep their data locally and send only model updates to a coordinating server. attacks the problem from the other side: the raw data never leaves the user's device at all. Both cost something, usually accuracy or compute, and this lesson is about what you get for that price.

How it works

Step 1: How models leak, membership inference

The basic attack is membership inferenceAn attack that decides, from a model's outputs, whether a given record was in the model's training set.: given a record and access to the model, decide whether that record was in the training data. Shokri et al. (arXiv 1610.05820) introduced it against commercial "machine learning as a service" models, including one trained on a hospital discharge dataset, "whose membership is sensitive from the privacy perspective".

Their attack turns ML against itself. The attacker trains several shadow modelsModels the attacker trains on data similar to the target's, so the attacker knows which records were members and can learn what member and non-member outputs look like. on data like the target's, where membership is known, then trains an attack classifier on the shadow models' outputs labelled "in" or "out". The signal it learns is the gap between how confidently a model answers records it trained on and records it has never seen.

A much simpler version works almost as well: compute the model's loss on the record and say "member" if it is low. Yeom et al. (arXiv 1709.01604) prove that this kind of attack succeeds in proportion to the generalization gap: overfitting is sufficient for membership inference. They also show it is not necessary, so a model that generalizes well can still leak.

Cheap defences help less than you would hope. In Shokri et al.'s tests, returning only the top 1 or 3 classes or rounding probabilities barely moved the attack. Even returning only the predicted label left the attack at 66 percent accuracy on their purchase dataset and 73 percent on the hospital data, against 50 percent for guessing. Strong L2 regularization cut it to near 50 percent, but on the hospital data it also cut the model's own test accuracy from 0.55 to 0.35.

Step 2: Differential privacy, the guarantee

Call two datasets neighbouringTwo datasets that differ in exactly one record: one includes a particular person's data and the other does not. if they differ in one person's record. A randomized algorithm $M$ is $(\epsilon ,\delta )$-differentially private if, for every pair of neighbouring datasets $d,d^{\prime }$ and every set of possible outputs $S$:

(the form used by Abadi et al., arXiv 1607.00133). In words: whatever the output is, it was almost as likely without you. $\epsilon$ is the privacy budgetThe epsilon in differential privacy: a bound on how much any one person's data can change the probability of any output. Smaller is more private; budgets add up across repeated releases.; $\delta$ is a small probability that the guarantee fails, which Abadi et al. note should preferably be smaller than one over the dataset size.

The simplest mechanism adds noise to a count. A counting query changes by at most 1 when one person is added (its sensitivityThe largest amount one person's data can change a query's answer; noise is scaled to it. is 1), so releasing the count plus Laplace noiseNoise from the Laplace (double exponential) distribution, whose density falls off as exp(-|x|/b); with scale b = sensitivity / epsilon it gives epsilon-differential privacy. of scale $1/\epsilon$ is $\epsilon$-DP.

What does $\epsilon$ mean to an attacker? Suppose they know everyone else's data and must guess whether you are in, with even odds beforehand. The definition (with $\delta =0$) caps their accuracy at

For $\epsilon =1$: $e/(1+e)=2.718/3.718=0.73$, so no attack can do better than 73 percent. For $\epsilon =0.1$ the cap is 52.5 percent, close to a coin flip. For $\epsilon =8$ it is 99.97 percent, which is why large budgets are a much weaker promise than they sound. Budgets also add up: answering $k$ queries at $\epsilon$ each costs up to $k\epsilon$ in total.

Step 3: DP-SGD, privacy for model training

A model is just a very complicated answer computed from the data, so the same idea applies to training. DP-SGDDifferentially private stochastic gradient descent: clip each example's gradient to a maximum norm, add Gaussian noise to the sum, then take the step; a privacy accountant tracks the total epsilon. (Abadi et al.) changes each gradient step in two ways:

1. Clip each example's gradient to norm at most $C$. Now no single person can push the model by more than $C$, which bounds the sensitivity.
2. Add Gaussian noise $\mathcal{N}(0,{\sigma }^2{C}^{2}I)$ to the sum of clipped gradients, then divide by the expected batch size $L$ and take the step.

Each batch includes each example with probability $q=L/N$, which helps: an example that was not sampled cannot leak in that step. A privacy accountantThe bookkeeping that adds up the privacy cost of every noisy training step to give the total (epsilon, delta) for the trained model. adds up the cost over all $T$ steps. Tight accounting matters: Abadi et al. report that with $q=0.01$, $\sigma =4$, $\delta =10^{-5}$ and $T=10,000$ steps, their moments accountant gives $\epsilon \approx 1.26$, where the standard strong composition theorem would give $\epsilon \approx 9.34$ for the same training run.

The price is accuracy. On MNIST, their non-private baseline reached 98.30 percent; with DP-SGD they report 97, 95 and 90 percent test accuracy at $\epsilon$ = 8, 2 and 0.5 (all with $\delta =10^{-5}$). On CIFAR-10, using convolution layers pre-trained on the public CIFAR-100 dataset, the non-private baseline was about 80 percent and the private model reached 73 percent at $\epsilon =8$. Using public data for the parts of the model that need no private data is a standard trick.

Step 4: Federated learning, keep the data on the device

McMahan et al. (arXiv 1602.05629) named this federated averagingThe standard federated learning algorithm: sampled clients run several local SGD epochs and the server takes a data-size-weighted average of their models. (FedAvg). Three knobs control it: $C$, the fraction of clients used per round; $E$, the number of local passes over each client's data; and $B$, the local minibatch size. With $E=1$ and the whole local dataset as one batch it is plain distributed gradient descent; letting each phone do more local work before sending is what cut the number of communication rounds by 10 to 100 times in their experiments. Too many local epochs can make training plateau or diverge, because each phone's data is small, unbalanced and unlike the others (non-IID).

Federated learning is not a privacy guarantee by itself. Model updates are computed from the data, and Zhu, Liu and Han (arXiv 1906.08935) showed that shared gradients can be inverted to recover training images pixel by pixel and text token by token. Production systems therefore stack defences: secure aggregationA cryptographic protocol that lets a server learn the sum of many clients' updates without seeing any individual update. so the server only sees the sum, and differential privacy at the user level so that even the sum barely depends on any one user. McMahan et al. (arXiv 1710.06963) added user-level DP to FedAvg for language models and found that with a large enough number of users the private models were close to the non-private ones; the cost moved from accuracy to extra computation.

Try it

1. Start at epsilon 1 with one answer. The two curves overlap heavily: the attacker is right 69.7 percent of the time, under the 73.1 percent cap, and the typical error of the released count is 1.00 person. Now drag epsilon to 0.1: the attacker drops to 52.4 percent, but the typical error is 10 people on a count of 40. Privacy and accuracy are the same dial.
2. Set epsilon to 3. The answer is accurate (typical error 0.33) and the attacker is right 88.8 percent of the time. A single-digit epsilon can still mean a lot of leakage for one person.
3. Set epsilon back to 1 and average 10 answers. The error shrinks to 0.35, the attacker reaches 87.3 percent, and the total budget is 10, where the guarantee says almost nothing. Now set epsilon to 0.1 with 10 answers: the total budget is back to 1.0 and the attacker falls to 54.6 percent. Repeated queries spend the budget; that is why systems track it.

Going deeper

Code: attack an overfit model, then train it with DP-SGD

A logistic regression with as many features as training examples (500 each), so it overfits. The attack is the loss threshold: low loss means member. Then the same model is trained with DP-SGD (Poisson sampling at $q=0.1$, clip norm 1, 1,000 steps) at three noise levels, with $\epsilon$ computed by a Renyi-DP accountant for the sampled Gaussian mechanism and $\delta =10^{-5}$. The attack is scored by AUC, where 0.5 means it cannot tell members from non-members.

# Membership inference against an overfit model, and DP-SGD as the defence.
import math
import numpy as np
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)
d, n = 500, 500                                       # as many features as examples: overfits
w_true = np.zeros(d); w_true[:10] = 1.0
X = rng.normal(0, 1, (2 * n, d)) / np.sqrt(d)         # each row has norm about 1
y = (X @ w_true * np.sqrt(d / 10) + rng.normal(0, 0.5, 2 * n) > 0).astype(float)
Xin, yin, Xout, yout = X[:n], y[:n], X[n:], y[n:]     # members / non-members
sig = lambda z: 1 / (1 + np.exp(-z))
q, T, C, lr = 0.1, 1000, 1.0, 4.0                     # sample rate, steps, clip norm, step size

def train(sigma):
    w, r = np.zeros(d), np.random.default_rng(1)
    for _ in range(T):
        i = np.flatnonzero(r.random(n) < q)             # Poisson sampling, about 50 per batch
        g = (sig(Xin[i] @ w) - yin[i])[:, None] * Xin[i]  # one gradient per example
        if sigma is None:
            w -= lr * g.mean(0); continue
        g /= np.maximum(1, np.linalg.norm(g, axis=1) / C)[:, None]   # clip each to norm C
        w -= lr * (g.sum(0) + r.normal(0, sigma * C, d)) / (q * n)   # add noise, average
    return w

def epsilon(sigma, delta=1e-5):                       # RDP accountant, integer orders a
    best = math.inf
    for a in range(2, 128):
        t = [math.log(math.comb(a, k)) + (a - k) * math.log(1 - q) + k * math.log(q)
             + (k * k - k) / (2 * sigma ** 2) for k in range(a + 1)]
        m = max(t); log_A = m + math.log(sum(math.exp(v - m) for v in t))
        best = min(best, (T * log_A + math.log(1 / delta)) / (a - 1))
    return best

loss = lambda w, X, y: np.logaddexp(0, -(2 * y - 1) * (X @ w))   # log loss per example
for sigma in (None, 1.0, 2.0, 4.0):
    w = train(sigma)
    acc_in, acc_out = ((Xin @ w > 0) == yin).mean(), ((Xout @ w > 0) == yout).mean()
    score = -np.r_[loss(w, Xin, yin), loss(w, Xout, yout)]   # attack: low loss = member
    auc = roc_auc_score(np.r_[np.ones(n), np.zeros(n)], score)
    name = "no privacy    " if sigma is None else f"sigma {sigma} eps {epsilon(sigma):4.1f}"
    print(f"{name}: train acc {acc_in:.2f}  test acc {acc_out:.2f}  attack AUC {auc:.2f}")

no privacy    : train acc 0.99  test acc 0.65  attack AUC 0.76
sigma 1.0 eps 28.5: train acc 0.91  test acc 0.64  attack AUC 0.66
sigma 2.0 eps  9.8: train acc 0.80  test acc 0.60  attack AUC 0.60
sigma 4.0 eps  4.2: train acc 0.69  test acc 0.58  attack AUC 0.56

• No privacy: train accuracy 0.99 against test accuracy 0.65, and the loss attack reaches an AUC of 0.76. The generalization gap is the leak.
• sigma 1 ($\epsilon$ 28.5): the attack falls to 0.66 even though $\epsilon$ this large promises almost nothing formally. Clipping and noise act as a regularizer.
• sigma 4 ($\epsilon$ 4.2): the attack is down to 0.56, and test accuracy has fallen from 0.65 to 0.58. On a dataset this small, privacy is expensive; the larger the dataset, the less noise each person needs.

The accountant here is a simple one (integer orders only). Libraries such as Opacus for PyTorch and TensorFlow Privacy implement DP-SGD with tighter accountants and efficient per-example gradients; use them rather than hand-rolled code in production.

Choosing epsilon and auditing it

• There is no universal right epsilon. Single digits are common in research; the bound $e^{\epsilon }/(1+e^{\epsilon })$ is a good sanity check of what a number means to an attacker.
• Say what is protected. Example-level DP protects one record; if one user contributes many records, you need user-level DP, as in federated training.
• Audit empirically. Run membership inference against your own model before release, on held-out members and non-members. An AUC well above 0.5 is a finding even when a formal guarantee exists, because it may mean the implementation or the threat model is wrong.
• Removing names is not anonymization. Combinations of ordinary attributes can re-identify people when joined with other data, which is why DP bounds what any output reveals instead of trusting the data to be anonymous.

Trade-offs

Common mistakes

• Assuming a model is safe to publish because the training data is not published.
• Treating federated learning as private by itself, without secure aggregation or DP on the updates.
• Quoting an epsilon without delta, the number of steps, or whether it is per example or per user.
• Answering many queries or retraining many times without adding up the privacy budget.
• Tuning hyperparameters on the private data many times and ignoring that this also leaks.

In the wild

Check yourself

Recap