Robustness and adversarial examples

The big idea

A digit classifier is 96 percent accurate on its test set. Change every pixel by a fifth of its range, in a direction you choose carefully, and accuracy falls to 11 percent. Change every pixel by half that amount in random directions and the model barely notices. The images still look the same to a person. The model was never "seeing" digits the way you do; it was adding up thousands of small pieces of evidence, and an attacker can nudge all of them at once.

Think of a jury that decides by adding up hundreds of small clues. No single clue is decisive, so tampering with one changes nothing. But a lawyer who knows which way each clue leans can shade every one of them slightly, and the total swings. That is an adversarial exampleAn input changed by a small, deliberately chosen perturbation so that a model gives a wrong output, usually with high confidence, while a person sees no meaningful change..

Adversaries are one way a model meets inputs it was not trained for. The other is the world itself: cameras change, customers change, a pandemic changes shopping. RobustnessA model's ability to keep its accuracy when inputs are perturbed, corrupted or drawn from a different distribution than the training data. is about both, and about testing for them before deployment instead of discovering them in production.

How it works

Step 1: Small perturbations, big errors

Szegedy et al. (arXiv 1312.6199) first showed that "an imperceptible non-random perturbation" of a test image can change a network's prediction arbitrarily. The early guess was that deep networks are too non-linear. Goodfellow, Shlens and Szegedy (arXiv 1412.6572) argued the opposite: they are too linear.

Their argument fits in one line. A linear score is $w^{\top }x$. Change every input by at most $\epsilon$ (a max-norm constraintA bound on the largest change to any single input feature, written as the infinity norm of the perturbation being at most epsilon.) and the biggest possible change in the score comes from moving each feature in the direction of its weight's sign:

where $n$ is the number of inputs and $m$ the average weight magnitude. The per-pixel change stays at $\epsilon$, but the effect on the score grows with $n$. With $n=1,000$ pixels, $m=0.1$ and $\epsilon =0.04$ (about 10 of 255 grey levels), the score moves by $0.04\times 0.1\times 1000=4$: enough to take a sigmoid output from 0.95 to 0.27. Random noise of the same size points the wrong way about half the time and mostly cancels, so its effect grows only like $\sqrt{n}$.

Step 2: The fast gradient sign method

For a non-linear model you do not know $w$, but you can use the gradient of the loss with respect to the input, which backpropagation gives you at the cost of one backward pass. Goodfellow et al. linearize the loss around $x$ and take the best max-norm step:

This is the fast gradient sign methodFGSM: an attack that adds epsilon times the sign of the loss gradient with respect to the input, the best single step under a max-norm limit if the loss were linear. (FGSM). Their famous example: GoogLeNet labels a panda image "panda" with 57.7 percent confidence; after adding a perturbation with $\epsilon =0.007$, the size of the smallest step in an 8-bit image, it says "gibbon" with 99.3 percent confidence. On MNIST with $\epsilon =0.25$, a shallow softmax classifier had a 99.9 percent error rate on FGSM examples with average confidence 79.3 percent, and a maxout network 89.4 percent error at 97.6 percent confidence.

Stronger attacks iterate. PGDProjected gradient descent attack: repeat small FGSM-style steps, projecting back into the allowed epsilon box after each one. takes many small sign steps and clips back into the $\epsilon$ box after each; Madry et al. (arXiv 1706.06083) treat it as a "universal first-order adversary" and use it both to evaluate and to train robust models. Attacks also work without gradients, by querying a model and transferring examples from a substitute model, and they survive the physical world: Eykholt et al. (arXiv 1707.08945) put black and white stickers on a real stop sign and caused targeted misclassification in 100 percent of lab images and 84.8 percent of video frames captured from a moving vehicle.

Step 3: Adversarial training

The most reliable defence is to train on the attack. Adversarial trainingTraining on adversarial examples generated against the current model at each step, alongside or instead of clean examples, so the model learns to resist them. generates FGSM or PGD examples against the current model at every step and adds them to the batch with their true labels:

Madry et al. frame it this way: an inner attacker maximises the loss inside the box, the outer training loop minimises it. Goodfellow et al. report that adversarial training cut a maxout network's error on FGSM examples from 89.4 to 17.9 percent. It costs extra compute per step (one extra backward pass for FGSM, several for PGD), often a little clean accuracy, and it protects only against the threat model you trained for: robustness to small max-norm changes says nothing about rotations, stickers or a new camera.

Defences that hide or break the gradient, such as adding a non-differentiable preprocessing step, usually fail against an attacker who adapts to them. Evaluate a defence with an attacker who knows it is there.

Step 4: Distribution shift, the everyday robustness problem

Most production failures need no attacker. Distribution shiftA difference between the data a model was trained on and the data it sees in use, in the inputs, the labels, or the relationship between them. means the test data no longer comes from the training distribution. Three kinds to name:

• Covariate shift: the inputs change but the rule linking them to labels does not (a new camera, a new market, older customers).
• Label shift: the class balance changes (fraud rates in a holiday season).
• Concept drift: the rule itself changes (what counts as spam, how fraudsters behave).

Even careful replication shifts the data. Recht et al. (arXiv 1902.10811) rebuilt the CIFAR-10 and ImageNet test sets by following the original collection process, and a broad range of models lost 3 to 15 percent accuracy on CIFAR-10 and 11 to 14 percent on ImageNet. Their analysis points to the models' inability to generalize to slightly harder images, not to overfitting of the old test sets.

The dangerous part is that confidence does not drop with accuracy. Softmax outputs stay high on inputs far from anything seen in training, so monitor input statistics and accuracy on fresh labels, not just the model's own confidence. Monitoring for drift in depth is covered in the MLOps module (ML-15.07, Monitoring, drift and retraining).

Step 5: Robustness testing before release

Test the cases your test set does not contain, and give each one a pass or fail threshold:

1. Slices. Accuracy per segment (device, region, language, lighting). Average accuracy hides weak slices.
2. Corruptions. Blur, noise, compression, brightness, occlusion at several strengths. ImageNet-C (Hendrycks and Dietterich, arXiv 1903.12261) packages 15 corruption types at five severity levels as a benchmark.
3. Behavioural tests. The CheckList method (Ribeiro et al., arXiv 2005.04118) for NLP uses minimum functionality tests, invariance tests (a change that should not alter the output, such as swapping a name) and directional tests (a change that should move it a known way, such as adding a negation). Its authors found critical failures in a commercial sentiment model that had been extensively tested.
4. Adversarial evaluation. Accuracy under FGSM and PGD at the $\epsilon$ that matches your threat model, using an attack that knows any defences.
5. Shifted holdouts. Data from a later time period, a new site or a new device, kept out of training entirely.

Try it

1. Start with the defaults: 100 pixels and a maximum change of 0.040 per pixel (10 of 255 grey levels). The worst-case change moves the score by 0.38 and P(class A) only drops from 0.95 to 0.93. The two pixel grids look identical.
2. Drag the pixel count to 1,000 without touching the change per pixel. The worst case now moves the score by 4.00 and P(class A) falls to 0.27: the decision flips. Random noise of exactly the same size moves it by -0.05. Find where the violet line crosses the red one: about 760 pixels.
3. Keep 1,000 pixels and lower the change to 0.010. The shift is 1.00 and P(class A) stays at 0.88. The size of the attack needed shrinks as dimension grows, which is why images with hundreds of thousands of pixels are so easy to fool.

Going deeper

Code: FGSM, adversarial training and a one-pixel shift

A softmax classifier on scikit-learn's 8 by 8 digits (pixels scaled to 0 to 1). FGSM uses the exact input gradient of the cross-entropy loss. A second model is adversarially trained with FGSM at $\epsilon =0.1$. Then the standard model meets random noise and a test set in which every image is shifted one pixel to the right.

# FGSM against a softmax classifier on 8x8 digits, adversarial training, and a shifted test set.
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True)
X = X / 16.0                                              # pixels in [0, 1]
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
Y = np.eye(10)[ytr]

def probs(W, b, X):
    z = X @ W + b; z -= z.max(1, keepdims=True); e = np.exp(z); return e / e.sum(1, keepdims=True)

def fgsm(W, b, X, y, eps):                                # x + eps * sign(grad_x loss)
    g = (probs(W, b, X) - np.eye(10)[y]) @ W.T            # exact input gradient of cross-entropy
    return np.clip(X + eps * np.sign(g), 0, 1)

def train(eps_adv=0.0, steps=2000, lr=0.5, lam=1e-3):
    W, b = np.zeros((64, 10)), np.zeros(10)
    for _ in range(steps):
        Xb = Xtr if eps_adv == 0 else np.r_[Xtr, fgsm(W, b, Xtr, ytr, eps_adv)]
        Yb = Y if eps_adv == 0 else np.r_[Y, Y]
        G = (probs(W, b, Xb) - Yb) / len(Xb)
        W -= lr * (Xb.T @ G + lam * W); b -= lr * G.sum(0)
    return W, b

acc = lambda W, b, X: (probs(W, b, X).argmax(1) == yte).mean()
models = {"standard": train(), "adv-trained": train(eps_adv=0.1)}
for name, (W, b) in models.items():
    row = " ".join(f"eps {e:.2f}: {acc(W, b, fgsm(W, b, Xte, yte, e)):.2f}" for e in (0, 0.05, 0.1, 0.2))
    print(f"{name:12s} {row}")

rng = np.random.default_rng(0)                            # random noise of the same size as FGSM
W, b = models["standard"]
print(f"random +-0.10 noise: {acc(W, b, np.clip(Xte + 0.1 * rng.choice([-1, 1], Xte.shape), 0, 1)):.2f}")
shift = np.roll(Xte.reshape(-1, 8, 8), 1, axis=2).reshape(-1, 64)   # every image moved 1 pixel right
print(f"shifted 1 pixel:     {acc(W, b, shift):.2f}")
conf = lambda X: probs(W, b, X).max(1).mean()
print(f"mean confidence: clean {conf(Xte):.2f}, shifted {conf(shift):.2f}, FGSM 0.2 {conf(fgsm(W, b, Xte, yte, 0.2)):.2f}")

standard     eps 0.00: 0.96 eps 0.05: 0.86 eps 0.10: 0.64 eps 0.20: 0.11
adv-trained  eps 0.00: 0.95 eps 0.05: 0.88 eps 0.10: 0.72 eps 0.20: 0.27
random +-0.10 noise: 0.95
shifted 1 pixel:     0.37
mean confidence: clean 0.87, shifted 0.66, FGSM 0.2 0.60

• FGSM: the standard model goes from 0.96 clean accuracy to 0.64 at $\epsilon =0.1$ and 0.11 at $\epsilon =0.2$, barely better than guessing among ten classes. Random noise of size 0.1 leaves it at 0.95.
• Adversarial training buys robustness near the training $\epsilon$ (0.72 instead of 0.64 at 0.1) for one point of clean accuracy (0.95). Beyond the trained $\epsilon$ it still collapses (0.27 at 0.2).
• A one-pixel shift, a change no person would notice, drops accuracy to 0.37, yet the mean confidence is still 0.66. Nobody attacked anything: the training data simply never had digits in that position. Data augmentation with small shifts would fix this far more cheaply than any adversarial defence.

Certified robustness and out-of-distribution detection

Two research directions go beyond empirical testing. Certified methods, such as interval bounds or randomized smoothing, prove that no perturbation within $\epsilon$ can change a given prediction, at the price of extra compute and usually lower accuracy. Out-of-distribution detectionDetecting inputs that are unlike the training data, so the system can abstain, ask a human or fall back to a safer rule instead of predicting. adds a second question to every prediction, "does this input look like my training data?", and routes doubtful inputs to a fallback or a human. Neither is a substitute for testing on realistic shifted data.

Trade-offs

Common mistakes

• Reporting robustness against random noise and calling the model robust to attacks; worst-case perturbations are far stronger.
• Evaluating a defence only with the attack it was designed against, instead of an adaptive attacker.
• Trusting softmax confidence as a sign that the input is familiar.
• Testing on a random split of the same data and assuming production will look the same.
• Adversarially training for one $\epsilon$ and threat model, then claiming general robustness.

In the wild

Where you will see this: content moderation, spam and fraud systems face users who probe and adapt to the model ( Design a fraud detection system and Design a spam / content moderation classifier design systems around it); self-driving perception teams test against physical attacks like the stop-sign stickers of Eykholt et al. and against weather, glare and new camera hardware; and image and text platforms run corruption and behavioural test suites such as ImageNet-C and CheckList before shipping models. Recht et al.'s rebuilt ImageNet test set is a reminder that even a faithful copy of the data pipeline produces a measurable shift.

Check yourself

Recap