Data augmentation

The big idea

A cat photographed from a slightly different angle, in dimmer light, or with its tail cut off by the frame, is still a cat. Your model should know that, but if every training photo shows cats centred and well lit, it never learns it. Data augmentationCreating extra training examples by applying transformations to existing ones that do not change their correct label, such as flips, crops and colour changes. creates those variations for you: each time an image is used for training, it is randomly transformed in a way that keeps its label true.

Think of a teacher preparing flashcards. Instead of showing the same card ten times, they redraw it each time: a little bigger, a little to the left, in a different colour pen. The student learns the letter, not the card. Augmentation does the same, and acts as a strong regularizer ( Regularization for deep nets ): the network cannot memorize exact pixels when the pixels keep changing.

How it works

Step 1: Geometric transforms

• Random crops and shifts. Cut a random region (for example 224 by 224 out of a slightly larger image) or shift by a few pixels. The object appears at different positions and scales, so the model stops relying on exact location.
• Flips. Mirror left to right. A mirrored dog is still a dog, so for most photos this doubles the data for free.
• Small rotations and scaling. A few degrees of tilt, a little zoom in or out, as a phone camera would produce.

Step 2: Photometric transforms

Colour jitterRandomly changing an image's brightness, contrast, saturation or hue during training so the model becomes robust to lighting and camera differences. changes brightness, contrast, saturation and hue by small random amounts. It teaches the model that the class does not depend on the camera, the time of day or the white balance. Blur and noise do the same for focus and sensor quality.

Step 3: Erasing and mixing

CutoutAn augmentation that blanks out a random square region of the input image during training. blanks a random square of the image. The paper that introduced it describes exactly this, randomly masking out square regions of the input during training. The network can no longer rely on one telltale patch (a dog's nose, a watermark) and must use the whole object.

MixupAn augmentation that blends two training images and their one-hot labels with the same weight lambda, drawn from a Beta distribution. goes further: it blends two images and blends their labels with the same weight.

With lambda = 0.7, a 3 and an 8 become one image whose target is 70 percent "3" and 30 percent "8". The mixup paper reports that this improves generalization on ImageNet-2012, CIFAR-10 and CIFAR-100 among others, and reduces memorization of corrupt labels. It is the one technique here that deliberately changes the label, and it does so honestly, in proportion.

Step 4: The label-preserving rule

Every augmentation must keep the label true for your task. That depends on the task, not the technique:

• A mirrored 3, b or d is a different symbol, so no horizontal flips for digits or text. In the code, flip augmentation lowers clean test accuracy from 0.953 to 0.918.
• Rotating a 6 by 180 degrees makes a 9. Rotating a satellite image is fine; rotating a street scene puts the sky at the bottom.
• In medical images, left and right can matter, and colour can carry the diagnosis, so flips and colour jitter may be wrong.
• For object detection and segmentation, the boxes and masks must be transformed together with the image.

Augment only the training set. Validation and test images stay untouched, so they measure the real task.

Going deeper

Code: augmentation on 300 digits

import numpy as np
from sklearn.datasets import load_digits
from sklearn.neural_network import MLPClassifier

imgs, y = load_digits(return_X_y=True); imgs = imgs.reshape(-1, 8, 8) / 16.0
rng = np.random.default_rng(0)

# Label-preserving transforms for an 8x8 digit
def shift(im):    # random crop-like shift by up to 1 pixel, zero fill
    dy, dx = rng.integers(-1, 2, 2)
    return np.roll(np.pad(im, 1), (dy, dx), (0, 1))[1:-1, 1:-1]
def jitter(im):   # brightness and contrast change
    return np.clip(im * rng.uniform(0.7, 1.3) + rng.uniform(-0.1, 0.1), 0, 1)
def cutout(im):   # blank a random 3x3 square
    r, c = rng.integers(0, 6, 2); im = im.copy(); im[r:r+3, c:c+3] = 0; return im
hflip = lambda im: im[:, ::-1]   # NOT label-preserving for digits: a mirrored 7 is not a 7

idx = rng.permutation(len(y)); tr, te = idx[:300], idx[300:]       # only 300 training images
test_sets = {"clean test": imgs[te], "shifted test": np.array([shift(im) for im in imgs[te]])}

def run(name, aug, copies=4):
    X = [imgs[tr]] + [np.array([aug(im) for im in imgs[tr]]) for _ in range(copies)] if aug else [imgs[tr]]
    X = np.concatenate(X).reshape(-1, 64); Y = np.tile(y[tr], len(X) // len(tr))
    m = MLPClassifier((64,), max_iter=800, random_state=0).fit(X, Y)
    scores = "  ".join(f"{k} {m.score(v.reshape(-1, 64), y[te]):.3f}" for k, v in test_sets.items())
    print(f"{name:24} train size {len(X):5}   {scores}")

run("no augmentation", None)
run("shift", shift)
run("shift + jitter + cutout", lambda im: cutout(jitter(shift(im))))
run("horizontal flip", hflip)

# mixup: blend two images AND their one-hot labels, lambda ~ Beta(alpha, alpha)
lam = 0.7                                   # one draw; in training, sample a new lambda per pair
a, b = tr[y[tr] == 3][0], tr[y[tr] == 8][0]
x_mix = lam * imgs[a] + (1 - lam) * imgs[b]
y_mix = lam * np.eye(10)[y[a]] + (1 - lam) * np.eye(10)[y[b]]
print(f"\nmixup of digit {y[a]} and digit {y[b]}, lambda {lam:.2f}: label {np.round(y_mix, 2)}")

no augmentation          train size   300   clean test 0.953  shifted test 0.381
shift                    train size  1500   clean test 0.927  shifted test 0.866
shift + jitter + cutout  train size  1500   clean test 0.939  shifted test 0.797
horizontal flip          train size  1500   clean test 0.918  shifted test 0.369

mixup of digit 3 and digit 8, lambda 0.70: label [0.  0.  0.  0.7 0.  0.  0.  0.  0.3 0. ]

The model trained on 300 plain images scores 0.953 on clean test digits but only 0.381 on the same test digits shifted by up to one pixel: it learned exact positions. With shift augmentation it scores 0.866 on shifted digits, at a small cost on clean ones (0.927), because 8 by 8 digits are already perfectly centred. Adding jitter and cutout on top gives 0.939 clean and 0.797 shifted. Horizontal flips make things worse on both, because they teach the model mirrored digits that never occur. Augmentation is not free accuracy; it buys robustness to the variations you choose, so choose the ones your real data will have.

Online augmentation

In practice augmentation is applied on the fly by the data loader: each epoch sees a freshly transformed version of every image, so the network effectively never sees the same input twice. Libraries such as torchvision's transforms and Keras preprocessing layers chain random transforms into a pipeline that runs on CPU workers or the GPU while the model trains. Automated policies (searching over which transforms and how strong) and test-time augmentation (averaging predictions over a few transformed copies of a test image) build on the same idea.

Trade-offs

Common mistakes

• Using transforms that change the label: flipping text, rotating 6 into 9, mirroring left-right medical scans.
• Augmenting the validation or test set, so the reported score no longer reflects real inputs.
• Forgetting to transform bounding boxes or masks along with the image.
• Using augmentation so strong that training images no longer resemble real ones, so the model underfits.
• Augmenting once and saving a fixed set of copies to disk, instead of drawing new random transforms every epoch. (The code above does this only to keep the example short.)

In the wild

Random crops and horizontal flips are part of the standard ImageNet training recipe for CNNs (the ResNet paper samples a 224 by 224 crop from each image or its horizontal flip), and photo apps, self-driving perception and industrial inspection systems all rely on augmentation to cope with lighting, angle and camera differences. Domain-specific pipelines add their own label-safe transforms: rotations for satellite tiles, elastic deformations for handwriting and microscopy, and masking blocks of time steps and frequency channels for speech spectrograms (SpecAugment).

Check yourself

Recap