Transfer learning and fine-tuning

The big idea

You have 600 labelled photos of defects on a factory line. Training a ResNet from scratch on 600 images would overfit badly: it has millions of weights and almost nothing to learn them from. But a ResNet trained on ImageNet's 1.2 million photos has already learned to detect edges, textures, shapes and object parts. Those features are useful for almost any photo, including yours. Transfer learningReusing a model trained on one task or dataset (the source) as the starting point for a different task (the target), instead of training from random weights. starts from that trained network and adapts it to your task.

Think of hiring an experienced photographer to shoot product images for your shop. You do not teach them what light and focus are; you only show them your products and your style. A few days of adjustment beats years of training a beginner. The pretrained network is the experienced hire, and your small dataset is the short induction.

How it works

Step 1: Take a pretrained backbone

Recall from Pooling and CNN architecture that a CNN is a backbone (convolution stages that turn pixels into features) plus a head (layers that turn features into a decision). A pretrained modelA network whose weights were already trained on a large dataset, such as ImageNet, and are published for reuse. comes with weights learned on a large dataset. Its early layers detect generic patterns such as edges and colour blobs ( Convolutions ); later layers respond to more task-specific combinations. Libraries publish these weights: in torchvision, calling resnet50 with ResNet50_Weights.DEFAULT downloads a ResNet-50 trained on ImageNet, and weights.transforms() gives the exact resizing and normalization it expects ( Images as tensors ).

Step 2: Replace the head

The pretrained head predicts the source classes, for example ImageNet's 1,000. Remove it and attach a new, randomly initialized layer with one output per class of your task: a head replacementSwapping a pretrained model's final classification layer for a new one sized for the target task's classes.. For ResNet-50 that means a new linear layer from its 2,048 pooled features to your class count.

Step 3: Freeze the backbone and train the head

FreezingMarking parameters as not trainable, so the optimizer leaves their pretrained values unchanged. means excluding the backbone's weights from the optimizer, so only the new head learns. This is feature extractionUsing a frozen pretrained network purely to compute features, and training only a new head on top of them., row 1 of Figure 1. It is fast (no gradients through the backbone), needs very little data, and cannot damage the pretrained features. Train the head until the validation loss stops improving.

The order matters. A fresh head starts random, so its early gradients are large and noisy. If the backbone were trainable at that point, those gradients would flow back and wreck the pretrained features. The Keras transfer-learning guide states this directly: fine-tune only after the model with frozen layers has been trained to convergence.

Step 4: Unfreeze, carefully

Fine-tuningContinuing to train some or all of a pretrained network's weights on the target task, usually with a much smaller learning rate. then unfreezes some layers and keeps training, usually starting with the top block (row 2) and possibly moving down to all layers (row 3). Use a much smaller learning rate than for training from scratch: you want to nudge good weights, not overwrite them. Watch validation loss closely; with a small dataset and millions of trainable weights, overfitting comes fast ( Regularization for deep nets ).

When to unfreeze more:

• More target data: more weights can be trained without overfitting.
• Images that differ from the source: medical scans, satellite images or microscopy look little like everyday photos, so later, task-specific layers need to adapt.
• Validation accuracy has plateaued with the frozen backbone, and you have the compute to spare.

Going deeper

Code: when transfer helps, and when it does not

import numpy as np
from sklearn.datasets import load_digits

X, y = load_digits(return_X_y=True); X = X / 16.0
relu = lambda z: np.maximum(0, z)

def train(X, y, W1, b1, lr_body, steps=600, lr=0.5, seed=0):
    """MLP body 64 -> 64, plus a NEW random head. lr_body=0 keeps the body frozen."""
    W1, b1, k = W1.copy(), b1.copy(), y.max() + 1
    W2, b2, Y = np.random.default_rng(seed).normal(0, 0.1, (64, k)), np.zeros(k), np.eye(k)[y]
    for _ in range(steps):
        H = relu(X @ W1 + b1); Z = H @ W2 + b2
        P = np.exp(Z - Z.max(1, keepdims=True)); P /= P.sum(1, keepdims=True)
        G = (P - Y) / len(X); dH = (G @ W2.T) * (H > 0)     # softmax cross-entropy gradients
        W2 -= lr * H.T @ G; b2 -= lr * G.sum(0)
        W1 -= lr_body * X.T @ dH; b1 -= lr_body * dH.sum(0)
    return (lambda Xn: (relu(Xn @ W1 + b1) @ W2 + b2).argmax(1)), W1, b1

def experiment(name, Xs, ys, Xt, yt, sizes):
    W0, b0 = np.random.default_rng(1).normal(0, 0.15, (64, 64)), np.zeros(64)   # random body
    _, Wp, bp = train(Xs, ys, W0, b0, lr_body=0.5, steps=800)                    # pretrained body
    print(f"{name}\n{'labels/class':>14} {'scratch':>8} {'frozen+head':>12} {'fine-tune':>10}")
    for n in sizes:                       # few labels on the target; average of 5 random draws
        acc = np.zeros(3)
        for s in range(5):
            g = np.random.default_rng(s)
            idx = np.concatenate([g.choice(np.where(yt == c)[0], n, replace=False) for c in np.unique(yt)])
            test = np.setdiff1d(np.arange(len(yt)), idx)
            runs = [train(Xt[idx], yt[idx], W0, b0, lr_body=0.5, seed=s),    # train everything from scratch
                    train(Xt[idx], yt[idx], Wp, bp, lr_body=0.0, seed=s),    # frozen body, new head only
                    train(Xt[idx], yt[idx], Wp, bp, lr_body=0.05, seed=s)]   # fine-tune body with small lr
            acc += [(m(Xt[test]) == yt[test]).mean() / 5 for m, _, _ in runs]
        print(f"{n:>14} {acc[0]:>8.3f} {acc[1]:>12.3f} {acc[2]:>10.3f}")

# A) Narrow source task: features learned on digits 0-4, reused for new classes 5-9
experiment("A) source: digits 0-4 -> target: digits 5-9", X[y < 5], y[y < 5], X[y >= 5], y[y >= 5] - 5, [3, 10, 30])

# B) Same digits, new domain: pretrain on 900 clean images, target is the other images with heavy noise
g = np.random.default_rng(0); perm = g.permutation(len(y)); a, b = perm[:900], perm[900:]
Xnoisy = np.clip(X[b] + g.normal(0, 0.3, X[b].shape), 0, 1)
experiment("\nB) source: clean digits -> target: noisy digits", X[a], y[a], Xnoisy, y[b], [2, 5, 20])

A) source: digits 0-4 -> target: digits 5-9
  labels/class  scratch  frozen+head  fine-tune
             3    0.894        0.798      0.839
            10    0.935        0.886      0.913
            30    0.961        0.924      0.948

B) source: clean digits -> target: noisy digits
  labels/class  scratch  frozen+head  fine-tune
             2    0.558        0.729      0.730
             5    0.697        0.791      0.797
            20    0.796        0.835      0.836

The code compares three strategies on small labelled sets, using a tiny one-hidden-layer network on 8 by 8 digits, so treat it as a model of the idea, not of ImageNet-scale results. In case A the body was pretrained on digits 0 to 4 and reused for the new classes 5 to 9, and training from scratch wins at every size: features learned for five classes in a shallow net are too narrow to help. In case B the body was pretrained on 900 clean digit images and reused on the other images with heavy noise added. With 2 labels per class, the frozen pretrained body reaches 0.729 accuracy against 0.558 from scratch; the gap shrinks to 0.835 versus 0.796 at 20 labels per class, and fine-tuning with a small body learning rate edges past frozen features.

That pair is the whole lesson in miniature. Transfer pays off when the source task is broad or close to yours and labels are scarce; its advantage shrinks as your own data grows; and a narrow, unrelated source can be worse than nothing. ImageNet backbones transfer so well because 1,000 diverse classes force general features.

The same steps in PyTorch

The runnable code above is NumPy; with PyTorch and torchvision the recipe looks like this:

import torch, torch.nn as nn
from torchvision.models import resnet50, ResNet50_Weights

weights = ResNet50_Weights.DEFAULT
model = resnet50(weights=weights)
preprocess = weights.transforms()                    # the resize + normalization it was trained with

for p in model.parameters():                         # 1) freeze the backbone
    p.requires_grad = False
model.fc = nn.Linear(model.fc.in_features, 5)        # 2) new head for 5 classes (trainable)
opt = torch.optim.AdamW(model.fc.parameters(), lr=1e-3)
# ... train the head until validation loss stops improving ...

for p in model.layer4.parameters():                  # 3) unfreeze the last stage
    p.requires_grad = True
opt = torch.optim.AdamW([
    {"params": model.layer4.parameters(), "lr": 1e-5},   # small lr for pretrained weights
    {"params": model.fc.parameters(), "lr": 1e-4},
])

Batch normalization in a frozen backbone

Batch normalization layers ( Weight initialization and normalization ) keep running statistics that are updated in training mode even when their weights are frozen. If your small batches have different statistics, those updates can quietly damage the pretrained features. The Keras guide recommends keeping BatchNormalization layers in inference mode when fine-tuning; in PyTorch the equivalent is calling .eval() on the backbone's normalization layers.

Trade-offs

Common mistakes

• Unfreezing the backbone while the new head is still random, so large early gradients wreck the pretrained features.
• Fine-tuning with the same learning rate used for training from scratch.
• Preprocessing inputs differently from the pretrained model's training (size, normalization, RGB vs BGR).
• Letting batch normalization statistics update on tiny batches during fine-tuning.
• Assuming any pretrained model helps. A source unrelated to your data can be no better, or worse, than random weights.

In the wild

Transfer learning is the default way vision models are built in practice. Medical imaging, retail product recognition, agriculture and industrial inspection teams typically start from a backbone pretrained on ImageNet or a larger dataset, rather than from random weights. Model hubs such as torchvision, Keras Applications and Hugging Face publish pretrained weights together with their preprocessing, and the same pattern of pretrain on a broad task, then fine-tune on a narrow one, is how large language models are adapted too.

Check yourself

Recap