Machine Learning ML-15.08

Model compression

Make models smaller and faster: quantization to fewer bits, pruning unimportant weights, and knowledge distillation into a small student.

Advanced 14 min Last reviewed 2026-10-16

The big idea

The model that wins offline is often too big to serve: too slow for the latency budget, too large for a phone, too expensive per request at millions of requests a day. Compression shrinks a trained model while keeping most of its accuracy. There are three main tools, and they attack different kinds of waste.

Quantization stores each number with fewer bits. Most weights do not need 32 bits of precision.
Pruning removes weights that barely matter. Trained networks have many near-zero connections.
Distillation trains a small model to imitate the big one. The big model's full output distribution carries more information than the labels it was trained on.

Think of a photo library. You can save each photo at lower quality (quantization), crop away the empty sky (pruning), or ask a skilled artist to redraw the scene as a small sketch that keeps what matters (distillation). Each saves space in a different way, and they combine.

How it works

Quantization Map each weight (and often each activation) to a small integer plus a shared scale. 8 bits is 4 times smaller than 32; 4 bits is 8 times smaller.
Pruning Set the smallest-magnitude weights to zero, then fine-tune the rest to recover accuracy. Unstructured zeros save memory only with sparse storage, and save time only on hardware or kernels that skip them.
Distillation Train a new, smaller network to match the big model's output probabilities on a transfer set. The student is an ordinary dense model, so it is fast everywhere.

Step 1: Quantization

A 32-bit float can represent about 4 billion values; the weights of one layer usually span a narrow range. Quantization maps that range onto a few integer levels. The scheme from Jacob and colleagues at Google (2017), designed for integer-only inference, is an affine map

r = S (q - Z)

where $r$ is the real value, $q$ the integer, $S$ a positive scale and $Z$ the zero point, the integer that represents real 0. Worked example with 8 bits: a layer's weights span $- 0.5$ to $0.7$ . There are $2^{8} - 1 = 255$ steps, so $S = 1.2/255 \approx 0.0047$ and $Z = round (0.5/0.0047) = 106$ . The weight $0.3$ becomes $q = round (0.3/0.0047) + 106 = 170$ and is read back as $0.0047 \times (170 - 106) \approx 0.301$ .

Two ways to get there:

Post-training quantization (PTQ): quantize a finished model, using a small calibration set to pick scales for activations. Fast and usually enough at 8 bits.
Quantization-aware training (QAT): simulate the rounding during training so the model learns weights that survive it. Needed when accuracy drops too much, typically at lower bit widths or on already compact models.

Large language models push further: GPTQ (2022) quantized 175-billion-parameter GPT models to 3 or 4 bits per weight in about four GPU hours, with negligible accuracy degradation, and reported end-to-end inference speed-ups over FP16 of about 3.25x on A100 GPUs and 4.5x on A6000s. Lower precision also helps during training, the subject of the next lesson ( GPUs and distributed training ).

Step 2: Pruning

Pruning exploits redundancy. Han and colleagues (2015) used three steps: train the network to learn which connections are important, prune the unimportant ones, and retrain the rest. On ImageNet this cut AlexNet from 61 million to 6.7 million parameters (9x) and VGG-16 from 138 million to 10.3 million (13x), without loss of accuracy.

The catch is speed. Unstructured pruning leaves zeros scattered through dense matrices; standard hardware multiplies them anyway. To turn sparsity into latency you need sparse kernels, hardware support for a fixed sparsity pattern, or structured pruning, which removes whole neurons, channels or attention heads so the remaining matrices are simply smaller.

Step 3: Knowledge distillation

Temperature 1 For a handwritten 2 the teacher's logits are 9, 5, 4, 1 and 0 for the classes 2, 3, 7, 1 and 0. At T = 1 almost all probability sits on 2; the rest is too small to learn from.
Temperature 4 Dividing the logits by 4 before the softmax keeps the order but spreads the mass: this 2 looks somewhat like a 3, a bit like a 7, and little like a 1. That similarity structure is what the student learns.

Hinton, Vinyals and Dean's paper, linked below, starts from an observation: a trained classifier assigns small probabilities to all wrong answers, and their relative sizes say a lot about how it generalizes. Their example: an image of a BMW may have a very small chance of being mistaken for a garbage truck, but that is still many times more likely than mistaking it for a carrot. One-hot labels throw this away.

Knowledge distillation trains the student on the teacher's soft targets, produced with a softmax at a raised temperature $T$ :

p_{i} = \frac{exp ( z _{i} / T )}{\sum _{j} exp ( z _{j} / T )}

The student uses the same high temperature during distillation and $T = 1$ afterwards. The paper combines two losses, cross-entropy with the soft targets and, with a considerably lower weight, cross-entropy with the true labels, and multiplies the soft-target term by $T^{2}$ , because its gradients scale as $1/ T^{2}$ . In the high-temperature limit, distillation becomes matching the teacher's logits.

The results that made the idea famous: on MNIST a large regularized net made 67 test errors and a small net trained normally made 146; the same small net trained to match the large net's soft targets at temperature 20 made 74. On a speech recognition acoustic model, a single distilled model reached 60.8 percent frame accuracy against 58.9 percent for the baseline and 61.1 percent for the 10-model ensemble it was distilled from, with the same word error rate as the ensemble, 10.7 percent. DistilBERT later applied the idea to language models: 40 percent smaller than BERT, 60 percent faster, keeping 97 percent of its language understanding capabilities.

Going deeper

Code: all three on one network

compress.pyruns with python3 + numpy + scipy + scikit-learn

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier, MLPRegressor
from scipy.ndimage import shift
X, y = load_digits(return_X_y=True); X = X / 16
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=0)
teacher = MLPClassifier((512, 512), alpha=1e-3, max_iter=600, random_state=0).fit(Xtr, ytr)
W, b = [w.copy() for w in teacher.coefs_], [c.copy() for c in teacher.intercepts_]

def logits(X, W, b):                                    # forward pass, returning pre-softmax scores
    for i, (w, c) in enumerate(zip(W, b)):
        X = X @ w + c
        if i < len(W) - 1: X = np.maximum(X, 0)
    return X
acc = lambda W, b: (logits(Xte, W, b).argmax(1) == yte).mean()
n_params = sum(w.size + c.size for w, c in zip(W, b))
print(f"teacher: {n_params:,} parameters, {n_params * 4 / 1e6:.2f} MB in float32, accuracy {acc(W, b):.3f}")

def quantize(w, bits):                                  # affine: real = scale * (q - zero_point)
    levels = 2 ** bits - 1; scale = (w.max() - w.min()) / levels
    zero = np.round(-w.min() / scale)
    return scale * (np.clip(np.round(w / scale) + zero, 0, levels) - zero)
for bits in [8, 4, 3, 2]:
    print(f"quantized to {bits} bits: {n_params * bits / 8 / 1e6:.2f} MB, accuracy {acc([quantize(w, bits) for w in W], b):.3f}")

for frac in [0.5, 0.8, 0.9, 0.97]:                      # magnitude pruning, no retraining
    cut = np.quantile(np.abs(np.concatenate([w.ravel() for w in W])), frac)
    print(f"pruned {frac:.0%} of weights: accuracy {acc([np.where(np.abs(w) < cut, 0, w) for w in W], b):.3f}")

# Distillation: a 16-unit student, trained on hard labels or on the teacher's logits (soft targets).
jitter = np.concatenate([shift(Xtr.reshape(-1, 8, 8), (0, dy, dx), order=0).reshape(len(Xtr), 64)
                         for dy in (-1, 0, 1) for dx in (-1, 0, 1)])  # transfer set: shifted copies
small = dict(hidden_layer_sizes=(16,), alpha=1e-3, max_iter=2000, random_state=0)
hard = MLPClassifier(**small).fit(Xtr, ytr)
hard9 = MLPClassifier(**small).fit(jitter, np.tile(ytr, 9))                   # same shifted copies, true labels
soft = MLPRegressor(**small).fit(jitter, logits(jitter, W, b) - logits(jitter, W, b).mean(1, keepdims=True))
s_params = 64 * 16 + 16 + 16 * 10 + 10
print(f"student ({s_params:,} parameters) on hard labels: accuracy {hard.score(Xte, yte):.3f}")
print(f"same student, hard labels on shifted copies:  accuracy {hard9.score(Xte, yte):.3f}")
print(f"student distilled from teacher logits:        accuracy {(soft.predict(Xte).argmax(1) == yte).mean():.3f}")

Output

teacher: 301,066 parameters, 1.20 MB in float32, accuracy 0.978
quantized to 8 bits: 0.30 MB, accuracy 0.978
quantized to 4 bits: 0.15 MB, accuracy 0.978
quantized to 3 bits: 0.11 MB, accuracy 0.976
quantized to 2 bits: 0.08 MB, accuracy 0.944
pruned 50% of weights: accuracy 0.975
pruned 80% of weights: accuracy 0.972
pruned 90% of weights: accuracy 0.929
pruned 97% of weights: accuracy 0.484
student (1,210 parameters) on hard labels: accuracy 0.961
same student, hard labels on shifted copies:  accuracy 0.893
student distilled from teacher logits:        accuracy 0.972

The teacher is a 512-512 MLP on scikit-learn's 8x8 digits: 301,066 parameters, 1.20 MB in float32, 97.8 percent test accuracy.

Quantization (weights only, one scale per matrix, no retraining): 8 bits and 4 bits both keep 97.8 percent at 0.30 MB and 0.15 MB; 3 bits gives 97.6 percent; at 2 bits accuracy falls to 94.4 percent. Most precision was unnecessary until the very end.
Pruning by magnitude, without fine-tuning: 50 percent of weights gone costs 0.3 points (97.5), 80 percent costs 0.6 (97.2), 90 percent costs almost 5 (92.9), and 97 percent collapses to 48.4. Fine-tuning after pruning, as Han and colleagues did, recovers much of the loss at high sparsity.
Distillation into a 16-unit student with 1,210 parameters, 250 times fewer: trained on the true labels it scores 96.1 percent. Trained to match the teacher's logits (the high-temperature limit) on a transfer set of shifted copies of the training images, it scores 97.2 percent. The control matters: the same student trained on the true labels of those shifted copies scores only 89.3 percent, so the gain comes from the teacher's soft targets, not from the extra images.

Accuracy, latency and cost

Compression is judged on the serving metric, not on size alone. Measure accuracy on the slices that matter (rare classes often suffer first), latency at p99 on the real hardware, and cost per thousand predictions. Gains from different techniques multiply: a distilled student can also be quantized. And re-check calibration: a quantized or distilled model can keep its accuracy while its probabilities drift, which matters whenever thresholds or prices are set on them.

Trade-offs

Option	Wins when	Loses when
Post-training quantization	Minutes of work; 4x smaller at 8 bits; integer hardware runs it fast	Accuracy can drop at 4 bits and below without QAT
Quantization-aware training	Best accuracy at low bit widths	Needs the training pipeline and more compute
Unstructured pruning	Very high sparsity with little accuracy loss after fine-tuning	Little speed-up on standard dense hardware
Structured pruning	Real speed-ups: smaller dense matrices	Larger accuracy hit per parameter removed
Distillation	Small, fast dense model; can absorb an ensemble	A full training run; needs a good transfer set

Common mistakes

Reporting model size reductions without measuring latency on the target hardware.
Pruning to high sparsity and expecting a speed-up on dense GPU kernels.
Quantizing without a calibration set that looks like real traffic, so activation ranges are wrong.
Checking only overall accuracy after compression, not rare classes or calibration.
Distilling on a transfer set that does not cover the inputs the student will see in production.

In the wild

Google

Distilling the knowledge in a neural network (Hinton, Vinyals, Dean, 2015) (opens in a new tab)

The paper argues that training and deployment have different needs, so a cumbersome model or ensemble can be trained to extract structure from data and its knowledge then transferred to a small model for deployment. It trains the small model on the cumbersome model's soft targets at a raised softmax temperature, combined with a lower-weighted hard-label loss. A small MNIST net went from 146 to 74 test errors this way, against 67 for the large net, and distilling a 10-model speech ensemble into a single acoustic model transferred more than 80 percent of the ensemble's frame accuracy gain.

Check yourself

A layer's weights span -0.5 to 0.7 and are quantized to 8 bits with r = S(q - Z). What is the scale S?
Show answer

B. 1.2 / 255, about 0.0047 The range 1.2 is split into 2^8 - 1 = 255 steps.
You prune 90 percent of a model's weights (unstructured) and latency on a standard GPU barely changes. Why?
Show answer

B. Dense kernels still multiply the zeros; you need sparse kernels, hardware support or structured pruning Scattered zeros save memory with sparse storage, but compute only drops if the hardware or kernel skips them.
Why does distillation use a temperature above 1 when producing soft targets?
Show answer

B. To spread probability over the wrong classes so their relative sizes, which encode similarity, carry a usable training signal At T = 1 the wrong-class probabilities are tiny; raising T exposes their ratios. The paper multiplies the soft loss by T^2 to keep gradients balanced.
In the code, why include a student trained on true labels of the shifted copies?
Show answer

B. As a control: it shows the distilled student's gain comes from the teacher's soft targets, not from the extra images That control scored 89.3 percent against 97.2 percent for the distilled student.

Recap

Quantization stores weights (and activations) as low-bit integers with a scale and zero point: r = S(q - Z). 8 bits is usually nearly free.
Pruning removes small weights and fine-tunes; unstructured sparsity saves memory but only saves time with support for skipping zeros.
Distillation trains a small student on a teacher's temperature-softened outputs, which carry how the teacher ranks wrong answers.
Techniques combine: distill, then quantize.
Judge compression on accuracy by slice, calibration, p99 latency on real hardware and cost per prediction.