Distillation and synthetic data

The big idea

A frontier model can do your task well, but it is slow and expensive to call millions of times. A small model is fast and cheap but not good enough. DistillationTraining a smaller student model to imitate a larger teacher model's outputs, so that it keeps much of the teacher's skill at a fraction of the cost. closes the gap: let the big model, the teacher, produce answers for many inputs, and train the small model, the student, to reproduce them.

The same trick generalises. Any time good training data is scarce, a strong model can write more of it: instructions and answers, worked solutions, rare edge cases, dialogues in a niche domain. This is synthetic dataTraining data generated by a model rather than collected from people or the world, for example instructions, answers or worked solutions written by a strong model.. It is now a normal part of training pipelines, and its main danger is quality. Generated data must be filtered, and a model trained over and over on model output, with nothing fresh, slowly forgets the rare parts of the world.

Think of an apprentice and a master chef. The apprentice learns far faster by watching the master cook thousands of dishes than from a recipe book with fifty recipes. But if apprentices only ever learn from previous apprentices, each generation a little sloppier and a little less adventurous, the unusual dishes vanish from the kitchen within a few generations.

How it works

Teacher and student

In the classic form, the student is trained to match the teacher's full probability distribution over outputs, not just its top answer. Those soft targetsA teacher model's full probability distribution over possible outputs, used as a training target. It carries more information than a single correct label, such as which wrong answers are nearly right. carry extra information: a teacher that says "cat 0.7, lynx 0.25, car 0.0001" is telling the student that cats and lynxes look alike. A temperature above 1 softens the distribution further so this structure is easier to learn.

With language models you often only get text back from the teacher, especially through an API. Then distillation is simply supervised fine-tuning on teacher-written answers, sometimes including its reasoning traces ( Reasoning models and test-time compute ). It still works remarkably well, because the expensive part, knowing what a good answer looks like, comes from the teacher.

Generating synthetic data

A typical pipeline starts from a small set of seed tasks written by people, asks a strong model to propose many new tasks in the same spirit, then asks it to answer them. Variety comes from deliberately varying topics, difficulty, personas and formats in the generation prompts. For skills with checkable answers, such as math or code, you can generate many attempts and keep only those that pass.

Filtering is most of the work

Raw generated data is full of near-duplicates, wrong answers and bland, samey phrasing. Common filters:

• Correctness: run code against tests, check math answers, or have a second model verify.
• Deduplication: drop examples too similar to each other or to your evaluation sets.
• Quality scoring: an LLM judge ( LLM-as-judge ) or a classifier rates helpfulness or difficulty; keep the top slice.
• Diversity: cap how many examples come from any one topic or template.

A smaller, cleaner set usually beats a larger, noisier one.

Model collapse

Generative models under-represent rare things: when they sample, the common cases come out often and the rare ones seldom or never. Train a new model on that output and the tails get thinner; repeat for a few generations and they vanish. Researchers call this model collapseThe gradual, compounding loss of diversity and rare content when generative models are trained repeatedly on data produced by earlier models, without enough fresh real data.. The defence is to keep fresh real data in the mix, filter for diversity, and track where your data came from.

Try it

Try this:

1. At the defaults (200 samples per generation, no real data), press "Run 10 generations" twice. After 10 generations 13 of the 20 words survive; after 20 only 9 do, and the rare half of the vocabulary, 16 percent of real text, has vanished completely.
2. Reset and switch sampling to top-p 0.9, which never emits the least likely words. After 10 generations only 3 words remain. Truncated sampling, common in production, speeds collapse up.
3. Reset, go back to full sampling and set fresh real data to 10 percent. Run 50 generations: about 17 words survive and the rare half still holds roughly its real share, because real data keeps reintroducing them.

Each word that hits zero is marked with a red x. A count model cannot recover a word it never sees again, which is the core of the problem.

Going deeper

Soft labels and unlabelled inputs, in code

A teacher trained on plenty of data labels 3,000 unlabelled inputs. A student trained on those labels nearly matches the teacher, while one trained only on the 60 human labels you could afford falls far behind.

# Distillation in miniature: a student learns from a teacher's outputs on unlabelled inputs.
import numpy as np

rng = np.random.default_rng(1)
d, k = 20, 5                                          # features, classes
W_true = rng.normal(size=(d, k))

def make(n):                                          # inputs and noisy "human" labels
    X = rng.normal(size=(n, d))
    p = np.exp(X @ W_true); p /= p.sum(1, keepdims=True)
    y = np.array([rng.choice(k, p=row) for row in p])
    return X, y

def softmax(z):
    z = z - z.max(1, keepdims=True)
    e = np.exp(z); return e / e.sum(1, keepdims=True)

def train(X, targets, steps=600, lr=0.5):            # softmax regression on target probabilities
    W = np.zeros((d, k))
    for _ in range(steps):
        W -= lr * X.T @ (softmax(X @ W) - targets) / len(X)
    return W

onehot = lambda y: np.eye(k)[y]
X_test, _ = make(5000)
truth = (X_test @ W_true).argmax(1)
acc = lambda W: (np.argmax(X_test @ W, 1) == truth).mean()
show = lambda name, W: print(f"{name:36}accuracy {acc(W):.3f}")

X_big, y_big = make(5000)                             # plenty of labelled data
teacher = train(X_big, onehot(y_big))
X_small, y_small = make(60)                           # all the labels you can afford
X_pool = rng.normal(size=(3000, d))                   # unlabelled inputs (prompts)

show("teacher, 5,000 human labels", teacher)
show("student, 60 human labels", train(X_small, onehot(y_small)))
hard = onehot(np.argmax(X_pool @ teacher, 1))         # teacher's single best answer
show("student, 3,000 teacher hard labels", train(X_pool, hard))
soft = softmax(X_pool @ teacher)                      # teacher's full probabilities
show("student, 3,000 teacher soft labels", train(X_pool, soft))

teacher, 5,000 human labels         accuracy 0.964
student, 60 human labels            accuracy 0.572
student, 3,000 teacher hard labels  accuracy 0.948
student, 3,000 teacher soft labels  accuracy 0.963

The student jumps from 0.572 to 0.948 accuracy with the teacher's hard labels, and to 0.963 with its soft probabilities, matching the teacher. In this toy the student has the same shape as the teacher; in practice it is much smaller, and it keeps much, not all, of the teacher's skill on the task it was distilled for.

What students lose

Distilled students are strong on the distribution they were trained on and weaker outside it. They can copy the teacher's style without its knowledge, sounding confident while being wrong more often. Evaluate a student on held-out, real inputs ( Building an eval set ), not only on more teacher output.

Rules and licences

Some model providers' terms restrict using their outputs to train competing models. Check the terms of any model you use as a teacher, and record which model and prompts produced each dataset.

Contamination

Synthetic data can accidentally include, or closely paraphrase, items from public benchmarks the generator saw during pretraining. Deduplicate against your evaluation sets, or your scores will look better than the model is.

Trade-offs

Common mistakes

• Training on everything the teacher produced, without checking correctness or removing duplicates.
• Generating thousands of examples from one prompt template and getting thousands of near-identical examples.
• Evaluating the student only on teacher-generated data, which hides how it does on real inputs.
• Repeatedly training on your own model's outputs with no fresh real data, and slowly losing rare behaviours.
• Ignoring the terms of service of the model used as the teacher.

In the wild

Check yourself

Recap