Supervised fine-tuning and instruction tuning

The big idea

A base model has read enough text to know how to answer almost anything; it just does not know that answering is its job. Supervised fine-tuningTraining a pretrained model on curated prompt and response examples so it learns to follow instructions in a consistent format. Often called SFT or instruction tuning. (SFT), also called instruction tuningSupervised fine-tuning on a collection of instructions paired with good responses, so a model learns to follow instructions in general rather than one task., shows it many examples of the job: here is a request, here is what a good assistant says. Training is the same next-token prediction as pretraining, just on these examples.

Three details make it work. Conversations are written in a fixed chat templateThe exact text format, with special tokens marking roles and turn boundaries, that a chat model was trained on. Prompts must use the same template at inference., so the model always knows who is speaking. The loss is computed only on the assistant's replies, so the model learns to answer, not to imitate users. And the examples are carefully chosen, because the model copies whatever habits they contain.

Think of an apprentice chef who has read every cookbook. On day one in a real kitchen, they still need to learn the rhythm of service: take the order slip, cook exactly that dish, plate it the house way. A few weeks of shadowing a great chef teaches this; shadowing a sloppy one teaches sloppiness just as efficiently.

How it works

Instruction datasets

An instruction datasetA collection of prompts paired with high-quality responses, covering many kinds of tasks, used for supervised fine-tuning. is a list of conversations: a prompt (sometimes with a system message and earlier turns) and the desired response. Good datasets cover many task types, such as answering questions, summarising, rewriting, coding, extracting fields, brainstorming, multi-turn follow-ups and polite refusals, written in the style you want.

Sources include human-written demonstrations, conversations rewritten by experts, existing datasets converted into instruction form, and responses generated by a stronger model and then filtered ( Distillation and synthetic data ).

Chat templates

The model sees one flat token sequence, so roles and turn boundaries are marked with special tokens. A made-up template might render a conversation like this:

<|system|> You are a concise assistant. <|end|>
<|user|> What is the capital of France? <|end|>
<|assistant|> Paris. <|end|>

Every model family defines its own template, and the model only behaves well with the one it was trained on. Libraries store the template with the tokenizer, so in practice you pass a list of messages and let the library render them. Getting this wrong, for example a missing end-of-turn token, is one of the most common reasons a fine-tuned model rambles or never stops.

Loss masking

The training objective is next-token prediction, but if you train on every token, the model also learns to predict the user's messages and the system prompt. That wastes capacity and can make the model start writing the user's side of the conversation. So the loss is maskedExcluding some tokens, usually the prompt, from the training loss so the model learns only to produce the response.: computed only on the assistant tokens. The model still reads the prompt as context; it just is not graded on it. In common training code, masked positions get the label $-100$, which the cross-entropy loss ignores.

Quality over quantity

SFT teaches style and behaviour more than knowledge ( The LLM training pipeline ), and it teaches them by imitation. The model copies everything in the examples, including mistakes, hedging, verbosity and invented facts. A thousand clean, diverse, carefully written examples can beat a hundred thousand noisy ones, and a dataset full of confident made-up answers teaches the model to make things up confidently.

Practical rules: deduplicate, check facts in responses, keep a consistent style, cover the edge cases you care about (including when to say "I don't know"), and hold out an evaluation set before training ( Building an eval set ).

Try it

Render a conversation, mark which tokens are trained on, and compare the two losses:

# Format a conversation with a chat template and mask the loss to the reply.
import numpy as np

conversation = [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "What is the capital of France?"},
    {"role": "assistant", "content": "Paris."},
]

def render(msgs):
    # A made-up template: every model family defines its own special tokens.
    tokens, train = [], []
    for m in msgs:
        head = [f"<|{m['role']}|>"]
        body = m["content"].split() + ["<|end|>"]
        tokens += head + body
        learn = m["role"] == "assistant"          # loss only on the reply
        train += [False] * len(head) + [learn] * len(body)
    return tokens, np.array(train)

tokens, mask = render(conversation)
for t, m in zip(tokens, mask):
    print(f"{t:<16} {'LOSS' if m else '-'}")
print(f"{mask.sum()} of {len(tokens)} tokens are trained on")

# Toy per-token losses from an imaginary model (negative log probabilities).
rng = np.random.default_rng(1)
nll = rng.uniform(0.5, 3.0, size=len(tokens))
print(f"loss over all tokens:    {nll.mean():.3f}")
print(f"loss over reply only:    {nll[mask].mean():.3f}")
# In Hugging Face style code, masked positions get the label -100,
# which the cross-entropy loss ignores.
vocab = {t: i for i, t in enumerate(sorted(set(tokens)))}
ids = np.array([vocab[t] for t in tokens])
labels = np.where(mask, ids, -100)
print("input ids: ", ids[-5:].tolist())
print("labels:    ", labels[-5:].tolist(), "(last 5 positions)")

<|system|>       -
You              -
are              -
a                -
concise          -
assistant.       -
<|end|>          -
<|user|>         -
What             -
is               -
the              -
capital          -
of               -
France?          -
<|end|>          -
<|assistant|>    -
Paris.           LOSS
<|end|>          LOSS
2 of 18 tokens are trained on
loss over all tokens:    1.723
loss over reply only:    1.171
input ids:  [4, 1, 0, 5, 1]
labels:     [-100, -100, -100, 5, 1] (last 5 positions)

Only 2 of the 18 tokens, "Paris." and the end marker, are trained on. With the same imaginary per-token losses, averaging over everything gives 1.723 and averaging over the reply gives 1.171: masking changes what the optimiser is pushed to improve. The last lines show the usual encoding, where every masked label is $-100$.

Try this on your own copy:

1. Add a second user and assistant turn. Should the first assistant reply be trained on too? (Usually yes: every assistant turn is a target.)
2. Remove "<|end|>" from the assistant body. What would a model trained this way do at the end of its answer?
3. Change learn to train on every token and see how the trained-on count jumps. In a real dataset with long prompts and short answers, most of the gradient would go into predicting prompts.

Going deeper

How much data, how many epochs

SFT datasets typically run from a few thousand to a few hundred thousand conversations, trained for one to a few epochs at a learning rate much lower than pretraining. Too many epochs on a small set leads to memorising responses word for word and losing generality. Watch loss on a held-out set and look at samples.

Catastrophic forgetting

Fine-tuning on a narrow task can erode abilities the model had, a problem called catastrophic forgettingWhen training a model on new data makes it lose abilities or knowledge it had before.. Mixing in some general instruction data, using lower learning rates, or training small adapters instead of all weights ( Parameter-efficient fine-tuning: LoRA and QLoRA ) all help.

Packing

Short conversations waste compute when padded to a fixed length. PackingConcatenating several short training examples into one full-length sequence, with attention and loss masks so they do not see each other. concatenates several examples into one sequence, with attention masks so examples do not see each other and loss masks so only replies count.

Trade-offs

Common mistakes

• Using a different chat template at inference than in training, or forgetting the end-of-turn token.
• Training on prompts as well as replies by accident, then wondering why the model writes the user's next message.
• Collecting lots of mediocre examples instead of fewer excellent ones.
• Expecting SFT to teach a large body of new facts. Use retrieval for knowledge.
• Training several epochs on a small dataset without checking a held-out set.

In the wild

Check yourself

Recap