The LLM training pipeline

The big idea

Ask a freshly pretrained model "What is the capital of France?" and it may answer "Paris". It may equally reply with "What is the capital of Germany? What is the capital of Spain?", because on the web, a question is often followed by more questions, as in a quiz. A pretrained model is a superb text continuer, not an assistant. It has knowledge and skills, but no idea that it is supposed to help you.

Assistants are built in stages. PretrainingThe first and by far the largest training stage: next-token prediction on a huge text corpus, producing a base model with broad knowledge but no instruction-following behaviour. creates a capable but undirected base modelA model straight out of pretraining, before any fine-tuning. It continues text rather than following instructions.. Supervised fine-tuningTraining a pretrained model on curated prompt and response examples so it learns to follow instructions in a consistent format. Often called SFT or instruction tuning. teaches it the shape of a helpful answer. Preference tuningTraining that pushes a model toward responses people (or an AI judge) prefer, using comparisons between responses. RLHF and DPO are the main methods. makes answers more helpful and better aligned with what people want. Safety training reduces harmful outputs. Each later stage uses a tiny fraction of the compute of pretraining, but changes the model's behaviour dramatically.

Think of training a doctor. Years of reading textbooks and journals give broad knowledge (pretraining). Then a residency where they watch and copy how experienced doctors handle real patients (supervised fine-tuning). Then feedback on their own work, "this explanation was clearer than that one" (preference tuning). And throughout, firm rules about what they must never do (safety training). The knowledge comes mostly from the reading; the bedside manner comes from the later stages.

How it works

Stage 1: pretraining

Next-token prediction on trillions of tokens of web pages, books, code and other text ( Pretraining: next-token prediction ). This is where nearly all the compute goes, and where the model picks up grammar, facts, reasoning patterns and coding skill. The output, the base model, can do a lot if you set up the text cleverly (a few examples in a prompt, Few-shot prompting and examples ), but it does not reliably follow instructions, it has no consistent persona, and it will happily continue harmful text.

Stage 2: supervised fine-tuning (SFT)

Continue training on a much smaller set of example conversations: a prompt, and the response a good assistant would give. The training objective is still next-token prediction, but only on curated responses, so the model learns how to respond: answer the question, follow the requested format, ask for clarification when needed, stop when done. The dataset is tiny compared with pretraining, often thousands to a few hundred thousand examples, and quality matters far more than size ( Supervised fine-tuning and instruction tuning ).

Stage 3: preference tuning

Writing the perfect answer is hard; comparing two answers is easy. So the next stage collects comparisons: for a prompt, which of two responses is better? Two families of methods learn from those comparisons:

• RLHF: train a reward model to predict which response people prefer, then use reinforcement learning to make the model produce high-reward responses ( RLHF ).
• DPO and relatives: learn directly from the preference pairs with a simple loss, skipping the reward model ( DPO and other preference methods ).

Preference tuning improves things that are hard to demonstrate but easy to judge: helpfulness, tone, honesty about uncertainty, how much detail to give. The judgments can come from people or from an AI guided by written principles ( Constitutional AI and RLAIF ).

Stage 4: safety training

The model also has to decline harmful requests without refusing legitimate ones. Safety training adds examples and preferences targeted at risky areas, red-teaming results and policy rules. In practice it is woven into SFT and preference tuning rather than being a separate step at the end, and it continues after release as new failures are found ( Guardrails ).

What each stage changes

The later stages mostly change behaviour, not knowledge. A useful mental model: pretraining decides what the model can do, post-training decides what it does by default. That is why fine-tuning is a poor way to teach a model new facts, and a good way to teach it a format, style or habit ( Prompting vs RAG vs fine-tuning: choosing ).

Many modern pipelines add more stages: training on long documents to extend the context window, reinforcement learning on problems with checkable answers to build reasoning ( Reasoning models and test-time compute ), tool-use training, and distillation from bigger models ( Distillation and synthetic data ).

Try it

Going deeper

Where the compute goes

Pretraining dominates the bill. Post-training datasets are measured in thousands to millions of examples instead of trillions of tokens, so each later stage costs a small fraction of pretraining. That asymmetry is why companies and researchers can fine-tune open-weights base models on modest hardware, especially with parameter-efficient methods ( Parameter-efficient fine-tuning: LoRA and QLoRA ).

The alignment tax

Post-training can make a model slightly worse at some raw capabilities, for example certain benchmarks, while making it much more useful. This cost is called the alignment taxAny loss of capability on some tasks caused by alignment training such as RLHF. Methods try to keep it small.. Methods try to keep it small, for example by mixing some pretraining data back in during preference tuning.

Base vs instruct models

Open-weights releases often ship both. Use the instruct (or chat) model for almost everything. Use the base model when you want to fine-tune it yourself from a neutral starting point, or for pure text continuation where an assistant persona gets in the way.

Trade-offs

Common mistakes

• Expecting fine-tuning to add large amounts of new knowledge. It mostly changes behaviour; use retrieval for knowledge ( Why RAG? ).
• Prompting a base model like a chat model and concluding it is broken.
• Thinking safety is a filter bolted on at the end. It is trained into the model's behaviour across stages.
• Assuming every stage is always present. Some open models skip preference tuning; some pipelines add reasoning or tool-use stages.

In the wild

Ouyang et al., 2022 (paper)

Training language models to follow instructions with human feedback (InstructGPT) (opens in a new tab)

The paper starts from the observation that making language models bigger does not by itself make them better at following a user's intent. It fine-tunes GPT-3 first with supervised learning on human demonstrations, then with reinforcement learning from human rankings of model outputs. Outputs from the resulting 1.3 billion parameter model were preferred by people over outputs from the 175 billion parameter GPT-3, despite 100 times fewer parameters, with improved truthfulness and less toxic output. It is the template for the pipeline in this lesson.

Check yourself

Recap