Training, Fine-tuning and Alignment

A pretrained language model can continue any text, but it does not yet answer questions, follow instructions or decline harmful requests. Those behaviours are trained in after pretraining, in a series of much smaller stages. This module walks through them in the order a model goes through them: supervised fine-tuning on example conversations, then preference tuning with human or AI feedback, then safety training.

You will see how RLHF uses a reward model and reinforcement learning, why DPO gets similar results with a single classification-style loss, and how written principles and AI feedback replace much of the human labelling. The practical lessons cover parameter-efficient fine-tuning with LoRA and QLoRA and a decision guide for choosing between prompting, retrieval and fine-tuning. The module closes with reasoning models that spend more compute at answer time, and with distillation and synthetic data, which make small models punch above their weight.