DPO and other preference methods

The big idea

RLHF works, but it is a lot of machinery: a reward model to train, a reinforcement learning loop that samples from the model during training, and four models in memory at once. Direct preference optimisationDirect preference optimisation: a method that trains a language model directly on chosen and rejected response pairs with a classification-style loss, skipping the separate reward model and RL loop. (DPO) gets to a similar place with a single supervised-style loss on the same preference pairs.

The key insight is mathematical: for the RLHF objective (maximise reward, minus a KL penalty to a reference model), the best possible policy can be written in closed form in terms of the reward. Flip that around and the reward can be written in terms of the policy. So instead of learning a reward model and then a policy, you can train the policy directly so that it implicitly is the reward model that explains the preferences.

Think of learning to cook from a food critic. The RLHF way: first build a model of the critic's taste from their reviews, then practise cooking to please that model. The DPO way: look at pairs of dishes the critic compared and, for each pair, nudge your own habits toward the winner and away from the loser, a little more strongly when you were getting it wrong. No separate critic model; your own cooking habits become the record of their taste.

How it works

The loss

For a prompt $x$ with a chosen response $y_w$ and a rejected one $y_l$, compute how much more likely each response has become under the policy ${\pi }_{\theta }$ than under the frozen reference ${\pi }_{\text{ref}}$ (usually the SFT model). Then ask the chosen one to have gained more than the rejected one:

This is the Bradley-Terry loss from RLHF's reward model ( RLHF ), with the reward replaced by $\beta \log \frac{{\pi }_{\theta }}{{\pi }_{\text{ref}}}$, the implicit rewardIn DPO, the quantity beta times the log ratio of the policy's and the reference model's probability of a response. DPO training makes it behave like a reward model.. A log probability of a whole response is just the sum of the log probabilities of its tokens, which a single forward pass gives you.

What the gradient does

Each pair pushes the chosen response's probability up and the rejected response's probability down. The push is weighted by how wrong the model currently is: if the implicit reward already ranks the pair correctly by a wide margin, the sigmoid is near 1 and the update is tiny; if it ranks them the wrong way, the update is large. Nothing is sampled from the model during training; you only score the fixed pairs.

The role of beta

$\beta$ plays the same role as the KL coefficient in RLHF: it controls how far the policy may move from the reference. A small $\beta$ means a given probability shift counts for little, so the model must move further to satisfy the preferences, drifting far from the reference. A large $\beta$ keeps it close. Values between about 0.1 and 0.5 are common starting points.

Try it

DPO on a toy policy that chooses among six canned responses, trained on 400 noisy preference pairs:

# Direct preference optimisation on a toy "policy" over 6 possible responses.
import numpy as np

responses = ["helpful", "helpful+long", "terse", "off-topic", "rude", "wrong"]
utility = np.array([2.0, 1.5, 0.5, -1.0, -2.0, -1.5])   # hidden human taste
ref_logits = np.array([0.5, 0.2, 0.8, 0.3, 0.0, 0.6])   # the SFT model
softmax = lambda z: np.exp(z - z.max()) / np.exp(z - z.max()).sum()

# Preference pairs (winner, loser), labelled by noisy people.
rng = np.random.default_rng(0)
pairs = []
for _ in range(400):
    a, b = rng.choice(6, 2, replace=False)
    p_a = 1 / (1 + np.exp(-(utility[a] - utility[b])))
    pairs.append((a, b) if rng.random() < p_a else (b, a))

W, L = np.array(pairs).T                        # winners, losers

def train_dpo(beta, steps=2000):
    theta, ref_logp = ref_logits.copy(), np.log(softmax(ref_logits))
    for _ in range(steps):
        ratio = np.log(softmax(theta)) - ref_logp    # log pi / pi_ref
        m = beta * (ratio[W] - ratio[L])             # implicit reward margin
        coef = 1 - 1 / (1 + np.exp(-m))              # from -log sigmoid(m)
        g = np.bincount(W, coef, 6) - np.bincount(L, coef, 6)
        theta += 0.5 * g / len(pairs)                # step size scaled by 1/beta
    return softmax(theta)

ref = softmax(ref_logits)
print("reference: ", " ".join(f"{r}={p:.2f}" for r, p in zip(responses, ref)))
for beta in (0.1, 0.5, 2.0):
    pol = train_dpo(beta)
    kl = np.sum(pol * np.log(pol / ref))
    print(f"beta={beta:<4} top: {responses[pol.argmax()]:<8} "
          f"p(helpful)={pol[0]:.2f}  p(rude)={pol[4]:.3f}  KL={kl:.2f}")

reference:  helpful=0.18 helpful+long=0.13 terse=0.24 off-topic=0.15 rude=0.11 wrong=0.20
beta=0.1  top: helpful  p(helpful)=0.99  p(rude)=0.000  KL=1.67
beta=0.5  top: helpful  p(helpful)=0.72  p(rude)=0.000  KL=1.03
beta=2.0  top: helpful  p(helpful)=0.37  p(rude)=0.035  KL=0.21

The reference model is fairly flat, with "terse" the most likely at 0.24. After DPO, "helpful" wins at every $\beta$, but how far the policy moves depends on $\beta$:

• $\beta =2.0$: a gentle shift, "helpful" at 0.37, KL from the reference 0.21.
• $\beta =0.5$: "helpful" at 0.72, KL 1.03.
• $\beta =0.1$: "helpful" at 0.99 and KL 1.67. The model has collapsed onto one answer, crushing "helpful+long", which people also liked.

That last line is a real DPO behaviour: with a small $\beta$, the loss keeps rewarding ever larger margins, and the policy can become overconfident and lose diversity.

Try this on your own copy: set the reference logits to all zeros (a uniform reference) and compare the KL values; then flip 30 percent of the pairs at random to simulate noisy labellers and see how $\beta$ affects robustness.

Going deeper

Other members of the family

DPO set off many variants, each tweaking one assumption:

• IPO changes the loss so that margins cannot grow without limit, reducing the overfitting seen at small $\beta$.
• KTO learns from unpaired thumbs-up and thumbs-down labels, which are easier to collect from product feedback than side-by-side comparisons.
• ORPO and SimPO remove the reference model, reducing memory, and ORPO folds preference learning into supervised fine-tuning.
• Online or iterative DPO generates fresh responses from the current policy, gets them judged, and repeats, recovering some of RL's ability to learn from the model's own outputs.

Why RL has not gone away

DPO learns only from the fixed pairs you give it. RL methods sample from the current model and learn from what it actually produces, so they can improve on situations the preference data never covered. For goals with an automatic score, such as passing unit tests or getting a maths answer right, RL on those rewards is how reasoning models are trained ( Reasoning models and test-time compute ). In practice many pipelines use both: DPO-style losses for cheap, stable preference tuning, and RL where a reliable reward exists.

Data still decides the outcome

DPO removes machinery, not the need for good preferences. Pairs where the chosen and rejected responses differ in many ways at once (length, tone and correctness) teach a muddle. Pairs that differ in the property you care about teach it cleanly. And the pairs should come from a model similar to the one being trained; preferences over another model's style of response transfer less well.

Trade-offs

Common mistakes

• Assuming DPO cannot drift. Small beta and many epochs can move the model far from its reference and collapse its diversity.
• Using preference pairs generated by a very different model and expecting clean improvements.
• Building pairs where chosen responses are simply longer, then wondering why the model becomes verbose.
• Forgetting to run SFT first. DPO starts from, and measures drift against, a reasonable instruction-following model.
• Reporting training loss as success. Evaluate on fresh prompts with human or judge-model comparisons ( LLM-as-judge ).

In the wild

Check yourself

Recap