RLHF

The big idea

Supervised fine-tuning can only teach what someone wrote down. But people are much better at judging answers than at writing perfect ones: given two replies, most of us can say which is clearer, more honest or more useful, even if we could not have written either. RLHFReinforcement learning from human feedback: train a reward model on human comparisons of responses, then optimise the language model with RL to score highly on it, with a penalty for drifting from the starting model. turns those judgments into a training signal.

It works in three steps. Collect human comparisons of model responses. Train a reward modelA model that takes a prompt and a response and outputs a single score predicting how much people would prefer that response. to predict which response people prefer. Then use reinforcement learning to make the language model produce responses the reward model scores highly, while a penalty stops it from drifting too far from where it started.

Think of a stand-up comedian testing material. They cannot ask the audience to write better jokes, but they can watch which jokes get laughs. Over many nights they learn what lands. The danger is also familiar: a comedian who chases laughs too hard ends up with cheap tricks that work on the room but are not actually good comedy. In RLHF that failure is called reward hackingWhen a model being optimised finds ways to score highly on a reward model that do not reflect real quality, such as excessive length or flattery..

How it works

Step 1: preference data

Take a set of prompts. For each, sample several responses from the SFT model and ask people to rank them, or to pick the better of two. Rankings are turned into pairs: every (preferred, rejected) combination. Comparisons are faster and more consistent than writing answers, and they can capture qualities like tone and level of detail that are hard to specify in instructions.

Step 2: the reward model

The reward model is usually a copy of the language model with its output layer replaced by a single number: the score for a (prompt, response) pair. It is trained so that preferred responses score higher than rejected ones, with the Bradley-TerryA model of pairwise comparisons in which the probability that A beats B is the sigmoid of the difference in their scores. The standard loss for reward models. loss:

where $y_w$ is the preferred response, $y_l$ the rejected one, and $\sigma$ the sigmoid. If the reward model gives the winner a score 2 higher, $\sigma (2)=0.88$ and the loss is 0.13; if it has them the wrong way round by 2, the loss is 2.13. Only score differences matter, so the absolute values are arbitrary.

Step 3: optimise with PPO and a KL penalty

Now treat the language model as a policyIn reinforcement learning, the rule that chooses actions. For a language model in RLHF, the model itself, choosing one token at a time.: given a prompt, it generates a response, and the reward model scores it. PPO ( Policy gradients and actor-critic, PPO ) adjusts the weights so high-scoring responses become more likely.

Left alone, the policy would drift wherever the reward model's quirks lead. So the reward is reduced by a penalty for moving away from the original SFT model, measured by the KL divergenceKullback-Leibler divergence: a measure of how different one probability distribution is from another. In RLHF it measures how far the policy has drifted from the reference model. between the two models' token probabilities:

$\beta$ controls the leash. Too small and the model wanders into reward hacking; too large and it barely improves on SFT. Some setups also mix in a little pretraining data during this stage to reduce the alignment tax ( The LLM training pipeline ).

Why reward hacking happens

The reward model is a learned approximation of human judgment, trained on a limited set of comparisons. Push hard enough on any approximation and you find its blind spots: if raters slightly preferred longer answers, the policy learns that longer is always better; if they liked confident tone, it learns to sound confident even when wrong. The reward model's score keeps going up while the real quality goes down. This is Goodhart's lawThe observation that when a measure becomes a target, it stops being a good measure. In RLHF, optimising a reward model too hard breaks its correlation with real quality. in action.

Try it

Try this:

1. At the default penalty 0.2, training settles near the green line: true reward 1.34, close to the best possible 1.36 on this curve.
2. Lower the penalty to 0.10. The reward model's score climbs to 4.05, but true reward falls to 1.12: the policy is exploiting the reward model.
3. Lower it to 0.05. The reward model reports 7.20, its highest yet, while true reward is negative: the model is now worse than the one you started with. Raise the penalty to 0.5 and the opposite happens: safe, but you leave improvement on the table.

The curves are illustrative: the true-reward shape uses the functional form fitted in the paper below, with made-up coefficients.

The same story with a real (tiny) reward model, trained on simulated preferences and then over-optimised with best-of-n sampling:

# Train a reward model from pairwise preferences, then over-optimise it.
import numpy as np

rng = np.random.default_rng(0)
def true_reward(q, L):   # what people want: quality, moderate length
    return q + 0.6 * L - 0.3 * L ** 2

# 1. Preference data: pairs of responses with features (quality q, length L).
n = 2000
qa, qb = rng.normal(size=n), rng.normal(size=n)
La, Lb = rng.uniform(-1, 1, n), rng.uniform(-1, 1, n)  # modest lengths
diff = true_reward(qa, La) - true_reward(qb, Lb)
a_wins = rng.random(n) < 1 / (1 + np.exp(-diff))  # noisy human choice

# 2. Reward model r(x) = w . [q, L], trained with the Bradley-Terry loss
#    -log sigmoid(r(chosen) - r(rejected)).
X = np.where(a_wins[:, None], 1, -1) * np.c_[qa - qb, La - Lb]
w = np.zeros(2)
for _ in range(500):
    p = 1 / (1 + np.exp(-X @ w))
    w += 0.5 * X.T @ (1 - p) / n           # gradient ascent on log-likelihood
print(f"reward model weights: quality {w[0]:.2f}, length {w[1]:.2f}")
print(f"agrees with the training preferences: {np.mean(X @ w > 0):.1%}")

# 3. "Policy optimisation" by best-of-n: sample n responses, keep the one the
#    reward model likes best. The policy can produce much longer answers.
def best_of(nc, trials=2000):
    q, L = rng.normal(size=(trials, nc)), rng.normal(0, 2, size=(trials, nc))
    pick = np.argmax(w[0] * q + w[1] * L, axis=1)
    i = np.arange(trials)
    proxy = (w[0] * q + w[1] * L)[i, pick]
    return proxy.mean(), true_reward(q, L)[i, pick].mean()

for nc in (1, 4, 16, 64, 256):
    proxy, gold = best_of(nc)
    print(f"best of {nc:3d}: reward model {proxy:5.2f}  true reward {gold:5.2f}")

reward model weights: quality 1.04, length 0.67
agrees with the training preferences: 73.7%
best of   1: reward model -0.02  true reward -1.21
best of   4: reward model  1.76  true reward -0.01
best of  16: reward model  2.99  true reward -0.26
best of  64: reward model  4.00  true reward -1.06
best of 256: reward model  4.80  true reward -2.24

The simulated people value quality and moderate length; in the labelled data lengths are modest, so the learned reward model simply concludes "longer is better" (weight 0.67). Picking the best of 4 responses by reward model raises true reward from -1.21 to -0.01. Picking the best of 256 pushes the reward model's score to 4.80 while true reward falls to -2.24: it now selects extremely long answers the reward model never saw in training.

Going deeper

Why RL instead of just training on the best samples?

You could sample many responses, keep the highest-scoring and fine-tune on them (best-of-n or rejection sampling fine-tuning). That works and is often used. RL adds the ability to learn from every sample, good or bad, and to assign credit within a response. The cost is complexity: PPO for language models juggles four models at once (policy, reference, reward model and a value model), which is memory-hungry and finicky to tune. That complexity motivated simpler methods such as DPO ( DPO and other preference methods ).

Reward model quality is the ceiling

The policy can only become as good as the reward model can judge. Teams invest heavily in rater guidelines, rater agreement checks, and diverse prompts. Reward models also need to be retrained as the policy improves, because a better policy produces responses unlike anything in the original comparison data.

What RLHF tends to change

Typical effects are better instruction following, better calibrated refusals, more helpful formatting, and fewer blatant errors. Typical side effects are verbosity, sycophancy (agreeing with the user because raters liked it), and a uniform "assistant voice". These are reward-model quirks being optimised, and are monitored with evaluations ( Why evals are the core skill ).

Trade-offs

Common mistakes

• Trusting the reward model's score as the measure of success. Always evaluate with fresh human or held-out judgments.
• Setting the KL penalty too low to get a bigger reward number, then shipping a verbose, sycophantic model.
• Collecting preferences on narrow prompts, then optimising on broad ones where the reward model is guessing.
• Thinking RLHF adds knowledge. It reshapes behaviour the base model already supports.
• Ignoring rater disagreement. If raters disagree often, the reward model is learning noise.

In the wild

Check yourself

Recap