Reasoning models and test-time compute

The big idea

A plain chat model answers the way you blurt out a reply in conversation: one token after another, with no chance to work things out first. That is fine for "what is the capital of France?" and poor for a tricky probability puzzle or a bug hidden three functions deep.

A reasoning modelA language model trained to produce a long chain of intermediate reasoning (often hidden from the user) before its final answer, usually with reinforcement learning on problems whose answers can be checked. is trained to write a long private scratchpad before it answers: try an approach, check it, notice a mistake, back up, try again. Every extra token of that scratchpad is extra computation spent on this one question. Spending computation when the question is asked, rather than only during training, is called test-time computeComputation spent while answering a request, for example by generating a longer chain of reasoning or sampling several answers and picking one. Also called inference-time compute., and on hard problems more of it buys more accuracy.

Think of two exam rules. In one, you must say your answer the moment you read the question. In the other, you get scratch paper and ten minutes. The second student is the same person, with the same knowledge, but will get far more of the hard questions right, and will be slower on every question, easy ones included.

How it works

Thinking tokens are computation

A transformer does a fixed amount of work per token: one pass through all its layers. A problem that needs ten dependent steps of logic cannot be squeezed into one token's worth of work. Writing the steps out lets each new token build on the ones before it, so a longer output is literally more serial computation. That is why asking a model to "think step by step" helps ( Chain of thought and step-by-step reasoning ), and why a model trained to do it well, at length, helps much more.

Reinforcement learning on verifiable rewards

How do you teach a model to reason well when nobody can write down the ideal chain of thought for every problem? You grade only the outcome. Take problems whose answers can be checked automatically: math with a known final number, code with unit tests, puzzles with a unique solution. For each problem the model generates several attempts. A checker gives each a reward, 1 if the answer is right and 0 if not. A policy-gradient update then makes the token choices in successful attempts more likely and those in failed attempts less likely. This is RLVRReinforcement learning on verifiable rewards: training a model with rewards computed by a program that checks the final answer, such as exact match on a math result or passing unit tests, instead of a learned reward model..

Compared with RLHF ( RLHF ), the reward comes from a program, not from a learned model of human preference. A checker cannot be flattered, so there is less room for reward hacking, though not none: a model can learn to game weak tests, or to output a format the checker misreads.

A popular recipe scores each attempt against the other attempts at the same problem: its advantageIn reinforcement learning, how much better an action turned out than expected. Group-relative methods use the reward minus the average reward of other attempts at the same problem. is its reward minus the group's average. If 3 of 8 attempts are right, the right ones get a positive push and the wrong ones a negative one; if all 8 are right or all are wrong, nothing is learned from that problem. No separate value network is needed.

What emerges

Nobody tells the model how to reason. Behaviours like re-checking an answer, noticing an error ("wait, that is wrong") and trying another approach appear because they lead to more rewarded answers. Training runs also typically show responses getting longer as the model learns that thinking more pays off on hard problems.

Other ways to spend test-time compute

A longer chain of thought is one way. Others work with any model:

• Majority vote (self-consistencySampling several reasoning paths for the same question and returning the most common final answer. Also called majority voting.): sample N answers and return the most common one.
• Best-of-N with a checker: sample N answers and return one that passes the tests or a verifier model's check.
• Search over steps: expand partial solutions and let a verifier that scores each step decide which ones to continue.

All of them multiply token cost. A parallel method costs N times the tokens but little extra wall-clock time; a longer chain costs both.

Try it

Try this:

1. At the defaults (one attempt right 40 percent of the time, wrong answers spread over three options), majority vote climbs from 40 percent to about 60 percent at N = 9 and 84 percent at N = 31. A perfect checker does far better: 99 percent at N = 9, because it only needs one right answer among the samples.
2. Switch to "one common wrong answer" and set accuracy to 30 percent. Now the most common wrong answer (49 percent of attempts) beats the right one, and voting makes things worse: about 11 percent at N = 31. Voting amplifies whatever answer is most likely, right or wrong.
3. Look at the cost line. Nine samples of 2,000 tokens cost 9 times the tokens. Run in parallel they take about as long as one; generated one after another, like a single long chain, they take 6 minutes at the assumed speed.

Going deeper

A toy run of RLVR

The policy below chooses how many thinking tokens to spend on easy and hard problems. The only feedback is a checker's right or wrong, minus a tiny cost per token, with group-relative advantages over 8 attempts. The success rates are made up; the learning rule is the real idea.

# Toy RL on verifiable rewards: a policy learns how long to "think" per problem type.
# The only signal is a checker that says right or wrong. No human labels, no reward model.
import numpy as np

rng = np.random.default_rng(0)
LENGTHS = np.array([0, 200, 1000, 4000])           # thinking tokens the policy can spend
P_RIGHT = {"easy": np.array([0.60, 0.95, 0.97, 0.97]),   # made-up: chance the final answer
           "hard": np.array([0.02, 0.15, 0.55, 0.85])}   # passes the checker
COST = 0.00005                                      # small penalty per thinking token
logits = {k: np.zeros(4) for k in P_RIGHT}          # start: every length equally likely

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

def report(tag):
    for kind, z in logits.items():
        pi = softmax(z)
        acc, toks = pi @ P_RIGHT[kind], pi @ LENGTHS
        print(f"{tag:6} {kind}: P(length)={np.round(pi, 2)}  accuracy {acc:.2f}  tokens {toks:5.0f}")

report("before")
for step in range(400):
    kind = ["easy", "hard"][step % 2]
    pi = softmax(logits[kind])
    picks = rng.choice(4, size=8, p=pi)             # a group of 8 attempts at one problem
    right = rng.random(8) < P_RIGHT[kind][picks]    # the checker grades each final answer
    reward = right - COST * LENGTHS[picks]
    adv = (reward - reward.mean()) / (reward.std() + 1e-6)   # better or worse than the group
    for a, g in zip(picks, adv):                    # policy gradient: push up good choices
        grad = -pi.copy()
        grad[a] += 1
        logits[kind] += 0.05 * g * grad
report("after")

before easy: P(length)=[0.25 0.25 0.25 0.25]  accuracy 0.87  tokens  1300
before hard: P(length)=[0.25 0.25 0.25 0.25]  accuracy 0.39  tokens  1300
after  easy: P(length)=[0.   0.99 0.   0.  ]  accuracy 0.95  tokens   209
after  hard: P(length)=[0.   0.01 0.03 0.96]  accuracy 0.83  tokens  3863

Starting from a uniform choice, the policy learns to answer easy problems after about 200 tokens and to spend about 4,000 on hard ones, raising hard-problem accuracy from 0.39 to 0.83 without wasting tokens on easy ones. Nobody labelled the right length; it fell out of the reward.

Why only some tasks improve

RLVR needs answers a program can check, so the gains are largest in math, coding and formal logic. They often transfer partly to other domains, but for open-ended writing there is no exact checker, and teams fall back to learned reward models or AI judges ( LLM-as-judge ) with their usual weaknesses.

Compute-optimal thinking

More thinking is not always better. On easy questions it adds latency and cost for nothing, and very long chains can wander or second-guess a correct answer. Research on test-time scaling finds the best strategy depends on the difficulty of the prompt, which is why many products let you set a thinking budget or effort level, or let the model decide how long to think.

Distilling reasoning

Long reasoning traces from a big reasoning model can be used as supervised fine-tuning data for a smaller model, which then reasons far better than it would by RL on its own at that size. This is distillation ( Distillation and synthetic data ).

When to use a reasoning mode

Use one when correctness matters more than speed and the task has real multi-step structure: math, tricky code changes, planning, analysing a long contract for conflicts, agent steps that are expensive to get wrong. Skip it for lookups, short rewrites, classification, chat small talk, and anything with a tight latency budget ( Latency and cost budgets ). A common design routes easy requests to a fast model and hard ones to a reasoning model ( Routing, cascades and cost optimization ).

Trade-offs

Common mistakes

• Turning on maximum reasoning effort for every request, then wondering why latency and bills tripled.
• Assuming a reasoning model is better at everything. On simple tasks it is mostly slower.
• Trusting the visible reasoning text as an exact account of how the answer was produced. It is generated text, useful but not a guaranteed explanation.
• Majority voting on a task where the model's most common answer is wrong: more samples lock in the error.
• Using weak tests as the reward and getting a model that passes tests rather than solving problems.

In the wild

Check yourself

Recap