Chain of thought and step-by-step reasoning

The big idea

Ask someone "a shop sells pens in packs of 12 for 3 dollars; how much for 50 pens?" and demand an instant number, and you may hear "12.50". Let them work it out loud, "50 pens needs 5 packs, because 4 packs is only 48", and they get 15. The working is not decoration. It is where the answer comes from.

Language models behave the same way. A model writes one token at a time, and each token gets the same fixed amount of computation. If the answer needs several dependent steps, asking for the answer immediately forces the model to jump. Asking it to write the steps first, chain of thoughtPrompting a model to write out intermediate reasoning steps before its final answer, which often improves accuracy on multi-step problems., puts each intermediate result into the context, where the next tokens can build on it.

It is not free. Reasoning tokens cost money and time on every call, and on simple tasks they buy nothing. The skill is knowing when to ask for it, how to get a clean answer out of it, and when to skip it.

How it works

Eliciting reasoning

There are three common ways to get it:

• Zero-shot: add an instruction such as "Think step by step before answering." This one sentence gave large gains on early instruction-following models.
• Few-shot: show examples whose answers include the worked reasoning, so the model copies the pattern ( Few-shot prompting and examples ). This is how chain of thought was first demonstrated.
• Structured: tell the model what steps to take ("first list the constraints, then check each option, then decide"). This gives you control over the kind of reasoning, which helps when you know how the problem should be solved.

Models trained for reasoning ( Reasoning models and test-time compute ) do this by themselves, often in a separate thinking section. Prompting for it still helps with instruction-tuned models, and structured guidance can still help reasoning models.

Separate the reasoning from the answer

If a program uses the result, ask for the reasoning first and the answer in a fixed place last: a final line Answer: 15, or reasoning inside <thinking> tags and the answer inside <answer> tags. Then parse only the answer part. Two rules matter:

• Reasoning before the answer. If the answer comes first, it was produced before any reasoning existed, and the "reasoning" is just a justification written afterwards.
• Never parse the whole text. The working often contains other numbers, such as the wrong 12.5 the model rejected along the way.

Self-consistency

A model sampling at a non-zero temperature ( Decoding and sampling ) can take different reasoning paths on different runs. Self-consistencySampling several chain-of-thought answers to the same question and returning the most common final answer. runs the same prompt several times and takes a majority vote on the final answers. Wrong paths tend to disagree with each other, while right paths tend to land on the same answer. It multiplies cost by the number of samples and, as Reasoning models and test-time compute shows, it only helps when the right answer is more likely than any single wrong one.

When reasoning is wasted

Step-by-step reasoning helps on problems with several dependent steps: arithmetic word problems, logic, multi-constraint planning, careful comparisons, code that needs tracing. It does little for lookups, simple classification, extraction, rewriting or chit-chat, where the answer does not depend on intermediate results. There it adds latency and cost, and sometimes hurts, as the model talks itself out of an obvious answer. Measure it on your own eval set.

Try it

Accuracy here is whatever your own eval set measured; the lab does the arithmetic. Prices and speeds are adjustable assumptions.

Try this:

1. At the defaults, reasoning lifts accuracy from 78 to 86 percent: 8,000 more right answers per day at 100,000 calls, for about 400 extra dollars a day, or 0.05 dollars per extra correct answer. Each answer takes 7.3 seconds instead of 0.7.
2. Set both accuracies to 95 percent, as on a simple classification task. Step by step now only adds latency and cost. The lab tells you to answer directly.
3. Raise extra reasoning tokens to 3,000. The cost per extra correct answer rises more than sevenfold. Is an extra right answer worth that much in your product? That is the real question, and only you can answer it.

Going deeper

Parsing and voting in code

The five completions below are hand-written to look like sampled outputs. The parser reads only the final Answer: line, and the vote fixes the first sample's mistake.

# Ask for reasoning, then a final line you can parse; vote across several samples.
# The five completions below are written by hand to look like sampled model outputs.
import re
from collections import Counter

PROMPT = """A shop sells pens in packs of 12 for 3 dollars. Ana needs 50 pens.
How much does she pay? Think step by step, then end with a line 'Answer: <number>'."""

samples = [
    "Each pen costs 3 / 12 = 0.25. 50 x 0.25 = 12.5.\nAnswer: 12.5",           # ignores packs
    "4 packs give 48 pens, not enough, so 5 packs. Cost 5 * 3 = 15 dollars.\nAnswer: 15",
    "50 / 12 = 4.17, so she needs 5 packs. 5 x 3 = 15.\nAnswer: 15",
    "She needs ceil(50 / 12) = 5 packs, 15 dollars in total.\nAnswer: $15",
    "50 pens is about 4 packs. 4 x 3 = 12.\nAnswer: 12",                       # rounds down
]

def final_answer(text):
    """Take only the last 'Answer:' line; ignore the reasoning above it."""
    m = re.findall(r"^Answer:\s*\$?([-\d.]+)\s*$", text, flags=re.M)
    return float(m[-1]) if m else None

answers = [final_answer(s) for s in samples]
print("parsed:", answers)
print("first sample alone:", answers[0])
votes = Counter(a for a in answers if a is not None)
best, n = votes.most_common(1)[0]
print(f"majority vote: {best} ({n} of {len(answers)} samples agree)")

parsed: [12.5, 15.0, 15.0, 15.0, 12.0]
first sample alone: 12.5
majority vote: 15.0 (3 of 5 samples agree)

Taking the first sample alone would have returned 12.5. Three of five paths agree on 15, so the vote returns it. Note that the parser also handles $15: real outputs drift from the requested format, and the final-line rule makes that drift easy to handle.

Is the reasoning faithful?

The written reasoning is generated text, not a trace of the model's internal computation. It usually tracks what the model does, but models can reach an answer for one reason and write a different, plausible-looking explanation. Use the reasoning for debugging and for improving answers; do not treat it as a guaranteed explanation, especially for audits.

Hiding reasoning from users

In a product, you usually show only the final answer. Generate the reasoning in a separate section and strip it before display. Keep it in logs, since it is useful for error analysis ( Error analysis ), but treat it as possibly containing data the user should not see, such as parts of the system prompt.

Trade-offs

Common mistakes

• Asking for the answer first and the reasoning after, which makes the reasoning a justification, not a cause.
• Parsing the whole output and picking up a number from the working instead of the final answer.
• Turning on step-by-step reasoning everywhere, including tasks where it only adds latency.
• Showing raw reasoning to end users, including half-formed ideas and internal instructions.
• Treating the written reasoning as a faithful explanation of how the model decided.

In the wild

Check yourself

Recap