Decoding and sampling

The big idea

A language model does not output text. At each step it outputs a probability for every token in its vocabulary: after "The cat sat on the", maybe 39 percent "mat", 21 percent "floor", 16 percent "sofa", and tiny amounts for "moon" or "theorem". Something has to turn that list into one actual token, append it, and ask again. That something is the decodingThe procedure that turns a model's next-token probabilities into an actual output token, step by step: greedy, beam search, or sampling with settings such as temperature, top-k and top-p. strategy, and it changes the output as much as the model does.

Always taking the single most likely token sounds safest, but it produces bland, repetitive text. Picking randomly in proportion to the probabilities gives lively, varied text, but now and then it picks "theorem". Practical decoding sits in between: sample, but reshape and trim the distribution first.

Think of ordering at a restaurant you know well. Always ordering your top favourite is predictable and gets boring. Picking a dish completely at random will eventually land you something you hate. Most people pick among their few favourites, sometimes adventurously, sometimes cautiously depending on the mood. Temperature is the mood; top-k and top-p are the shortlist.

How it works

Greedy decoding

Greedy decodingDecoding that always picks the single most likely next token. Deterministic but prone to bland, repetitive text. takes the most likely token every time. It is deterministic and fast, and fine for short, factual outputs like a classification label or a number. For longer text it tends to fall into loops ("the best way to learn is to learn is to learn..."), because once a phrase is likely, repeating it stays likely.

Beam search

Beam searchDecoding that keeps the B most probable partial sequences at each step and finally returns the most probable complete one. keeps the $B$ most probable partial sequences instead of one, extends each by every token, keeps the best $B$ again, and finally returns the most probable complete sequence. It finds higher-probability outputs than greedy and was standard for translation and speech recognition, where there is one right answer. For open-ended writing it makes the blandness worse: the most probable text is not the most human-like text.

Temperature

The model's raw scores are logitsThe raw, unnormalised scores a model outputs for each token before softmax turns them into probabilities.. Before softmax, divide them by a temperatureA decoding setting that divides logits before softmax: below 1 sharpens the distribution toward the top tokens, above 1 flattens it, and 0 means greedy. $T$:

$T=1$ leaves the model's distribution unchanged. $T<1$ sharpens it: with $T=0.5$, "mat" goes from 39 to 63 percent. $T>1$ flattens it: at 2.0 "mat" drops to 24 percent and even "theorem" gets a chance. As $T\to 0$ it becomes greedy.

Top-k and top-p

Even at a sensible temperature, the long tail of thousands of unlikely tokens adds up, and occasionally sampling from it derails the text. Two cut-offs fix this:

• Top-kSampling only from the k most likely tokens, after renormalising their probabilities to sum to 1. keeps the $k$ most likely tokens and renormalises. With $k=3$: mat 0.51, floor 0.28, sofa 0.21.
• Top-pNucleus sampling: keep the smallest set of most likely tokens whose probabilities add up to at least p, then renormalise and sample from it., also called nucleus sampling, keeps the smallest set of top tokens whose probabilities add up to at least $p$. With $p=0.8$: mat, floor, sofa and bed (0.39 + 0.21 + 0.16 + 0.11 = 0.87), and the rest are cut.

Top-p adapts to the situation: when the model is confident, the nucleus is one or two tokens; when many continuations are plausible, it widens. Top-k uses the same $k$ whether the model is sure or not.

Try it

Try this:

1. Press Sample 20 a few times at the defaults. Mat comes up most often, but floor, sofa and bed appear regularly, and once in a while something odd. Now set temperature to 0: every draw is mat.
2. Set temperature to 2 and look at the bars: the distribution is much flatter and moon, piano and even theorem get real probability. Sample 20 and see the variety, wanted and unwanted.
3. Keep temperature 2 and set top-p to 0.8. The tail is cut, so you keep variety among plausible tokens without ever drawing theorem. This combination, a moderate-to-high temperature plus a top-p cut, is the usual recipe for creative text.

The same numbers in Python:

# Temperature, top-k and top-p on one next-token distribution.
import numpy as np

tokens = ["mat", "floor", "sofa", "bed", "roof",
          "chair", "table", "moon", "piano", "theorem"]
logits = np.array([3.2, 2.6, 2.3, 1.9, 1.2, 1.0, 0.6, -0.5, -1.0, -2.5])

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

def top_k(p, k):
    keep = np.argsort(-p)[:k]
    q = np.zeros_like(p); q[keep] = p[keep]
    return q / q.sum()

def top_p(p, threshold):
    order = np.argsort(-p)
    cum = np.cumsum(p[order])
    n = int(np.searchsorted(cum, threshold - 1e-9) + 1)  # smallest set >= p
    q = np.zeros_like(p); q[order[:n]] = p[order[:n]]
    return q / q.sum()

show = lambda name, p: print(f"{name:<16}" + " ".join(
    f"{t}:{v:.2f}" for t, v in zip(tokens, p) if v >= 0.005))

show("T = 1.0", softmax(logits))
show("T = 0.5", softmax(logits / 0.5))
show("T = 2.0", softmax(logits / 2.0))
show("top-k = 3", top_k(softmax(logits), 3))
show("top-p = 0.8", top_p(softmax(logits), 0.8))

rng = np.random.default_rng(0)
p = softmax(logits)
draws = rng.choice(len(tokens), size=1000, p=p)
print("\n1,000 samples at T = 1:", {tokens[i]: int((draws == i).sum())
                                    for i in np.argsort(-p)[:5]})
print("greedy always picks:", tokens[int(np.argmax(logits))])

T = 1.0         mat:0.39 floor:0.21 sofa:0.16 bed:0.11 roof:0.05 chair:0.04 table:0.03 moon:0.01 piano:0.01
T = 0.5         mat:0.63 floor:0.19 sofa:0.10 bed:0.05 roof:0.01 chair:0.01
T = 2.0         mat:0.24 floor:0.18 sofa:0.15 bed:0.12 roof:0.09 chair:0.08 table:0.06 moon:0.04 piano:0.03 theorem:0.01
top-k = 3       mat:0.51 floor:0.28 sofa:0.21
top-p = 0.8     mat:0.45 floor:0.25 sofa:0.18 bed:0.12

1,000 samples at T = 1: {'mat': 355, 'floor': 225, 'sofa': 157, 'bed': 124, 'roof': 50}
greedy always picks: mat

In 1,000 samples at temperature 1, mat came up 355 times and floor 225 times, close to their probabilities of 0.39 and 0.21. Randomness is not noise to be eliminated: it is a direct readout of the model's uncertainty.

Going deeper

Repetition penalties

Models sometimes repeat phrases. Repetition or frequency penalties lower the logits of tokens that already appeared, in proportion to how often (frequency penalty) or just once if present at all (presence penalty). They help with loops but can backfire: code and lists legitimately repeat tokens, and penalising them produces odd synonyms. Use them sparingly.

Why outputs vary, and determinism

The same prompt gives different answers because decoding samples. To make outputs repeatable, set temperature to 0 or fix a random seed where the API allows it. Even then, hosted models may not be perfectly deterministic: GPU arithmetic in different batch sizes and orders can change the last digits of the logits, and when two tokens are nearly tied, that flips the choice and everything after it. For tests, compare meaning rather than exact strings, and run evaluations several times ( Building an eval set ).

Choosing settings

• Extraction, classification, code, maths: temperature 0 to 0.3. You want the most likely answer.
• General assistant answers: around 0.7 with top-p around 0.9 is a common starting point.
• Brainstorming and fiction: higher temperature, with top-p to cut the tail.
• Generating many candidates to pick from ( Chain of thought and step-by-step reasoning ): a higher temperature increases diversity between them.

Some providers ask you to adjust either temperature or top-p, not both, and some reasoning models fix these settings. Check the documentation for the model you use.

Trade-offs

Common mistakes

• Turning temperature up to make a model "smarter" or more accurate. It makes it more random, not more knowledgeable.
• Using a high temperature for structured output such as JSON, then blaming the model for broken syntax.
• Expecting temperature 0 to guarantee byte-identical outputs across calls on a hosted API.
• Stacking heavy repetition penalties on code generation, which needs to repeat variable names and keywords.
• Judging a prompt by one sample. With sampling on, look at several outputs before deciding it works.

In the wild

Check yourself

Recap