Skip to content
learn
  1. Learn
  2. Artificial Intelligence
  3. Transformers and LLM Internals

AI AI-05.07

Pretraining: next-token prediction

The next-token objective, pretraining data, compute = 6ND, scaling laws, compute-optimal training after Chinchilla, and the emergent abilities debate.

Intermediate 14 min Last reviewed

The big idea

A freshly built transformer knows nothing: its weights are random. PretrainingThe first and most expensive training stage of a language model: next-token prediction over a very large, broad text corpus, without human labels. fills it with knowledge using one deceptively simple task: given some text, predict the next token. Do that over trillions of tokens of web pages, books, code and papers, and the model learns spelling, grammar, facts, the style of legal contracts, how Python functions are structured and how arguments unfold, because every one of those things helps predict what comes next.

Why does such a narrow task teach so much? Because to predict the next word of "The capital of Australia is", you need geography; to predict the next line of a proof, you need some mathematics; to predict a character's reply in a novel, you need a model of what that character wants. The objective is simple, but doing it well requires everything.

Think of a student who prepares for every exam in the world by reading every textbook with a card covering the next word, guessing it, then sliding the card to check. Billions of guesses later, they have absorbed a vast amount, even though nobody ever taught them a single subject directly.

How it works

One pretraining step A batch of text sequences is fed to the model. At every position the model outputs a probability for every vocabulary token. The loss is the cross-entropy between those predictions and the actual next tokens. Backpropagation computes gradients, and the optimizer updates all weights. The loop repeats over trillions of tokens. gradients text batch sequences of tokens model predict every next token cross-entropy vs. actual next tokens update AdamW step Figure 1. The pretraining loop: text in, a next-token prediction at every position, cross-entropy against the real next tokens, gradients, a weight update (glowing: the return path). Repeated for hundreds of thousands of steps.

Step 1: The objective

For a sequence of tokens , the model outputs at each position a probability distribution over the vocabulary for token . The loss is the average cross-entropyThe negative log of the probability the model assigned to the correct answer, averaged over examples. 0 means certain and right; large means confident and wrong.:

If the model gives the true next token probability 0.5, that position costs ; at 0.9 it costs 0.11. Thanks to the causal mask ( Self-attention, step by step ), one forward pass makes a prediction at every position at once, so a 2,000-token sequence is 2,000 training examples.

Step 2: The data

Pretraining corpora are mixtures: filtered web pages, books, encyclopaedias, scientific papers, code, forums, and increasingly synthetic text. Much of the work is cleaning, which means removing duplicates, spam, boilerplate and personal data, filtering for quality, and choosing mixture weights: how often to sample each source. More code in the mix tends to help programming and structured reasoning; more of one language helps that language. Data quality turns out to matter as much as quantity, and many labs consider their data pipeline their most important asset.

Step 3: Compute, parameters and tokens

Training cost is measured in floating-point operations (FLOPs). A useful approximation for a transformer with parameters trained on tokens is

about 2 FLOPs per parameter per token for the forward pass and 4 for the backward pass. A 70-billion-parameter model trained on 1.4 trillion tokens needs about FLOPs.

Step 4: Scaling laws

In 2020, researchers found that a language model's loss falls as a smooth power law in model size, data and compute, with trends spanning more than seven orders of magnitude. Architecture details such as width versus depth mattered little within a wide range. This made progress predictable: train a family of small models, fit the curve, and forecast how a model 100 times bigger will do before spending the money.

Step 5: Compute-optimal training

Given a fixed compute budget, should you train a bigger model on fewer tokens or a smaller model on more? The Chinchilla study fitted hundreds of training runs and concluded that model size and training tokens should grow together: for every doubling of model size, double the tokens. Its table of estimates works out to roughly 20 training tokens per parameter, for example 1 billion parameters on 20.2 billion tokens. Many earlier large models had been trained on too little data for their size.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

The curve uses the paper's own fitted formula, , with .

Try this:

  1. At the default budget (about FLOPs), your pink model has 282 billion parameters and gets only 333 billion tokens: predicted loss 1.986. Slide the model size down to about 71 billion: it now trains on 1.3 trillion tokens and the loss drops to 1.939. Same compute, smaller model, better result. This is roughly the Gopher versus Chinchilla comparison from the paper.
  2. Notice the curve is a valley. Too small and the model lacks capacity; too large and it cannot see enough data for the budget.
  3. Move the compute budget from 1e19 up to 1e25 and watch the optimum move right. Both the best model size and the best number of tokens grow. This fitted formula is one of three methods in the paper and leans toward smaller models on more tokens than its headline table (the tokens-per-parameter ratio it reports climbs from about 32 to over 100 across this range), so treat the exact ratio as an estimate.

Next-token prediction in miniature, with a model that just counts which character follows which:

next-token.pyruns with python3 + numpy
# Next-token prediction in miniature: a character bigram "language model"
# trained by counting, scored with cross-entropy, then the compute rule C = 6ND.
import numpy as np

text = ("the cat sat on the mat. the dog sat on the log. "
        "the cat saw the dog. the dog saw the cat. ") * 20
chars = sorted(set(text))
ix = {c: i for i, c in enumerate(chars)}
V = len(chars)
ids = np.array([ix[c] for c in text])
x, y = ids[:-1], ids[1:]                 # every position is a training example
split = int(0.9 * len(x))

counts = np.ones((V, V))                 # add-one smoothing
np.add.at(counts, (x[:split], y[:split]), 1)
P = counts / counts.sum(1, keepdims=True)   # P(next char | current char)

def loss(xs, ys):                        # average cross-entropy in nats
    return -np.mean(np.log(P[xs, ys]))

print(f"vocabulary {V} chars, {len(x):,} training examples from one text")
print(f"uniform guess loss : {np.log(V):.3f} nats")
print(f"bigram model loss  : {loss(x[split:], y[split:]):.3f} nats (held out)")

rng = np.random.default_rng(1)
c, out = ix["t"], "t"
for _ in range(40):                      # generate by sampling the next char
    c = rng.choice(V, p=P[c]); out += chars[c]
print("sample:", repr(out))

N, D = 70e9, 1.4e12                      # parameters, training tokens
print(f"\ntraining compute for N={N:.0e}, D={D:.1e}: 6ND = {6 * N * D:.2e} FLOPs")

Output

vocabulary 15 chars, 1,799 training examples from one text
uniform guess loss : 2.708 nats
bigram model loss  : 0.800 nats (held out)
sample: 'thheg on sathe log don sthe e the the sam'

training compute for N=7e+10, D=1.4e+12: 6ND = 5.88e+23 FLOPs

Even this tiny "model" cuts the loss from 2.708 nats (uniform guessing over 15 characters) to 0.800 on held-out text, and its samples start to look like the training text. A transformer does the same thing with far more context and capacity.

Going deeper

Beyond compute-optimal

Compute-optimal means the best model for a fixed training budget. But a model is trained once and then served billions of times, and serving cost grows with model size. So many current models are deliberately over-trained: smaller than compute-optimal and trained on far more tokens than 20 per parameter, which costs more to train but less to run. A practical limit is appearing too: the supply of high-quality text is finite, which drives interest in synthetic data and multiple passes over the best sources.

The emergent abilities debate

Some abilities, such as multi-step arithmetic, seemed to appear suddenly once models crossed a certain size, which was called emergence and raised worries about unpredictable jumps. A later analysis argued that much of this sharpness comes from the metric: an all-or-nothing score such as exact match jumps from 0 to 1, even when the underlying per-token probabilities improve smoothly. Measured with continuous metrics, many "emergent" abilities improved gradually. The debate continues, but it is a useful lesson in evaluation: how you measure shapes what you see ( Why evals are the core skill ).

What pretraining does not give you

A pretrained base model is a superb text continuer, not an assistant. Ask it a question and it may continue with three more questions, because that is what follows a question on many web pages. Instruction tuning and preference training ( The LLM training pipeline ) turn it into a helpful assistant, and they use a tiny fraction of the pretraining compute.

Trade-offs

OptionWins whenLoses when
Compute-optimal (about 20 tokens per parameter) Best model for a fixed training budget Larger model than necessary to serve, so higher inference cost
Over-trained small model Cheap, fast inference; fits on smaller hardware More training compute per point of quality; needs much more data
Huge model on little data Nothing, usually: an under-trained model wastes compute Worse loss than a smaller model at the same budget

Common mistakes

  • Thinking a base model "wants" to answer questions. It continues text; helpfulness comes from later training.
  • Comparing models by parameter count alone. Training tokens and data quality matter just as much.
  • Reading the 20-tokens-per-parameter rule as a law. It is an estimate for minimising training compute, from one study's setup.
  • Treating a sudden jump on an exact-match benchmark as proof of a sudden change in capability, without checking smoother metrics.
  • Forgetting deduplication: repeated documents inflate apparent data size and encourage memorisation.

In the wild

Hoffmann et al., 2022 (paper)

Training Compute-Optimal Large Language Models (Chinchilla) (opens in a new tab)

The authors trained over 400 language models from 70 million to over 16 billion parameters on 5 to 500 billion tokens, and concluded that model size and training tokens should be scaled equally. To test it they trained Chinchilla, 70 billion parameters on 1.4 trillion tokens, using the same compute as the 280-billion-parameter Gopher with 4 times more data. Chinchilla outperformed Gopher and several larger models across many tasks, while being cheaper to fine-tune and serve.

Andrej Karpathy

nanoGPT (opens in a new tab)

A small, readable codebase for training GPT-style models, described as the simplest, fastest repository for training and fine-tuning medium-sized GPTs. Its README trains a character-level Shakespeare model in about 3 minutes on one A100 GPU and reproduces the 124-million-parameter GPT-2 on OpenWebText in about 4 days on an 8 x A100 node. The README now marks it as deprecated in favour of a newer project, nanochat, but it remains a clear reference for the pretraining loop.

Check yourself

  1. A model assigns probability 0.25 to the correct next token. What is the cross-entropy loss at that position, in nats?

    Show answer

    C. About 1.39 -ln(0.25) = ln 4 = 1.386.

  2. Roughly how many FLOPs does it take to train a 7-billion-parameter model on 2 trillion tokens?

    Show answer

    B. 8.4 x 10^22 C = 6ND = 6 x 7 x 10^9 x 2 x 10^12 = 8.4 x 10^22 FLOPs.

  3. What was the main conclusion of the Chinchilla study?

    Show answer

    B. For a fixed compute budget, scale model size and training tokens together; many large models were under-trained Doubling the model should come with doubling the tokens. A 70B model on 1.4T tokens beat a 280B model trained with the same compute.

  4. Why do many deployed models train on far more than 20 tokens per parameter?

    Show answer

    B. Serving cost depends on model size, so a smaller, longer-trained model is cheaper to run at scale Compute-optimal minimises training cost. When inference dominates total cost, over-training a smaller model is the better deal.

  5. An ability jumps from 0 percent to 60 percent exact-match accuracy between two model sizes. What should you check before calling it emergent?

    Show answer

    B. Whether a continuous metric, such as the probability of the right answer, improved smoothly across sizes All-or-nothing metrics can turn smooth underlying improvement into apparent jumps.

Recap

  • Pretraining is next-token prediction with cross-entropy over trillions of tokens; every position is a training example.
  • Data mixtures and cleaning matter as much as data volume.
  • Training compute is about 6ND FLOPs for N parameters and D tokens.
  • Loss falls as a smooth power law in model size, data and compute, which makes scaling predictable.
  • Compute-optimal training scales parameters and tokens together; deployed models are often over-trained for cheaper inference.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up