Skip to content
learn
  1. Learn
  2. Artificial Intelligence
  3. Transformers and LLM Internals

AI AI-05.01

Tokenization

Why language models read subword tokens, how byte-pair encoding learns them, vocabulary trade-offs, letter-counting quirks and tokens as the unit of cost.

Beginner 14 min Last reviewed

The big idea

A language model never sees letters or words. Before any text reaches it, a tokenizerThe component that splits text into tokens from a fixed vocabulary and maps each token to an integer id; it is trained once, before the model. chops the text into pieces called tokensThe unit of text a language model reads and writes: a whole word, part of a word, a single character or byte, or punctuation, each with an integer id. and replaces each piece with an integer. "the model reads" might become three tokens, the, model, reads, and then three numbers. The model only ever works with those numbers.

What should a piece be? Whole words are meaningful, but there are too many of them, and new ones appear every day: names, typos, product codes, "photosynthesis". Single characters can spell anything, but then even a short paragraph is a very long sequence, and the model wastes effort relearning how letters form words. Modern tokenizers sit in between, using subwordsPieces of words: common words stay whole, rarer words are split into frequent fragments, and anything can still be spelled from small pieces.: common words become one token, rare words are split into familiar fragments, and anything at all can still be spelled out from small pieces.

Think of a typesetter's case of metal type. Besides single letters, a printer kept ready-made blocks for the most common words and endings, because setting "the" or "-ing" letter by letter all day is slow. Unusual words were still set letter by letter. A subword tokenizer is that type case, chosen automatically from how often pieces appear.

How it works

From text to token ids The text 'the tokenizer' is split into subword tokens: underscore the, underscore token, izer. Each token is looked up in the vocabulary and replaced by its integer id. Only the ids enter the model. text "the tokenizer" tokens _the | _token | izer token ids [262, 11241, 7509] model Figure 1. Text becomes subword tokens, and tokens become integer ids from a fixed vocabulary. The model only ever sees the ids (glowing path). The ids here are illustrative.

Step 1: Pick the starting alphabet

The most popular subword method is byte-pair encodingA tokenization method that starts from single characters or bytes and repeatedly merges the most frequent adjacent pair into a new token, until the vocabulary reaches a target size. (BPE). Training starts from the smallest units. Many tokenizers start from the 256 possible byte values, so every possible input, any language, emoji or binary junk, can be represented and nothing is ever "unknown". Words are usually split on spaces first, with the space kept as part of the following token (shown here as _).

Step 2: Count adjacent pairs, merge the most frequent

Look at every pair of neighbouring symbols across the training text, weighted by how often each word occurs. Take the most frequent pair, say e followed by s, and add es to the vocabulary as a new token. Replace every occurrence of the pair with the new token. Repeat. Each repetition is one merge, and the ordered list of merges is the tokenizer.

Here are the first steps on the classic toy corpus: low 5 times, lower 2 times, newest 6 times and widest 3 times, with _ marking the end of a word.

Byte-pair encoding, merge by merge

  1. Start Every word is split into characters: l o w _ (x5), l o w e r _ (x2), n e w e s t _ (x6), w i d e s t _ (x3). The vocabulary is just the characters.
  2. Merge 1: t + _ The pairs e s, s t and t _ each appear 9 times (6 in newest + 3 in widest). Ties are broken by a fixed rule, here alphabetical, so t _ becomes the new token t_.
  3. Merges 2 and 3: s + t_, then e + st_ Each also appears 9 times. After three merges the ending est_ is a single token, shared by newest and widest.
  4. Merges 4 and 5: o + w, then l + ow low appears 5 + 2 = 7 times (low and lower), so low becomes one token.
  5. Merges 6 to 8 w + est_, n + e, then ne + west_: the frequent word newest is now a single token.
  6. Use the merges To tokenize new text, apply the merges in the order they were learned. lowest, never seen in training, becomes low + est_: two familiar pieces.

Step 3: Stop at a target vocabulary size

Keep merging until the vocabulary reaches a chosen size. Large language models typically use vocabularies of tens of thousands to a few hundred thousand tokens. Each token then gets an integer id, and the model gets one row in its embedding table per id ( Embeddings and positional encoding ).

Step 4: Tokenize new text

At use time the tokenizer applies the learned merges, in order, to the new text. Frequent words collapse into single tokens; rare words stay split into several pieces. The tokenizer is trained once, before the model, and then frozen: the model's vocabulary can never change without retraining the model.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Start at 0 merges: every character is its own token and "the model reads the tokens" is 27 tokens long. Slide up to 60: the same text is 8 tokens, because those words are frequent in the training corpus.
  2. Pick a rare word. Even at the maximum number of merges, "photosynthesis" stays in many small pieces: the corpus never contained it, so no merges cover it. Rare words cost more tokens.
  3. Pick numbers. The digits show ? instead of an id: this tiny corpus contained no digits, so they are not in its base vocabulary. That is exactly why real tokenizers start from all 256 bytes.

The same algorithm in Python, on the classic toy corpus from the steps above:

bpe.pyruns with python3 (standard library)
# Byte-pair encoding from scratch: learn merges from a tiny corpus,
# then use them to tokenize words it has and has not seen.
from collections import Counter

corpus = {"low": 5, "lower": 2, "newest": 6, "widest": 3}   # word: count

# Start: every word is a sequence of characters plus an end-of-word marker.
words = {tuple(w) + ("_",): c for w, c in corpus.items()}

def pair_counts(words):
    pairs = Counter()
    for syms, c in words.items():
        for a, b in zip(syms, syms[1:]):
            pairs[a, b] += c
    return pairs

def merge(words, pair):
    out = {}
    for syms, c in words.items():
        s, i = [], 0
        while i < len(syms):
            if i + 1 < len(syms) and (syms[i], syms[i + 1]) == pair:
                s.append(syms[i] + syms[i + 1]); i += 2
            else:
                s.append(syms[i]); i += 1
        out[tuple(s)] = c
    return out

merges = []
for step in range(10):
    pairs = pair_counts(words)
    best = max(pairs, key=lambda p: (pairs[p], p))   # most frequent pair
    merges.append(best)
    words = merge(words, best)
    print(f"merge {step + 1:>2}: {best[0]!r:>8} + {best[1]!r:<6} "
          f"(seen {pairs[best]} times)")

def tokenize(word):
    syms = {tuple(word) + ("_",): 1}
    for m in merges:                       # apply merges in learned order
        syms = merge(syms, m)
    return list(next(iter(syms)))

for w in ["newest", "lowest", "widen", "slower"]:
    print(f"{w:>7} -> {tokenize(w)}")

Output

merge  1:      't' + '_'    (seen 9 times)
merge  2:      's' + 't_'   (seen 9 times)
merge  3:      'e' + 'st_'  (seen 9 times)
merge  4:      'o' + 'w'    (seen 7 times)
merge  5:      'l' + 'ow'   (seen 7 times)
merge  6:      'w' + 'est_' (seen 6 times)
merge  7:      'n' + 'e'    (seen 6 times)
merge  8:     'ne' + 'west_' (seen 6 times)
merge  9:    'low' + '_'    (seen 5 times)
merge 10:      'w' + 'i'    (seen 3 times)
 newest -> ['newest_']
 lowest -> ['low', 'est_']
  widen -> ['wi', 'd', 'e', 'n', '_']
 slower -> ['s', 'low', 'e', 'r', '_']

lowest and slower never appeared in training, yet both are tokenized with the learned piece low. widen has no learned ending, so it falls back to single characters.

Going deeper

Vocabulary size is a trade-off

A bigger vocabulary means more words fit in one token, so sequences are shorter, which makes the model faster and lets more text fit in its context window. But every token needs a row in the embedding table and in the output layer, and rare tokens are seen too few times in training to learn good vectors. A smaller vocabulary keeps the tables small and every token well trained, but makes sequences longer. Multilingual models need large vocabularies, otherwise languages that were rare in the tokenizer's training data get split into many tiny pieces.

Why models struggle to count letters

Ask how many times "r" appears in "strawberry". In the lab, with 60 merges, the word becomes _s t ra w b er r y: two of the three r's are hidden inside ra and er. A real model sees token ids, and the id for a multi-letter token does not directly expose which letters it contains. The model has to have learned the spelling of each token from indirect evidence. The same effect makes reversing words, counting characters and some arithmetic harder than they look: numbers are often split into chunks in ways that do not line up with place value. Asking the model to spell the word out letter by letter first, or to use a code tool ( Function calling and tool use ), sidesteps the problem.

Tokens are the unit of cost and limits

Everything about a hosted model is counted in tokens: the context window (how much input and output fit at once, Context windows and the KV cache ), rate limits, and price, usually billed per million input tokens and per million output tokens at different rates. The same sentence can take a different number of tokens in different models, and text in languages other than English, code and numbers often takes more tokens per word.

Other subword methods

BPE has relatives. WordPiece, used by BERT, picks merges by how much they improve the likelihood of the training text rather than by raw frequency. Unigram tokenization, in the SentencePiece library, starts with a large vocabulary and prunes it. All produce the same kind of result: a fixed vocabulary of subword pieces. Some research models skip tokenization and read raw bytes, paying for it with much longer sequences.

Trade-offs

OptionWins whenLoses when
Word-level tokens Short sequences; each token is a meaningful word Huge vocabulary; unknown words, typos and new names cannot be represented
Character or byte level Tiny vocabulary; nothing is ever unknown; spelling is visible Very long sequences, so slower and less text fits in the context
Subword (BPE and relatives) Common words whole, rare words in pieces, nothing unknown with a byte base Letter-level tasks get harder; token counts vary by language and domain

Common mistakes

  • Estimating cost or context use from word or character counts instead of running the actual tokenizer.
  • Assuming every model uses the same tokenizer. Token counts, and so prices and limits, differ between models.
  • Forgetting that leading spaces are part of tokens, so "hello" and " hello" are different tokens. This matters when you build prompts by string concatenation.
  • Expecting reliable letter counting, string reversal or digit-level arithmetic from a model without letting it spell things out or use a tool.
  • Changing or extending the tokenizer of a trained model without retraining or at least fine-tuning the embeddings for the new tokens.

In the wild

Sennrich, Haddow and Birch, 2016 (paper)

Neural Machine Translation of Rare Words with Subword Units (opens in a new tab)

Translation models of the time used a fixed vocabulary, but translation is an open-vocabulary problem: names, compounds and loanwords keep appearing. This paper proposed encoding rare and unknown words as sequences of subword units, adapting byte pair encoding compression to choose the units. It reports gains over a dictionary back-off baseline of 1.1 BLEU for English-German and 1.3 BLEU for English-Russian on WMT 15. This is the origin of BPE as a tokenizer for neural language models.

Check yourself

  1. What does one byte-pair encoding training step do?

    Show answer

    B. Finds the most frequent adjacent pair of symbols and merges it into a new token BPE counts neighbouring pairs across the corpus, weighted by word frequency, and adds the most frequent pair as a new token, repeating until the vocabulary is big enough.

  2. Why do many tokenizers start from the 256 byte values?

    Show answer

    B. So any input, in any language or format, can be represented and no token is ever unknown With all bytes in the base vocabulary, the worst case is spelling something byte by byte. Nothing falls outside the vocabulary.

  3. A model miscounts the r's in 'strawberry'. What is the most likely reason?

    Show answer

    B. It sees subword tokens such as 'ra' and 'er', whose ids do not directly expose their letters Letters are hidden inside multi-letter tokens. Spelling the word out first, or using a code tool, makes the letters visible.

  4. You grow the vocabulary from 32,000 to 256,000 tokens. What is the main benefit and the main cost?

    Show answer

    A. Shorter sequences; bigger embedding and output tables and rarer tokens More merges mean more text per token, so shorter sequences, but each token needs its own rows in the embedding and output layers, and rare tokens get little training.

  5. Your app's prompts are in Japanese and English. Why should you measure token counts separately?

    Show answer

    B. The same amount of text can need a different number of tokens per language, which changes cost and context use Tokenizers are trained on a mix of text, and languages that were less common in that mix are often split into more tokens per word.

Recap

  • Models read token ids, not letters or words; the tokenizer is trained once and then frozen.
  • Subword tokens keep common words whole and split rare words, so nothing is unknown.
  • BPE repeatedly merges the most frequent adjacent pair; the ordered merge list is the tokenizer.
  • Vocabulary size trades sequence length against table size and rare-token quality.
  • Tokens explain letter-counting quirks and are the unit of context limits and pricing.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up