Skip to content
learn
  1. Learn
  2. Artificial Intelligence
  3. Transformers and LLM Internals

AI AI-05.06

Encoder, decoder and decoder-only models

BERT-style encoders, T5-style encoder-decoders and GPT-style decoders: how their attention masks differ, how each is trained, and which fits which task.

Intermediate 10 min Last reviewed

The big idea

All transformers are built from the same block. What separates the three big families is mostly who is allowed to look at whom, and what the model is trained to predict.

  • An encoder-onlyA transformer whose tokens attend in both directions over the whole input. It produces a contextual vector per token and is used for understanding tasks such as classification and embeddings. BERT is the classic example. model reads the whole input at once, every token seeing every other. It is built to understand: classify, tag, embed.
  • A decoder-onlyA transformer that generates text left to right with causal self-attention, each token seeing only earlier tokens. GPT-style models, including most modern chat assistants, are decoder-only. model reads left to right, each token seeing only the past. It is built to generate: continue text one token at a time.
  • An encoder-decoderA transformer with an encoder that reads the full input bidirectionally and a decoder that generates the output causally while attending to the encoder's outputs through cross-attention. T5 and the original transformer are examples. model does both: an encoder reads the input fully, then a decoder writes an output while consulting it. It is built to transform one sequence into another.

Think of three kinds of reader. A proofreader (encoder) reads the whole page before marking anything. A storyteller (decoder) speaks one word at a time and cannot take back what was said. A translator (encoder-decoder) first reads the whole source sentence, then writes the translation word by word, glancing back at the source as needed.

How it works

Three transformer families side by side Left: an encoder stack reading 'the cat sat' with arrows in both directions, producing a vector per token for classification or embeddings. Middle: an encoder and a decoder connected by cross-attention; the decoder writes 'le chat' while reading the encoder output. Right: a decoder-only stack reading tokens left to right and predicting the next token; it glows. the cat sat whole input encoder bidirectional vector per token classify, tag, embed the cat sat encoder decoder causal le chat ... output text the cat prefix so far decoder causal next token: sat cross-attention Figure 1. Encoder-only (left) sees the whole input both ways and outputs vectors. Encoder-decoder (middle) reads fully, then generates while cross-attending to the input. Decoder-only (right, glowing) reads strictly left to right and predicts the next token; this design dominates modern language models.

Encoder-only: bidirectional understanding

An encoder applies self-attention with no mask, so the representation of "bank" in "I sat by the bank of the river" can use "river", which comes later. The output is one contextual vector per token. Since nothing is generated, it cannot be trained by predicting the next token: that would be trivial when every token can see the answer. Instead BERT was trained with masked language modelling: hide about 15 percent of tokens and predict them from both sides. For a task, you add a small head on top, for example a classifier on a summary token, and fine-tune. Encoders power classification, named-entity tagging, and most of the embedding models from Sentence and document embeddings .

Decoder-only: left-to-right generation

A decoder applies causal self-attention, and is trained on next-token prediction over huge amounts of text ( Pretraining: next-token prediction ). Every position of every training sequence is a training example, so the data is used very efficiently. To generate, it predicts a token, appends it to the input, and repeats. Any task can be posed as text continuation: "Translate to French: the cat sat ->", "Summarise: ...", "Q: ... A:". That generality, plus the fact that it scales smoothly, is why GPT-style decoder-only models dominate modern large language models and chat assistants.

Encoder-decoder: sequence to sequence

The original transformer was built for translation and had both halves. The encoder reads the source bidirectionally. The decoder generates the target causally, and each decoder block has an extra cross-attentionAttention in which queries come from one sequence (the decoder) and keys and values from another (the encoder output), letting the output consult the input. layer whose queries come from the decoder and whose keys and values come from the encoder output. T5 trained this design on many tasks recast as text-to-text, and it remains strong for translation, summarisation and speech recognition, where input and output are clearly separate.

Try it

The difference really is in the masks:

masks.pyruns with python3 + numpy
# The three transformer families differ mainly in who may attend to whom.
# 1 = may attend, . = blocked. Rows are the attending token.
import numpy as np

def show(name, m, rows, cols):
    print(name)
    print("        " + " ".join(f"{c:>6}" for c in cols))
    for r, line in zip(rows, m):
        print(f"{r:>7} " + " ".join(f"{'1' if v else '.':>6}" for v in line))
    print()

src = ["the", "cat", "sat"]
tgt = ["<s>", "le", "chat"]
n = len(src)

# Encoder-only (BERT-style): every token sees every token, both directions.
show("encoder: bidirectional self-attention", np.ones((n, n)), src, src)

# Decoder-only (GPT-style): each token sees itself and the past only.
show("decoder-only: causal self-attention", np.tril(np.ones((n, n))), src, src)

# Encoder-decoder (T5-style): the decoder attends causally to its own
# tokens and fully to every encoder output (cross-attention).
show("encoder-decoder: decoder self-attention", np.tril(np.ones((n, n))), tgt, tgt)
show("encoder-decoder: cross-attention", np.ones((n, n)), tgt, src)

Output

encoder: bidirectional self-attention
           the    cat    sat
    the      1      1      1
    cat      1      1      1
    sat      1      1      1

decoder-only: causal self-attention
           the    cat    sat
    the      1      .      .
    cat      1      1      .
    sat      1      1      1

encoder-decoder: decoder self-attention
           <s>     le   chat
    <s>      1      .      .
     le      1      1      .
   chat      1      1      1

encoder-decoder: cross-attention
           the    cat    sat
    <s>      1      1      1
     le      1      1      1
   chat      1      1      1

Sort these tasks by the family that is the most natural fit today:

Which model family fits each task best?

Show answers
  • Embed 10 million support tickets for semantic search: Encoder-only. Embedding models are usually encoders: bidirectional attention gives a good vector for the whole text, and nothing needs generating.
  • Tag every token in a contract as a person, organisation or date: Encoder-only. Token labelling needs each token's full context in both directions and no generation.
  • A general chat assistant that writes, codes and answers questions: Decoder-only. Open-ended generation across many tasks is exactly what large decoder-only models are trained for.
  • Autocomplete the next few words in an editor: Decoder-only. Continuing a prefix left to right is next-token prediction.
  • Transcribe speech audio into text: Encoder-decoder. A separate input (audio) is encoded fully, then text is generated while cross-attending to it, as in classic speech recognition models.
  • A dedicated model that translates sentences between two languages: Encoder-decoder. Translation was the original encoder-decoder task: read the whole source, then write the target.

Note that a large decoder-only model can do every one of these tasks, given the right prompt. The sorter asks for the most natural, efficient fit.

Going deeper

Why decoders won the scaling race

Next-token prediction needs no labels, every token is a training signal, and one model handles any task phrased as text. Decoders also serve generation efficiently with a KV cache ( Context windows and the KV cache ). Encoder-decoders split parameters between two stacks and need a clear input/output boundary, which chat conversations do not have. Encoders cannot generate fluently at all.

Encoders are still everywhere

For classification, retrieval, reranking and embeddings, a small encoder is often faster, cheaper and as accurate as a large decoder, because it reads the input once with full context and outputs exactly what is needed. Most retrieval systems in Why RAG? run on encoders.

Hybrids

Variants blur the lines. A prefix language model lets a decoder attend bidirectionally over a prompt and causally over the continuation. Many decoder-only models are also turned into embedding models by fine-tuning them with a contrastive objective. And multimodal models often pair a vision encoder with a decoder language model ( Multimodal models ).

Trade-offs

OptionWins whenLoses when
Encoder-only Classification, tagging, embeddings, reranking; small and fast Cannot generate fluent text; needs a task head and fine-tuning
Encoder-decoder Clear input to output mappings: translation, summarisation, speech to text Two stacks to train and serve; awkward for multi-turn chat
Decoder-only General-purpose generation and chat; scales well; any task as text Larger and slower for simple understanding tasks; sees its input causally

Common mistakes

  • Using a huge chat model for a high-volume classification or embedding job that a small encoder would do faster and cheaper.
  • Trying to make an encoder like BERT generate text by sampling from its masked-token predictions.
  • Assuming "decoder" always means a model with cross-attention. In decoder-only models there is no encoder and no cross-attention, just causal self-attention.
  • Forgetting that a decoder-only model reads the prompt causally too: earlier prompt tokens cannot see later ones, which is one reason instruction placement matters ( Prompting fundamentals ).

In the wild

Devlin et al., 2018 (paper)

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (opens in a new tab)

BERT pre-trains deep bidirectional representations by conditioning on both left and right context in all layers. The pre-trained model is fine-tuned with just one additional output layer per task, and the paper reports new state-of-the-art results on eleven language understanding tasks, including question answering and language inference. It made encoder-only pre-training plus fine-tuning the standard recipe for understanding tasks.

Raffel et al., 2019 (paper)

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5) (opens in a new tab)

T5 reformulates every text problem, from classification to translation to summarisation, as feeding text in and getting text out, using an encoder-decoder transformer. The authors systematically compared pre-training objectives, architectures, datasets and transfer methods, combined the best choices with scale and a new corpus, and released the dataset, models and code.

Check yourself

  1. What makes an encoder 'bidirectional'?

    Show answer

    B. Its self-attention has no causal mask, so every token can attend to tokens before and after it Without a mask, each token's representation uses context from both sides of it.

  2. Why can't BERT be trained by plain next-token prediction?

    Show answer

    B. Every token can see the whole input, including the token it should predict, so the task would be trivial Bidirectional attention would leak the answer. BERT instead hides tokens and predicts them (masked language modelling).

  3. In an encoder-decoder model, where do the keys and values of cross-attention come from?

    Show answer

    B. The encoder's output vectors Queries come from the decoder; keys and values come from the encoder output, so the decoder can consult the full input.

  4. You must embed 50 million product descriptions nightly on a tight budget. Which family is the natural choice?

    Show answer

    B. A small encoder-only embedding model Embedding needs no generation; a small encoder reads each text once with full context and is far cheaper per item.

Recap

  • Encoder-only models attend in both directions and output contextual vectors: classification, tagging, embeddings.
  • Decoder-only models attend causally and generate the next token; they dominate modern LLMs.
  • Encoder-decoders read the input fully, then generate while cross-attending to it: translation, summarisation, speech.
  • The families differ mainly in attention masks and training objectives, not in the block itself.
  • Pick the smallest family that fits: encoders still win many understanding and retrieval jobs.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up