AI AI-05.04
Multi-head attention
Why transformers run several attention heads in parallel, how heads split the width, concatenate and project back, and what heads really learn.
The big idea
One attention step gives each token one set of weights: one opinion about which other tokens matter. But a word usually needs several kinds of context at once. In "The chef who trained in Lyon cooked the fish", the verb "cooked" wants its subject ("chef", several words back), its object ("fish"), and perhaps the tense of nearby words. A single softmax row has to compromise between all of these, and averaging very different things blurs them.
Multi-head attentionRunning several attention operations in parallel on the same tokens, each with its own query, key and value projections, then concatenating their outputs and projecting back to the model width. runs several attention operations side by side. Each headOne of the parallel attention operations in multi-head attention, with its own learned query, key and value projections into a smaller subspace. has its own query, key and value projections, so it can learn its own notion of "relevant". One head might link verbs to subjects, another might track the previous token, another might connect a pronoun to its noun. Their results are then stitched together.
Think of a panel of specialist reviewers reading the same manuscript. The grammar reviewer, the fact checker and the style editor each mark different passages as important. The final report combines all their notes. Asking one generalist to do all three at once would produce a vaguer report.
How it works
Step 1: Split the width into heads
With model width and heads, each head works in dimensions. A model with width 4,096 and 32 heads gives each head 128 dimensions. In practice the projections are computed as one big matrix multiplication and then reshaped into heads, so splitting costs nothing extra.
Step 2: Attend in each head independently
Each head runs exactly the attention from the previous lesson on its own slice:
All heads run in parallel, as one batched operation on the GPU. Each produces its own weight table, so different heads can attend to completely different tokens.
Step 3: Concatenate and project
Put the outputs side by side, which restores width , and multiply by a learned output matrix :
lets the heads' findings mix: information gathered by the "subject" head and the "object" head is combined into one vector per token that goes on to the rest of the block.
The cost is the same as one big head
Splitting does not add parameters. , , and are each , so the layer has weights whether you use 1 head or 32. With width 4,096 that is million weights per attention layer. Multi-head attention spends the same budget on several narrow lookups instead of one wide one.
Try it
The whole computation in numpy, with 4 tokens, width 8 and 2 heads:
# Multi-head attention: split the model width into h heads, attend in
# each head separately, concatenate, project back.
import numpy as np
rng = np.random.default_rng(0)
n, d_model, h = 4, 8, 2 # 4 tokens, width 8, 2 heads
d_head = d_model // h # each head works in 4 dimensions
x = rng.normal(size=(n, d_model))
W_Q, W_K, W_V, W_O = (rng.normal(size=(d_model, d_model)) * 0.5
for _ in range(4))
def softmax(s):
e = np.exp(s - s.max(axis=-1, keepdims=True))
return e / e.sum(axis=-1, keepdims=True)
def split(m): # (n, d_model) -> (h, n, d_head)
return m.reshape(n, h, d_head).transpose(1, 0, 2)
Q, K, V = split(x @ W_Q), split(x @ W_K), split(x @ W_V)
mask = np.triu(np.full((n, n), -np.inf), k=1)
A = softmax(Q @ K.transpose(0, 2, 1) / np.sqrt(d_head) + mask) # (h, n, n)
heads = A @ V # (h, n, d_head)
out = heads.transpose(1, 0, 2).reshape(n, d_model) @ W_O # (n, d_model)
np.set_printoptions(precision=2, suppress=True)
print("Q per head:", Q.shape, " weights:", A.shape, " output:", out.shape)
for i in range(h):
print(f"head {i}, last token attends:", A[i, -1])
print(f"parameters in W_Q, W_K, W_V, W_O: {4 * d_model * d_model}"
f" (the same as one big head of width {d_model})")Output
Q per head: (2, 4, 4) weights: (2, 4, 4) output: (4, 8)
head 0, last token attends: [0.09 0.51 0.1 0.31]
head 1, last token attends: [0.35 0.12 0.3 0.23]
parameters in W_Q, W_K, W_V, W_O: 256 (the same as one big head of width 8) Try this:
- Look at the two rows printed for the last token. Same tokens, same input, but head 0 puts 0.51 on the second token while head 1 spreads its attention much more evenly. The heads disagree, which is the point.
- Follow the shapes: queries are
(2, 4, 4), two heads with 4 tokens and 4 dimensions each; the output is back to(4, 8), one full-width vector per token. - Change
hto 1, 4 or 8 and rerun (width 8 is divisible by all of them). The parameter count stays 256; only the number of separate attention patterns changes.
Going deeper
What heads learn, and the caveats
Researchers who inspected trained models found heads with recognisable jobs. In BERT, some heads attend to delimiter tokens, some to a fixed offset such as the previous or next token, some broadly over the whole sentence, and some track syntax, such as linking verbs to their direct objects, or coreference. In decoder models, "induction heads" that copy patterns seen earlier in the context have been studied as a mechanism behind in-context learning.
But heads are not neat modules. Many are redundant: a large share can be removed from a trained model at test time with little loss, and some layers work with a single head. Roles are often fuzzy or shared across heads, and as the previous lesson warned, attention patterns are hints rather than explanations. Interpretability research ( Alignment and interpretability, the big picture ) goes deeper.
Sharing keys and values: MQA and GQA
During generation, every head's keys and values for past tokens are stored in the KV cache, and that memory, not compute, often limits speed ( Context windows and the KV cache ). Multi-query attention (MQA) keeps many query heads but shares a single key and value head among them. Grouped-query attention (GQA) is the middle ground: query heads are split into groups, and each group shares one key and value head. Both shrink the cache by the sharing factor in exchange for a small quality cost, and GQA is common in current open-weights models.
Choosing the number of heads
More heads give more independent patterns but each one gets fewer dimensions, and a head that is too narrow cannot represent much in its queries and keys. Head dimensions of 64 or 128 are typical, and the head count is usually chosen from the width to land there.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Single head | Simplest; full width for one pattern | One averaged attention pattern per token; must compromise between relationships |
| Multi-head (MHA) | Several relationships per layer at the same parameter cost | Largest KV cache; many heads end up redundant |
| Grouped-query (GQA) | Much smaller KV cache, faster decoding, little quality loss | Key and value heads are shared, so slightly less expressive |
| Multi-query (MQA) | Smallest KV cache | Most quality loss of the three; can make training less stable |
Common mistakes
- Thinking more heads means more parameters. With the width fixed, the attention layer has weights regardless of head count.
- Forgetting the output projection . Without it, the heads' outputs sit side by side in separate slots and never mix within the layer.
- Scaling by instead of . The scores in each head are dot products over dimensions.
- Assigning a single clean job to each head ("head 5 is the grammar head") from a few attention maps.
In the wild
Clark et al., 2019 (paper)
What Does BERT Look At? An Analysis of BERT's Attention (opens in a new tab)The authors analysed the attention heads of a trained BERT model. They report heads that attend to delimiter tokens, to specific positional offsets, or broadly over the whole sentence, with heads in the same layer often behaving similarly, and other heads that pick out syntactic relations such as direct objects of verbs and determiners of nouns with high accuracy, plus heads that capture coreference.
Michel, Levy and Neubig, 2019 (paper)
Are Sixteen Heads Really Better than One? (opens in a new tab)This paper found that a large percentage of attention heads can be removed at test time without significantly hurting performance, and that some layers can be reduced to a single head. It suggests that the benefit of many heads comes partly from training dynamics: they help a model learn, even if many are not needed afterwards.
Check yourself
-
A model has width 2,048 and 16 heads. What is the dimension of each head's queries and keys?
Show answer
B. 128 d_head = d_model / h = 2,048 / 16 = 128.
-
Going from 8 heads to 16 heads at the same width changes the attention layer's parameter count how?
Show answer
C. Leaves it the same W_Q, W_K, W_V and W_O stay d_model x d_model. Heads only change how that width is divided.
-
What is W_O for?
Show answer
B. Mixing the concatenated outputs of the heads back into one d_model-wide vector per token Concatenation just places head outputs side by side; W_O combines them so information from different heads can interact.
-
Why do many recent models use grouped-query attention?
Show answer
B. Sharing key and value heads shrinks the KV cache, which speeds up generation Generation is often limited by KV cache memory. GQA stores fewer key and value heads at a small quality cost.
Recap
- Multi-head attention runs several attention operations in parallel, each in its own subspace of width d_model / h.
- Each head has its own query, key and value projections, so it can attend to different tokens for different reasons.
- Head outputs are concatenated and mixed by an output projection W_O.
- Heads cost no extra parameters; some learn recognisable patterns, but many are redundant.
- MQA and GQA share keys and values across heads to shrink the KV cache.
Next up
- Build production LLM apps learning path