Skip to content
learn
  1. Learn
  2. Artificial Intelligence
  3. Transformers and LLM Internals

AI AI-05.03

Self-attention, step by step

Queries, keys and values, dot-product scores, scaling by sqrt(d), softmax, weighted sums and causal masking, worked through by hand with three tokens.

Intermediate 22 min Last reviewed

The big idea

Read the sentence "The animal didn't cross the street because it was too tired." What does "it" refer to? You know instantly: the animal, because streets do not get tired. To understand one word you looked at other words and decided which ones mattered. A word's meaning depends on its context.

Self-attentionA mechanism in which every token computes a weighted average of information from the other tokens in the same sequence, with weights based on how relevant each one is. gives every token that ability. Each token asks a question ("I am a pronoun, which noun am I about?"), every token advertises what it offers ("I am an animate noun"), and the asking token pulls in information from the tokens whose advertisement best matches its question. After one round of attention, the vector for "it" contains information about "animal".

Think of a meeting where everyone holds up a card describing their expertise. You have a question, so you scan the room, pay most attention to the two people whose cards match your question, a little to a few others, and none to the rest. Then you combine what they tell you, weighted by how much you listened to each. That is one attention step, run for every person in the room at once.

How it works

Each token's vector is turned into three new vectors by three learned matrices:

  • a queryIn attention, the vector a token uses to ask what it is looking for; it is compared with every key. : what this token is looking for;
  • a keyIn attention, the vector a token uses to advertise what it contains; queries are compared against it. : what this token offers, used for matching;
  • a valueIn attention, the vector a token hands over when it is attended to; the output is a weighted sum of values. : what this token actually hands over if attended to.

Keys decide who gets attention; values decide what is passed on. Separating them lets a token be found for one reason and contribute something else.

Attention from the token sat to the tokens the, cat and sat The query of sat is compared with the keys of the, cat and sat. The scores become weights 0.07, 0.62 and 0.31 after scaling and softmax. The output for sat is the weighted sum of the three value vectors; the link to cat is thickest and glows. the weight 0.07 cat weight 0.62 sat weight 0.31 query of sat output = 0.07 v_the + 0.62 v_cat + 0.31 v_sat Figure 1. The query of 'sat' is matched against every key. Softmax turns the scores into the weights shown on each token, and the output is the weighted sum of the values. Here 'sat' takes 62 percent of its information from 'cat' (glowing). Numbers from the worked example below.

Step 1: Score every query against every key

The relevance of token to token is the dot product : large when the two vectors point the same way. Doing this for all pairs at once is one matrix product, , giving an table of scores for tokens.

Step 2: Scale by the square root of the dimension

Divide every score by , where is the length of the key vectors. Why? A dot product sums terms, so with random-ish vectors its typical size grows like . With 128-dimensional keys, raw scores would be about 11 times larger than with 1-dimensional ones. Huge scores push softmax into a corner where one weight is almost 1, the rest almost 0, and the gradients nearly vanish, so learning stalls. Dividing by keeps scores in a healthy range whatever the dimension.

Step 3: Softmax turns scores into weights

SoftmaxA function that turns a list of scores into positive weights that sum to 1: exponentiate each score and divide by the total. exponentiates each score and divides by the total, so each row becomes positive weights that sum to 1:

Exponentiating exaggerates differences: a score higher by 1 gets about 2.7 times the weight.

Step 4: Weighted sum of values

Each token's output is the weighted average of all value vectors, using its row of weights. In one formula, the whole layer is:

Step 5: Causal masking

A language model is trained to predict the next token, so a token must not see tokens that come after it, or training would be trivially easy and useless for generation. A causal maskA mask that sets the attention scores from each token to all later tokens to minus infinity, so their softmax weights become 0 and a token can only attend to itself and earlier tokens. sets every score above the diagonal of the table to before the softmax. Since , future tokens get exactly zero weight. Encoder models used for understanding, such as BERT, skip the mask and let every token see the whole input ( Encoder, decoder and decoder-only models ).

A full numeric example with three tokens

Take the sequence "the cat sat" with tiny 2-dimensional vectors. In a real model , and come from the learned matrices; here they are written down directly so you can follow the arithmetic.

Attention for 'sat', by hand

  1. The vectors Queries: the [1, 0], cat [0, 1], sat [1, 2]. Keys: the [1, 0], cat [0, 2], sat [1, 1]. Values: the [0, 1], cat [4, 0], sat [1, 3]. The query of sat roughly means 'looking for a noun'; the key of cat roughly means 'I am a noun'.
  2. Dot products q_sat . k_the = 1x1 + 2x0 = 1. q_sat . k_cat = 1x0 + 2x2 = 4. q_sat . k_sat = 1x1 + 2x1 = 3. Raw scores: [1, 4, 3].
  3. Scale d = 2, so divide by sqrt(2), about 1.414: [0.71, 2.83, 2.12].
  4. Mask sat is the last token, so nothing is in its future and nothing is masked. For cat, the score against sat would be set to minus infinity; for the, both later scores would be.
  5. Softmax e^0.71 = 2.03, e^2.83 = 16.92, e^2.12 = 8.34. Total 27.29. Weights: 0.07, 0.62, 0.31. They sum to 1.
  6. Weighted sum of values 0.07 x [0, 1] + 0.62 x [4, 0] + 0.31 x [1, 3] = [2.79, 0.99]. The new vector for sat is dominated by the value of cat.

The same in numpy, for all three rows at once:

attention.pyruns with python3 + numpy
# Self-attention by hand for three tokens: "the cat sat".
import numpy as np

np.set_printoptions(precision=2, suppress=True)
tokens = ["the", "cat", "sat"]

# In a real model Q, K and V come from multiplying token vectors by
# learned matrices W_Q, W_K, W_V. Here we write them down directly (d = 2).
Q = np.array([[1.0, 0.0],    # the
              [0.0, 1.0],    # cat
              [1.0, 2.0]])   # sat: "looking for a noun"
K = np.array([[1.0, 0.0],    # the
              [0.0, 2.0],    # cat: "I am a noun"
              [1.0, 1.0]])   # sat
V = np.array([[0.0, 1.0],
              [4.0, 0.0],
              [1.0, 3.0]])

def softmax(s):
    e = np.exp(s - s.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

d = Q.shape[1]
scores = Q @ K.T / np.sqrt(d)          # every query against every key
print("scaled scores:\n", scores)

mask = np.triu(np.ones((3, 3)), k=1)   # 1 above the diagonal = future
scores = np.where(mask == 1, -np.inf, scores)
W = softmax(scores)                    # each row sums to 1
print("causal attention weights:\n", W)

out = W @ V                            # weighted sum of value vectors
for t, row in zip(tokens, out):
    print(f"{t:>4} -> {row}")

Output

scaled scores:
 [[0.71 0.   0.71]
 [0.   1.41 0.71]
 [0.71 2.83 2.12]]
causal attention weights:
 [[1.   0.   0.  ]
 [0.2  0.8  0.  ]
 [0.07 0.62 0.31]]
 the -> [0. 1.]
 cat -> [3.22 0.2 ]
 sat -> [2.79 0.99]

Row 1 is [1, 0, 0]: with the causal mask, the first token can only attend to itself. Row 2, "cat", splits 0.2 and 0.8 between "the" and itself. Row 3 is the hand calculation above.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. With sat selected, the row reads 0.07, 0.62, 0.31 and a masked x, matching the worked example. Drag the key of the (pink) toward the amber query of sat and watch the weight move from cat to the.
  2. Set vector length to 3. The same directions now give much larger dot products, and sat puts 89 percent on cat. Then set divide by sqrt(d) to no: 95 percent. Long vectors and missing scaling make softmax nearly winner-take-all.
  3. Select the and switch the mask to none. Without the causal mask the first token attends to later tokens too (0.375 to itself, 0.375 to sat): fine for an encoder, cheating for a model learning to predict the next token.

Going deeper

Attention is a soft dictionary lookup

A Python dictionary takes a key and returns exactly one value. Attention is a soft version: the query is compared with every key, and instead of returning one value it returns a blend of all values weighted by how well each key matched. Because every step is differentiable, the model can learn what to ask for (), how to be found () and what to hand over () by gradient descent.

The cost: quadratic in length

The score table has entries. Double the context and the table quadruples: 1,000 tokens give 1 million scores per head per layer; 100,000 tokens give 10 billion. This is the central cost of long contexts, and the reason for the KV cache and the long-context tricks in Context windows and the KV cache . Optimised kernels such as FlashAttention compute exactly the same result without storing the whole table in slow memory.

What attention does not do

Attention mixes information between tokens, but it is only a weighted average: by itself it cannot do much computation on what it gathers. That is the job of the feed-forward network that follows it in every transformer block ( The transformer block ). One attention operation also has only one set of weights per token; running several in parallel lets a model track several relationships at once, which is multi-head attention ( Multi-head attention ).

Attention weights are not explanations

It is tempting to read the weight matrix as "what the model was thinking about". Be careful. Values can be small or cancel out, information has already mixed in earlier layers, and many heads are combined later. Attention maps are a useful hint for debugging, not proof of why a model produced an answer.

Trade-offs

OptionWins whenLoses when
Full self-attention Any token can use any other directly, in one step; highly parallel on GPUs Compute and memory grow with the square of the sequence length
Recurrent networks (RNN, LSTM) Constant memory per step; natural for streams Information from far back must survive many steps; sequential, so slow to train
Local or sliding-window attention Linear cost in length; good for long documents Distant tokens connect only indirectly, through several layers

Common mistakes

  • Forgetting the scaling, which makes softmax saturate and training unstable as dimensions grow.
  • Applying the causal mask after the softmax instead of before. Zeroing weights after softmax leaves rows that no longer sum to 1; mask scores with first.
  • Confusing keys and values. Keys decide who is attended to; values are what gets passed on.
  • Assuming the weights are symmetric. Token A's attention to B () is generally different from B's attention to A ().
  • Treating attention maps as faithful explanations of model behaviour.

In the wild

Vaswani et al., 2017 (paper)

Attention Is All You Need (opens in a new tab)

The paper introduced the transformer, built entirely from attention and feed-forward layers with no recurrence. It defines scaled dot-product attention as softmax of Q times K transposed over the square root of d_k, times V, and explains the scaling: for large d_k the dot products grow large and push softmax into regions with extremely small gradients. Everything in this lesson is that one equation.

Jay Alammar

The Illustrated Transformer (opens in a new tab)

A widely used visual walkthrough of the original transformer. It explains self-attention with the sentence "The animal didn't cross the street because it was too tired", derives query, key and value vectors from learned matrices (64-dimensional from 512-dimensional embeddings in the original model), and divides the scores by 8, the square root of 64, before the softmax. A good second explanation if you want more pictures.

Check yourself

  1. In attention, what decides how much token i draws from token j?

    Show answer

    B. The dot product of query i and key j, after scaling and softmax Queries are matched against keys; softmax over those scores gives the weights. Values only determine what is passed on.

  2. Raw scores for one token are [2, 0, 0]. What are the softmax weights, roughly?

    Show answer

    B. [0.79, 0.11, 0.11] e^2 = 7.39 and e^0 = 1, total 9.39. Weights are 7.39 / 9.39 = 0.79 and 1 / 9.39 = 0.11 each.

  3. Why are scores divided by sqrt(d_k)?

    Show answer

    B. Dot products grow with dimension; without scaling, softmax saturates and gradients become tiny The typical size of a dot product of d_k-dimensional vectors grows like sqrt(d_k). Dividing keeps softmax in a range where it still learns.

  4. In a 4-token sequence with a causal mask, how many entries of the 4 x 4 weight matrix are forced to zero?

    Show answer

    C. 6 All entries above the diagonal: 3 + 2 + 1 = 6. The 10 entries on and below the diagonal remain.

  5. A team zeros out the future weights after softmax instead of masking scores before it. What goes wrong?

    Show answer

    B. Rows no longer sum to 1, so the outputs are scaled incorrectly Softmax had already spread probability onto future tokens; removing it afterwards leaves rows summing to less than 1. Mask with minus infinity before softmax.

Recap

  • Each token produces a query (what it looks for), a key (how it is found) and a value (what it hands over).
  • Scores are query-key dot products, scaled by sqrt(d_k) so softmax does not saturate.
  • Softmax turns each row of scores into weights summing to 1; the output is the weighted sum of values.
  • A causal mask sets future scores to minus infinity so language models cannot peek ahead.
  • The n x n score table makes attention quadratic in sequence length.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up