AI AI-05.02
Embeddings and positional encoding
How token ids become vectors through an embedding table, why attention needs position injected, and how sinusoidal, learned and rotary (RoPE) positions work.
The big idea
The tokenizer hands the model a list of integers such as [3, 7, 3]. Integers are just labels: token 7 is not "more" than token 3. The first thing a transformer does is swap each id for a learned vector, its token embedding, so that tokens with related meanings can start out near each other.
That alone loses something vital. "Dog bites man" and "man bites dog" contain the same three tokens, and the attention mechanism at the heart of the transformer ( Self-attention, step by step ) treats its input as an unordered set: shuffle the tokens and each one computes exactly the same thing. So the model must be told where each token is. That is positional encodingInformation about each token's place in the sequence, added to or mixed into its vector so that order-blind attention can tell positions apart..
Think of a pile of photos from a holiday. Each photo shows what happened, but you cannot tell the story until you write the time on the back of each one. Token embeddings are the photos; positional encoding is the timestamp.
How it works
Step 1: The embedding table
The model holds an embedding tableA learned matrix with one row per vocabulary token; looking up a token id returns its row, the token's embedding vector.: a matrix with one row per token in the vocabulary and one column per model dimension. With a vocabulary of 50,000 and a model width of 4,096, that is 50,000 x 4,096 = about 205 million numbers. Looking up token 3 simply returns row 3. Mathematically this is a one-hot vector times the matrix, but nobody computes it that way; it is an array index.
The rows start random and are learned during training like every other weight. Many models reuse the same matrix at the very end, to turn the final vectors back into scores over the vocabulary (called weight tying), which the original transformer did too.
Step 2: Why order must be injected
Attention compares every token with every other token using dot products. Nothing in that computation knows that token 2 came before token 3. If the same token appears twice, as dog does in "dog bites dog", both copies start with identical vectors and would be treated identically. Position must be mixed into the vectors so that the two copies, and the two word orders, become different.
Step 3: Sinusoidal positions (the original recipe)
The original transformer added a fixed vector to each token embedding, built from sine and cosine waves at different frequencies:
The first dimensions oscillate fast, changing noticeably from one position to the next; later dimensions oscillate so slowly that they distinguish only distant positions. Together they give every position a unique fingerprint, like the hands of a clock: the second hand tells nearby moments apart, the hour hand tells far-apart ones apart. For example, at position 5 dimension 0 is .
The authors chose these waves because a fixed offset between two positions corresponds to a simple linear transformation of their encodings, which they hoped would make relative positions easy to learn. They also tried learned position vectors and found the results nearly identical, but kept sinusoids in the hope of handling sequences longer than those seen in training.
Step 4: Learned positions
The simplest alternative is a second table: one learned vector per position, up to a maximum length, added to the token embedding. Early GPT and BERT models worked this way. It is flexible, but it has a hard ceiling: there is simply no vector for position 2,049 if training stopped at 2,048.
Step 5: Rotary position embeddings (RoPE)
What attention really needs is relative position: how far apart a query and a key are. RoPERotary position embedding: encodes position by rotating each pair of query and key dimensions by an angle proportional to the token's position, so their dot product depends on relative distance. builds that in directly. Instead of adding a vector to the embedding, it rotates the query and key vectors inside every attention layer. Each pair of dimensions is treated as a 2D point and rotated by an angle of position x a frequency, with fast and slow frequencies as in the sinusoids.
The trick is in the dot product. Rotate the query by and the key by , and their dot product depends only on the difference, :
So a query at position 105 looking at a key at 102 scores exactly like a query at 5 looking at a key at 2. RoPE is used by most current open-weights language models, and its frequencies can be rescaled to stretch a model to longer contexts ( Context windows and the KV cache ).
Try it
Try this:
- In sinusoidal positions, move the position slider slowly. The left columns of the selected row flicker from step to step while the right columns barely change: fast and slow waves.
- Switch to RoPE rotation. With the query at 5 and the key at 2 the score is 2.235. Move both sliders up by 10, to 15 and 12: the arrows swing round, but the score stays 2.235, because the offset is still 3.
- Set both positions equal (2 and 2, or 5 and 5): the score is always 2.47. Now try an offset of 9 in each direction, 9 and 0, then 0 and 9: the scores, -2.48 and -2.46, differ slightly, so the model can tell "9 before" from "9 after".
All three ideas in numpy, including a check that RoPE scores depend only on the offset:
# Token embeddings, sinusoidal positions and RoPE in a few lines of numpy.
import numpy as np
rng = np.random.default_rng(0)
vocab, d = 10, 8
E = rng.normal(size=(vocab, d)) # embedding table: a row per token id
ids = np.array([3, 7, 3]) # "dog bites dog": token 3 twice
x = E[ids] # lookup = pick rows
one_hot = np.eye(vocab)[ids]
print("lookup equals one-hot @ E:", np.allclose(x, one_hot @ E))
print("same token, same vector: ", np.allclose(x[0], x[2]))
def sinusoidal(n_pos, d):
pos = np.arange(n_pos)[:, None]
freq = 1 / 10000 ** (np.arange(0, d, 2) / d)
pe = np.zeros((n_pos, d))
pe[:, 0::2] = np.sin(pos * freq)
pe[:, 1::2] = np.cos(pos * freq)
return pe
h = x + sinusoidal(3, d) # add position to each token
print("after adding positions: ", np.allclose(h[0], h[2]))
# RoPE: rotate each pair of dimensions by an angle that grows
# with position.
def rope(v, pos, base=10000):
out = v.copy()
for i in range(0, len(v), 2):
a = pos / base ** (i / len(v))
c, s = np.cos(a), np.sin(a)
x0, x1 = v[i], v[i + 1]
out[i], out[i + 1] = c * x0 - s * x1, s * x0 + c * x1
return out
q, k = rng.normal(size=d), rng.normal(size=d)
for m, n in [(5, 2), (105, 102), (1005, 1002), (5, 4)]:
score = rope(q, m) @ rope(k, n)
print(f"query at {m:>4}, key at {n:>4} (offset {m - n}):"
f" score {score:+.4f}")Output
lookup equals one-hot @ E: True
same token, same vector: True
after adding positions: False
query at 5, key at 2 (offset 3): score +0.9782
query at 105, key at 102 (offset 3): score +0.9782
query at 1005, key at 1002 (offset 3): score +0.9782
query at 5, key at 4 (offset 1): score -0.8987 Positions 5 and 2, 105 and 102, and 1005 and 1002 give exactly the same score, +0.9782. Change the offset to 1 and the score changes.
Going deeper
Absolute versus relative
Sinusoidal and learned encodings are absolute: they label each token with its own position and leave the model to work out distances. RoPE and methods such as ALiBi, which subtracts a penalty from attention scores in proportion to distance, are relative: the attention score itself is shaped by how far apart two tokens are. Relative schemes generalise better to positions the model has rarely seen, which matters for long contexts.
Where the information lives
Additive encodings mix position into the residual stream once, at the bottom, and every layer must carry it upwards. RoPE re-applies position inside every attention layer and touches only queries and keys, not values, so the content being moved around is not polluted with position codes. This is one reason it became the default.
Embedding size and cost
The embedding table is often one of the largest single matrices in a small model, because it scales with vocabulary size times width. In large models the transformer blocks dominate the parameter count instead ( The transformer block ).
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Sinusoidal (fixed) | No parameters; defined for any position | Absolute; in practice extrapolates poorly far beyond training lengths |
| Learned absolute | Simple; the model fits whatever position pattern helps | Hard maximum length; positions near the limit get less training |
| RoPE (rotary) | Relative distances built into attention; context can be extended by rescaling | Still degrades beyond trained lengths without care; slightly more complex |
| ALiBi (distance penalty) | Very simple relative bias; extrapolates gracefully | Fixed preference for nearby tokens that some tasks do not want |
Common mistakes
- Thinking attention knows word order by itself. Without positional information, a transformer sees a bag of tokens.
- Assuming the embedding row for a token carries its position. The same token at different positions has the same token embedding; position comes from a separate mechanism.
- Running a model on sequences much longer than it was trained on and expecting it to work because the position formula is defined there.
- Applying RoPE to value vectors, or to queries but not keys. The relative property comes from rotating both queries and keys.
In the wild
Su et al., 2021 (paper)
RoFormer: Enhanced Transformer with Rotary Position Embedding (opens in a new tab)The paper surveys ways of feeding position into transformers and proposes Rotary Position Embedding, which encodes absolute position with a rotation matrix while making self-attention depend explicitly on relative position. It highlights flexibility in sequence length and a decay of inter-token dependency as distance grows. The authors evaluated it on long-text classification benchmarks, and the method is now standard in open-weights language models.
Vaswani et al., 2017 (paper)
Attention Is All You Need (opens in a new tab)Because the transformer has no recurrence or convolution, the authors had to inject position explicitly. They added sine and cosine encodings of different frequencies to the input embeddings, reported that learned position embeddings gave nearly identical results, and kept the sinusoids because they may let the model extrapolate to longer sequences. They also shared one weight matrix between the embedding layers and the output layer.
Check yourself
-
What does an embedding lookup for token id 7 return?
Show answer
B. Row 7 of a learned matrix with one row per vocabulary token The embedding table holds one learned vector per token. Lookup is indexing, equivalent to multiplying a one-hot vector by the table.
-
Why does a transformer need positional information at all?
Show answer
B. Attention compares tokens with dot products and ignores their order, so 'dog bites man' and 'man bites dog' would look the same Self-attention is permutation-invariant. Order must be added to the vectors, or built into the attention scores.
-
With RoPE, a query at position 40 and a key at 37 score s. What does a query at 140 with a key at 137 score, for the same q and k?
Show answer
A. s RoPE rotates both by angles proportional to their positions; the dot product depends only on the difference, 3 in both cases.
-
A model with learned absolute positions was trained up to 2,048 tokens. What happens at token 3,000?
Show answer
B. There is no learned position vector for it, so the model cannot represent that position without changes A learned table has a fixed number of rows. Beyond it, there is nothing to look up; this hard limit is a key weakness of learned absolute positions.
Recap
- Token ids index a learned embedding table: one vector per vocabulary token.
- Attention ignores order, so position must be injected or built into the attention scores.
- Sinusoidal encodings add waves of many frequencies; learned encodings add a trained vector per position up to a maximum.
- RoPE rotates queries and keys by position, so attention scores depend on relative distance.
- Relative schemes such as RoPE dominate today and are the starting point for long-context extensions.
Next up
- Build production LLM apps learning path