Seq2seq and attention

The big idea

Translation does not map one word to one word. "the black cat sleeps" becomes "le chat noir dort": the adjective moves, and in other sentences words split, merge or disappear. Input and output are sequences of different lengths and different orders. A sequence-to-sequenceA model that reads an input sequence with an encoder and generates an output sequence with a decoder, one token at a time; also called an encoder-decoder. (seq2seq) model handles this with two networks: an encoder that reads the source sentence and a decoder that writes the translation one word at a time.

The first seq2seq models made the encoder squeeze the whole sentence into one vector before the decoder started. Imagine an interpreter who must listen to an entire speech, memorize it, and only then translate, never allowed to look back. Short speeches go fine; long ones fall apart. Now give the interpreter the transcript and let them glance at the relevant line for every sentence they produce. That glance is attentionA mechanism that lets a model compute, for each output step, a weighted average of all input positions, with weights from a softmax over learned relevance scores., and it became the core of the transformer.

How it works

Step 1: Encoder and decoder

The encoder is an RNN, usually an LSTM or GRU ( LSTM and GRU ), that reads the source and produces a hidden state per word. The decoder is a second RNN that generates the target one token at a time. At each step it takes the previous output word, updates its state and predicts a softmax distribution ( Multi-class and multi-label classification ) over the target vocabulary. It stops when it emits an end-of-sentence token, so the output length is free.

During training the decoder is fed the correct previous word rather than its own guess, a trick called teacher forcingTraining a sequence decoder by feeding it the true previous token at each step instead of its own prediction, which makes training faster and more stable.. At inference time it must use its own predictions, usually with beam search to keep several candidate translations alive.

Sutskever, Vinyals and Le (2014) showed this works at scale: a multilayer LSTM maps the input sequence to a vector of fixed dimensionality and another deep LSTM decodes the target from it. On the WMT'14 English to French task they report a BLEU score of 34.8, against 33.3 for a phrase-based statistical system. They also found that reversing the order of the source words improved results markedly, because it created many short-term dependencies between source and target. That trick is a hint: the model struggled to connect words that were far apart.

Step 2: The bottleneck

Bahdanau, Cho and Bengio (2014) named the problem. They conjectured that "the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture". A 300-dimensional vector has to hold a 5-word sentence and a 50-word sentence alike. In their experiments the basic encoder-decoder's quality dropped sharply as sentences got longer, while their attention model showed, in their words for the largest version, "no performance deterioration even with sentences of length 50 or more".

Step 3: Attention as a soft lookup

Their fix: keep all the encoder states $h_1,\dots ,h_n$ (they used a bidirectional RNN, so each state describes its word in the context of the whole sentence), and let the decoder look up what it needs at every output step $i$.

1. Score each encoder state against the decoder's current state $s_{i-1}$: $e_{ij}=a(s_{i-1},h_j)$, where $a$ is a small learned function.
2. Normalize the scores into weights with a softmax: ${\alpha }_{ij}=\frac{\exp (e_{ij})}{{\sum }_k\exp (e_{ik})}$. They are positive and sum to 1.
3. Read the weighted average, the context vectorThe weighted average of encoder states, using the attention weights, that the decoder receives at one output step.: $c_i={\sum }_j{\alpha }_{ij}{h}_j$, and feed it to the decoder with its state.

A worked example. Writing "noir", suppose the scores for the four English words are 1, 3, 0.5 and 0. The exponentials are 2.72, 20.09, 1.65 and 1, summing to 25.45, so the weights are 0.11, 0.79, 0.06 and 0.04. The context is 79 percent the encoder state of "black". It is a soft lookupA lookup that returns a weighted blend of all stored values according to how well each key matches the query, instead of exactly one value.: a dictionary lookup returns exactly one entry, attention returns a blend weighted by relevance. Because every step is differentiable, the scoring function is trained jointly with everything else by backpropagation, with no alignment labels: the model learns where to look because looking in the right place lowers the translation loss.

Step 4: Scoring functions

Bahdanau's $a$ is a feed-forward network on both states, often called additive attentionAttention whose score is a small feed-forward network applied to the decoder state and an encoder state, as in Bahdanau et al. 2014.. Luong, Pham and Manning (2015) compared simpler choices for the score of decoder state $h_t$ against encoder state ${\bar{h}}_s$:

• dot: $h_t^{\top }{\bar{h}}_s$, no parameters at all;
• general: $h_t^{\top }{W}_a{\bar{h}}_s$, a learned bilinear form;
• concat: $v_a^{\top }\tanh (W_a[h_t;{\bar{h}}_s])$, essentially the additive form.

They observed that dot "works well for the global attention" (attending over all source words). Transformers use the dot product, divided by the square root of the vector size so the softmax does not saturate as vectors get longer ( Self-attention, step by step ).

Alignment heatmaps

Stack the weight vectors for every output word and you get a matrix: one row per target word, one column per source word. Plotted as a heatmap, it shows which source words the model used for each output word, an alignmentA matrix of attention weights with one row per output token and one column per input token, often drawn as a heatmap to show which inputs each output used. learned without any alignment labels. For mostly monotone language pairs it runs down the diagonal, with detours where word order differs. Bahdanau et al. show a sentence where French puts the adjectives after the noun and the model correctly aligns "zone" with "Area", jumping over the two words "European Economic".

Try it

1. Step through the rows with Next. Most of the bright cells sit on the diagonal, but "chat" and "noir" cross over "black cat", and "chaise" and "rouge" cross over "red chair". That crossing is the reordering a fixed left-to-right alignment could not express.
2. Select "la". Part of its weight goes to "chair": in French the article must agree with the noun's gender, so the model needs to know the noun before choosing "la" over "le".
3. Drag the temperature to 0.2: every row becomes almost one-hot, a hard alignment. Drag it to 4: the weights spread out and the context becomes a blur of the whole sentence, which is close to the one-vector bottleneck again.

Going deeper

Code: attention and the bottleneck in numpy

import numpy as np
rng = np.random.default_rng(0)
src = "the black cat sleeps on the red chair".split()
tgt = "le chat noir dort sur la chaise rouge".split()
align = [0, 2, 1, 3, 4, 5, 7, 6]            # which source word each target word translates

# Pretend encoder: one 16-dim state per source position (word vector + a little context noise)
vocab = {w: rng.normal(0, 1, 16) for w in sorted(set(src))}
H = np.array([vocab[w] + 0.3 * rng.normal(0, 1, 16) for w in src])

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

# Pretend decoder: its query at step i resembles the encoder state it needs (a trained model learns this)
Q = np.array([H[j] + 0.5 * rng.normal(0, 1, 16) for j in align])

print("target   top source word   weight")
for i, q in enumerate(Q):
    a = softmax(H @ q / np.sqrt(16))       # scaled dot-product scores -> weights over the 8 source words
    context = a @ H                        # the context vector: a weighted average of encoder states
    j = int(a.argmax())
    print(f"{tgt[i]:8s} {src[j]:17s} {a[j]:.2f}")

# The bottleneck: from ONE fixed vector, which source words can you still recover?
def recover(v):                            # nearest source position by cosine similarity
    return int(np.argmax(H @ v / np.linalg.norm(H, axis=1)))
fixed = H.mean(axis=0)                     # a single summary vector for the whole sentence
hits_fixed = sum(recover(fixed) == j for j in align)
hits_attn = sum(recover(softmax(H @ q / 4) @ H) == j for q, j in zip(Q, align))
print(f"words recovered from one fixed vector: {hits_fixed} of {len(tgt)}")
print(f"words recovered with attention:        {hits_attn} of {len(tgt)}")

target   top source word   weight
le       the               0.59
chat     cat               0.62
noir     black             0.69
dort     sleeps            0.63
sur      on                0.73
la       the               0.46
chaise   chair             0.62
rouge    red               0.94
words recovered from one fixed vector: 1 of 8
words recovered with attention:        8 of 8

The encoder states here are synthetic (a vector per word plus noise), and each decoder query is built to resemble the state it needs, which is what training would learn to do. Scaled dot-product scores and a softmax then put the largest weight on the right English word for every French word, including the swapped adjective-noun pairs.

The last two lines measure the bottleneck. Average all encoder states into one fixed vector and ask which source word it is closest to: you get one answer for the whole sentence, so only 1 of the 8 target words finds its source. With attention, each output step builds its own context, and all 8 recover the right source word. A real fixed-vector model is trained to pack more than an average, but the vector stays the same size however long the sentence is.

From attention to transformers

Rename the pieces. The decoder state that asks the question is a queryIn attention, the vector that represents what the current position is looking for; it is compared against every key.; what it is compared against is a key; what gets averaged is a value. In Bahdanau's model the keys and values are both the encoder states. Two steps lead from here to the transformer:

• Self-attention: let every word of a sentence attend to every other word of the same sentence. Each word gets a context-dependent representation, which is the fix for the one-vector-per-word limit of static embeddings ( Word embeddings ).
• Drop the recurrence: if every position can reach every other position directly, the RNN is no longer needed. Vaswani et al. (2017) proposed "the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely", and reported translation models that were "more parallelizable and requiring significantly less time to train".

The path between word 1 and word 50 goes from 49 recurrent steps to one attention step, and all positions are computed at once on a GPU. The AI track picks up from here: Self-attention, step by step builds self-attention step by step and The transformer block assembles the transformer block.

Reading heatmaps with care

Heatmaps are a useful debugging view, for example to spot a model that skips or repeats part of the source. But attention weights are not a faithful explanation of a prediction. Jain and Wallace (2019) found learned attention weights frequently uncorrelated with gradient-based measures of feature importance, and very different attention distributions that yield equivalent predictions. Treat a heatmap as a hint about information flow, not proof of why the model decided.

Trade-offs

Common mistakes

• Thinking attention picks one source word. It returns a weighted average; one-hot weights are a special case.
• Forgetting that the weights for each output step sum to 1, so "more attention on X" always means less on everything else.
• Treating alignment heatmaps as ground-truth explanations of the model's decision.
• Training with teacher forcing only and being surprised that errors compound when the decoder must consume its own outputs.
• Comparing dot-product scores across vectors of very different sizes without scaling, so the softmax saturates.

In the wild

Attention helped make neural machine translation work at production scale. Google's 2016 neural machine translation paper describes a deep LSTM encoder-decoder with 8 encoder and 8 decoder layers using attention and residual connections, the RNN-plus-attention design of this lesson at production scale. Other sequence-to-sequence tasks adopted the same encoder-decoder-with-attention pattern, and every large language model today is built from the self-attention that grew out of it.

Check yourself

Recap