Skip to content
learn
  1. Learn
  2. Artificial Intelligence
  3. Transformers and LLM Internals

AI AI-05.05

The transformer block

Attention plus a feed-forward network, wrapped in residual connections and layer norm: the residual stream, pre-norm, stacking, and parameter-count math.

Intermediate 14 min Last reviewed

The big idea

Attention lets tokens exchange information, but it only computes weighted averages. A model also needs to think about what it gathered: combine features, apply facts it has memorised, transform "subject of a verb" plus "plural" into a prediction. A transformer blockThe repeating unit of a transformer: multi-head attention followed by a position-wise feed-forward network, each wrapped in a residual connection and layer normalisation. pairs the two: an attention layer, where tokens talk to each other, followed by a feed-forward networkA small two-layer neural network applied to each token's vector separately: widen (often 4x), apply a nonlinearity, narrow back. Also called the MLP., where each token thinks on its own. A model is just this block stacked dozens of times.

The glue is the residual streamThe sequence of token vectors that flows through a transformer; each sub-layer reads from it and adds its output back, rather than replacing it.. Each token has one vector that flows from the bottom of the model to the top. Sub-layers do not replace it; they read from it and add their result back. Information that no layer touches passes straight through.

Think of a shared document passed down a line of editors. No editor rewrites the whole thing. Each reads it, adds a few notes in the margin, and passes it on. The document at the end contains the original text plus every editor's contributions, and any editor can build on the notes of the ones before.

How it works

A pre-norm transformer block with the residual stream as a highway On the left, the residual stream runs upward from the input token vectors through two addition points to the next block, glowing. On the right, a branch reads the stream through a layer norm into multi-head attention and adds the result back at the first addition point; a second branch reads through another layer norm into the feed-forward network and adds back at the second addition point. add add token vectors from the block below + + to the next block layer norm multi-head attention tokens talk layer norm feed-forward network each token thinks residual stream Figure 1. The residual stream (left, glowing) runs straight up through the block. Each sub-layer reads a normalised copy, computes, and adds its result back at a + junction: attention first (tokens talk), then the feed-forward network (each token thinks).

Step 1: Attention sub-layer

Take the stream , normalise a copy, run multi-head attention on it, and add the result back:

This is where information moves between positions: "it" picks up features of "animal", a verb picks up its subject.

Step 2: Feed-forward sub-layer

Normalise again and apply the feed-forward network (often called the MLP) to every token separately:

widens each vector, typically by 4x, is a nonlinearity such as GELU, and narrows it back. Nothing crosses between tokens here. Studies of trained models suggest that much of the factual knowledge, "Paris is in France", is stored in these layers, which act a bit like a huge learned lookup table keyed on patterns in the stream.

Step 3: Layer normalisation

Layer normalisationAn operation that rescales each token's vector to mean 0 and standard deviation 1 across its features, then applies a learned scale and shift. It keeps activations in a stable range. rescales each token's vector to mean 0 and variance 1 (then applies a learned scale and shift). Without it, the size of the stream would drift as dozens of layers keep adding to it, and training would become unstable. Many current models use RMSNorm, a cheaper variant that skips subtracting the mean.

Where the norm goes matters. The original transformer normalised after each addition (post-norm). Most modern models normalise the input of each sub-layer and leave the stream itself untouched (pre-norm, as in Figure 1). Pre-norm keeps a clean path from the top of the model to the bottom, so gradients flow easily and deep stacks train reliably, usually with less careful learning-rate warm-up.

Step 4: Stack them

A model is an embedding layer, then identical blocks (each with its own weights), then a final layer norm and an output layer that turns each token's final vector into scores over the vocabulary. Small models have around a dozen blocks; large ones have many dozens. Here is the canonical full encoder-decoder layout from the literature, drawn in the pre-norm style used today. The decoder-only models behind chat assistants keep just the right-hand column without the cross-attention ( Encoder, decoder and decoder-only models ).

The full transformer architecture: on the left an encoder stack of Nx layers, each with layer norm, multi-headed self-attention, an addition, layer norm, a feed-forward network and another addition; on the right a decoder stack with masked self-attention, cross-attention to the encoder output and a feed-forward network, ending in a norm, a linear layer and predictions.
The full encoder-decoder transformer in pre-norm form. Each dashed box is a sub-layer with its residual connection (the arrow around it into a + circle). Image: dvgodoy, CC BY 4.0 via Wikimedia Commons, CC BY 4.0, via Wikimedia Commons

Try it

One pre-norm block in numpy, plus a parameter count for two model shapes:

block.pyruns with python3 + numpy
# A pre-norm transformer block in numpy, then the parameter count of a stack.
import numpy as np

rng = np.random.default_rng(0)
n, d = 4, 16                                   # 4 tokens, width 16

def layer_norm(x, eps=1e-5):
    mu = x.mean(-1, keepdims=True)
    var = x.var(-1, keepdims=True)
    return (x - mu) / np.sqrt(var + eps)       # (learned scale/shift omitted)

def attention(x, Wq, Wk, Wv, Wo):              # single head, causal
    q, k, v = x @ Wq, x @ Wk, x @ Wv
    s = q @ k.T / np.sqrt(d) + np.triu(np.full((n, n), -np.inf), 1)
    w = np.exp(s - s.max(1, keepdims=True))
    return (w / w.sum(1, keepdims=True)) @ v @ Wo

def mlp(x, W1, W2):                            # widen 4x, nonlinearity, narrow
    return np.maximum(0, x @ W1) @ W2

W = lambda *shape: rng.normal(size=shape) * 0.1
Wq, Wk, Wv, Wo = W(d, d), W(d, d), W(d, d), W(d, d)
W1, W2 = W(d, 4 * d), W(4 * d, d)

x = rng.normal(size=(n, d))                    # the residual stream
x1 = x + attention(layer_norm(x), Wq, Wk, Wv, Wo)  # read, add back
x2 = x1 + mlp(layer_norm(x1), W1, W2)              # read, add back
print("stream shape stays", x.shape, "->", x2.shape)
print(f"size of what each sub-layer added: attention "
      f"{np.linalg.norm(x1 - x):.2f}, MLP {np.linalg.norm(x2 - x1):.2f}, "
      f"stream {np.linalg.norm(x):.2f}")

def params(d, layers, vocab, ctx):
    attn, ffn = 4 * d * d, 2 * d * 4 * d       # ignoring biases and norms
    emb = vocab * d + ctx * d
    return attn * layers, ffn * layers, emb

for d_, L in [(768, 12), (4096, 32)]:
    a, f, e = params(d_, L, 50_257, 1024)
    total = a + f + e
    print(f"width {d_:>4}, {L} layers: attention {a / 1e6:7.1f}M  "
          f"MLP {f / 1e6:7.1f}M  embeddings {e / 1e6:5.1f}M  "
          f"total {total / 1e6:7.1f}M  (MLP share {f / total:.0%})")

Output

stream shape stays (4, 16) -> (4, 16)
size of what each sub-layer added: attention 0.79, MLP 1.79, stream 9.74
width  768, 12 layers: attention    28.3M  MLP    56.6M  embeddings  39.4M  total   124.3M  (MLP share 46%)
width 4096, 32 layers: attention  2147.5M  MLP  4295.0M  embeddings 210.0M  total  6652.5M  (MLP share 65%)

Try this:

  1. Check the shapes: the stream goes in as (4, 16) and comes out as (4, 16). Every block preserves the shape, which is why blocks can be stacked any number of times.
  2. Compare the sizes in the second line: each sub-layer adds a modest update to a stream that is much larger. Blocks refine the stream rather than replace it.
  3. Read the parameter counts. Use the formulas below to check one by hand before reading on.

Going deeper

Parameter-count math

Ignoring small biases and norm weights, one block has:

  • attention: , each , so ;
  • feed-forward: is and is , so ;
  • total per block, and for blocks.

Add the token embedding table, vocabulary x , and learned position vectors if the model has them. For width 768, 12 blocks, a 50,257-token vocabulary and 1,024 positions: million in the blocks plus 39.4 million in embeddings, about 124 million. At width 4,096 with 32 blocks the blocks hold about 6.4 billion parameters and the embeddings only 0.2 billion.

Where the parameters live in two model shapes On a log scale: for width 768 with 12 layers, attention has 28.3 million parameters, feed-forward 56.6 million and embeddings 39.4 million. For width 4096 with 32 layers, attention has 2,147.5 million, feed-forward 4,295.0 million and embeddings 210.0 million. millions of parameters width 768: attention 28.3M width 768: feed-forward 56.6M width 768: embeddings 39.4M width 4096: attention 2,147.5M width 4096: feed-forward 4,295.0M width 4096: embeddings 210.0M Figure 2. Parameter counts from the code above (log scale). Feed-forward layers always hold twice the attention weights; as models grow, embeddings become a small share and the feed-forward layers dominate (glowing).

Two useful consequences. A forward pass costs roughly 2 floating-point operations per parameter per token (one multiply and one add), which is how training and serving costs are estimated ( Pretraining: next-token prediction , How LLM inference works ). And since the feed-forward layers hold most weights, they are the natural place to add capacity cheaply with a mixture of experts ( Mixture of experts ).

Why residuals matter so much

Without residual connections, a signal would pass through dozens of transformations in sequence, and gradients would shrink or explode on the way back. With them, the gradient has a direct path through the additions to every layer, so a 100-block model is trainable. Residuals also make each block's job easier: it only needs to learn a useful change to the stream, and if it has nothing to add it can learn to add almost nothing.

Reading the stream

Interpretability researchers treat the residual stream as a shared communication channel: early layers write features, later layers read them. Projecting the stream of an intermediate layer through the output layer often already gives sensible next-token guesses, which sharpen as you go up the stack.

Trade-offs

OptionWins whenLoses when
Post-norm (original) Can reach slightly better results when it trains well Harder to train deep stacks; needs careful learning-rate warm-up
Pre-norm (common today) Stable training of very deep models; clean residual path Stream magnitude grows with depth; a final norm is required
Wider blocks, fewer layers More parallel work per layer; fewer sequential steps Less depth for multi-step composition of features
Narrower blocks, more layers More sequential processing steps Slower per token at inference; harder to parallelise

Common mistakes

  • Thinking attention holds most of the parameters. In a standard block the feed-forward network has twice as many.
  • Forgetting the residual connections when implementing a block from scratch; a deep stack without them barely trains.
  • Mixing up where the layer norm goes in pre-norm: it normalises the sub-layer's input, not the stream itself.
  • Counting parameters as and forgetting the embedding table, which dominates in small models with large vocabularies.
  • Assuming each block does a distinct, human-readable job. Features are spread across many layers.

In the wild

Where you will see this: every modern language model, from small on-device models to frontier models, is a stack of blocks like this one, usually decoder-only, pre-norm, with RMSNorm, a gated feed-forward variant and rotary positions. Vision transformers apply the same block to patches of an image, and speech and protein models reuse it unchanged. When a model card lists "layers", "hidden size" and "heads", it is describing these blocks.

Check yourself

  1. What does the residual connection do in a transformer block?

    Show answer

    B. Adds each sub-layer's output to its input, so information and gradients can pass straight through x <- x + f(LN(x)). The stream keeps everything that was there and gains the sub-layer's update.

  2. In one standard block of width d, how many weights do the feed-forward layers have?

    Show answer

    B. 8d^2 W_1 is d x 4d and W_2 is 4d x d: 4d^2 + 4d^2 = 8d^2, twice the attention's 4d^2.

  3. Which sub-layer moves information between token positions?

    Show answer

    C. Multi-head attention Attention is the only part where tokens read from other tokens. The feed-forward network and norms work on each token separately.

  4. A model has width 1,024, 24 blocks and a 32,000-token vocabulary. Roughly how many parameters, ignoring positions, biases and norms?

    Show answer

    B. About 335 million Blocks: 12 x 1,024^2 x 24 = 302 million. Embeddings: 32,000 x 1,024 = 33 million. Total about 335 million.

Recap

  • A block is attention (tokens talk) then a feed-forward network (each token thinks), each with a residual connection and layer norm.
  • The residual stream carries each token's vector up the stack; sub-layers read from it and add back.
  • Pre-norm normalises each sub-layer's input and keeps a clean residual path, which makes deep stacks stable.
  • A block has about 12 d^2 weights, two thirds of them in the feed-forward layers.
  • Models are this block stacked dozens of times, plus embeddings at the bottom and an output layer at the top.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up