Skip to content
learn
  1. Learn
  2. Artificial Intelligence
  3. Transformers and LLM Internals

AI-M05

Transformers and LLM Internals

What happens inside a large language model: tokens, embeddings, self-attention, transformer blocks, pretraining, sampling, the KV cache, experts and multimodal inputs, ending with a tiny GPT you build yourself.

12 lessonsabout 185 min

Large language models look like magic from the outside and like a stack of simple, well-understood parts from the inside. This module opens the box. You start where the model starts, with tokenization, then follow a token through the embedding table and positional encoding into self-attention, which you will compute by hand with three tokens before scaling it up to multiple heads and full transformer blocks.

With the architecture in place you will see how pretraining on next-token prediction turns that architecture into a capable model, why scaling laws made bigger models predictable, and how decoding settings such as temperature and top-p shape every answer. The advanced lessons cover what limits long contexts and makes generation memory-bound (the KV cache), how mixture-of-experts models get more capacity for the same compute, and how images and audio enter the same token stream. The module ends with a code-along that builds a tiny character-level GPT from scratch, tying every lesson together.

Lessons

  1. AI-05.01 Tokenization Language models read tokens, not characters or words. Beginner 14 min
  2. AI-05.02 Embeddings and positional encoding Each token id is looked up in an embedding table to get a vector. Intermediate 12 min
  3. AI-05.03 Self-attention, step by step Self-attention lets every token gather information from other tokens. Intermediate 22 min
  4. AI-05.04 Multi-head attention Multi-head attention runs several attention operations in parallel, each with its own query, key and value projections, so different heads can focus on different relationships, such as syntax or references. Intermediate 10 min
  5. AI-05.05 The transformer block A transformer block applies attention, then a feed-forward network applied to each token, each wrapped with a residual connection and layer normalization. Intermediate 14 min
  6. AI-05.06 Encoder, decoder and decoder-only models Encoder-only models like BERT see the whole input in both directions and excel at understanding tasks such as classification and embeddings. Intermediate 10 min
  7. AI-05.07 Pretraining: next-token prediction Pretraining teaches a model to predict the next token over huge amounts of text, and doing that well forces it to learn grammar, facts and patterns of reasoning. Intermediate 14 min
  8. AI-05.08 Decoding and sampling At each step a language model outputs a probability for every token. Beginner 12 min
  9. AI-05.09 Context windows and the KV cache Attention compares every token with every other, so cost grows quadratically with context length. Advanced 14 min
  10. AI-05.10 Mixture of experts A mixture-of-experts layer contains many feed-forward experts and a router that sends each token to only a few of them. Advanced 10 min
  11. AI-05.11 Multimodal models Multimodal models handle images, audio or video alongside text. Advanced 12 min
  12. AI-05.12 Build a tiny GPT from scratch Building a tiny GPT from scratch ties everything together: tokenize text, embed tokens with positions, stack transformer blocks with causal self-attention, predict the next token with cross-entropy loss, train with AdamW, and sample text with temperature. Advanced 40 min

Start with lesson 1