How LLM inference works

The big idea

You send a 2,000-token prompt to a model and ask for a 300-token answer. The first word appears after a fraction of a second, then the rest streams in at a steady pace. Those are two different jobs with two different bottlenecks. Reading the prompt is one big parallel computation. Writing the answer is 300 small steps, and each one has to haul the entire model through the chip's memory system to produce a single token.

Picture a scribe working from a huge reference book kept in a vault. To read your request, they bring the book out once and go through all of it in one sitting. To write the reply, they must fetch the whole book from the vault for every single word, glance at it briefly, write the word, and put it back. The fetching, not the writing, takes the time. And if 30 people are waiting, the scribe should write one word for each of them on every trip, because the trip costs the same either way.

That scribe is the whole story of LLM serving: decoding is limited by how fast weights move from memory, and batching many users onto each trip is how servers get their throughput.

How it works

Prefill and decode

PrefillThe first phase of generation: the whole prompt is processed in one parallel pass, filling the KV cache and producing the first token. Usually limited by compute. runs every prompt token through the model at once. It is one big batch of matrix multiplications, the kind of work accelerators are built for.

DecodeThe second phase of generation: one token at a time, each step reading all weights and the whole KV cache to produce a single token. Usually limited by memory bandwidth. then produces the answer one token per step. Each step depends on the token before it, so the steps cannot run in parallel for one sequence. Thanks to the KV cache, each step only computes the new token's keys and values, but it still has to read every weight of the model.

The three numbers that matter

• TTFTTime to first token: how long a user waits from sending a request until the first output token appears. Driven mostly by prefill and queueing., time to first token: the wait before anything appears. It grows with prompt length and with time spent queueing.
• TPOTTime per output token: the average time between consecutive output tokens for one request, which sets the streaming speed a user sees. Also called inter-token latency., time per output token: the streaming pace. A TPOT of 100 ms is 10 tokens per second.
• ThroughputThe number of output tokens a server generates per second across all requests. It sets how many users one machine can serve and so the cost per token.: total output tokens per second across all users on a server. It decides your cost per token.

End-to-end latency is roughly $\text{TTFT}+\text{TPOT}\times \text{output tokens}$. With a TTFT of 0.3 s, a TPOT of 20 ms and a 500-token answer: $0.3+0.02\times 500=10.3$ seconds. For a chat interface, the 0.3 s and the steady stream matter most, since users read while it writes (see Streaming responses to users, later in this module). For a batch job that summarises a million documents, nobody watches the stream and throughput is all that counts.

Why decode is memory-bound

A forward pass costs about $2N$ floating-point operations (FLOPs) per token, where $N$ is the parameter count: one multiply and one add per weight. Take an 8-billion-parameter model in 16-bit precision, on an accelerator that moves 3 TB/s from memory and computes 1,000 TFLOPS (round numbers of the same order as today's data-centre GPUs).

• Moving the weights: $8\times 10^9\times 2$ bytes $=16$ GB per step. At 3 TB/s that takes $16/3000\approx 5.3$ ms.
• Computing one token: $2\times 8\times 10^9=16$ GFLOP. At 1,000 TFLOPS that takes $0.016$ ms.

The compute units finish in 0.016 ms and then wait more than 5 ms for the next weights. Decode for one user is memory-boundLimited by how fast data can be moved from memory rather than by how fast the chip can compute. Faster arithmetic does not help; moving fewer bytes does.: its speed is set by bytes moved, not FLOPs. That caps this setup at about $1/5.3\text{ ms}\approx 190$ tokens per second for a single user, however fast the arithmetic is.

The useful ratio is arithmetic intensityFLOPs performed per byte moved from memory. Below the hardware's ratio of compute to bandwidth, work is memory-bound; above it, compute-bound.: FLOPs done per byte read. Decode for one sequence does 2 FLOPs per 2-byte weight, so about 1 FLOP per byte. The accelerator can do $1000/3\approx 333$ FLOPs in the time it reads one byte. Anything far below 333 leaves compute idle.

Prefill is compute-bound

Prefill reads the weights once and uses them for all 2,000 prompt tokens: $2\times 8\times 10^9\times 2000=32$ TFLOP, about 32 ms of compute, against 5.3 ms of weight reading. Now arithmetic is the bottleneck. This is why TTFT grows roughly in proportion to prompt length once prompts are long, and why a server processes prompt tokens far faster than it generates output tokens: 2,000 prompt tokens take about 32 ms, while 2,000 output tokens for one user would take over 10 seconds.

Batching: sharing the trip

If 32 users are decoding at once, the server reads the weights once per step and computes one token for each of them. The read still takes 5.3 ms; the compute grows to 0.5 ms. Each user still sees about 190 tokens per second, but the server now produces about 6,000. Throughput grows almost linearly with batch size until the batch is large enough (around 333 here) that compute becomes the bottleneck, after which each user's TPOT starts to rise.

So batch size is a dial between per-user speed and total throughput. In practice the KV cache limits the batch first: every sequence in the batch needs its own cache in GPU memory (the next lesson, GPU memory math, does that arithmetic). Smarter batching and fewer bytes per weight, both later in this module, push the limits further.

Try it

1. With the defaults (8B, 16-bit, batch 1), compare the two bars. How much of each decode step is the chip actually computing?
2. Switch to 4-bit weights. Which number changes, TTFT or TPOT, and why?
3. Raise the batch from 1 to 128, then to 512. Watch throughput, TPOT and TTFT. At what point does adding users start to slow each user down?

The same estimate in Python, for a range of batch sizes:

# Roofline estimate for decoding with an 8B-parameter model in 16-bit weights
# on an illustrative accelerator: 3 TB/s memory bandwidth, 1,000 TFLOPS.
# One decode step reads every weight once and does about 2 FLOPs per
# parameter per sequence in the batch. KV cache traffic is ignored here.
BANDWIDTH = 3e12      # bytes per second
COMPUTE = 1e15        # FLOPs per second
PARAMS = 8e9
BYTES_PER_PARAM = 2   # 16-bit

weight_bytes = PARAMS * BYTES_PER_PARAM
read_time = weight_bytes / BANDWIDTH            # same for any batch size
print(f"weights: {weight_bytes / 1e9:.0f} GB, read time per step: {read_time * 1e3:.2f} ms")
print(f"compute needs {COMPUTE / BANDWIDTH:.0f} FLOPs per byte to stay busy\n")

print(f"{'batch':>6}{'FLOP/byte':>11}{'TPOT ms':>9}{'tok/s/user':>12}{'tok/s total':>13}")
for batch in [1, 8, 32, 128, 256, 512, 1024]:
    compute_time = 2 * PARAMS * batch / COMPUTE
    tpot = max(read_time, compute_time)         # the slower one wins
    intensity = 2 * PARAMS * batch / weight_bytes
    bound = "memory" if read_time >= compute_time else "compute"
    print(f"{batch:>6}{intensity:>11.0f}{tpot * 1e3:>9.2f}{1 / tpot:>12.0f}"
          f"{batch / tpot:>13,.0f}  {bound}")

weights: 16 GB, read time per step: 5.33 ms
compute needs 333 FLOPs per byte to stay busy

 batch  FLOP/byte  TPOT ms  tok/s/user  tok/s total
     1          1     5.33         188          188  memory
     8          8     5.33         188        1,500  memory
    32         32     5.33         188        6,000  memory
   128        128     5.33         188       24,000  memory
   256        256     5.33         188       48,000  memory
   512        512     8.19         122       62,500  compute
  1024       1024    16.38          61       62,500  compute

Up to a batch of 256, every user gets the same 5.33 ms per token while total throughput rises 256-fold, because the weight read is shared. At 512 the step becomes compute-bound: throughput stops growing and each user slows down. Real servers hit KV cache memory limits and inefficiencies well before this ideal point.

Going deeper

Measuring how well you use the hardware

Two utilisation metrics tell you how close a server is to these limits:

• Model bandwidth utilisation (MBU), for memory-bound decode: achieved bandwidth divided by peak bandwidth, where achieved bandwidth is (weight bytes + KV cache bytes) / TPOT. A 7B model in 16-bit with a TPOT of 14 ms moves 14 GB in 14 ms, which is 1 TB/s; on hardware with a 2 TB/s peak, MBU is 50 percent.
• Model FLOPs utilisation (MFU), for compute-bound work such as prefill and large batches: achieved FLOPs per second divided by peak.

What this estimate leaves out

• The KV cache is read too. Each decode step reads every cached key and value of every sequence in the batch. At long contexts and large batches this traffic can rival the weights themselves ( Context windows and the KV cache ), so TPOT grows with context length.
• Attention FLOPs grow with context. The $2N$ rule ignores attention's own arithmetic, which grows with the number of tokens already in context.
• Prefill and decode compete. On a busy server, a new request's prefill can stall the decode steps of everyone else. Serving systems split long prefills into chunks, or run prefill and decode on separate machines, to keep TPOT smooth.
• Queueing. Under load, TTFT includes waiting for a slot, which often dominates.

Bigger models across several GPUs

A model too large for one GPU is split across several. Each GPU then reads only its share of the weights per step, so the combined bandwidth rises, but the GPUs must exchange activations at every layer. The PaLM inference work in the case study below shows how the best way to split a model depends on whether you optimise for latency or throughput.

Trade-offs

Common mistakes

• Quoting one "tokens per second" number without saying whether it is per user or across the whole server.
• Expecting a GPU with more FLOPS to speed up single-user decoding, when bandwidth is the limit.
• Ignoring TTFT for long prompts, so a chat feature feels slow even though streaming is fast.
• Benchmarking with one request at a time and assuming the same latency under production load.
• Forgetting the KV cache when estimating how many users fit on one GPU.

In the wild

Check yourself

Recap