GPU memory math

The big idea

You have one GPU with 80 GB of memory and an 8-billion-parameter model. Will it run? Easily. How many users can it serve at once? That is the real question, and the answer can be 200 or 3 depending on how long their conversations are. The weights are a fixed cost you pay once. Every active request then needs its own KV cacheThe stored keys and values of every earlier token, per layer, kept during generation so each new token only computes its own key and value., and that is what runs out.

Think of a restaurant kitchen. The ovens and fridges (the weights) take a fixed share of the floor. Every table being served needs its own counter space for its order in progress (the KV cache), and a table with a long order needs more counter. When the counter is full, new tables wait at the door, however fast the cooks are.

The arithmetic is simple enough to do on a napkin, and doing it before you pick hardware or promise a context length saves a lot of surprises.

How it works

Weights: parameters times bytes

Each parameter is stored as a number, and the number format decides the bytes:

Models are served in 16-bit by default; 32-bit is rare for inference. A handy rule of thumb: a 7-billion-parameter model needs about 14 GB in 16-bit. Low-bit formats also store small scale factors alongside the weights. If one 16-bit scale is kept per group of 128 weights, that adds $16/128=0.125$ bits per weight, so "4-bit" is really about 4.1 bits. Quantization for inference, later in this module, covers how these formats trade memory for quality.

Serving needs far less than training. There are no gradients and no optimizer state, which is where most of the 16 bytes per parameter of training ( Parameter-efficient fine-tuning: LoRA and QLoRA ) goes.

KV cache: per token, per request, per batch

For every token it has seen, each request keeps one key and one value vector per layer and per key/value headIn grouped-query attention, several query heads share one key and value head. The number of key/value heads, not query heads, sets the KV cache size.:

Take a model shaped like a common 8B open-weights model: 32 layers, 8 key/value heads of dimension 128, cached in 16-bit.

• Per token: $2\times 32\times 8\times 128\times 2=131,072$ bytes, which is 128 KiB.
• Per request of 8,192 tokens: $131,072\times 8,192\approx 1.07$ GB.
• For 52 such requests: about 56 GB, more than three times the weights.

A 70B model of the same family has 80 layers with the same 8 key/value heads, so 320 KiB per token and about 2.7 GB per 8,192-token request.

How many requests fit?

Put the pieces together:

On one 80 GB GPU, with a 10 percent reserve (8 GB) and 16 GB of weights, 56 GB is left for the cache:

• At 2,048 tokens per request (0.27 GB each): 208 requests.
• At 8,192 tokens (1.07 GB): 52.
• At 131,072 tokens (17.2 GB): 3.

Same GPU, same model, a 70-fold difference in capacity. This is the number that decides your cost per user, and it is why How LLM inference works said the KV cache, not compute, usually caps the batch size first. The 10 percent reserve is an illustrative figure for activations, the runtime and fragmentation; real servers measure theirs at startup.

When the model does not fit on one GPU

A 70B model in 16-bit needs 140 GB of weights, more than any single 80 GB GPU. You can split it across several GPUs with tensor parallelismSplitting each layer's weight matrices across several GPUs, which compute their parts in parallel and exchange results at every layer. It divides weight and KV cache memory per GPU., where each GPU holds a slice of every layer, including a slice of the KV cache. Memory adds up, but not usefully at first:

• 2 GPUs (160 GB): 144 GB after the reserve, 140 GB of weights, 4 GB left. One 8,192-token request fits.
• 4 GPUs (320 GB): 148 GB left, about 55 requests.
• 1 GPU with 4-bit weights (35 GB): 37 GB left, about 13 requests, if the quality loss is acceptable.

The first configuration "runs the model" in the sense that it loads. It is useless for serving. Always check the room left for the cache, not just whether the weights fit.

Try it

1. Start at the defaults (8B, 16-bit, one 80 GB GPU, 8,192 tokens, 16 requests). Raise the requests until the bar turns red. Then move tokens per request to 32,768. What happens to the maximum?
2. Switch to the 70B model on 2 GPUs. The weights fit, but how many requests can you serve? Try 4 GPUs, then 1 GPU with 4-bit weights.
3. On 4 GPUs with the 70B model, switch the KV cache to 8-bit. Which part of the bar changes, and how does the maximum move?

The same arithmetic in Python:

# How many requests fit on the GPUs? Weights + KV cache + a reserve.
# Model shapes: 8 key/value heads of dimension 128 (grouped-query attention).
MODELS = {"8B": (8e9, 32), "70B": (70e9, 80)}   # parameters, layers
KV_HEADS, HEAD_DIM = 8, 128
RESERVE = 0.10   # activations, runtime, fragmentation (illustrative)

def weights_gb(params, bits):
    return params * bits / 8 / 1e9

def kv_bytes_per_token(layers, bits=16):
    return 2 * layers * KV_HEADS * HEAD_DIM * bits / 8   # K and V

def max_requests(model, w_bits, gpus, gpu_gb, tokens, kv_bits=16):
    params, layers = MODELS[model]
    budget = gpus * gpu_gb * (1 - RESERVE) - weights_gb(params, w_bits)
    per_req = kv_bytes_per_token(layers, kv_bits) * tokens / 1e9
    return max(0, int(budget // per_req)), budget, per_req

for bits in (16, 8, 4):
    print(f"8B at {bits:>2}-bit: {weights_gb(8e9, bits):5.1f} GB of weights")
print(f"KV per token: 8B {kv_bytes_per_token(32) / 1024:.0f} KiB,"
      f" 70B {kv_bytes_per_token(80) / 1024:.0f} KiB\n")

print("8B, 16-bit, one 80 GB GPU")
for tokens in (2048, 8192, 32768, 131072):
    n, budget, per_req = max_requests("8B", 16, 1, 80, tokens)
    print(f"  {tokens:>7,} tokens: {per_req:6.2f} GB each -> max {n:>3}"
          f" in {budget:.0f} GB")

print("\n70B at 8,192 tokens per request")
for w_bits, gpus, kv_bits in [(16, 2, 16), (16, 4, 16), (16, 4, 8), (4, 1, 16)]:
    n, budget, _ = max_requests("70B", w_bits, gpus, 80, 8192, kv_bits)
    print(f"  {w_bits:>2}-bit weights, {gpus} x 80 GB, {kv_bits}-bit KV:"
          f" {budget:5.0f} GB left -> max {n:>3}")

8B at 16-bit:  16.0 GB of weights
8B at  8-bit:   8.0 GB of weights
8B at  4-bit:   4.0 GB of weights
KV per token: 8B 128 KiB, 70B 320 KiB

8B, 16-bit, one 80 GB GPU
    2,048 tokens:   0.27 GB each -> max 208 in 56 GB
    8,192 tokens:   1.07 GB each -> max  52 in 56 GB
   32,768 tokens:   4.29 GB each -> max  13 in 56 GB
  131,072 tokens:  17.18 GB each -> max   3 in 56 GB

70B at 8,192 tokens per request
  16-bit weights, 2 x 80 GB, 16-bit KV:     4 GB left -> max   1
  16-bit weights, 4 x 80 GB, 16-bit KV:   148 GB left -> max  55
  16-bit weights, 4 x 80 GB, 8-bit KV:   148 GB left -> max 110
   4-bit weights, 1 x 80 GB, 16-bit KV:    37 GB left -> max  13

Note how each halving of KV cache precision doubles the number of requests, while halving weight precision only helps by the bytes it frees. When the weights are a small share of memory, cache tricks matter more; when the weights barely fit, shrinking them is what makes serving possible at all.

Going deeper

Reserving for the maximum wastes memory

The formula above assumes each request uses exactly its token count. A server does not know how long an answer will be. If it reserves cache for the maximum context of every request up front, a request that ends after 300 tokens leaves most of its reservation unused. In the 8B example, reserving 32,768 tokens per request admits only 13 requests, even if most of them finish within 2,048 tokens and 208 could have fit. Allocating the cache in small blocks as tokens arrive fixes this, which is the idea behind PagedAttention in the next lesson, Batching and PagedAttention.

What happens when the cache fills mid-generation

Requests grow by one token per step, so a batch that fit at the start can outgrow memory. A server then has three choices: refuse new requests (queue them), pause a running request and free its cache to recompute or reload it later (preemption), or crash with an out-of-memory error. Good servers queue and preempt; the cost is extra latency for the requests that wait.

Other ways to buy room

• Smaller KV cache per token: grouped-query or multi-query attention (built into the model, Multi-head attention ), or an 8-bit KV cache (a serving setting, check quality on your evals).
• Shorter contexts: cap the maximum length, trim conversation history, or retrieve a few passages instead of pasting documents ( Why RAG? ).
• Shared prefixes: requests that start with the same system prompt can share one copy of its cache (prefix caching, covered with other speedups later in this module).
• More GPUs: more memory for the cache, but each GPU added through tensor parallelism also adds communication at every layer.

Trade-offs

Common mistakes

• Checking only that the weights fit, then discovering the server can hold one request.
• Mixing up query heads and key/value heads in the KV formula, which overestimates the cache by the group size.
• Promising a long maximum context without working out how few such requests fit at once.
• Forgetting the reserve for activations and the runtime, so the server runs out of memory at peak load.
• Mixing GB (10^9 bytes) and GiB (2^30 bytes) and being off by 7 percent at the worst moment.

In the wild

Serving engines run this calculation for you. vLLM, for example, pre-allocates its KV cache from the share of GPU memory set by its gpu_memory_utilization setting, and the size of that cache then decides how many requests it can batch.

Check yourself

Recap