LLM Inference and Serving

A model that works in a notebook still has to answer thousands of people quickly and at a cost that makes sense. Serving is where that happens, and it follows a few physical limits: how fast the chip can move bytes from memory, how much memory it has, and how many users can share each step.

You will start with the two phases of generation and the latency numbers users feel, then do the memory arithmetic that decides how many requests fit on a GPU. From there you will see how servers keep the GPU busy with continuous batching and paged KV caches, how quantization and speculative decoding make each step cheaper, how routing and caching cut the bill, and how streaming makes a slow answer feel fast.