Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Design Case Studies

System Design HLD-19.23

Design an LLM chat service (ChatGPT-like)

Design an LLM chat app: SSE token streaming, a GPU pool with continuous batching, conversation storage, per-tier rate limits, caching and safety.

Advanced 22 min Last reviewed

The problem

A user types a question and, within a second, words start to appear, a few at a time, until a full answer has streamed in. The conversation is saved, so they can come back tomorrow and continue it. Free users get a modest allowance; paying users get more and get served first when the system is busy. Harmful requests and harmful answers are caught on the way.

Most of that sounds like an ordinary web app, and much of it is. What makes this design different is the expensive part in the middle. Every reply is generated by a large model running on GPUs, one token at a time, for many seconds. GPUs are scarce and costly, so the design revolves around keeping them busy, queueing fairly when they are full, and streaming each token to the user the moment it exists. This case study builds on the inference mechanics in How LLM inference works ; here the focus is the system around the model.

Requirements

Functional

  • Send a message in a conversation and receive the reply as a stream of tokens.
  • Stop a reply mid-stream; regenerate a reply; edit an earlier message and branch from it.
  • List, open and delete past conversations.
  • Plans (tiers) with different usage limits and priority.
  • Check inputs and outputs against a safety policy.

Non-functional

  • Time to first tokenThe time from sending a request to receiving the first generated token. It is what makes a chat app feel responsive. (TTFT) under about 1 second at the median while the system is not overloaded.
  • Streaming speed comfortably faster than people read, steady rather than bursty.
  • High availability for the chat path; conversation history is never lost once a reply completes.
  • Cost efficiency: GPU utilization is the main lever, so the design must keep GPUs busy without letting latency explode.

Out of scope

  • Training and fine-tuning the model, and the internals of the inference engine beyond what affects capacity.

Estimates

These numbers are illustrative assumptions to practice the arithmetic ( Back-of-the-envelope estimation ), not figures from any real product.

  • Traffic: 10 million daily active users sending 10 messages a day is 100 million requests a day, or 100,000,000 / 86,400 = about 1,160 per second on average. Assume a peak of 3x: about 3,500 per second.
  • Output tokens: at an average of 400 output tokens per reply, the peak is 3,500 x 400 = 1.4 million tokens per second to generate.
  • Open streams: if each user sees 40 tokens per second, a 400-token reply streams for 10 seconds. By Little's law, concurrent streams = arrival rate x duration = 3,500 x 10 = 35,000 open SSE connections at peak.
  • GPUs: suppose one inference replica (a model copy on one or more GPUs) runs 16 streams at 40 tokens per second each, so 640 tokens per second. Then 1,400,000 / 640 = about 2,190 replicas at peak. Even with generous assumptions, this is the cost center by orders of magnitude.
  • Prompt tokens: every request resends the conversation so far. At 3,000 prompt tokens on average, prefill handles 3,500 x 3,000 = 10.5 million prompt tokens per second, which is why reusing work across turns matters.
  • Storage: a message pair of about 500 tokens is roughly 2 KB of text (a common rule of thumb is about 4 characters per English token). 100 million x 2 KB = 200 GB a day, about 73 TB a year before replication. Large, but ordinary for a sharded database.

API

POST /v1/conversations  -> { conversation_id }

POST /v1/conversations/{id}/messages
  { content, parent_message_id, model }
  -> 200 text/event-stream
  event: start  data: {"message_id": "m_812"}
  event: token  data: {"text": "Sure"}
  event: token  data: {"text": ", here"}
  ...
  event: done   data: {"finish": "stop",
                       "usage": {"in": 2950, "out": 412}}

POST /v1/conversations/{id}/messages/{mid}/stop -> 202
GET  /v1/conversations?cursor=...  -> page of conversations
GET  /v1/conversations/{id}        -> message tree

Errors: 429 + Retry-After when a tier limit is hit;
        event: error if the stream fails midway.

The reply comes back as Server-Sent EventsServer-Sent Events: a long-lived HTTP response with content type text/event-stream, over which the server pushes a sequence of small text events to the client. on the same request that sent the message ( Real-time: polling, long polling, SSE and WebSockets ). Stop is a separate request because SSE is one-way.

Data model

  • conversations (partitioned by user_id): conversation_id, user_id, title, model, created_at, updated_at. Listing a user's conversations is one partition read, newest first.
  • messages (partitioned by conversation_id, sorted by a time-ordered message_id): parent_id, role (user, assistant, system), content, model, tokens_in, tokens_out, status (streaming, complete, stopped, blocked). The parent_id makes the conversation a tree, so edits and regenerations are new branches rather than overwrites.
  • usage counters (in-memory store keyed by user and window): tokens and requests used, for rate limits. A durable usage ledger is written asynchronously for billing.
  • prefix cache (on the GPUs, ephemeral): attention keys and values for recent prompts, reused when the next turn starts with the same text.

High-level design

LLM chat service: request, inference and stream paths Request path: the client sends a message to the API gateway, which authenticates the user and applies per-tier rate limits, then to the chat service. The chat service loads the conversation history from the conversation database, runs input safety checks and sends the assembled prompt to the router and queue. Inference path: the router assigns the request to a replica in the GPU inference pool, which batches requests continuously and returns tokens to the chat service as they are generated. Stream path: the chat service streams tokens to the client over Server-Sent Events, runs output safety checks on the text, saves the reply to the conversation database and records token usage asynchronously for billing. message load, save check prompt assign tokens SSE stream usage Client API gateway auth, tier limits Chat service history, prompt, stream Conversation DB Safety filters input and output Router and queue priority by tier GPU inference pool continuous batching Usage metering async, billing Figure 1. The GPU pool is the scarce resource in the middle. Everything before it decides who may use it and with what prompt; everything after it streams, checks and stores what it produces.
  1. Request path The gateway authenticates the user and checks their tier's limits. The chat service loads the conversation, checks the new message against the safety policy, assembles the prompt and hands it to the router's queue.
  2. Inference path The router picks a replica with room in its batch, preferring one that already caches this conversation's prompt prefix. The replica generates tokens and returns each one as soon as it exists.
  3. Stream path The chat service forwards tokens to the client over SSE, checks the growing text against the output policy, saves the finished reply, and records token usage asynchronously for limits and billing.

Deep dives

Deep dive 1: Streaming tokens over SSE

A 400-token reply takes about 10 seconds to generate. Waiting for all of it would make the app feel broken, so tokens are pushed as they are produced. SSE fits well: the client already sends a request, the server needs to push only one way, and it is plain HTTP, so it passes through load balancers, proxies and CDNs that understand HTTP. WebSockets would also work, but the two-way channel buys little here, since stop and new messages can be ordinary requests.

Details that matter at scale:

  • Coalescing: flushing one network write per token works, but grouping tokens every 30 to 50 ms cuts the number of writes and events with no visible difference to the reader.
  • Long-lived connections: 35,000 streams are open at peak, each for seconds. Load balancer and proxy idle timeouts must exceed the longest silence (a slow first token, a long tool call); a periodic comment line (: keepalive) keeps idle connections open.
  • Cancellation: the stop button, and also a closed tab, must reach the GPU. A generation nobody reads still holds a batch slot and its memory, so the chat service cancels the request at the replica as soon as the client goes away.
  • Disconnects: the reply keeps being written to the database as it streams (or at least at the end), so a client that reconnects fetches the finished message instead of losing it.

Deep dive 2: The GPU pool and continuous batching

Inference has two phases ( How LLM inference works ). Prefill processes the whole prompt in parallel and produces the first token; decode then generates one token per step, and each step must read the model weights from GPU memory. Reading the weights costs about the same whether the step produces one token or tokens for 16 users, so serving many requests in the same step, called batching, multiplies throughput.

The catch is that replies have very different lengths. With static batching, a replica takes a batch, runs it until the longest reply finishes, then takes the next batch; slots whose reply ended early sit idle. With continuous batchingAn inference scheduling method that adds new requests to a running batch, and removes finished ones, at every decode step, so batch slots never wait for the longest request., a slot is refilled at the very next step after its reply finishes. Here is one replica with 4 slots and 12 requests:

batching.pyruns with python3 (standard library)
# One GPU replica with 4 batch slots serves 12 queued chat requests.
# Each decode step produces one token for every request in the batch.
# Static batching refills only when the whole batch is done; continuous
# batching refills a slot the moment its request finishes.
import random

random.seed(3)
lengths = [random.randint(20, 400) for _ in range(12)]  # output tokens
SLOTS = 4

def serve(continuous):
    queue, active, finish, step, busy = list(range(12)), {}, {}, 0, 0
    while queue or active:
        if continuous or not active:          # when may we admit?
            while queue and len(active) < SLOTS:
                i = queue.pop(0)
                active[i] = lengths[i]
        step += 1                              # one decode step
        busy += len(active)                    # slots doing useful work
        for i in list(active):
            active[i] -= 1
            if active[i] == 0:
                del active[i]
                finish[i] = step
    mean = sum(finish.values()) / len(finish)
    return step, busy / (step * SLOTS), mean

print("output lengths:", lengths)
for name, mode in (("static", False), ("continuous", True)):
    steps, util, mean = serve(mode)
    print(f"{name:10}  total steps {steps:4}  slot use {util:4.0%}  "
          f"mean finish step {mean:5.0f}")

Output

output lengths: [141, 323, 298, 86, 209, 329, 262, 340, 317, 53, 330, 26]
static      total steps  993  slot use  68%  mean finish step   555
continuous  total steps  853  slot use  80%  mean finish step   451

Continuous batching finished the same 12 replies in 853 steps instead of 993, kept slots busy 80 percent of the time instead of 68, and each reply finished on average about 100 steps sooner. On a real fleet the same idea shows up as more users per GPU, which is the single biggest cost lever ( Batching and PagedAttention ).

Batch size is not free. The number of slots is limited by GPU memory, since each active request keeps a KV cache that grows with its context ( GPU memory math ). And a bigger batch makes every decode step a little slower, so each user's stream slows down. Choosing slots per replica is choosing a point between per-user speed and total throughput, which the lab below lets you explore.

Deep dive 3: Router, queueing and admission control

Requests do not go straight to a GPU. A router with a queue in front of it ( Why async? Queues explained ) decides where and when each request runs:

  • Placement: send the request to a replica with free batch slots and enough free KV cache memory for its prompt plus its maximum output.
  • Prefix affinity: the system prompt and the conversation so far are the same as last turn plus one message. If the request lands on the replica that served the previous turn, that replica's prefix cacheAttention keys and values computed for a prompt prefix, kept in GPU memory so a later request that starts with the same text can skip recomputing them. lets prefill skip the shared part. This turns 3,000 prompt tokens of work into a few hundred, cutting TTFT and GPU load. Hashing the conversation id onto replicas gives affinity; falling back to any replica when that one is full keeps it a hint, not a rule.
  • Priority: when the pool is full, paid tiers are dequeued first, and free requests wait longer or are routed to a smaller model.
  • Admission control: if the expected queue wait would push TTFT past a few seconds, reject now with a 429 and a Retry-After rather than accept a request that will feel broken anyway. Unbounded queues in front of a saturated pool only convert overload into timeouts.

Autoscaling helps with the daily curve but not with spikes: a new GPU node must be provisioned and load tens of gigabytes of weights, which takes minutes. So the pool runs with headroom, sheds lower-priority load first, and scales on queue depth and KV memory use rather than CPU.

Deep dive 4: Conversation storage and context assembly

Each turn, the chat service reads the conversation's messages along the current branch, writes the user's message with status complete, and creates the assistant message with status streaming before inference begins. When the stream ends it updates the content, token counts and final status (complete, stopped or blocked). Because messages are keyed by conversation and sorted by a time-ordered id, both the read and the writes hit one partition.

The prompt the model sees is assembled, not stored: a system prompt, then as much recent history as fits the model's context window, then the new message. Long conversations need a policy: drop the oldest turns, or replace them with a summary generated by a smaller model in the background. Conversation titles are generated the same way, asynchronously, after the first exchange. Neither job sits on the path to the first token.

Deep dive 5: Rate limits per tier

Limits protect the GPU pool and implement the business model. They differ from the limiters in Rate limiting algorithms in one way: the cost of a request is measured in tokens, and the output token count is unknown until the reply ends.

  • Keep a token bucket per user in a fast shared store, refilled at the tier's rate, plus a cap on concurrent streams per user.
  • On admission, reserve the prompt tokens plus the request's maximum output tokens. When the reply finishes, refund what was not used. This prevents many parallel requests from overspending a nearly empty bucket.
  • Apply cheap request-count limits at the gateway first, so abusive clients are turned away before any database or GPU work.
  • Record final usage asynchronously to a durable ledger for billing; the in-memory counters are for enforcement only.

Deep dive 6: Caching and safety filters

Caching works differently here. Whole responses are rarely reusable: prompts include personal history and sampling makes outputs vary, so an exact-match response cache only pays off for fixed, repeated prompts such as a suggested starter question. The cache that matters is the prefix cache from deep dive 3, plus ordinary caching of conversation lists and static assets. Semantic caching (reusing answers to similar questions) saves GPU time but risks returning an answer to a slightly different question, so use it only where that is acceptable.

Safety checks run on both sides of the model. Input checks are fast classifiers run before inference, so a blocked request never consumes GPU time. Output checks are harder because the text is streaming: the chat service runs the classifier over a sliding window of recent text every few dozen tokens. Checking more often delays the stream less but costs more; if a check fails, the stream stops, the visible text is replaced with a notice and the message status becomes blocked. Decide in advance whether a filter outage fails closed (stop serving) or fails open (serve and review later), since that is a policy choice, not a technical one.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Set batch slots to 1. How many requests per second can 50 replicas serve, and how fast does each user's stream run? Now set slots to 16 and compare both numbers.
  2. Raise the request rate slowly until continuous batching crosses the 1 second goal. Notice how flat the curve is until it suddenly is not. What does that say about running the pool at 95 percent utilization?
  3. Double the mean reply length to 800 tokens. How many replicas do you need to get back under the goal at 60 requests per second?

Bottlenecks and failure modes

FailureImpactMitigation
GPU pool saturatedTTFT climbs, then requests time outAdmission control with 429; tier priority; headroom; smaller fallback model
Replica crash mid-streamReply cut offError event to the client; save partial text; one-click regenerate
Abandoned streamsSlots held for nobodyCancel generation on stop and on disconnect
Proxy idle timeoutsStreams dropped on slow repliesKeepalive comments; timeouts longer than the worst gap
Prefix cache missesSlower TTFT, more prefill loadConversation affinity in the router
Safety filter outageUnsafe output or no serviceDecided fail-open or fail-closed policy; alerts
Usage counter store downLimits not enforcedLocal per-node fallback limits; reconcile from the ledger

Trade-offs we made

OptionWins whenLoses when
SSE for streaming (chosen) Plain HTTP, one-way push is all the reply needs, works with existing proxies Stop and other actions need separate requests
WebSockets Two-way channel for voice or collaborative features More connection state; harder through some proxies
Continuous batching with many slots (chosen) Most users per GPU, lowest cost Each stream is slower; KV memory limits slots
Small batches Fastest per-user streaming Far fewer users per GPU, much higher cost
Reject early when full (chosen) Fast, honest failure; protects paying tiers Some users see errors at peak
Unbounded queue No request is refused Everyone times out under sustained overload

Extensions

  • Tool use: the model calls search, code execution or other APIs mid-reply, which adds pauses to the stream and new failure modes.
  • Files and retrieval: uploaded documents are chunked and embedded, and relevant pieces are added to the prompt.
  • Multiple models: separate GPU pools per model, with a router that sends easy requests to a small model and escalates hard ones.
  • Batch API: offline jobs at a discount that fill the GPU pool's idle hours.
  • Multi-region: inference pools in several regions, routing by capacity as much as by distance.

In the wild

Where you will see this: hosted chat assistants, coding assistants inside editors, customer support bots and any product that exposes a large model to many users at once. The shape repeats: a conventional web tier for identity, limits and storage, a token stream over SSE to the browser, and a scarce GPU pool behind a router that batches continuously and sheds load before it melts. The open-source inference servers that implement continuous batching and paged KV memory are covered in Batching and PagedAttention .

Check yourself

  1. Why does continuous batching serve more users per GPU than static batching?

    Show answer

    B. Finished requests free their slot at the next decode step instead of waiting for the longest reply in the batch Reply lengths vary a lot. Static batches idle the slots whose replies ended early; continuous batching refills them immediately, raising slot utilization.

  2. A user closes the tab while a 2,000-token reply is generating. What should happen?

    Show answer

    B. The chat service cancels the generation at the replica, freeing the batch slot and its KV memory A generation nobody reads still consumes a slot and GPU memory. Cancel it; save whatever text exists if you want the partial reply in history.

  3. Why does the router try to send a conversation's next turn to the same replica?

    Show answer

    B. That replica may still hold the KV cache for the conversation's prompt prefix, so prefill can skip most of it Each turn's prompt is the previous prompt plus a little more. Prefix caching avoids recomputing thousands of tokens, cutting time to first token and GPU work.

  4. Token-based rate limits reserve the request's maximum output tokens at admission. Why?

    Show answer

    B. The real output length is unknown until the end, so reserving and refunding stops parallel requests from overspending the budget Without a reservation, a user with little budget left could start many requests at once, each of which looks affordable on its own.

  5. The pool is at 95 percent utilization and TTFT is fine. What is the risk?

    Show answer

    B. Queue wait rises steeply near full utilization, and new GPU capacity takes minutes to add, so a small spike causes long waits Queueing delay grows sharply as utilization approaches 100 percent. Keep headroom and use admission control, because autoscaling GPUs is slow.

Recap

  • Replies stream token by token over SSE on the same request; stop and disconnect must cancel generation at the GPU.
  • The GPU pool is the cost center: continuous batching keeps slots busy, while slots per replica trade per-user speed for throughput.
  • A router places requests by free memory and prefix-cache affinity, prioritizes by tier and rejects early when the queue is too long.
  • Conversations are a tree of messages partitioned by conversation; prompts are assembled per turn to fit the context window.
  • Rate limits count tokens with reserve-and-refund; safety checks run before inference and on the streaming output.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up