Streaming responses to users

The big idea

A model takes 12 seconds to write a 300-token answer. If your app waits for the whole thing, the user stares at a spinner for 12 seconds and wonders if it broke. If you show each token as it arrives, the first words appear in half a second and the answer writes itself while they read. The total time is identical. The experience is not.

Think of a waiter who brings each dish as soon as it is ready, instead of holding the whole meal until the last plate is done. Nobody gets their food sooner overall, but nobody sits at an empty table either.

StreamingSending a model's output to the client piece by piece as it is generated, instead of waiting for the complete response. turns the long wait into time to first token, usually well under a second, and it gives users something just as valuable: the chance to see early that an answer is going the wrong way and stop it.

How it works

Server-Sent Events

Most model APIs stream with Server-Sent EventsA simple standard for a server to push a stream of text events to a client over one long-lived HTTP response with the content type text/event-stream. (SSE). The server answers with the content type text/event-stream and keeps the response open. Each event is a few lines of text, ended by a blank line:

event: content_block_delta
data: {"delta": " Settings"}

event: content_block_delta
data: {"delta": ","}

SSE suits model output well: the data flows one way (server to client), it is plain HTTP, so it works through ordinary load balancers and proxies, and each event is just text. WebSockets also work, but their two-way channel is more than a single answer needs.

Two practical details:

• The browser's built-in EventSource only opens GET requests without a body, while a chat request usually needs a POST with the conversation in it. So chat clients typically read the stream with fetch and parse the events themselves, as in the code below.
• Events do not arrive one per network packet. One chunk may hold half an event, or three. The client must buffer text and split on the blank line that ends each event.

What arrives in the stream

The stream carries more than text. Typical event types are: a start event, many small text deltas, events for tool calls (with their arguments arriving in pieces), a final event with the stop reason and token usage, and occasional keep-alive pings. Errors can also arrive mid-stream, after the HTTP status 200 has already been sent, so the client must handle an error event, not just a failed request.

Rendering partial output

Appending raw text is easy. Rendering it well takes a little care:

• Markdown arrives half-finished. A code block opens with three backticks and closes many tokens later. Re-render the message from the full text so far on each update, and treat an unclosed block as open rather than letting the rest of the page break.
• Do not act on half a structure. Tool-call arguments and JSON output stream in pieces. Wait for the block's stop event before parsing or running anything.
• Update the screen on a timer, not per token. Batching updates every few dozen milliseconds avoids re-rendering hundreds of times a second.
• Keep the scroll position sensible. Auto-scroll only if the user is already at the bottom, so they can read earlier text while the answer grows.
• Accessibility. A screen reader announcing every token is unusable. Announce when the answer is complete, or in sentence-sized pieces.

Cancellation

A Stop button must stop the work, not just the display. In the browser, abort the fetch (with an AbortController). On the server, detect that the client disconnected and cancel the upstream model request. If the server keeps reading from the model after the user has left, you pay for tokens nobody will see, and on your own GPUs the request keeps occupying a batch slot and KV cache memory ( Batching and PagedAttention ) that another user could have used.

Every hop must pass the cancellation on: browser to app server, app server to any queue or worker, worker to the model. A proxy that buffers the response, or a worker that ignores disconnects, silently breaks the chain. Test it: press Stop, then check the server logs to see whether generation actually ended.

Try it

1. Keep the defaults and move the time slider from 0 to 100 percent. When does each panel first show anything?
2. Set time to first token to 3,000 ms. How much does streaming help now? Which part of the wait can only be fixed on the server ( How LLM inference works )?
3. Move the time to 20 percent and press Stop with a server that keeps going. How many tokens are wasted, and who pays for them?

A minimal stream parser with partial rendering and a Stop:

# Parse a Server-Sent Events stream of tokens, as a chat client does.
# Network chunks do not respect event boundaries, so the client must buffer.
import json

tokens = ["Open", " Settings", ",", " then", " Security", ".", " Choose", " Reset", "."]
stream = "".join(f"data: {json.dumps({'delta': t})}\n\n" for t in tokens) + "data: [DONE]\n\n"
chunks = [stream[i:i + 23] for i in range(0, len(stream), 23)]   # arbitrary split
print(f"{len(tokens)} events arrive in {len(chunks)} chunks, e.g. {chunks[1]!r}\n")

def read_events(chunks, stop_after=None):
    buffer, text, seen = "", "", 0
    for chunk in chunks:
        buffer += chunk
        while "\n\n" in buffer:                      # a blank line ends one event
            event, buffer = buffer.split("\n\n", 1)
            data = "\n".join(l[6:] for l in event.split("\n") if l.startswith("data: "))
            if data == "[DONE]":
                return text, "finished"
            text += json.loads(data)["delta"]
            seen += 1
            print(f"  render: {text!r}")             # partial rendering
            if stop_after and seen == stop_after:
                return text, "cancelled: close the connection"
    return text, "stream ended without [DONE]: show an error"

print("full answer:")
print(read_events(chunks))
print("\nuser presses Stop after 4 tokens:")
print(read_events(chunks, stop_after=4))

9 events arrive in 11 chunks, e.g. '\n\ndata: {"delta": " Set'

full answer:
  render: 'Open'
  render: 'Open Settings'
  render: 'Open Settings,'
  render: 'Open Settings, then'
  render: 'Open Settings, then Security'
  render: 'Open Settings, then Security.'
  render: 'Open Settings, then Security. Choose'
  render: 'Open Settings, then Security. Choose Reset'
  render: 'Open Settings, then Security. Choose Reset.'
('Open Settings, then Security. Choose Reset.', 'finished')

user presses Stop after 4 tokens:
  render: 'Open'
  render: 'Open Settings'
  render: 'Open Settings,'
  render: 'Open Settings, then'
('Open Settings, then', 'cancelled: close the connection')

Notice that the 9 events arrive in 11 chunks that cut through the middle of events, which is why the parser buffers until it sees a blank line. A stream that ends without its final event should be shown as an error, not as a complete answer.

Going deeper

Proxies and timeouts

Streams fail in production in boring ways. A reverse proxy or CDN may buffer the whole response before forwarding it, which silently turns streaming off; disable response buffering for streaming routes. Idle timeouts on load balancers can cut a long stream, especially while the model is thinking before its first token; keep-alive pings help. Compression can also delay small events. Test streaming through your real infrastructure, not just against a local server.

Resuming and multiple tabs

SSE has an event id and a Last-Event-ID header for resuming after a dropped connection, but most model APIs cannot resume a generation halfway. If the connection drops, apps usually either retry the whole request or store the partial answer and let the user continue. Without HTTP/2, browsers also limit how many SSE connections one site can keep open (MDN notes a limit of 6 per browser and domain), which matters if users open many tabs.

Streaming and safety checks

If you check model output with a filter (for safety or for leaked data), streaming complicates it: by the time the filter sees a problem, part of the answer is on the user's screen. Options include running fast checks on each sentence before forwarding it, holding back a short buffer, or retracting a message when a later check fails. Each trades a little latency for safety.

Trade-offs

Common mistakes

• Making the Stop button only hide the text while the server keeps generating and billing.
• Parsing each network chunk as one event, which breaks as soon as an event is split.
• Running a tool call or parsing JSON before its block has finished streaming.
• Deploying behind a proxy that buffers responses, so streaming works locally and not in production.
• Treating a stream that ended without its final event as a complete answer.

In the wild

Chat assistants generally stream their answers, and model APIs commonly offer a streaming mode, usually over Server-Sent Events, with a stop or cancel control in the interface. The details that differ between products are the ones in this lesson: how partial Markdown and code are rendered, whether tool calls appear as they stream, and whether pressing Stop keeps the partial answer in the conversation.

Check yourself

Recap