System Design HLD-19.07
Design a chat system (WhatsApp/Slack)
Design a chat system: WebSocket gateways, routing to the recipient's connection, per-conversation ordering, receipts, presence, offline push and groups.
The problem
A chat app feels simple: type, press send, and the message appears on your friend's phone a moment later, marked delivered and then read. Behind that, the system must keep tens of millions of phones and browsers connected at once, find which of a thousand servers holds the recipient's connection, store every message durably, show messages in the same order to everyone in the conversation, and reach people whose phones are asleep.
What makes chat different from most web systems is the long-lived connection. A request-response API can send any request to any server. A chat server holds open sockets, so it has state: to deliver a message to Bob, you must find the specific server Bob is connected to right now, and that changes every time his train goes through a tunnel.
Requirements
Functional
- One-to-one and group conversations (groups up to a few hundred members; large channels are an extension).
- Send and receive text messages in real time; media through separate upload.
- Message history on every device, in the same order for everyone.
- Sent, delivered and read receipts; online and last-seen presence.
- Push notifications for users who are offline.
Non-functional
- Scale: 500 million daily users, 40 messages per user per day, 100 million connections at peak.
- Latency: under 500 ms end to end when both users are online.
- Durability: a message acknowledged to the sender is never lost.
- Ordering: messages within a conversation appear in one agreed order on all devices.
Out of scope
- Voice and video calls, message search, and the cryptography of end-to-end encryption beyond its design consequences.
Estimates
- Messages: 500M x 40 = 20 billion a day, about 230,000 per second on average and roughly 700,000 at a 3x peak.
- Storage: at about 200 bytes per message with metadata, 4 TB a day, roughly 1.5 PB a year before replication. Writes dominate and reads are mostly recent, which points to a write-optimized store partitioned by conversation.
- Connections: if a gateway server holds 100,000 open WebSockets (a planning figure limited mostly by memory; measure yours), 100 million connections need 1,000 gateways, plus headroom for a region failing over.
- Presence: with heartbeats every 30 seconds, 100 million connections send over 3 million heartbeats per second. Presence must be handled at the gateways and in memory, never written to a database per heartbeat.
API
Clients talk to gateways over one WebSocket carrying small framed messages:
client -> server SEND { conv: "c_91", client_msg_id: "a3", body: "see you at 6" }
server -> client ACK { client_msg_id: "a3", seq: 42, ts: ... }
server -> client MSG { conv: "c_91", seq: 42, from: "alice", body: "..." }
client -> server RECEIPT { conv: "c_91", delivered_up_to: 42, read_up_to: 40 }
client -> server SYNC { conv: "c_91", after_seq: 37 } # after reconnecting
History, conversation lists and media uploads use an ordinary HTTPS API.
Data model
- Messages, in a wide-column store ( NoSQL families ), partition key
(conversation_id, bucket)and clustering keyseq, where the bucket is a time window such as ten days. A conversation's recent messages are one sequential read, and very active conversations do not grow one partition forever. - Conversations and membership:
conversation_id -> members, anduser_id -> conversationssorted by last activity, for the inbox screen. - Session registry (in memory, for example Redis):
user_id -> [(device, gateway_id)], with a short TTL refreshed by heartbeats. - Receipts: per member per conversation,
last_delivered_seqandlast_read_seq. Two numbers, not one row per message.
High-level design
- Send path Alice's client sends the message with a client-generated id. The message service assigns the conversation's next sequence number, writes the message to the store, and only then acknowledges Alice with the sequence number. A retried SEND with the same client id gets the same ACK, not a duplicate.
- Deliver path The service looks up Bob's live sessions in the registry and forwards the message to each gateway holding one. Gateway B pushes it down Bob's socket; Bob's client sends back a receipt saying it has everything up to seq 42, which is relayed to Alice as 'delivered'.
- Offline path Bob has no live session. The message is already stored, so nothing is lost; the push service sends a notification through Apple's or Google's push gateway. When Bob opens the app it reconnects and syncs everything after the last sequence number it has.
Deep dives
Deep dive 1: Gateways and routing to a connection
Gateways do as little as possible: terminate TLS and WebSockets, authenticate the connection, keep heartbeats, and pass frames to and from the message service. Keeping them thin means they rarely need deploying, which matters because every deploy disconnects users. Clients connect through a load balancer that picks any gateway ( L4 vs L7 load balancing ), so the system must record where each user landed: the gateway writes user -> gateway into the session registry on connect and refreshes it with each heartbeat, so entries for dead gateways expire on their own.
To deliver, the message service looks up the recipient's sessions and sends the message to those gateways, over persistent internal connections or a per-gateway queue. A user with a phone and a laptop has two sessions and gets both copies. The alternative, a pub/sub topic per user that gateways subscribe to, trades the registry lookup for a pub/sub system that must handle millions of topics. Either works; the registry is simpler to reason about.
When a gateway dies, its 100,000 clients reconnect within seconds, ideally with jittered backoff so they do not stampede the survivors ( Timeouts, retries, backoff and jitter ). Nothing is lost, because messages were stored before delivery and each client syncs from its last sequence number.
Deep dive 2: Storage, ordering and exactly-once display
Chat history is write-heavy, append-only and read mostly by "latest messages in this conversation", which is exactly what a wide-column store partitioned by conversation is good at ( Storage engines: B-tree vs LSM-tree ). Ordering is defined per conversation: the message service assigns each conversation a monotonically increasing sequence number, for example by routing all of a conversation's writes to one owner using consistent hashing, which assigns seq from an in-memory counter persisted with the message. Global ordering across conversations is not needed and would be a bottleneck.
The network still delivers out of order and twice: reconnects replay, gateways retry. So clients keep a small reorder buffer and dedupe by message id. Here is a client receiving seven frames, with a gap and a duplicate:
# Per-conversation ordering and dedupe on the client.
# The server gives each message in a conversation a sequence number.
# Messages may arrive out of order (reconnects) or twice (retries).
arrivals = [ # (seq, client_msg_id, text), in arrival order
(1, "a1", "hi"), (2, "b1", "hey!"), (4, "a3", "see you at 6"),
(3, "a2", "dinner?"), (4, "a3", "see you at 6"), (6, "b3", "ok"),
(5, "b2", "sounds good"),
]
shown, buffer, seen = [], {}, set()
next_seq = 1
for seq, cid, text in arrivals:
if cid in seen:
print(f"seq {seq}: duplicate {cid}, dropped")
continue
seen.add(cid)
buffer[seq] = text
if seq != next_seq:
print(f"seq {seq}: gap, waiting for seq {next_seq}")
while next_seq in buffer: # release everything now in order
shown.append(buffer.pop(next_seq))
next_seq += 1
print("displayed:", " | ".join(shown))
print("last seq read:", next_seq - 1, "(sent to server as the read receipt)")Output
seq 4: gap, waiting for seq 3
seq 4: duplicate a3, dropped
seq 6: gap, waiting for seq 5
displayed: hi | hey! | dinner? | see you at 6 | sounds good | ok
last seq read: 6 (sent to server as the read receipt) The client held seq 4 until seq 3 arrived, dropped the duplicate, and displayed everything in order. If a gap does not fill within a second or two, the client asks the server with SYNC after_seq. The same idea, at-least-once delivery plus idempotent handling, gives users effectively exactly-once display ( Delivery semantics ). On the send side, the client_msg_id makes retries safe: the service remembers recent client ids per conversation and returns the original ACK.
Deep dive 3: Receipts, presence and groups
Receipts are cumulative: "delivered up to 42" and "read up to 40" per member, two numbers per conversation. That is far cheaper than a status row per message, and for groups the sender's "read by 5 of 8" is a count over members' read_up_to.
Presence is the noisiest data in the system and the least important to persist. Gateways know who is connected; they publish online and offline transitions (not heartbeats) to a presence service, which fans them out only to users who currently have that contact on screen. Debounce flapping connections: wait a few seconds before announcing "offline". For large groups, show presence only on request.
Group fan-out: a message to a group of 200 is stored once in the conversation's partition, then delivered to each online member's gateway. The fan-out is on delivery, not on storage, unlike the news feed's timeline fan-out ( Design a news feed (Twitter/Instagram) ). Very large channels (tens of thousands of members) switch to a pull model: clients are told "channel X has new messages" and fetch them.
End-to-end encryption, briefly
With end-to-end encryption, clients encrypt messages with keys the server never sees, so the server stores and routes ciphertext. The architecture above does not change, but anything that needs to read content on the server, such as search, previews in push notifications, or spam filtering, must move to the client or go away. Group messages are encrypted for each member's devices, which makes adding devices and members a key-distribution problem.
Try it
Try this:
- Set 1 million connected users and a 5-second polling interval. How many requests per second does short polling cost compared with WebSockets?
- Lower the message rate to one per minute. Which approach wastes the most work when chats are quiet?
- Chat needs both directions (typing indicators, receipts). Which options handle client-to-server messages without a separate request?
Bottlenecks and failure modes
| Failure | Impact | Mitigation |
|---|---|---|
| Gateway crash | 100,000 clients disconnected | Jittered reconnect; sync from last seq; nothing lost |
| Message service owner for a conversation down | Sends to it stall | Reassign ownership via the hash ring; seq continues from the stored maximum |
| Message store slow | Sends not acknowledged | Clients retry with the same client id; backpressure at gateways |
| Hot group conversation | One partition and owner overloaded | Time buckets, pull model for huge channels |
| Presence storm (mass reconnect) | Millions of status updates | Debounce, send transitions only to interested viewers |
| Push provider outage | Offline users not notified | Retry with backoff; messages wait in storage for sync |
Trade-offs we made
| Option | Wins when | Loses when |
|---|---|---|
| WebSockets (chosen) | Two-way, low latency, one connection per device | Stateful servers; routing to connections needed |
| Store before acknowledging (chosen) | An acknowledged message is never lost | Send latency includes a durable write |
| Ordering per conversation (chosen) | Scales out; matches what users see | No global order across conversations |
| Session registry lookup (chosen) | Simple, explicit routing | Another in-memory system to keep available |
| Cumulative receipts (chosen) | Two numbers per member instead of a row per message | Cannot mark a single older message unread |
Extensions
- Large channels with thousands of members, delivered by notify-and-pull.
- Multi-region: users connect to the nearest region; each conversation has a home region that assigns sequence numbers.
- Message edits, deletes and reactions as new events in the same sequence, applied by clients.
- Search over history, which conflicts with end-to-end encryption.
In the wild
Slack
Real-time Messaging (opens in a new tab)Slack's real-time path is split across services that mirror this design. Channel Servers are stateful, in-memory and own channels by consistent hashing on the channel id (about 16 million channels per host at peak), while Gateway Servers, deployed across regions, hold users' WebSocket subscriptions and act as the interface between clients and Channel Servers. A sent message goes from the web backend through an Admin Server to the channel's Channel Server, which sends it to every subscribed Gateway Server and on to connected clients. Presence Servers track who is online, and the post says the system delivers messages across the world in 500 ms.
Discord
How Discord stores trillions of messages (opens in a new tab)Discord stores messages partitioned by channel and a time bucket, the same key this lesson uses. By 2022 its Cassandra cluster had grown to 177 nodes holding trillions of messages and suffered from hot partitions, garbage collection pauses and compaction backlogs. Discord moved to ScyllaDB and put Rust data services in front of the database that route requests by channel id with consistent hashing and coalesce identical concurrent requests into one query. After the migration the cluster went from 177 nodes to 72, and p99 latency for fetching historical messages dropped from 40 to 125 ms to 15 ms.
Check yourself
-
Why does a chat system need a session registry?
Show answer
B. Connections are long-lived, so delivery must find the specific gateway holding the recipient's socket Any server can take a request, but only one gateway holds Bob's open socket right now.
-
When is the sender's message acknowledged?
Show answer
B. After it is durably stored with its sequence number Acknowledging only after the durable write guarantees an acknowledged message is never lost.
-
A client receives seq 41, then 43. What should it do?
Show answer
B. Hold 43 briefly, and request seq 42 with a sync if it does not arrive Per-conversation sequence numbers let clients detect gaps and fill them, giving a consistent order.
-
How is a message to a 200-member group stored?
Show answer
B. Once in the conversation's partition, then delivered to each online member Fan-out happens on delivery; storage stays one copy per message.
-
Why are presence heartbeats not written to a database?
Show answer
B. Millions per second of low-value updates would overwhelm it; gateways track them in memory Only transitions matter, and only to users looking at that contact.
Recap
- Thin WebSocket gateways hold connections; a session registry maps users to gateways.
- Store each message with a per-conversation sequence number before acknowledging it.
- Clients reorder and dedupe by sequence and message id, and sync gaps after reconnects.
- Receipts are cumulative per member; presence lives in memory and sends only transitions.
- Groups store a message once and fan out on delivery; offline users get push notifications.
Next up
- Crack the system design round learning path