Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Design Case Studies

System Design HLD-19.07

Design a chat system (WhatsApp/Slack)

Design a chat system: WebSocket gateways, routing to the recipient's connection, per-conversation ordering, receipts, presence, offline push and groups.

Intermediate 25 min Last reviewed

The problem

A chat app feels simple: type, press send, and the message appears on your friend's phone a moment later, marked delivered and then read. Behind that, the system must keep tens of millions of phones and browsers connected at once, find which of a thousand servers holds the recipient's connection, store every message durably, show messages in the same order to everyone in the conversation, and reach people whose phones are asleep.

What makes chat different from most web systems is the long-lived connection. A request-response API can send any request to any server. A chat server holds open sockets, so it has state: to deliver a message to Bob, you must find the specific server Bob is connected to right now, and that changes every time his train goes through a tunnel.

Requirements

Functional

  • One-to-one and group conversations (groups up to a few hundred members; large channels are an extension).
  • Send and receive text messages in real time; media through separate upload.
  • Message history on every device, in the same order for everyone.
  • Sent, delivered and read receipts; online and last-seen presence.
  • Push notifications for users who are offline.

Non-functional

  • Scale: 500 million daily users, 40 messages per user per day, 100 million connections at peak.
  • Latency: under 500 ms end to end when both users are online.
  • Durability: a message acknowledged to the sender is never lost.
  • Ordering: messages within a conversation appear in one agreed order on all devices.

Out of scope

  • Voice and video calls, message search, and the cryptography of end-to-end encryption beyond its design consequences.

Estimates

  • Messages: 500M x 40 = 20 billion a day, about 230,000 per second on average and roughly 700,000 at a 3x peak.
  • Storage: at about 200 bytes per message with metadata, 4 TB a day, roughly 1.5 PB a year before replication. Writes dominate and reads are mostly recent, which points to a write-optimized store partitioned by conversation.
  • Connections: if a gateway server holds 100,000 open WebSockets (a planning figure limited mostly by memory; measure yours), 100 million connections need 1,000 gateways, plus headroom for a region failing over.
  • Presence: with heartbeats every 30 seconds, 100 million connections send over 3 million heartbeats per second. Presence must be handled at the gateways and in memory, never written to a database per heartbeat.

API

Clients talk to gateways over one WebSocket carrying small framed messages:

client -> server  SEND    { conv: "c_91", client_msg_id: "a3", body: "see you at 6" }
server -> client  ACK     { client_msg_id: "a3", seq: 42, ts: ... }
server -> client  MSG     { conv: "c_91", seq: 42, from: "alice", body: "..." }
client -> server  RECEIPT { conv: "c_91", delivered_up_to: 42, read_up_to: 40 }
client -> server  SYNC    { conv: "c_91", after_seq: 37 }      # after reconnecting

History, conversation lists and media uploads use an ordinary HTTPS API.

Data model

  • Messages, in a wide-column store ( NoSQL families ), partition key (conversation_id, bucket) and clustering key seq, where the bucket is a time window such as ten days. A conversation's recent messages are one sequential read, and very active conversations do not grow one partition forever.
  • Conversations and membership: conversation_id -> members, and user_id -> conversations sorted by last activity, for the inbox screen.
  • Session registry (in memory, for example Redis): user_id -> [(device, gateway_id)], with a short TTL refreshed by heartbeats.
  • Receipts: per member per conversation, last_delivered_seq and last_read_seq. Two numbers, not one row per message.

High-level design

Chat system: send, deliver and offline paths Send path: Alice's client sends over a WebSocket to gateway A, which forwards to the message service; it assigns the next sequence number for the conversation, stores the message in the message store and acknowledges Alice. Deliver path: the message service looks up Bob's gateway in the session registry and forwards the message to gateway B, which pushes it to Bob; Bob's client returns a delivered receipt. Offline path: if Bob has no live session, the message service asks the push service to notify his phone through APNs or FCM. SEND a3 append seq 42 find Bob MSG 42 Bob offline Alice Gateway A WebSockets Message service seq per conversation Message store key = conversation Session registry user -> gateway Gateway B WebSockets Bob Push service APNs / FCM phone asleep Figure 1. Gateways hold connections; the message service holds the logic. Messages are stored before they are acknowledged, then routed to wherever the recipient is connected, or to a push notification if nowhere.
  1. Send path Alice's client sends the message with a client-generated id. The message service assigns the conversation's next sequence number, writes the message to the store, and only then acknowledges Alice with the sequence number. A retried SEND with the same client id gets the same ACK, not a duplicate.
  2. Deliver path The service looks up Bob's live sessions in the registry and forwards the message to each gateway holding one. Gateway B pushes it down Bob's socket; Bob's client sends back a receipt saying it has everything up to seq 42, which is relayed to Alice as 'delivered'.
  3. Offline path Bob has no live session. The message is already stored, so nothing is lost; the push service sends a notification through Apple's or Google's push gateway. When Bob opens the app it reconnects and syncs everything after the last sequence number it has.

Deep dives

Deep dive 1: Gateways and routing to a connection

Gateways do as little as possible: terminate TLS and WebSockets, authenticate the connection, keep heartbeats, and pass frames to and from the message service. Keeping them thin means they rarely need deploying, which matters because every deploy disconnects users. Clients connect through a load balancer that picks any gateway ( L4 vs L7 load balancing ), so the system must record where each user landed: the gateway writes user -> gateway into the session registry on connect and refreshes it with each heartbeat, so entries for dead gateways expire on their own.

To deliver, the message service looks up the recipient's sessions and sends the message to those gateways, over persistent internal connections or a per-gateway queue. A user with a phone and a laptop has two sessions and gets both copies. The alternative, a pub/sub topic per user that gateways subscribe to, trades the registry lookup for a pub/sub system that must handle millions of topics. Either works; the registry is simpler to reason about.

When a gateway dies, its 100,000 clients reconnect within seconds, ideally with jittered backoff so they do not stampede the survivors ( Timeouts, retries, backoff and jitter ). Nothing is lost, because messages were stored before delivery and each client syncs from its last sequence number.

Deep dive 2: Storage, ordering and exactly-once display

Chat history is write-heavy, append-only and read mostly by "latest messages in this conversation", which is exactly what a wide-column store partitioned by conversation is good at ( Storage engines: B-tree vs LSM-tree ). Ordering is defined per conversation: the message service assigns each conversation a monotonically increasing sequence number, for example by routing all of a conversation's writes to one owner using consistent hashing, which assigns seq from an in-memory counter persisted with the message. Global ordering across conversations is not needed and would be a bottleneck.

The network still delivers out of order and twice: reconnects replay, gateways retry. So clients keep a small reorder buffer and dedupe by message id. Here is a client receiving seven frames, with a gap and a duplicate:

ordering.pyruns with python3 (standard library)
# Per-conversation ordering and dedupe on the client.
# The server gives each message in a conversation a sequence number.
# Messages may arrive out of order (reconnects) or twice (retries).
arrivals = [  # (seq, client_msg_id, text), in arrival order
    (1, "a1", "hi"), (2, "b1", "hey!"), (4, "a3", "see you at 6"),
    (3, "a2", "dinner?"), (4, "a3", "see you at 6"), (6, "b3", "ok"),
    (5, "b2", "sounds good"),
]

shown, buffer, seen = [], {}, set()
next_seq = 1
for seq, cid, text in arrivals:
    if cid in seen:
        print(f"seq {seq}: duplicate {cid}, dropped")
        continue
    seen.add(cid)
    buffer[seq] = text
    if seq != next_seq:
        print(f"seq {seq}: gap, waiting for seq {next_seq}")
    while next_seq in buffer:          # release everything now in order
        shown.append(buffer.pop(next_seq))
        next_seq += 1

print("displayed:", " | ".join(shown))
print("last seq read:", next_seq - 1, "(sent to server as the read receipt)")

Output

seq 4: gap, waiting for seq 3
seq 4: duplicate a3, dropped
seq 6: gap, waiting for seq 5
displayed: hi | hey! | dinner? | see you at 6 | sounds good | ok
last seq read: 6 (sent to server as the read receipt)

The client held seq 4 until seq 3 arrived, dropped the duplicate, and displayed everything in order. If a gap does not fill within a second or two, the client asks the server with SYNC after_seq. The same idea, at-least-once delivery plus idempotent handling, gives users effectively exactly-once display ( Delivery semantics ). On the send side, the client_msg_id makes retries safe: the service remembers recent client ids per conversation and returns the original ACK.

Deep dive 3: Receipts, presence and groups

Receipts are cumulative: "delivered up to 42" and "read up to 40" per member, two numbers per conversation. That is far cheaper than a status row per message, and for groups the sender's "read by 5 of 8" is a count over members' read_up_to.

Presence is the noisiest data in the system and the least important to persist. Gateways know who is connected; they publish online and offline transitions (not heartbeats) to a presence service, which fans them out only to users who currently have that contact on screen. Debounce flapping connections: wait a few seconds before announcing "offline". For large groups, show presence only on request.

Group fan-out: a message to a group of 200 is stored once in the conversation's partition, then delivered to each online member's gateway. The fan-out is on delivery, not on storage, unlike the news feed's timeline fan-out ( Design a news feed (Twitter/Instagram) ). Very large channels (tens of thousands of members) switch to a pull model: clients are told "channel X has new messages" and fetch them.

End-to-end encryption, briefly

With end-to-end encryption, clients encrypt messages with keys the server never sees, so the server stores and routes ciphertext. The architecture above does not change, but anything that needs to read content on the server, such as search, previews in push notifications, or spam filtering, must move to the client or go away. Group messages are encrypted for each member's devices, which makes adding devices and members a key-distribution problem.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Set 1 million connected users and a 5-second polling interval. How many requests per second does short polling cost compared with WebSockets?
  2. Lower the message rate to one per minute. Which approach wastes the most work when chats are quiet?
  3. Chat needs both directions (typing indicators, receipts). Which options handle client-to-server messages without a separate request?

Bottlenecks and failure modes

FailureImpactMitigation
Gateway crash100,000 clients disconnectedJittered reconnect; sync from last seq; nothing lost
Message service owner for a conversation downSends to it stallReassign ownership via the hash ring; seq continues from the stored maximum
Message store slowSends not acknowledgedClients retry with the same client id; backpressure at gateways
Hot group conversationOne partition and owner overloadedTime buckets, pull model for huge channels
Presence storm (mass reconnect)Millions of status updatesDebounce, send transitions only to interested viewers
Push provider outageOffline users not notifiedRetry with backoff; messages wait in storage for sync

Trade-offs we made

OptionWins whenLoses when
WebSockets (chosen) Two-way, low latency, one connection per device Stateful servers; routing to connections needed
Store before acknowledging (chosen) An acknowledged message is never lost Send latency includes a durable write
Ordering per conversation (chosen) Scales out; matches what users see No global order across conversations
Session registry lookup (chosen) Simple, explicit routing Another in-memory system to keep available
Cumulative receipts (chosen) Two numbers per member instead of a row per message Cannot mark a single older message unread

Extensions

  • Large channels with thousands of members, delivered by notify-and-pull.
  • Multi-region: users connect to the nearest region; each conversation has a home region that assigns sequence numbers.
  • Message edits, deletes and reactions as new events in the same sequence, applied by clients.
  • Search over history, which conflicts with end-to-end encryption.

In the wild

Slack

Real-time Messaging (opens in a new tab)

Slack's real-time path is split across services that mirror this design. Channel Servers are stateful, in-memory and own channels by consistent hashing on the channel id (about 16 million channels per host at peak), while Gateway Servers, deployed across regions, hold users' WebSocket subscriptions and act as the interface between clients and Channel Servers. A sent message goes from the web backend through an Admin Server to the channel's Channel Server, which sends it to every subscribed Gateway Server and on to connected clients. Presence Servers track who is online, and the post says the system delivers messages across the world in 500 ms.

Discord

How Discord stores trillions of messages (opens in a new tab)

Discord stores messages partitioned by channel and a time bucket, the same key this lesson uses. By 2022 its Cassandra cluster had grown to 177 nodes holding trillions of messages and suffered from hot partitions, garbage collection pauses and compaction backlogs. Discord moved to ScyllaDB and put Rust data services in front of the database that route requests by channel id with consistent hashing and coalesce identical concurrent requests into one query. After the migration the cluster went from 177 nodes to 72, and p99 latency for fetching historical messages dropped from 40 to 125 ms to 15 ms.

Check yourself

  1. Why does a chat system need a session registry?

    Show answer

    B. Connections are long-lived, so delivery must find the specific gateway holding the recipient's socket Any server can take a request, but only one gateway holds Bob's open socket right now.

  2. When is the sender's message acknowledged?

    Show answer

    B. After it is durably stored with its sequence number Acknowledging only after the durable write guarantees an acknowledged message is never lost.

  3. A client receives seq 41, then 43. What should it do?

    Show answer

    B. Hold 43 briefly, and request seq 42 with a sync if it does not arrive Per-conversation sequence numbers let clients detect gaps and fill them, giving a consistent order.

  4. How is a message to a 200-member group stored?

    Show answer

    B. Once in the conversation's partition, then delivered to each online member Fan-out happens on delivery; storage stays one copy per message.

  5. Why are presence heartbeats not written to a database?

    Show answer

    B. Millions per second of low-value updates would overwhelm it; gateways track them in memory Only transitions matter, and only to users looking at that contact.

Recap

  • Thin WebSocket gateways hold connections; a session registry maps users to gateways.
  • Store each message with a per-conversation sequence number before acknowledging it.
  • Clients reorder and dedupe by sequence and message id, and sync gaps after reconnects.
  • Receipts are cumulative per member; presence lives in memory and sends only transitions.
  • Groups store a message once and fan out on delivery; offline users get push notifications.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up