System Design HLD-19.08
Design a notification system
Design a notification service: push, SMS and email channels, templates, user preferences, priority queues, dedupe, per-user limits and provider failover.
The problem
Dozens of services want to tell users something: the payments service ("your card was charged"), security ("new login from Lisbon"), social ("Maya liked your photo"), marketing ("20 percent off this weekend"). Each could call Apple's push service, an email provider and an SMS gateway directly, but then every team would reimplement templates, unsubscribe rules and retries, and nobody could stop a user from getting forty notifications in an hour.
A notification system is the shared service in the middle. It accepts "notify this user about this event", decides whether, when and how to notify, renders the message, and delivers it through third-party providers that are slow, rate-limited and occasionally down. The interesting parts are not the sending; they are the rules before sending and the failure handling after.
Requirements
Functional
- Accept notification requests from internal services: user, category, template, data, optional channels.
- Channels: mobile push, email and SMS (in-app inbox as an extension).
- Respect user preferences: per-category opt-outs, preferred channels, quiet hours.
- Templates with localization.
- Track delivery status: sent, delivered, failed, opened.
Non-functional
- Scale: 100 million users; about 1 billion push, 100 million emails and 5 million SMS a day; marketing campaigns of 50 million in an hour.
- Latency: security and transactional notifications (one-time codes, fraud alerts) delivered within seconds even during a campaign.
- No duplicates from retries; no notifications to users who opted out.
- Survive any single provider being down.
Out of scope
- Choosing what to notify about (the calling services decide), and the content of marketing campaigns.
Estimates
- Push: 1 billion a day is about 11,600 per second on average.
- Campaign peak: 50 million in an hour is about 13,900 per second, on top of the normal load. Campaigns must not delay transactional messages, so they need separate queues and capacity. Whether each channel gets its own work queue or subscribes to one shared topic is the choice laid out in message queue vs pub/sub vs event streams .
- Email: 100 million a day, about 1,200 per second. SMS: 5 million a day, about 58 per second, but each costs real money, so SMS is reserved for categories that justify it.
- Delivery log: 1.1 billion records a day at about 500 bytes is about 550 GB a day; 30 days of retention is about 16 TB, in a store built for append-heavy writes.
API
POST /v1/notifications
Idempotency-Key: payments-charge-ch_1N8x
{ "user_id": "u_42", "category": "payment_receipt", "priority": "high",
"template": "charge_succeeded", "data": { "amount": "50.00", "currency": "EUR" } }
202 Accepted { "notification_id": "n_9f..." }
GET /v1/notifications/{id} -> status per channel
PUT /v1/users/{id}/preferences -> opt-outs, channels, quiet hours
The call returns 202 as soon as the request is validated and queued; delivery is asynchronous.
Data model
- Preferences, keyed by
user_id: opt-outs per category, channel order, quiet hours and time zone. Read on every notification, so cached. - Devices, keyed by
user_id: push tokens per device, email address, phone number. Tokens that providers report as invalid are removed. - Templates, keyed by
(template, locale, channel), versioned. - Delivery log, partitioned by
notification_idwith an index by user: one row per attempt with provider, status and timestamps. - Dedupe keys, in an in-memory store with a TTL of a day:
(user_id, idempotency_key).
High-level design
- Accept path The API checks the idempotency key, loads the user's preferences, applies opt-outs, quiet hours and per-user limits, and enqueues one message per chosen channel into a queue for that channel and priority. It answers 202 at once.
- Dispatch path Channel workers pull from their queues, high priority first, render the template in the user's language, and call the provider. Each provider has its own worker pool and rate limit, so a slow email provider never holds up push.
- Feedback path Every attempt is written to the delivery log. Temporary failures are retried with exponential backoff; permanent ones, such as an invalid push token, remove the token. After a few failures a message moves to a dead letter queue for inspection.
Deep dives
Deep dive 1: The rules before sending
Most of the product value is in deciding not to send. Run through what one batch of requests looks like after the rules:
# The decision steps a notification passes before any provider is called.
prefs = {"ana": {"marketing": False, "quiet": (22, 7)},
"raj": {"marketing": True, "quiet": None}}
CAP_PER_HOUR = {"marketing": 2, "social": 5, "security": None} # None: no cap
requests = [ # (hour, user, category, dedupe_key)
(9, "ana", "security", "login-77"),
(9, "ana", "security", "login-77"), # retried by the caller
(10, "ana", "marketing", "promo-1"),
(23, "ana", "social", "like-5"), # inside quiet hours
(14, "raj", "marketing", "promo-1"),
(14, "raj", "marketing", "promo-2"),
(14, "raj", "marketing", "promo-3"), # over the hourly cap
(14, "raj", "social", "like-9"),
]
def quiet(p, hour):
q = p["quiet"]
return q is not None and (hour >= q[0] or hour < q[1])
seen, sent_in_hour = set(), {}
for hour, user, cat, key in requests:
p, slot = prefs[user], (user, cat, hour)
if (user, key) in seen:
verdict = "drop: duplicate"
elif cat == "marketing" and not p["marketing"]:
verdict = "drop: user opted out"
elif cat != "security" and quiet(p, hour):
verdict = "delay: quiet hours, send at 07:00"
elif CAP_PER_HOUR[cat] and sent_in_hour.get(slot, 0) >= CAP_PER_HOUR[cat]:
verdict = "drop: per-user rate limit"
else:
sent_in_hour[slot] = sent_in_hour.get(slot, 0) + 1
verdict = "send"
seen.add((user, key))
print(f"{hour:02d}:00 {user:4} {cat:9} {key:9} -> {verdict}")Output
09:00 ana security login-77 -> send
09:00 ana security login-77 -> drop: duplicate
10:00 ana marketing promo-1 -> drop: user opted out
23:00 ana social like-5 -> delay: quiet hours, send at 07:00
14:00 raj marketing promo-1 -> send
14:00 raj marketing promo-2 -> send
14:00 raj marketing promo-3 -> drop: per-user rate limit
14:00 raj social like-9 -> send The order of the checks matters:
- Dedupe first: a caller that retries after a timeout sends the same idempotency key, and the second request is dropped ( Idempotency and safe retries ).
- Opt-outs by category, legally required for marketing in many places.
- Quiet hours in the user's time zone: delay, do not drop, and never delay security messages.
- Per-user rate limits by category, such as two marketing messages an hour, so a bug in a calling service cannot spam users ( Rate limiting algorithms ).
- Channel choice: push if the user has a device, otherwise email; SMS only for categories that warrant the cost.
Some products also batch: twenty "liked your photo" events in ten minutes become one "Maya and 19 others liked your photo". That needs a short holding window per user and category, keyed in the same in-memory store as the dedupe keys.
Deep dive 2: Queues by channel and priority
One shared queue would let a 50-million-message campaign sit in front of a one-time login code for half an hour. Separate queues per channel and priority give each kind of traffic its own lane ( Backpressure and flow control ):
- High priority (security codes, fraud alerts, payment receipts): small volume, dedicated workers, alerts on any queue age above a few seconds.
- Bulk (social and marketing): large volume, workers scaled to provider limits; campaigns are fed in gradually by a scheduler rather than dumped in at once.
- Per channel: email and SMS have their own queues and workers, because their providers have different rate limits and failure modes.
Queues also absorb provider slowdowns: if the email provider slows down, its queue grows while push carries on. Try it with the queue simulator below.
Deep dive 3: Providers fail, so plan for it
Third-party providers return errors, throttle you, and occasionally go down. Classify every response:
- Retryable (timeouts, 5xx, throttling): retry with exponential backoff and jitter ( Timeouts, retries, backoff and jitter ), respecting any retry-after hint.
- Permanent (invalid token, unsubscribed address, invalid number): do not retry; update the device or address records.
- Too many failures: move the message to a dead letter queue ( Retries, dead letter queues and poison messages ) and alert.
For SMS and email, contract with two providers and put a circuit breaker on each ( Circuit breakers and bulkheads ): when one's error rate crosses a threshold, route traffic to the other. Push is different, because each platform has exactly one gateway (Apple's for iOS, Google's for Android), so there is nothing to fail over to; queue and retry instead.
Retries create duplicates: if a provider accepted the message but the response was lost, a retry sends it twice. Pass a stable message id to providers that support idempotency, and accept that for push, rare duplicates are possible. That is the honest limit of at-least-once delivery to an external system ( Delivery semantics ).
Try it
Try this:
- Pick the 20-second spike, as when a campaign starts. How long does the backlog take to drain with 8 workers? With 12?
- Imagine a security code arriving at the peak of that backlog in the same queue. How long would it wait? That wait is why high-priority messages get their own queue.
- Set workers so that capacity is just below the producer rate on steady traffic. What happens to the queue over two minutes, and what would you alert on?
Bottlenecks and failure modes
| Failure | Impact | Mitigation |
|---|---|---|
| Campaign floods the system | Transactional messages delayed | Separate high-priority queues and workers; paced campaign feeding |
| Email or SMS provider down | Channel stalls | Second provider behind a circuit breaker |
| Push gateway throttling | Bulk push delayed | Backoff, per-provider rate limits, queue absorbs it |
| Caller retries | Duplicate notifications | Idempotency keys checked before enqueue |
| Calling service bug sends thousands | User spammed | Per-user, per-category rate limits |
| Stale push tokens | Wasted sends, provider complaints | Remove tokens reported invalid |
Trade-offs we made
| Option | Wins when | Loses when |
|---|---|---|
| Central notification service (chosen) | One place for preferences, limits, templates and providers | A shared dependency every team relies on |
| Queues per channel and priority (chosen) | Campaigns cannot delay security codes; failures isolated per channel | More queues and worker pools to run |
| Asynchronous 202 API (chosen) | Callers never wait for providers | Callers must query status if they care |
| At-least-once delivery | Messages are not lost when providers fail | Rare duplicates where providers cannot dedupe |
Extensions
- In-app inbox: store notifications per user and show them in the app, which is also the fallback when push is disabled.
- Digest batching: combine low-priority events into periodic summaries.
- Send-time optimization: deliver marketing when each user usually opens the app.
- Analytics: opens and clicks per template, fed back to calling teams.
In the wild
Where you will see this: every consumer app with push notifications runs a service of this shape, and cloud providers and specialised companies sell it as a managed product. The building blocks are always the same: one push gateway per mobile platform, a choice of email and SMS providers behind them, and a layer of preferences, limits and retries in front.
Check yourself
-
Why do high-priority notifications get their own queue?
Show answer
B. So a large campaign backlog never delays security codes or receipts A shared queue processes in order; separate lanes keep urgent messages fast whatever the bulk backlog.
-
A notification arrives at 23:30 local time for a user with quiet hours 22:00 to 07:00. It is a 'liked your photo' message. What happens?
Show answer
C. Delay it until 07:00 Quiet hours delay non-urgent notifications; security messages would still go out immediately.
-
The SMS provider's error rate jumps to 60 percent. What should the system do?
Show answer
B. Open its circuit breaker and route SMS to the second provider Two providers plus a circuit breaker turn a provider outage into a brief blip.
-
The payments service times out and retries the same notification request. How is a duplicate avoided?
Show answer
B. The idempotency key is checked before enqueueing, and the retry is dropped Dedupe at the entry point stops duplicates before any provider is involved.
Recap
- One shared service owns preferences, limits, templates and provider integrations.
- Most value is in the rules: dedupe, opt-outs, quiet hours, per-user limits, channel choice.
- Separate queues per channel and priority so bulk traffic never delays urgent messages.
- Classify provider errors: retry with backoff, drop permanent failures, dead-letter the rest.
- Use two providers with circuit breakers where possible; accept rare duplicates with push.
Next up
- Crack the system design round learning path