Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Design Case Studies

System Design HLD-19.08

Design a notification system

Design a notification service: push, SMS and email channels, templates, user preferences, priority queues, dedupe, per-user limits and provider failover.

Intermediate 18 min Last reviewed

The problem

Dozens of services want to tell users something: the payments service ("your card was charged"), security ("new login from Lisbon"), social ("Maya liked your photo"), marketing ("20 percent off this weekend"). Each could call Apple's push service, an email provider and an SMS gateway directly, but then every team would reimplement templates, unsubscribe rules and retries, and nobody could stop a user from getting forty notifications in an hour.

A notification system is the shared service in the middle. It accepts "notify this user about this event", decides whether, when and how to notify, renders the message, and delivers it through third-party providers that are slow, rate-limited and occasionally down. The interesting parts are not the sending; they are the rules before sending and the failure handling after.

Requirements

Functional

  • Accept notification requests from internal services: user, category, template, data, optional channels.
  • Channels: mobile push, email and SMS (in-app inbox as an extension).
  • Respect user preferences: per-category opt-outs, preferred channels, quiet hours.
  • Templates with localization.
  • Track delivery status: sent, delivered, failed, opened.

Non-functional

  • Scale: 100 million users; about 1 billion push, 100 million emails and 5 million SMS a day; marketing campaigns of 50 million in an hour.
  • Latency: security and transactional notifications (one-time codes, fraud alerts) delivered within seconds even during a campaign.
  • No duplicates from retries; no notifications to users who opted out.
  • Survive any single provider being down.

Out of scope

  • Choosing what to notify about (the calling services decide), and the content of marketing campaigns.

Estimates

  • Push: 1 billion a day is about 11,600 per second on average.
  • Campaign peak: 50 million in an hour is about 13,900 per second, on top of the normal load. Campaigns must not delay transactional messages, so they need separate queues and capacity. Whether each channel gets its own work queue or subscribes to one shared topic is the choice laid out in message queue vs pub/sub vs event streams .
  • Email: 100 million a day, about 1,200 per second. SMS: 5 million a day, about 58 per second, but each costs real money, so SMS is reserved for categories that justify it.
  • Delivery log: 1.1 billion records a day at about 500 bytes is about 550 GB a day; 30 days of retention is about 16 TB, in a store built for append-heavy writes.

API

POST /v1/notifications
  Idempotency-Key: payments-charge-ch_1N8x
  { "user_id": "u_42", "category": "payment_receipt", "priority": "high",
    "template": "charge_succeeded", "data": { "amount": "50.00", "currency": "EUR" } }
  202 Accepted { "notification_id": "n_9f..." }

GET /v1/notifications/{id}         -> status per channel
PUT /v1/users/{id}/preferences     -> opt-outs, channels, quiet hours

The call returns 202 as soon as the request is validated and queued; delivery is asynchronous.

Data model

  • Preferences, keyed by user_id: opt-outs per category, channel order, quiet hours and time zone. Read on every notification, so cached.
  • Devices, keyed by user_id: push tokens per device, email address, phone number. Tokens that providers report as invalid are removed.
  • Templates, keyed by (template, locale, channel), versioned.
  • Delivery log, partitioned by notification_id with an index by user: one row per attempt with provider, status and timestamps.
  • Dedupe keys, in an in-memory store with a TTL of a day: (user_id, idempotency_key).

High-level design

Notification system: accept, dispatch and feedback paths Accept path: internal services call the notification API, which checks the idempotency key, user preferences, quiet hours and per-user limits, then enqueues the notification into a queue chosen by channel and priority. Dispatch path: channel workers take messages, render templates and call the provider for that channel: APNs and FCM for push, an email provider, or one of two SMS providers. Feedback path: providers report delivery status, which is written to the delivery log; failures are retried with backoff and invalid tokens removed. prefs, limits status record attempt retry later Services payments, social Notification API rules + dedupe Preferences cached Push: high Push: bulk Email / SMS Channel workers render, send APNs / FCM Email provider SMS A / B failover Delivery log Figure 1. Decide first, then queue by channel and priority, then send. High-priority messages have their own queue so a campaign never delays a security code. Step through the three paths.
  1. Accept path The API checks the idempotency key, loads the user's preferences, applies opt-outs, quiet hours and per-user limits, and enqueues one message per chosen channel into a queue for that channel and priority. It answers 202 at once.
  2. Dispatch path Channel workers pull from their queues, high priority first, render the template in the user's language, and call the provider. Each provider has its own worker pool and rate limit, so a slow email provider never holds up push.
  3. Feedback path Every attempt is written to the delivery log. Temporary failures are retried with exponential backoff; permanent ones, such as an invalid push token, remove the token. After a few failures a message moves to a dead letter queue for inspection.

Deep dives

Deep dive 1: The rules before sending

Most of the product value is in deciding not to send. Run through what one batch of requests looks like after the rules:

pipeline.pyruns with python3 (standard library)
# The decision steps a notification passes before any provider is called.
prefs = {"ana": {"marketing": False, "quiet": (22, 7)},
         "raj": {"marketing": True, "quiet": None}}
CAP_PER_HOUR = {"marketing": 2, "social": 5, "security": None}  # None: no cap

requests = [  # (hour, user, category, dedupe_key)
    (9, "ana", "security", "login-77"),
    (9, "ana", "security", "login-77"),     # retried by the caller
    (10, "ana", "marketing", "promo-1"),
    (23, "ana", "social", "like-5"),        # inside quiet hours
    (14, "raj", "marketing", "promo-1"),
    (14, "raj", "marketing", "promo-2"),
    (14, "raj", "marketing", "promo-3"),    # over the hourly cap
    (14, "raj", "social", "like-9"),
]

def quiet(p, hour):
    q = p["quiet"]
    return q is not None and (hour >= q[0] or hour < q[1])

seen, sent_in_hour = set(), {}
for hour, user, cat, key in requests:
    p, slot = prefs[user], (user, cat, hour)
    if (user, key) in seen:
        verdict = "drop: duplicate"
    elif cat == "marketing" and not p["marketing"]:
        verdict = "drop: user opted out"
    elif cat != "security" and quiet(p, hour):
        verdict = "delay: quiet hours, send at 07:00"
    elif CAP_PER_HOUR[cat] and sent_in_hour.get(slot, 0) >= CAP_PER_HOUR[cat]:
        verdict = "drop: per-user rate limit"
    else:
        sent_in_hour[slot] = sent_in_hour.get(slot, 0) + 1
        verdict = "send"
    seen.add((user, key))
    print(f"{hour:02d}:00 {user:4} {cat:9} {key:9} -> {verdict}")

Output

09:00 ana  security  login-77  -> send
09:00 ana  security  login-77  -> drop: duplicate
10:00 ana  marketing promo-1   -> drop: user opted out
23:00 ana  social    like-5    -> delay: quiet hours, send at 07:00
14:00 raj  marketing promo-1   -> send
14:00 raj  marketing promo-2   -> send
14:00 raj  marketing promo-3   -> drop: per-user rate limit
14:00 raj  social    like-9    -> send

The order of the checks matters:

  1. Dedupe first: a caller that retries after a timeout sends the same idempotency key, and the second request is dropped ( Idempotency and safe retries ).
  2. Opt-outs by category, legally required for marketing in many places.
  3. Quiet hours in the user's time zone: delay, do not drop, and never delay security messages.
  4. Per-user rate limits by category, such as two marketing messages an hour, so a bug in a calling service cannot spam users ( Rate limiting algorithms ).
  5. Channel choice: push if the user has a device, otherwise email; SMS only for categories that warrant the cost.

Some products also batch: twenty "liked your photo" events in ten minutes become one "Maya and 19 others liked your photo". That needs a short holding window per user and category, keyed in the same in-memory store as the dedupe keys.

Deep dive 2: Queues by channel and priority

One shared queue would let a 50-million-message campaign sit in front of a one-time login code for half an hour. Separate queues per channel and priority give each kind of traffic its own lane ( Backpressure and flow control ):

  • High priority (security codes, fraud alerts, payment receipts): small volume, dedicated workers, alerts on any queue age above a few seconds.
  • Bulk (social and marketing): large volume, workers scaled to provider limits; campaigns are fed in gradually by a scheduler rather than dumped in at once.
  • Per channel: email and SMS have their own queues and workers, because their providers have different rate limits and failure modes.

Queues also absorb provider slowdowns: if the email provider slows down, its queue grows while push carries on. Try it with the queue simulator below.

Deep dive 3: Providers fail, so plan for it

Third-party providers return errors, throttle you, and occasionally go down. Classify every response:

For SMS and email, contract with two providers and put a circuit breaker on each ( Circuit breakers and bulkheads ): when one's error rate crosses a threshold, route traffic to the other. Push is different, because each platform has exactly one gateway (Apple's for iOS, Google's for Android), so there is nothing to fail over to; queue and retry instead.

Retries create duplicates: if a provider accepted the message but the response was lost, a retry sends it twice. Pass a stable message id to providers that support idempotency, and accept that for push, rare duplicates are possible. That is the honest limit of at-least-once delivery to an external system ( Delivery semantics ).

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Pick the 20-second spike, as when a campaign starts. How long does the backlog take to drain with 8 workers? With 12?
  2. Imagine a security code arriving at the peak of that backlog in the same queue. How long would it wait? That wait is why high-priority messages get their own queue.
  3. Set workers so that capacity is just below the producer rate on steady traffic. What happens to the queue over two minutes, and what would you alert on?

Bottlenecks and failure modes

FailureImpactMitigation
Campaign floods the systemTransactional messages delayedSeparate high-priority queues and workers; paced campaign feeding
Email or SMS provider downChannel stallsSecond provider behind a circuit breaker
Push gateway throttlingBulk push delayedBackoff, per-provider rate limits, queue absorbs it
Caller retriesDuplicate notificationsIdempotency keys checked before enqueue
Calling service bug sends thousandsUser spammedPer-user, per-category rate limits
Stale push tokensWasted sends, provider complaintsRemove tokens reported invalid

Trade-offs we made

OptionWins whenLoses when
Central notification service (chosen) One place for preferences, limits, templates and providers A shared dependency every team relies on
Queues per channel and priority (chosen) Campaigns cannot delay security codes; failures isolated per channel More queues and worker pools to run
Asynchronous 202 API (chosen) Callers never wait for providers Callers must query status if they care
At-least-once delivery Messages are not lost when providers fail Rare duplicates where providers cannot dedupe

Extensions

  • In-app inbox: store notifications per user and show them in the app, which is also the fallback when push is disabled.
  • Digest batching: combine low-priority events into periodic summaries.
  • Send-time optimization: deliver marketing when each user usually opens the app.
  • Analytics: opens and clicks per template, fed back to calling teams.

In the wild

Where you will see this: every consumer app with push notifications runs a service of this shape, and cloud providers and specialised companies sell it as a managed product. The building blocks are always the same: one push gateway per mobile platform, a choice of email and SMS providers behind them, and a layer of preferences, limits and retries in front.

Check yourself

  1. Why do high-priority notifications get their own queue?

    Show answer

    B. So a large campaign backlog never delays security codes or receipts A shared queue processes in order; separate lanes keep urgent messages fast whatever the bulk backlog.

  2. A notification arrives at 23:30 local time for a user with quiet hours 22:00 to 07:00. It is a 'liked your photo' message. What happens?

    Show answer

    C. Delay it until 07:00 Quiet hours delay non-urgent notifications; security messages would still go out immediately.

  3. The SMS provider's error rate jumps to 60 percent. What should the system do?

    Show answer

    B. Open its circuit breaker and route SMS to the second provider Two providers plus a circuit breaker turn a provider outage into a brief blip.

  4. The payments service times out and retries the same notification request. How is a duplicate avoided?

    Show answer

    B. The idempotency key is checked before enqueueing, and the retry is dropped Dedupe at the entry point stops duplicates before any provider is involved.

Recap

  • One shared service owns preferences, limits, templates and provider integrations.
  • Most value is in the rules: dedupe, opt-outs, quiet hours, per-user limits, channel choice.
  • Separate queues per channel and priority so bulk traffic never delays urgent messages.
  • Classify provider errors: retry with backoff, drop permanent failures, dead-letter the rest.
  • Use two providers with circuit breakers where possible; accept rare duplicates with push.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up