Skip to content
learn
  1. Learn
  2. High Level System Design
  3. APIs and Communication

System Design HLD-03.07

Webhooks

HTTP callbacks from one system to another: push versus pull, signed payloads, retries with backoff, replay protection and idempotent consumers.

Intermediate 8 min Last reviewed

The big idea

Your app sells subscriptions through a payment provider. When a customer's card is charged next month, how does your app find out? It could ask the provider every minute, "any new payments?", which is slow and wasteful for both sides. Or the provider could tell you the moment it happens. A webhookAn HTTP request a provider sends to a URL you registered whenever a specified event happens, carrying the event as JSON. is exactly that: you register a URL, and when an event occurs, the provider sends an HTTP POST describing it to your URL.

Webhooks are how payment processors, Git hosts, chat platforms and countless SaaS tools integrate with each other. They are simple to receive, but because your endpoint is a public URL that the internet can reach, and because the provider's deliveries can fail and be retried, a correct receiver needs three protections: verify the sender, tolerate duplicates, and respond fast.

Think of a doorbell instead of opening the door every few minutes to check for visitors. But anyone can press a doorbell, and the courier may ring twice.

How it works

Step 1: The delivery loop

Webhook delivery with signing, queueing and retries An event happens at the provider. The provider signs the payload and puts it on a delivery queue. A delivery worker POSTs it to the consumer's endpoint. If the endpoint fails or times out, the event goes back to the queue with exponential backoff and is retried. When the consumer responds 2xx, delivery stops. The consumer verifies the signature, records the event id, queues the work and returns quickly. Provider Your app fail: retry later, 2^n backoff invoice.paid event happens Delivery queue signed payloads Worker POST + timeout POST /webhooks verify, dedupe Job queue real work later 200 OK fast Figure 1. The provider signs each event and delivers it from a queue, retrying with growing delays until the consumer answers 2xx (the glowing retry loop). The consumer verifies, deduplicates, enqueues and acknowledges at once, doing the real work later.

Step 2: The provider's responsibilities

  • Sign every payload with a secret shared with the consumer, typically an HMAC of the timestamp and the body, sent in a header.
  • Retry failures (timeouts, connection errors, non-2xx responses) with exponential backoff over hours or days, then give up and surface the failure in a dashboard.
  • Include a unique event id so consumers can recognize repeats, and let consumers re-fetch the event or list missed ones through the normal API.

Step 3: The consumer's responsibilities

webhook-signature.pyruns with python3 (standard library)
import hashlib, hmac, json

SECRET = b"whsec_shared_between_provider_and_you"

def sign(body: bytes, timestamp: int) -> str:
    msg = f"{timestamp}.".encode() + body          # sign the time too, to stop replays
    return hmac.new(SECRET, msg, hashlib.sha256).hexdigest()

def verify(body: bytes, timestamp: int, signature: str, now: int, seen: set) -> str:
    if abs(now - timestamp) > 300:
        return "rejected: too old (possible replay)"
    if not hmac.compare_digest(sign(body, timestamp), signature):
        return "rejected: bad signature"
    event_id = json.loads(body)["id"]
    if event_id in seen:
        return "ignored: duplicate delivery"
    seen.add(event_id)
    return "accepted"

seen = set()
body = json.dumps({"id": "evt_42", "type": "invoice.paid", "amount": 1999}).encode()
sig = sign(body, 1_700_000_000)

print("1. genuine:        ", verify(body, 1_700_000_000, sig, 1_700_000_010, seen))
print("2. retry (same id):", verify(body, 1_700_000_000, sig, 1_700_000_070, seen))
tampered = body.replace(b"1999", b"1")
print("3. tampered body:  ", verify(tampered, 1_700_000_000, sig, 1_700_000_010, set()))
print("4. replayed later: ", verify(body, 1_700_000_000, sig, 1_700_009_000, set()))

Output

1. genuine:         accepted
2. retry (same id): ignored: duplicate delivery
3. tampered body:   rejected: bad signature
4. replayed later:  rejected: too old (possible replay)
  1. Verify the signature using a constant-time comparison. Anyone on the internet can POST to your URL; only the provider knows the secret. A tampered body fails the check.
  2. Reject old timestamps (here, more than 5 minutes off). Signing the timestamp stops an attacker from recording a genuine delivery and replaying it later.
  3. Deduplicate by event id. Retries mean the same event can arrive twice, or out of order. Record processed ids and make handlers idempotent, as in Idempotency and safe retries .
  4. Acknowledge fast, work later. Return 200 within a second or two after enqueuing the event. If you do slow work inline, the provider times out and retries, and you process it again.

Going deeper

Webhooks are hints, the API is the truth

Deliveries can be delayed, reordered or, after enough failures, dropped. Robust consumers treat a webhook as a prompt to look something up: on invoice.paid, fetch the invoice from the provider's API to get its current state, and run a periodic reconciliation job that lists recent events to catch anything missed.

Sending webhooks yourself

If you build a platform that sends webhooks, you inherit the provider side: a durable queue per customer endpoint, backoff, per-endpoint rate limits so one slow customer cannot clog everyone, signing secrets with rotation, and a log customers can inspect and replay. That is a messaging system in miniature; the patterns are in Retries, dead letter queues and poison messages .

Trade-offs

OptionWins whenLoses when
Webhooks (push) Near-real-time integrations with little load on either side Consumers behind firewalls or offline, and guaranteed completeness without reconciliation
Polling the provider's API (pull) Simple consumers, strict ordering, consumers that cannot expose a URL Freshness and efficiency: most polls find nothing
Webhooks plus periodic reconciliation Production integrations: fast in the normal case, complete in the bad case A little more code to build and run

Common mistakes

  • Skipping signature verification, so anyone can fake a "payment succeeded" event.
  • Doing slow work before responding, causing timeouts, retries and duplicate processing.
  • Assuming each event arrives exactly once and in order.
  • Comparing signatures with ==, which can leak timing information; use a constant-time compare.

Check yourself

  1. What is a webhook?

    Show answer

    B. An HTTP request a provider sends to your registered URL when an event happens The provider pushes events to you as HTTP POSTs instead of you asking repeatedly.

  2. Why does the provider sign the timestamp along with the body?

    Show answer

    B. So a recorded genuine delivery cannot be replayed much later The consumer rejects signatures on old timestamps, so captured deliveries expire quickly.

  3. Your handler takes 40 seconds per event and the provider times out after 10. What happens?

    Show answer

    B. The provider retries, so events are processed multiple times The provider sees a failure and retries. Acknowledge quickly and do the work asynchronously.

  4. How should a consumer handle a second delivery of the same event id?

    Show answer

    B. Recognize the id and skip the side effects Deduplicate by event id and keep handlers idempotent; a 2xx tells the provider to stop retrying.

Recap

  • A webhook is the provider POSTing an event to a URL you registered.
  • Providers sign payloads, retry with backoff and include unique event ids.
  • Consumers verify signatures and timestamps, deduplicate by id, and respond fast.
  • Do heavy work asynchronously after acknowledging.
  • Treat webhooks as hints and reconcile with the provider's API.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy