System Design HLD-03.07
Webhooks
HTTP callbacks from one system to another: push versus pull, signed payloads, retries with backoff, replay protection and idempotent consumers.
The big idea
Your app sells subscriptions through a payment provider. When a customer's card is charged next month, how does your app find out? It could ask the provider every minute, "any new payments?", which is slow and wasteful for both sides. Or the provider could tell you the moment it happens. A webhookAn HTTP request a provider sends to a URL you registered whenever a specified event happens, carrying the event as JSON. is exactly that: you register a URL, and when an event occurs, the provider sends an HTTP POST describing it to your URL.
Webhooks are how payment processors, Git hosts, chat platforms and countless SaaS tools integrate with each other. They are simple to receive, but because your endpoint is a public URL that the internet can reach, and because the provider's deliveries can fail and be retried, a correct receiver needs three protections: verify the sender, tolerate duplicates, and respond fast.
Think of a doorbell instead of opening the door every few minutes to check for visitors. But anyone can press a doorbell, and the courier may ring twice.
How it works
Step 1: The delivery loop
Step 2: The provider's responsibilities
- Sign every payload with a secret shared with the consumer, typically an HMAC of the timestamp and the body, sent in a header.
- Retry failures (timeouts, connection errors, non-2xx responses) with exponential backoff over hours or days, then give up and surface the failure in a dashboard.
- Include a unique event id so consumers can recognize repeats, and let consumers re-fetch the event or list missed ones through the normal API.
Step 3: The consumer's responsibilities
import hashlib, hmac, json
SECRET = b"whsec_shared_between_provider_and_you"
def sign(body: bytes, timestamp: int) -> str:
msg = f"{timestamp}.".encode() + body # sign the time too, to stop replays
return hmac.new(SECRET, msg, hashlib.sha256).hexdigest()
def verify(body: bytes, timestamp: int, signature: str, now: int, seen: set) -> str:
if abs(now - timestamp) > 300:
return "rejected: too old (possible replay)"
if not hmac.compare_digest(sign(body, timestamp), signature):
return "rejected: bad signature"
event_id = json.loads(body)["id"]
if event_id in seen:
return "ignored: duplicate delivery"
seen.add(event_id)
return "accepted"
seen = set()
body = json.dumps({"id": "evt_42", "type": "invoice.paid", "amount": 1999}).encode()
sig = sign(body, 1_700_000_000)
print("1. genuine: ", verify(body, 1_700_000_000, sig, 1_700_000_010, seen))
print("2. retry (same id):", verify(body, 1_700_000_000, sig, 1_700_000_070, seen))
tampered = body.replace(b"1999", b"1")
print("3. tampered body: ", verify(tampered, 1_700_000_000, sig, 1_700_000_010, set()))
print("4. replayed later: ", verify(body, 1_700_000_000, sig, 1_700_009_000, set()))Output
1. genuine: accepted
2. retry (same id): ignored: duplicate delivery
3. tampered body: rejected: bad signature
4. replayed later: rejected: too old (possible replay) - Verify the signature using a constant-time comparison. Anyone on the internet can POST to your URL; only the provider knows the secret. A tampered body fails the check.
- Reject old timestamps (here, more than 5 minutes off). Signing the timestamp stops an attacker from recording a genuine delivery and replaying it later.
- Deduplicate by event id. Retries mean the same event can arrive twice, or out of order. Record processed ids and make handlers idempotent, as in Idempotency and safe retries .
- Acknowledge fast, work later. Return
200within a second or two after enqueuing the event. If you do slow work inline, the provider times out and retries, and you process it again.
Going deeper
Webhooks are hints, the API is the truth
Deliveries can be delayed, reordered or, after enough failures, dropped. Robust consumers treat a webhook as a prompt to look something up: on invoice.paid, fetch the invoice from the provider's API to get its current state, and run a periodic reconciliation job that lists recent events to catch anything missed.
Sending webhooks yourself
If you build a platform that sends webhooks, you inherit the provider side: a durable queue per customer endpoint, backoff, per-endpoint rate limits so one slow customer cannot clog everyone, signing secrets with rotation, and a log customers can inspect and replay. That is a messaging system in miniature; the patterns are in Retries, dead letter queues and poison messages .
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Webhooks (push) | Near-real-time integrations with little load on either side | Consumers behind firewalls or offline, and guaranteed completeness without reconciliation |
| Polling the provider's API (pull) | Simple consumers, strict ordering, consumers that cannot expose a URL | Freshness and efficiency: most polls find nothing |
| Webhooks plus periodic reconciliation | Production integrations: fast in the normal case, complete in the bad case | A little more code to build and run |
Common mistakes
- Skipping signature verification, so anyone can fake a "payment succeeded" event.
- Doing slow work before responding, causing timeouts, retries and duplicate processing.
- Assuming each event arrives exactly once and in order.
- Comparing signatures with
==, which can leak timing information; use a constant-time compare.
Check yourself
-
What is a webhook?
Show answer
B. An HTTP request a provider sends to your registered URL when an event happens The provider pushes events to you as HTTP POSTs instead of you asking repeatedly.
-
Why does the provider sign the timestamp along with the body?
Show answer
B. So a recorded genuine delivery cannot be replayed much later The consumer rejects signatures on old timestamps, so captured deliveries expire quickly.
-
Your handler takes 40 seconds per event and the provider times out after 10. What happens?
Show answer
B. The provider retries, so events are processed multiple times The provider sees a failure and retries. Acknowledge quickly and do the work asynchronously.
-
How should a consumer handle a second delivery of the same event id?
Show answer
B. Recognize the id and skip the side effects Deduplicate by event id and keep handlers idempotent; a 2xx tells the provider to stop retrying.
Recap
- A webhook is the provider POSTing an event to a URL you registered.
- Providers sign payloads, retry with backoff and include unique event ids.
- Consumers verify signatures and timestamps, deduplicate by id, and respond fast.
- Do heavy work asynchronously after acknowledging.
- Treat webhooks as hints and reconcile with the provider's API.