System Design HLD-03.03
Idempotency and safe retries
Why retries double-charge customers, what idempotency means, and how idempotency keys make retries safe for effectively-once processing.
The big idea
A client sends "charge this card $50". The server charges the card, and then the response is lost: a phone switches networks, a load balancer times out, a connection resets. From the client's side the request simply failed. Did the charge happen or not? It cannot know. If it retries, the customer may pay twice. If it does not, the customer may not pay at all, or may think the purchase failed and buy again.
This is not a rare edge case. At scale, timeouts happen every minute, and every robust client retries. So every operation that changes something must be safe to repeat. An operation is idempotentAn operation is idempotent if performing it several times has the same effect as performing it once. if doing it twice has the same effect as doing it once. Reading is naturally idempotent, setting a value is too, but "charge $50" or "add one to the counter" is not, unless you make it so.
Think of a lift button. Pressing it five times calls the lift once. That is idempotent. A vending machine that drops a can for every press is not, and that is the machine you must protect.
How it works
Step 1: The double charge
Step 2: Idempotency keys
The standard fix has three parts:
- The client creates a unique key per logical operation, usually a random UUID, when the user presses "Pay". It sends the key in a header such as
Idempotency-Key, and reuses the same key for every retry of that operation. - The server checks the key before doing the work. If the key is new, it performs the operation and stores the response under the key. If the key has been seen, it skips the work and returns the stored response.
- Keys expire after a window long enough to cover any retries, often a day.
import uuid
ledger = [] # every charge the bank actually made
results = {} # idempotency key -> stored response
def charge(amount, key=None):
if key is not None and key in results:
return results[key] # a retry: replay the first answer
charge_id = f"ch_{len(ledger) + 1}"
ledger.append((charge_id, amount)) # the side effect happens here
response = {"id": charge_id, "amount": amount, "status": "succeeded"}
if key is not None:
results[key] = response # remember it under the key
return response
def client_pays(amount, lost_responses, use_key):
key = str(uuid.uuid4()) if use_key else None # one key per purchase, reused on retries
for attempt in range(1, lost_responses + 2):
resp = charge(amount, key)
if attempt > lost_responses: # this response finally arrives
return resp, attempt
for use_key in (False, True):
ledger.clear()
resp, attempts = client_pays(50, lost_responses=2, use_key=use_key)
print(f"key={'yes' if use_key else 'no '} attempts={attempts} "
f"charges={len(ledger)} total=${sum(a for _, a in ledger)} client saw {resp['id']}")Output
key=no attempts=3 charges=3 total=$150 client saw ch_3
key=yes attempts=3 charges=1 total=$50 client saw ch_1 Same client, same two lost responses. Without a key, three attempts mean three charges and $150. With a key, the first attempt charges and the two retries replay that result: one charge, $50, and the client finally sees ch_1, the charge that actually happened.
Step 3: The details that matter
- Atomicity. Saving the result and doing the work must succeed or fail together. If the server charges and crashes before saving the key, a retry charges again. Store the key in the same database transaction as the business change, or record the key first with an "in progress" state.
- Concurrent duplicates. Two copies of a request can arrive at the same moment. A unique constraint on the key, or a lock, makes the second wait or get
409 Conflictuntil the first finishes. - Same key, different request. If a client reuses a key with a different amount, that is a bug. Store a hash of the request with the key and reject mismatches with
422. - Scope. Keys are scoped to the caller (account or API key), so two customers cannot collide.
Step 4: At-least-once plus idempotency
Networks and queues offer at-least-once deliveryA delivery guarantee where every message arrives one or more times; duplicates are possible but losses are not.: retry until acknowledged, accept occasional duplicates. True exactly-once delivery across a network is not achievable in general. But at-least-once delivery combined with an idempotent receiver gives effectively once processing: the message may arrive twice, but its effect happens once. This is the foundation of reliable messaging, from payment APIs to Kafka consumers, which you will meet again in Delivery semantics .
Try it
Try this:
- With no key and 2 lost responses, count the charges in the ledger column.
- Turn the key on. The retries reach the server but take the short path to the stored result.
- Set lost responses to 0. Both modes behave the same: the key costs almost nothing when the network is healthy.
Going deeper
Natural idempotency
Some operations can be designed to be idempotent without keys. "Set the order status to shipped" is idempotent; "advance the status by one step" is not. "Add item A12 to cart with quantity 2" can be written as an upsert keyed by cart and item. Prefer absolute operations (set to X) over relative ones (add Y) wherever you can.
Retry politely
Safe retries still need limits. Clients should back off exponentially with random jitter, so that thousands of clients do not retry in lockstep against a struggling server. That is covered in Timeouts, retries, backoff and jitter .
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Idempotency keys with stored results | Payments, orders, any create action with side effects that clients retry | Storage for keys and results, and care with atomicity |
| Naturally idempotent operations (set, upsert) | Updates that can be expressed as setting a final state | Operations that are inherently additive, like charging money |
| No retries | Nothing in practice: failures become user-visible | Reliability: every network blip becomes an error or a lost action |
Common mistakes
- Generating a new key for each retry. The server then sees every retry as a new operation.
- Storing the key after the side effect in a separate step, so a crash in between causes a duplicate.
- Treating retries as rare. At scale, they happen constantly.
- Retrying non-idempotent operations automatically in a library or proxy without keys.
In the wild
Stripe
Designing robust and predictable APIs with idempotency (opens in a new tab)Brandur Leach's 2017 post explains why Stripe's API accepts an Idempotency-Key header on mutating requests: a client that charges a customer and loses the response cannot otherwise retry safely, and "accidentally calling it twice would lead to the customer being double-charged". The client sends a unique value identifying the operation, and the server uses it to recognize retries. The post pairs this with client-side retries using exponential backoff, waiting in proportion to 2 to the power of the number of failures, plus random jitter so that retries from many clients do not arrive at once.
Check yourself
-
Which operation is naturally idempotent?
Show answer
C. Set the order status to shipped Setting a final state gives the same result however many times you do it.
-
A client retries a payment with a NEW idempotency key each time. What happens?
Show answer
B. Each retry can charge again The server sees every key as a new operation. The same key must be reused for all retries of one operation.
-
Why must the key and the business change be saved atomically?
Show answer
B. If the server crashes between charging and saving the key, a retry would charge again Saving both in one transaction means a crash leaves either both or neither, never a charge without a key.
-
What does at-least-once delivery plus an idempotent receiver give you?
Show answer
B. Effectively-once processing: duplicates arrive but take effect once Messages may be delivered more than once, but the idempotent receiver makes the effect happen once.
Recap
- Responses get lost, so clients retry; non-idempotent retries cause duplicates like double charges.
- Idempotent means doing it twice has the same effect as doing it once.
- Idempotency keys: the client reuses one key per operation; the server stores and replays the result.
- Store the key atomically with the change, and handle concurrent duplicates.
- At-least-once delivery plus idempotency gives effectively-once processing.
Next up
- Crack the system design round learning path