Skip to content
learn
  1. Learn
  2. High Level System Design
  3. APIs and Communication

System Design HLD-03.03

Idempotency and safe retries

Why retries double-charge customers, what idempotency means, and how idempotency keys make retries safe for effectively-once processing.

Intermediate 12 min Last reviewed

The big idea

A client sends "charge this card $50". The server charges the card, and then the response is lost: a phone switches networks, a load balancer times out, a connection resets. From the client's side the request simply failed. Did the charge happen or not? It cannot know. If it retries, the customer may pay twice. If it does not, the customer may not pay at all, or may think the purchase failed and buy again.

This is not a rare edge case. At scale, timeouts happen every minute, and every robust client retries. So every operation that changes something must be safe to repeat. An operation is idempotentAn operation is idempotent if performing it several times has the same effect as performing it once. if doing it twice has the same effect as doing it once. Reading is naturally idempotent, setting a value is too, but "charge $50" or "add one to the counter" is not, unless you make it so.

Think of a lift button. Pressing it five times calls the lift once. That is idempotent. A vending machine that drops a can for every press is not, and that is the machine you must protect.

How it works

Step 1: The double charge

A retried payment with an idempotency key is processed once A sequence diagram: the client sends a charge with an idempotency key; the server finds the key is new, charges and stores the result; the response is lost. The client times out and retries with the same key; the server finds the key, loads the stored result and returns it without charging again. Client Payments API Key store results by key POST /charges $50, Idempotency-Key: 7f3a key 7f3a new? yes: charge, save result 201 Created (lost in the network) timeout: retry with the SAME key POST /charges $50, Idempotency-Key: 7f3a key 7f3a seen: load stored result 201 Created, charge ch_1 (replayed) Figure 1. The first response is lost, so the client retries with the same key. The server recognizes the key and replays the stored result instead of charging again: the short glowing path.

Step 2: Idempotency keys

The standard fix has three parts:

  1. The client creates a unique key per logical operation, usually a random UUID, when the user presses "Pay". It sends the key in a header such as Idempotency-Key, and reuses the same key for every retry of that operation.
  2. The server checks the key before doing the work. If the key is new, it performs the operation and stores the response under the key. If the key has been seen, it skips the work and returns the stored response.
  3. Keys expire after a window long enough to cover any retries, often a day.
idempotency.pyruns with python3 (standard library)
import uuid

ledger = []          # every charge the bank actually made
results = {}         # idempotency key -> stored response

def charge(amount, key=None):
    if key is not None and key in results:
        return results[key]                    # a retry: replay the first answer
    charge_id = f"ch_{len(ledger) + 1}"
    ledger.append((charge_id, amount))         # the side effect happens here
    response = {"id": charge_id, "amount": amount, "status": "succeeded"}
    if key is not None:
        results[key] = response                # remember it under the key
    return response

def client_pays(amount, lost_responses, use_key):
    key = str(uuid.uuid4()) if use_key else None   # one key per purchase, reused on retries
    for attempt in range(1, lost_responses + 2):
        resp = charge(amount, key)
        if attempt > lost_responses:               # this response finally arrives
            return resp, attempt

for use_key in (False, True):
    ledger.clear()
    resp, attempts = client_pays(50, lost_responses=2, use_key=use_key)
    print(f"key={'yes' if use_key else 'no '}  attempts={attempts}  "
          f"charges={len(ledger)}  total=${sum(a for _, a in ledger)}  client saw {resp['id']}")

Output

key=no   attempts=3  charges=3  total=$150  client saw ch_3
key=yes  attempts=3  charges=1  total=$50  client saw ch_1

Same client, same two lost responses. Without a key, three attempts mean three charges and $150. With a key, the first attempt charges and the two retries replay that result: one charge, $50, and the client finally sees ch_1, the charge that actually happened.

Step 3: The details that matter

  • Atomicity. Saving the result and doing the work must succeed or fail together. If the server charges and crashes before saving the key, a retry charges again. Store the key in the same database transaction as the business change, or record the key first with an "in progress" state.
  • Concurrent duplicates. Two copies of a request can arrive at the same moment. A unique constraint on the key, or a lock, makes the second wait or get 409 Conflict until the first finishes.
  • Same key, different request. If a client reuses a key with a different amount, that is a bug. Store a hash of the request with the key and reject mismatches with 422.
  • Scope. Keys are scoped to the caller (account or API key), so two customers cannot collide.

Step 4: At-least-once plus idempotency

Networks and queues offer at-least-once deliveryA delivery guarantee where every message arrives one or more times; duplicates are possible but losses are not.: retry until acknowledged, accept occasional duplicates. True exactly-once delivery across a network is not achievable in general. But at-least-once delivery combined with an idempotent receiver gives effectively once processing: the message may arrive twice, but its effect happens once. This is the foundation of reliable messaging, from payment APIs to Kafka consumers, which you will meet again in Delivery semantics .

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. With no key and 2 lost responses, count the charges in the ledger column.
  2. Turn the key on. The retries reach the server but take the short path to the stored result.
  3. Set lost responses to 0. Both modes behave the same: the key costs almost nothing when the network is healthy.

Going deeper

Natural idempotency

Some operations can be designed to be idempotent without keys. "Set the order status to shipped" is idempotent; "advance the status by one step" is not. "Add item A12 to cart with quantity 2" can be written as an upsert keyed by cart and item. Prefer absolute operations (set to X) over relative ones (add Y) wherever you can.

Retry politely

Safe retries still need limits. Clients should back off exponentially with random jitter, so that thousands of clients do not retry in lockstep against a struggling server. That is covered in Timeouts, retries, backoff and jitter .

Trade-offs

OptionWins whenLoses when
Idempotency keys with stored results Payments, orders, any create action with side effects that clients retry Storage for keys and results, and care with atomicity
Naturally idempotent operations (set, upsert) Updates that can be expressed as setting a final state Operations that are inherently additive, like charging money
No retries Nothing in practice: failures become user-visible Reliability: every network blip becomes an error or a lost action

Common mistakes

  • Generating a new key for each retry. The server then sees every retry as a new operation.
  • Storing the key after the side effect in a separate step, so a crash in between causes a duplicate.
  • Treating retries as rare. At scale, they happen constantly.
  • Retrying non-idempotent operations automatically in a library or proxy without keys.

In the wild

Stripe

Designing robust and predictable APIs with idempotency (opens in a new tab)

Brandur Leach's 2017 post explains why Stripe's API accepts an Idempotency-Key header on mutating requests: a client that charges a customer and loses the response cannot otherwise retry safely, and "accidentally calling it twice would lead to the customer being double-charged". The client sends a unique value identifying the operation, and the server uses it to recognize retries. The post pairs this with client-side retries using exponential backoff, waiting in proportion to 2 to the power of the number of failures, plus random jitter so that retries from many clients do not arrive at once.

Check yourself

  1. Which operation is naturally idempotent?

    Show answer

    C. Set the order status to shipped Setting a final state gives the same result however many times you do it.

  2. A client retries a payment with a NEW idempotency key each time. What happens?

    Show answer

    B. Each retry can charge again The server sees every key as a new operation. The same key must be reused for all retries of one operation.

  3. Why must the key and the business change be saved atomically?

    Show answer

    B. If the server crashes between charging and saving the key, a retry would charge again Saving both in one transaction means a crash leaves either both or neither, never a charge without a key.

  4. What does at-least-once delivery plus an idempotent receiver give you?

    Show answer

    B. Effectively-once processing: duplicates arrive but take effect once Messages may be delivered more than once, but the idempotent receiver makes the effect happen once.

Recap

  • Responses get lost, so clients retry; non-idempotent retries cause duplicates like double charges.
  • Idempotent means doing it twice has the same effect as doing it once.
  • Idempotency keys: the client reuses one key per operation; the server stores and replays the result.
  • Store the key atomically with the change, and handle concurrent duplicates.
  • At-least-once delivery plus idempotency gives effectively-once processing.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up