System Design HLD-19.15
Design a payment system
Design payments: a PSP flow with tokenized cards, idempotency keys, a payment state machine, a double-entry ledger and daily reconciliation.
The problem
A shopper clicks "Pay 50 dollars". The card is charged once, the order is confirmed, the seller is paid later minus a fee, and a year from now an auditor can trace every cent. That is the whole job, and almost every word in it is hard in a distributed system.
The traffic is modest compared to a news feed. What makes payments different is that mistakes cost real money and trust: charging twice, losing a charge, or showing "failed" for a payment that actually went through. The network between you and the card network is unreliable, so you will regularly not know whether a charge happened. The design is about making that uncertainty safe: idempotent retries, a strict state machine, an append-only ledger, and a daily check against the outside world.
Requirements
Functional
- Accept card payments for orders through an external payment service provider (PSP).
- Record every movement of money: charges, fees, refunds, payouts to sellers.
- Handle refunds and asynchronous outcomes (some payments complete minutes later, via callbacks).
- Pay sellers on a schedule.
Non-functional
- Correctness above all: never charge twice, never lose a charge, never let the books drift.
- Scale: 10 million payments a day, with sale-day peaks of about 10x the average.
- Auditability: every balance explainable from immutable records, kept for years.
- Security: card numbers never touch our servers.
Out of scope
- Building a card network or PSP, fraud models (a risk check is a hook), currency conversion and tax.
Estimates
- Payments: 10M a day is about 116 per second on average, so about 1,200 per second on a sale-day peak. A single well-tuned relational database handles this; scale is not the hard part.
- Ledger: each payment writes about 3 ledger entries (charge, seller share, fee), so 30 million entries a day. At about 200 bytes each, 6 GB a day, about 2.2 TB a year. Entries are never updated or deleted.
- Callbacks and reports: one webhook per payment from the PSP ( Webhooks ) and one settlement report per day listing everything it processed.
API
POST /v1/payments
Idempotency-Key: order-81723-attempt-1
{ order_id, amount: 5000, currency: "USD", payment_method_token: "tok_..." }
-> 201 { payment_id, status: "succeeded" | "pending" | "failed" }
GET /v1/payments/{payment_id}
POST /v1/payments/{payment_id}/refunds Idempotency-Key: ... { amount }
POST /v1/psp/webhooks # signed callbacks from the PSP
Amounts are integers in the smallest currency unit (5000 cents), never floating point. The client sends a token, not a card number: the card form is served by the PSP and returns a token, which keeps our systems out of most card-data compliance scope.
Data model
- Payments (relational, Transactions and ACID ):
payment_id, order_id, amount, currency, status, psp_reference, idempotency_key (unique), created_at, updated_at. - Idempotency keys:
key -> request hash, response, status, expires_at, so a retry with the same key returns the stored result. - Ledger entries (append-only):
entry_id, transaction_id, account, amount (signed), created_at. The entries of one transaction always sum to zero. - Accounts:
provider_receivable,merchant_payable:{seller},fee_revenue,refunds... A balance is the sum of an account's entries.
High-level design
- Charge path Card details go from the browser straight to the PSP, which returns a token. Checkout calls the payment service with the token and an idempotency key; the service stores the payment as pending, runs a risk check and asks the PSP to charge.
- Callback path The PSP reports the outcome synchronously or later by a signed webhook. The payment service verifies the signature, moves the payment to succeeded or failed, writes balanced ledger entries in the same transaction, and tells the order service.
- Reconciliation path Every day the PSP's settlement report is matched line by line against the ledger. Anything that does not match goes to a queue to be resolved, automatically where possible and by people otherwise.
Deep dives
Deep dive 1: Idempotency, end to end
Retries are unavoidable: the checkout times out, the user double-clicks, a worker restarts. Each of these must not become a second charge. Idempotency keys ( Idempotency and safe retries ) are used at two boundaries:
- Client to us: checkout sends an
Idempotency-Keyderived from the order. The payment service stores the key with a unique constraint in the same transaction as the payment row. A retry with the same key finds the row and returns its current result instead of creating a new payment. - Us to the PSP: we pass our own key (for example the payment id) to the provider, so if our call times out and we retry, the provider also recognizes the duplicate.
The sample below runs three payments against a provider that loses some replies. Payment p2's reply is lost and we retry; payment p3's reply is lost and our service crashes before retrying:
# A tiny payment service: retries with and without idempotency keys,
# a double-entry ledger, and reconciliation against the provider's report.
import itertools
class Provider: # the external payment provider
def __init__(self):
self.charges, self.ids = {}, itertools.count(1)
self.lose_reply = {"p2", "p3"} # replies lost on the way back
def charge(self, payment, amount, key=None):
ref = key or f"ch{next(self.ids)}" # no key: every call is new
self.charges.setdefault(ref, (payment, amount))
if payment in self.lose_reply:
self.lose_reply.discard(payment)
raise TimeoutError
return ref
def run(use_keys):
psp, ledger, state = Provider(), [], {}
# p3: the reply is lost and our service crashes before retrying
for payment, amount, retries in (("p1", 50, 1), ("p2", 20, 1),
("p3", 35, 0)):
key = f"pay-{payment}" if use_keys else None
for attempt in range(1 + retries):
try:
psp.charge(payment, amount, key)
state[payment] = "succeeded"
fee = round(amount * 0.03, 2) # 3 percent fee
ledger += [(payment, "provider_receivable", +amount),
(payment, "merchant_payable", -(amount - fee)),
(payment, "fee_revenue", -fee)]
break
except TimeoutError:
state[payment] = "unknown" # charged or not?
return psp, ledger, state
for use_keys in (False, True):
psp, ledger, state = run(use_keys)
print("with idempotency keys" if use_keys else "without keys")
charged = {}
for payment, amount in psp.charges.values():
charged[payment] = charged.get(payment, 0) + amount
print(" provider charged:", charged)
print(" our states: ", state)
print(" ledger sums to", round(sum(e[2] for e in ledger), 2))
booked = {p: a for p, acct, a in ledger
if acct == "provider_receivable"}
for p in sorted(charged): # the daily reconciliation
if charged[p] != booked.get(p, 0):
print(f" MISMATCH {p}: provider {charged[p]}, "
f"ledger {booked.get(p, 0)}")Output
without keys
provider charged: {'p1': 50, 'p2': 40, 'p3': 35}
our states: {'p1': 'succeeded', 'p2': 'succeeded', 'p3': 'unknown'}
ledger sums to 0.0
MISMATCH p2: provider 40, ledger 20
MISMATCH p3: provider 35, ledger 0
with idempotency keys
provider charged: {'p1': 50, 'p2': 20, 'p3': 35}
our states: {'p1': 'succeeded', 'p2': 'succeeded', 'p3': 'unknown'}
ledger sums to 0.0
MISMATCH p3: provider 35, ledger 0 Without keys, the retry of p2 created a second charge: the customer paid 40 dollars for a 20 dollar order. With keys, the provider recognized the retry and charged once. In both runs p3 is stuck in unknown: the customer was charged, but our ledger says nothing. Idempotency cannot fix a crash before the retry; that is what the next two deep dives handle.
Deep dive 2: A state machine with an unknown state
Every payment moves through explicit states: created -> pending -> succeeded | failed, plus refunded later. The important addition is honesty about timeouts: when the PSP call times out, the payment is unknown, not failed. Marking it failed and letting the user pay again is how double charges happen.
Unknown payments are resolved by a background job that retries the call with the same key or queries the PSP for the payment's status, and by webhooks, which may arrive minutes later. Transitions are conditional updates (UPDATE ... SET status = 'succeeded' WHERE status IN ('pending', 'unknown')), so a late webhook for a payment already marked succeeded changes nothing. Webhooks are verified by signature and are themselves processed idempotently, because providers deliver them at least once.
Deep dive 3: The double-entry ledger
A balance column that is incremented and decremented cannot explain itself. A double-entry ledger records every movement as a transaction of two or more entries that sum to zero. A 50 dollar charge with a 3 percent fee becomes:
provider_receivable +50.00(the PSP owes us)merchant_payable -48.50(we owe the seller)fee_revenue -1.50(our income)
Entries are append-only. A refund is a new transaction with opposite entries, never an edit. Any balance is the sum of its account's entries, and the sum of all entries is always zero, which the sample checks. If it ever is not, there is a bug, and you know immediately. The ledger entries are written in the same database transaction as the payment's state change, so there is never a succeeded payment without its entries.
Deep dive 4: Reconciliation
Your database is one record of the truth; the PSP's is another, and the bank's is a third. Every day the PSP publishes a settlement report of everything it processed. A reconciliation job ( Reconciliation and correctness jobs ) matches it line by line against the ledger:
- In both, amounts equal: done.
- At the PSP, not in our ledger (p3 in the sample): we charged but lost track. Post the missing entries and complete the order, or refund.
- In our ledger, not at the PSP: we think it succeeded but it did not. Investigate before shipping goods.
- Amounts differ: fees or currency handling disagree.
Reconciliation is what turns "exactly once" from a hope into a property you can check: retries and idempotency make duplicates rare, and reconciliation finds what is left.
Deep dive 5: Keeping card data out
Card-data security standards apply to every system that stores, processes or transmits card numbers. If the card form is hosted by the PSP and only a token reaches our servers, our checkout, payment service and databases never handle a card number, and the compliance scope shrinks to a small part of the system. The token can be charged only through our PSP account, so a leaked token is far less dangerous than a leaked card number.
Try it
Try this:
- Lose two responses in a row without a key. How many charges? Now turn the key on.
- In the sample, p3 stayed
unknown. List two ways the system learns the truth about it later. - Why is a refund a new ledger transaction rather than deleting the original entries?
Bottlenecks and failure modes
| Failure | Impact | Mitigation |
|---|---|---|
| PSP call times out | Charged or not? | unknown state; retry with same key or query status |
| Client retries or double-clicks | Duplicate payment | Idempotency key with a unique constraint |
| Webhook delivered twice or late | Double processing | Signature check; conditional state transitions |
| Crash between charge and ledger write | Money moved, books silent | Reconciliation against the PSP report |
| PSP outage | Payments fail | Second provider for new payments; queue non-urgent work |
| Floating-point amounts | Rounding drift | Integer minor units; currency stored with every amount |
Trade-offs we made
| Option | Wins when | Loses when |
|---|---|---|
| Relational database with transactions (chosen) | Atomic state change plus ledger entries; constraints | Harder to scale writes than a NoSQL store |
| Append-only double-entry ledger (chosen) | Every balance explainable; errors detectable | More rows; balances computed or cached |
| Tokenized cards via the PSP (chosen) | Card data never touches our systems | Tied to the provider's tokens |
| Synchronous charge at checkout (chosen) | Immediate answer for most payments | Must still handle async outcomes and timeouts |
Extensions
- Payouts to sellers on a schedule, with their own ledger transactions and bank reconciliation.
- Multiple PSPs with routing by cost, country or availability.
- Multi-currency accounts and foreign exchange entries.
- Chargebacks and disputes as their own state machine.
In the wild
Stripe
Designing robust and predictable APIs with idempotency (opens in a new tab)Stripe's 2017 post explains why its API accepts an Idempotency-Key header on mutating requests. It walks through three failures: the connection fails before reaching the server, the server fails midway through the operation, or the operation succeeds but the response is lost. With a key, a retried request returns the stored result of the original operation, and if the original was rolled back by an ACID database it is safe to retry wholesale. The post also recommends exponential backoff with random jitter so many clients do not retry at the same moment.
Check yourself
-
The call to the PSP times out. What state should the payment be in?
Show answer
B. Unknown, until a retry with the same key, a status query or a webhook resolves it Treating a timeout as failure invites a second payment for an order that may already be charged.
-
Why does a double-entry ledger make bugs visible?
Show answer
B. Every transaction's entries sum to zero, so any imbalance reveals an error The invariant is checkable at any time; a balance column cannot explain itself.
-
Reconciliation finds a charge at the PSP with no ledger entry. What most likely happened?
Show answer
B. We charged but crashed or lost the reply before recording it This is exactly the gap idempotency cannot close; reconciliation finds and fixes it.
-
Why should amounts be stored as integers of the smallest currency unit?
Show answer
B. Floating point cannot represent many decimal amounts exactly, causing rounding drift 0.1 + 0.2 is not 0.3 in floating point; cents as integers are exact.
Recap
- Use idempotency keys from client to us and from us to the PSP; store them with a unique constraint.
- A timeout means unknown, not failed; resolve it by retrying with the same key, querying, or webhooks.
- Record money in an append-only double-entry ledger, in the same transaction as the state change.
- Reconcile daily against the PSP's report to catch what retries and crashes leave behind.
- Tokenize cards through the PSP so card numbers never touch your systems.
Next up
- Crack the system design round learning path