System Design HLD-10.04
Reconciliation and correctness jobs
Why every system that moves money needs reconciliation: comparing independent ledgers, classifying breaks, and double-entry bookkeeping basics.
The big idea
You can use idempotency keys, outboxes, sagas and workflow engines, and you will still end up with mismatches. A bug ships. A webhook from the payment provider never arrives. An engineer fixes a row by hand. A saga's compensation fails three times and is forgotten. In a system that moves money or inventory, the question is not whether your records will drift from reality but how quickly you will notice.
ReconciliationRegularly comparing two independently produced records of the same activity, such as your ledger and a payment provider's report, and flagging every difference for investigation. is the safety net. A scheduled job compares your records with an independent source of truth and reports every difference. It does not prevent errors; it guarantees they are found, usually within a day, before they become a customer complaint, an audit finding or a growing hole in the accounts.
Think of checking your bank statement against your own record of spending. Each line should match. A charge you do not recognize, or a refund that never appeared, stands out only because you compared two lists that were written by different people.
How it works
Step 1: Pick an independent source
Reconciliation only works if the two sides were produced independently. Comparing your orders table against your own payments table catches some bugs, but both were written by your code, so a bug can corrupt both the same way. Stronger comparisons cross a boundary:
- Your payments ledger vs the payment provider's settlement report.
- Your bank-transfer records vs the bank statement.
- Your inventory counts vs the warehouse's stock report.
- Service A's view vs service B's view of the same saga, such as orders marked paid vs payments captured.
Step 2: Match, then classify the breaks
The job joins the two sides on a shared key, such as a payment id or idempotency key, and sorts every row into one of four buckets: matched, missing on their side, missing on our side, or matched with different values.
from collections import defaultdict
# Our ledger vs the payment provider's settlement report for one day (amounts in cents).
ours = {"pay-101": 5000, "pay-102": 1299, "pay-103": 870, "pay-104": 2500, "pay-105": 400}
provider = {"pay-101": 5000, "pay-102": 1299, "pay-104": 2050, "pay-105": 400, "pay-106": 999}
for pid in sorted(ours.keys() | provider.keys()):
a, b = ours.get(pid), provider.get(pid)
if a is None:
print(f"{pid}: MISSING IN OUR LEDGER provider settled {b}")
elif b is None:
print(f"{pid}: MISSING AT PROVIDER we recorded {a}")
elif a != b:
print(f"{pid}: AMOUNT MISMATCH ours {a} vs provider {b}")
print("matched:", sum(1 for p in ours if provider.get(p) == ours[p]), "of", len(ours | provider))
# Double entry: every movement is a debit (+) and an equal credit (-), so the books sum to zero.
entries = [("cash", +5000), ("revenue", -5000), # a sale
("revenue", +400), ("cash", -400), # a refund
("fees", +150), ("cash", -150)] # a provider fee
balances = defaultdict(int)
for account, amount in entries:
balances[account] += amount
print("balances:", dict(balances), "| total:", sum(balances.values()))
entries.append(("cash", +999)) # a bug writes one side only
print("after one-sided bug, total:", sum(a for _, a in entries), "-> the books no longer balance")Output
pay-103: MISSING AT PROVIDER we recorded 870
pay-104: AMOUNT MISMATCH ours 2500 vs provider 2050
pay-106: MISSING IN OUR LEDGER provider settled 999
matched: 3 of 6
balances: {'cash': 4450, 'revenue': -4600, 'fees': 150} | total: 0
after one-sided bug, total: 999 -> the books no longer balance Each break has typical causes. Missing at the provider: the charge request timed out and never actually happened, but we recorded it. Missing in our ledger: we charged the card but crashed before recording it, or the webhook was lost. Amount mismatch: currency conversion, a partial capture, or a fee netted out of the settlement. Many breaks are only timing: a payment captured at 23:59 settles tomorrow. Good jobs allow for this by re-checking unmatched rows on following days before raising an alarm.
Step 3: Repair and track
Every break that survives the timing window becomes a ticket with an owner, an age and an amount. Teams track the number and value of open breaks as a health metric and alert when they grow. Repairs go through the normal code paths, ideally a correcting entry rather than an edit to history, so the fix itself is auditable. A break that appears repeatedly points to a bug worth fixing at the source.
Step 4: Double-entry bookkeeping
Accountants solved a version of this problem centuries ago. In double-entry bookkeepingA bookkeeping method where every movement of value is recorded as two entries, a debit to one account and an equal credit to another, so the books always balance., every movement of value is recorded twice: a debit to one account and an equal credit to another. A sale debits cash and credits revenue; a fee debits fees and credits cash. Money never appears or vanishes; it only moves between accounts.
The payoff is a built-in invariant: across all accounts, debits equal credits, so the total is always zero. The second half of the code shows it: after a sale, a refund and a fee, the balances sum to zero; a bug that writes only one side of a movement makes the total non-zero, and a simple check catches it immediately. Ledgers built this way, with immutable entries and corrections recorded as new entries, are standard in payments systems because they make errors visible by construction.
Going deeper
Internal correctness jobs
The same idea applies beyond money. A job can check that every order marked shipped has a shipment, that cache contents match the database for a sample of keys, that two replicas or two services agree, or that every saga either completed or compensated within an hour. These invariant checks catch the long tail of bugs that unit tests and monitoring miss.
Scale
Comparing millions of rows a day is a batch data job: export both sides to a warehouse, join on the key, and write breaks to a table. Keep the job itself idempotent and re-runnable for any date, and keep the raw source files, because reconciliation is often how you answer an auditor months later.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Daily batch reconciliation | Settlement reports and statements that arrive daily; the standard for finance | Breaks are found up to a day later |
| Near real-time matching | High-value flows where a day is too long, such as payouts | More complex; many temporary breaks from timing |
| Internal invariant checks | Catching bugs between your own services and stores | Both sides share your code, so shared bugs slip through |
| Double-entry ledger | Money movements where every cent must be accounted for | More modeling work than a single balance column |
Common mistakes
- Trusting that idempotency and sagas make reconciliation unnecessary.
- Reconciling two records produced by the same code, which share the same bugs.
- Alerting on every timing difference instead of re-checking across a settlement window.
- Fixing breaks by editing rows in place, leaving no audit trail.
- Storing balances as a single mutable number with no ledger of movements behind it.
In the wild
Payment companies, banks and marketplaces all run daily reconciliation between their internal ledgers, payment processors and bank statements, and payment providers publish settlement and payout reports for exactly this purpose. Inventory-heavy businesses reconcile system stock against warehouse counts. Double-entry ledgers are the norm in financial software, from accounting packages to the internal ledgers of large payment platforms.
Check yourself
-
Why compare your ledger against the payment provider's report rather than against your own orders table?
Show answer
B. The provider's report was produced independently, so it catches bugs that affect both of your own tables Independent sources do not share your code's bugs, so differences reveal real errors.
-
A payment captured at 23:59 shows as 'missing at provider' in today's run. What is the right response?
Show answer
B. Treat it as a possible timing break and re-check it after the next settlement Settlement lags capture; good jobs re-check before raising an alert.
-
In double-entry bookkeeping, a bug records a sale's debit but not its credit. How is it detected?
Show answer
B. Debits no longer equal credits, so the books no longer sum to zero Every valid movement has two equal sides; a one-sided entry breaks the invariant.
-
How should a confirmed break be repaired?
Show answer
B. Record a correcting entry through normal code paths, with an owner and a reason Corrections as new entries keep the history auditable and the fix reviewable.
Recap
- Every system that moves money or inventory eventually drifts; reconciliation guarantees you notice.
- Compare against an independent source, match on a shared key, and classify breaks.
- Allow for timing differences, then give every remaining break an owner and track them.
- Repair with correcting entries, not edits, and fix repeated causes at the source.
- Double-entry bookkeeping makes many errors detectable because the books must always balance.