Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Distributed Transactions and Workflows

System Design HLD-10.04

Reconciliation and correctness jobs

Why every system that moves money needs reconciliation: comparing independent ledgers, classifying breaks, and double-entry bookkeeping basics.

Advanced 8 min Last reviewed

The big idea

You can use idempotency keys, outboxes, sagas and workflow engines, and you will still end up with mismatches. A bug ships. A webhook from the payment provider never arrives. An engineer fixes a row by hand. A saga's compensation fails three times and is forgotten. In a system that moves money or inventory, the question is not whether your records will drift from reality but how quickly you will notice.

ReconciliationRegularly comparing two independently produced records of the same activity, such as your ledger and a payment provider's report, and flagging every difference for investigation. is the safety net. A scheduled job compares your records with an independent source of truth and reports every difference. It does not prevent errors; it guarantees they are found, usually within a day, before they become a customer complaint, an audit finding or a growing hole in the accounts.

Think of checking your bank statement against your own record of spending. Each line should match. A charge you do not recognize, or a refund that never appeared, stands out only because you compared two lists that were written by different people.

How it works

Step 1: Pick an independent source

Reconciliation only works if the two sides were produced independently. Comparing your orders table against your own payments table catches some bugs, but both were written by your code, so a bug can corrupt both the same way. Stronger comparisons cross a boundary:

  • Your payments ledger vs the payment provider's settlement report.
  • Your bank-transfer records vs the bank statement.
  • Your inventory counts vs the warehouse's stock report.
  • Service A's view vs service B's view of the same saga, such as orders marked paid vs payments captured.

Step 2: Match, then classify the breaks

Comparing our ledger with the provider's report row by row Two columns. Our ledger lists pay-101 for 5000, pay-103 for 870 and pay-104 for 2500. The provider report lists pay-101 for 5000, pay-104 for 2050 and pay-106 for 999. pay-101 matches and is joined by a green line. pay-103 exists only in our ledger and glows red. pay-104 appears in both with different amounts and glows red. pay-106 exists only at the provider and glows red. match missing at provider amount mismatch missing in our ledger pay-101 5000 pay-103 870 pay-104 2500 (no record) pay-101 5000 (no record) pay-104 2050 pay-106 999 OUR LEDGER PROVIDER REPORT Figure 1. Rows are matched by payment id. pay-101 agrees. The other three are breaks: missing at the provider, an amount mismatch, and a settlement we never recorded. Each becomes a ticket with an owner.

The job joins the two sides on a shared key, such as a payment id or idempotency key, and sorts every row into one of four buckets: matched, missing on their side, missing on our side, or matched with different values.

reconcile.pyruns with python3 (standard library)
from collections import defaultdict

# Our ledger vs the payment provider's settlement report for one day (amounts in cents).
ours = {"pay-101": 5000, "pay-102": 1299, "pay-103": 870, "pay-104": 2500, "pay-105": 400}
provider = {"pay-101": 5000, "pay-102": 1299, "pay-104": 2050, "pay-105": 400, "pay-106": 999}

for pid in sorted(ours.keys() | provider.keys()):
    a, b = ours.get(pid), provider.get(pid)
    if a is None:
        print(f"{pid}: MISSING IN OUR LEDGER  provider settled {b}")
    elif b is None:
        print(f"{pid}: MISSING AT PROVIDER    we recorded {a}")
    elif a != b:
        print(f"{pid}: AMOUNT MISMATCH        ours {a} vs provider {b}")
print("matched:", sum(1 for p in ours if provider.get(p) == ours[p]), "of", len(ours | provider))

# Double entry: every movement is a debit (+) and an equal credit (-), so the books sum to zero.
entries = [("cash", +5000), ("revenue", -5000),                 # a sale
           ("revenue", +400), ("cash", -400),                   # a refund
           ("fees", +150), ("cash", -150)]                      # a provider fee
balances = defaultdict(int)
for account, amount in entries:
    balances[account] += amount
print("balances:", dict(balances), "| total:", sum(balances.values()))
entries.append(("cash", +999))                               # a bug writes one side only
print("after one-sided bug, total:", sum(a for _, a in entries), "-> the books no longer balance")

Output

pay-103: MISSING AT PROVIDER    we recorded 870
pay-104: AMOUNT MISMATCH        ours 2500 vs provider 2050
pay-106: MISSING IN OUR LEDGER  provider settled 999
matched: 3 of 6
balances: {'cash': 4450, 'revenue': -4600, 'fees': 150} | total: 0
after one-sided bug, total: 999 -> the books no longer balance

Each break has typical causes. Missing at the provider: the charge request timed out and never actually happened, but we recorded it. Missing in our ledger: we charged the card but crashed before recording it, or the webhook was lost. Amount mismatch: currency conversion, a partial capture, or a fee netted out of the settlement. Many breaks are only timing: a payment captured at 23:59 settles tomorrow. Good jobs allow for this by re-checking unmatched rows on following days before raising an alarm.

Step 3: Repair and track

Every break that survives the timing window becomes a ticket with an owner, an age and an amount. Teams track the number and value of open breaks as a health metric and alert when they grow. Repairs go through the normal code paths, ideally a correcting entry rather than an edit to history, so the fix itself is auditable. A break that appears repeatedly points to a bug worth fixing at the source.

Step 4: Double-entry bookkeeping

Accountants solved a version of this problem centuries ago. In double-entry bookkeepingA bookkeeping method where every movement of value is recorded as two entries, a debit to one account and an equal credit to another, so the books always balance., every movement of value is recorded twice: a debit to one account and an equal credit to another. A sale debits cash and credits revenue; a fee debits fees and credits cash. Money never appears or vanishes; it only moves between accounts.

The payoff is a built-in invariant: across all accounts, debits equal credits, so the total is always zero. The second half of the code shows it: after a sale, a refund and a fee, the balances sum to zero; a bug that writes only one side of a movement makes the total non-zero, and a simple check catches it immediately. Ledgers built this way, with immutable entries and corrections recorded as new entries, are standard in payments systems because they make errors visible by construction.

Going deeper

Internal correctness jobs

The same idea applies beyond money. A job can check that every order marked shipped has a shipment, that cache contents match the database for a sample of keys, that two replicas or two services agree, or that every saga either completed or compensated within an hour. These invariant checks catch the long tail of bugs that unit tests and monitoring miss.

Scale

Comparing millions of rows a day is a batch data job: export both sides to a warehouse, join on the key, and write breaks to a table. Keep the job itself idempotent and re-runnable for any date, and keep the raw source files, because reconciliation is often how you answer an auditor months later.

Trade-offs

OptionWins whenLoses when
Daily batch reconciliation Settlement reports and statements that arrive daily; the standard for finance Breaks are found up to a day later
Near real-time matching High-value flows where a day is too long, such as payouts More complex; many temporary breaks from timing
Internal invariant checks Catching bugs between your own services and stores Both sides share your code, so shared bugs slip through
Double-entry ledger Money movements where every cent must be accounted for More modeling work than a single balance column

Common mistakes

  • Trusting that idempotency and sagas make reconciliation unnecessary.
  • Reconciling two records produced by the same code, which share the same bugs.
  • Alerting on every timing difference instead of re-checking across a settlement window.
  • Fixing breaks by editing rows in place, leaving no audit trail.
  • Storing balances as a single mutable number with no ledger of movements behind it.

In the wild

Payment companies, banks and marketplaces all run daily reconciliation between their internal ledgers, payment processors and bank statements, and payment providers publish settlement and payout reports for exactly this purpose. Inventory-heavy businesses reconcile system stock against warehouse counts. Double-entry ledgers are the norm in financial software, from accounting packages to the internal ledgers of large payment platforms.

Check yourself

  1. Why compare your ledger against the payment provider's report rather than against your own orders table?

    Show answer

    B. The provider's report was produced independently, so it catches bugs that affect both of your own tables Independent sources do not share your code's bugs, so differences reveal real errors.

  2. A payment captured at 23:59 shows as 'missing at provider' in today's run. What is the right response?

    Show answer

    B. Treat it as a possible timing break and re-check it after the next settlement Settlement lags capture; good jobs re-check before raising an alert.

  3. In double-entry bookkeeping, a bug records a sale's debit but not its credit. How is it detected?

    Show answer

    B. Debits no longer equal credits, so the books no longer sum to zero Every valid movement has two equal sides; a one-sided entry breaks the invariant.

  4. How should a confirmed break be repaired?

    Show answer

    B. Record a correcting entry through normal code paths, with an owner and a reason Corrections as new entries keep the history auditable and the fix reviewable.

Recap

  • Every system that moves money or inventory eventually drifts; reconciliation guarantees you notice.
  • Compare against an independent source, match on a shared key, and classify breaks.
  • Allow for timing differences, then give every remaining break an owner and track them.
  • Repair with correcting entries, not edits, and fix repeated causes at the source.
  • Double-entry bookkeeping makes many errors detectable because the books must always balance.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy