System Design HLD-09.04
Delivery semantics
At-most-once, at-least-once and exactly-once delivery: where a crash loses or duplicates messages, and how dedupe and idempotency give effectively-once.
The big idea
A consumer takes a payment message, credits the account, and is about to tell the broker "done" when its machine dies. The broker never heard "done". Should it send the message again? If it does, the account may be credited twice. If it does not, and the crash had actually happened before the credit, the payment is lost. The broker cannot know which, because from the outside a crash before the work and a crash after it look the same.
Every messaging system resolves this the same way: it picks which failure it would rather have. Lose messages occasionally, or duplicate them occasionally. Those are the two practical delivery semanticsThe guarantee a messaging system gives about how many times a message is delivered or processed: at most once, at least once, or exactly once.. The third, exactly once, is what everyone wants and what almost no system can offer on its own across a network.
Think of mailing a cheque without tracking. If you never hear back, you can send another and risk paying twice, or not send one and risk never paying. The fix is not a better postal service. It is a cheque number the recipient checks before cashing.
How it works
Step 1: The order of "process" and "acknowledge" decides
A consumer does two things for each message: process it (update the database, send the email) and acknowledge it (commit the offset or ack the message, Kafka internals: the distributed log ). A crash can happen between them. Which one you do first decides what a crash costs you.
Step 2: At most once
At-most-onceEach message is processed zero or one times: it may be lost on failure but is never duplicated. delivery acknowledges first and processes second, or sends without waiting for any acknowledgement at all. A crash in between loses the message, but it is never processed twice. It is the cheapest option and fits data where a small loss does not matter: metrics samples, logs you can afford to drop, live location updates that the next update supersedes anyway.
Step 3: At least once
At-least-onceEach message is processed one or more times: it is never lost, but failures and retries can cause duplicates. delivery processes first and acknowledges second, and the producer retries until the broker confirms. Nothing is lost, but duplicates happen: a consumer crash before the ack causes a redelivery, and a producer whose confirmation got lost sends the same message again. This is the default for most queues and logs, and the right starting point for anything you cannot lose.
Step 4: Exactly once, and why it is really "effectively once"
Exactly-onceEach message affects the final result exactly one time, even if it was physically delivered more than once. would mean each message is delivered and processed once, no matter what fails. Across a network with independent failures, the sender can never be sure the other side got the message (the two generals problem, Why distributed systems are hard ), so physical delivery cannot be guaranteed exactly once.
What real systems achieve is effectively once: at-least-once delivery, plus a receiver that makes duplicates harmless. Two tools do it:
- Idempotent processing: design the operation so doing it twice equals doing it once. "Set status to SHIPPED" is idempotent; "add 50 to balance" is not. "Add 50 for payment pay-17 unless pay-17 was already applied" is ( Idempotency and safe retries ).
- A dedupe table: record every processed message id in the same database transaction as the work. A redelivered message finds its id already there and is skipped. Because the two writes commit together, there is no window where the work is done but the id is not recorded.
import sqlite3
# The broker redelivers "pay-17" because the consumer crashed before acknowledging it.
deliveries = ["pay-16", "pay-17", "pay-17", "pay-18"] # at-least-once: pay-17 arrives twice
amounts = {"pay-16": 30, "pay-17": 50, "pay-18": 20}
def run(dedupe):
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE account (balance INTEGER)")
db.execute("INSERT INTO account VALUES (0)")
db.execute("CREATE TABLE processed (msg_id TEXT PRIMARY KEY)") # the dedupe table
for msg in deliveries:
try:
with db: # one transaction
if dedupe:
db.execute("INSERT INTO processed VALUES (?)", (msg,)) # fails if seen
db.execute("UPDATE account SET balance = balance + ?", (amounts[msg],))
except sqlite3.IntegrityError:
print(f" skip duplicate {msg}")
return db.execute("SELECT balance FROM account").fetchone()[0]
print("expected balance:", sum(amounts.values()))
print("no dedupe :", run(dedupe=False))
print("with dedupe :")
print(" balance", run(dedupe=True))Output
expected balance: 100
no dedupe : 150
with dedupe :
skip duplicate pay-17
balance 100 Without the dedupe table, the redelivered pay-17 credits 50 twice. With it, the primary key on processed rejects the second insert, the transaction rolls back, and the balance is right.
Going deeper
Exactly-once inside one system
Some systems offer exactly-once within their own boundaries. Kafka's idempotent producer attaches a producer id and sequence number to each message so the broker discards retried duplicates. Kafka transactions let a consumer-producer write its output messages and its consumer offsets atomically, and readers set to read only committed data never see partial results. Stream processors such as Kafka Streams and Flink build exactly-once processing on these features. The guarantee holds while everything stays inside Kafka; the moment you call an external API or write to another database, you need idempotency or a dedupe table again (or the outbox pattern, Transactional outbox and change data capture ).
Dedupe windows
Some brokers deduplicate for you within a time window. Amazon SQS FIFO queues, for example, drop a message whose deduplication id was already seen in the previous 5 minutes. A window bounds memory, but a retry that arrives after the window passes straight through, so long-delayed retries still need idempotent consumers.
Choosing the message id
The id must identify the business event, not the delivery attempt. Generate it once where the event is created (an order id plus event type, or a UUID created before the first send) so every retry carries the same id.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| At most once | Telemetry, metrics, superseded updates where an occasional loss is fine | Anything you cannot afford to lose |
| At least once | The safe default; nothing lost, simple to run | Duplicates reach consumers, which must cope with them |
| At least once + idempotency or dedupe | Payments, orders, inventory: effectively-once results | Extra storage and design work; ids must be stable end to end |
| Broker transactions (exactly-once in Kafka) | Read-process-write pipelines that stay inside the broker | Throughput and complexity; guarantee ends at the system boundary |
Common mistakes
- Believing a vendor's "exactly once" covers side effects such as emails, card charges or calls to other services.
- Committing offsets before processing (often via auto-commit) and silently losing messages on crashes.
- Recording the message id in a different transaction from the work, recreating the duplicate window.
- Generating a new message id on each retry, so the dedupe table never matches.
- Making non-idempotent updates ("increment") in consumers that receive at-least-once delivery.
In the wild
Most managed queues document at-least-once delivery for their standard mode, Amazon SQS standard queues among them, and leave duplicates to the consumer. Payment APIs such as Stripe accept an idempotency key on requests so a retried charge is not applied twice. Kafka offers idempotent producers and transactions for exactly-once processing within Kafka, and stream processors such as Flink and Kafka Streams build on them.
Check yourself
-
A consumer commits its offset, then crashes before processing the message. What did you get?
Show answer
B. At-most-once: the message is lost The commit says 'done', so the message will not be redelivered, but the work never happened.
-
A consumer processes a message, then crashes before acknowledging. What happens next?
Show answer
B. The broker redelivers it and it is processed again Without an ack the broker assumes failure and redelivers, which causes a duplicate if the work had already happened.
-
Which consumer operation is naturally idempotent?
Show answer
C. Set order 42's status to SHIPPED Setting a value to a fixed state gives the same result no matter how many times it runs.
-
Why must the processed message id be stored in the same transaction as the business update?
Show answer
B. So the work and the record of doing it commit together; otherwise a crash between them recreates duplicates or losses Two separate writes reopen the crash window the dedupe table was meant to close.
Recap
- Whether you ack before or after processing decides if a crash loses or duplicates a message.
- At-most-once may lose messages; at-least-once may duplicate them.
- True exactly-once delivery across a network is impractical; aim for effectively-once results.
- Effectively once = at-least-once delivery + idempotent processing or a dedupe table in the same transaction.
- Broker exactly-once features stop at the broker's boundary; external side effects still need idempotency.
Next up
- Crack the system design round learning path