Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Asynchronous Processing and Messaging

System Design HLD-09.06

Retries, dead letter queues and poison messages

Handling messages that fail: transient vs permanent errors, retry queues with growing delays, dead letter queues, alerting and redrive.

Intermediate 10 min Last reviewed

The big idea

Some messages fail. Usually the cause is temporary: the database was briefly overloaded, a downstream API timed out. Try again a little later and it works. But some messages fail every single time: a malformed payload, a reference to a deleted customer, a bug that crashes on one particular input. Retrying those forever does nothing except waste capacity, and in an ordered queue it blocks everything behind them ( Ordering, partition keys and parallelism ). Such a message is called a poison messageA message that fails processing every time it is attempted, usually because of bad data or a bug, and can block or crash consumers if retried forever..

The standard answer has three parts: retry a few times with growing pauses, then move the message aside into a dead letter queue (DLQ)A separate queue or topic where messages go after they fail processing too many times, so they can be inspected, fixed and replayed without blocking the main flow., then have a human or a tool look at it, fix the cause, and send it back. The main flow keeps moving the whole time.

Think of a sorting office. A letter with a smudged address gets a second look, maybe a third. After that it does not go back on the conveyor belt to jam it again. It goes to the dead letter office, where someone works out where it was meant to go.

How it works

Main queue, retry queue and dead letter queue A producer sends messages to the main queue, which a consumer reads. Successful messages complete. A message that fails is sent to a retry queue with a delay that grows on each attempt, then loops back to the main queue. After the maximum number of attempts, the message is moved to the dead letter queue, which glows red and triggers an alert. After the bug is fixed, an operator redrives messages from the dead letter queue back to the main queue. success fail, attempts left after delay max attempts redrive after fix Producer Main queue Consumer Done Retry queue delay 10 s, 20 s, 40 s Dead letter queue alert on growth Figure 1. A failed message loops through the retry queue with growing delays (10 s, 20 s, 40 s). After the last attempt it moves to the DLQ, which raises an alert. Once the cause is fixed, the DLQ is redriven into the main queue.

Step 1: Transient or permanent?

Retrying only helps when the failure is transient: a timeout, a throttling response, a deadlock, a briefly unavailable dependency. A permanent failure, such as invalid data, a validation error or a missing record that will never appear, fails identically on every attempt.

When the consumer can tell the difference, it should: send permanent failures straight to the DLQ and retry only transient ones. Often it cannot tell for sure, which is why every retry policy also needs a limit.

Step 2: Retry with growing delays

Retrying immediately rarely helps: if the database is overloaded, hitting it again in a millisecond makes it worse. Retries use exponential backoffWaiting longer after each failed attempt, typically doubling the delay, often with random jitter so many clients do not retry in sync.: wait 10 seconds, then 20, then 40, usually with some random jitter so thousands of failed messages do not all come back at the same instant ( Timeouts, retries, backoff and jitter ).

Where the waiting happens matters:

  • In the consumer (sleep and try again): simple, but the consumer is stuck holding that message, and in a log it blocks the partition.
  • In a retry queue or topic: the consumer publishes the failed message, with an attempt counter, to a separate retry queue and moves on. A retry consumer delivers it back after the delay. Many teams chain several, such as orders-retry-10s, orders-retry-1m, orders-retry-10m. The main flow never waits.
  • In the broker: some queues handle it for you. In Amazon SQS, a message that is not deleted becomes visible again after a visibility timeout, and its receive count goes up each time.

Step 3: Dead letter queue

After the maximum number of attempts, the message moves to the DLQ, together with enough context to debug it: the original payload, the error, the attempt count, timestamps, and where it came from. Then the consumer acknowledges the original so the main flow continues.

dlq.pyruns with python3 (standard library)
from collections import deque

MAX_ATTEMPTS = 4
BASE_DELAY = 10                                  # seconds; doubles on each retry

def handle(msg, fixed):
    if msg["body"] == "qty=-3" and not fixed:   # bad data: fails every time (a poison message)
        raise ValueError("negative quantity")
    if msg["body"] == "qty=2" and msg["attempt"] == 1:
        raise TimeoutError("inventory service slow")   # transient: succeeds on retry
    return "ok"

def run(queue, fixed=False):
    retry, dlq, now = [], [], 0
    while queue or retry:
        if not queue:                            # nothing ready: jump to the next retry time
            retry.sort(key=lambda m: m["due"])
            now = retry[0]["due"]
            queue.append(retry.pop(0))
        msg = queue.popleft()
        try:
            handle(msg, fixed)
            print(f"t={now:>3}s  {msg['id']} attempt {msg['attempt']}: ok")
        except Exception as e:
            if msg["attempt"] >= MAX_ATTEMPTS:
                dlq.append(msg)
                print(f"t={now:>3}s  {msg['id']} attempt {msg['attempt']}: {e} -> DLQ")
            else:
                delay = BASE_DELAY * 2 ** (msg["attempt"] - 1)
                retry.append({**msg, "attempt": msg["attempt"] + 1, "due": now + delay})
                print(f"t={now:>3}s  {msg['id']} attempt {msg['attempt']}: {e}, retry in {delay}s")
    return dlq

msgs = [{"id": f"m{i}", "body": b, "attempt": 1} for i, b in enumerate(["qty=1", "qty=-3", "qty=2"])]
dlq = run(deque(msgs))
print("-- consumer fixed to handle it, redrive the DLQ --")
run(deque({**m, "attempt": 1} for m in dlq), fixed=True)

Output

t=  0s  m0 attempt 1: ok
t=  0s  m1 attempt 1: negative quantity, retry in 10s
t=  0s  m2 attempt 1: inventory service slow, retry in 10s
t= 10s  m1 attempt 2: negative quantity, retry in 20s
t= 10s  m2 attempt 2: ok
t= 30s  m1 attempt 3: negative quantity, retry in 40s
t= 70s  m1 attempt 4: negative quantity -> DLQ
-- consumer fixed to handle it, redrive the DLQ --
t=  0s  m1 attempt 1: ok

Watch the two kinds of failure. m2 hits a timeout once and succeeds on its first retry. m1 has a negative quantity, a permanent problem, so it fails four times with delays of 10, 20 and 40 seconds and lands in the DLQ. Meanwhile m0 and m2 were never held up. Once the consumer is fixed, redriving the DLQ processes m1 successfully.

Step 4: Alert, fix, redrive

A DLQ nobody watches is just a slower way to lose data. Make it operational:

  • Alert on DLQ depth and on the age of its oldest message. Any non-empty DLQ for a payments flow deserves a page; for analytics, a daily report may do.
  • Inspect and fix the cause: deploy the bug fix, repair the bad data, or decide the message should be discarded, and record that decision.
  • Redrive: move messages back to the main queue for reprocessing. Because they will now be processed after newer messages, consumers must handle out-of-order and duplicate arrivals, which is one more reason for idempotent consumers ( Delivery semantics ).

Going deeper

Retries and ordering

Moving a failed message to a retry topic lets later messages overtake it. If they are about the same entity, such as order 42 shipped overtaking order 42 paid, the consumer may apply them in the wrong order. Options: route all subsequent messages for a blocked key to the retry path too until it clears, or use per-entity version numbers so the consumer can detect and hold events that arrive early ( Ordering, partition keys and parallelism ).

Retry storms

When a shared dependency goes down, every message fails at once and every retry fires on schedule, multiplying the load just as the dependency tries to recover. Cap total retry rate, add jitter, and consider pausing consumption entirely when a circuit breaker reports the dependency is down ( Circuit breakers and bulkheads ). Messages wait safely in the queue instead of burning through their attempts and filling the DLQ.

Trade-offs

OptionWins whenLoses when
Retry in the consumer (sleep) Low volume, short transient errors, simple code Blocks the consumer and, in a log, the partition behind it
Retry queues with growing delays Keeping the main flow moving while transient errors clear More topics to run; later messages can overtake retried ones
Dead letter queue after N attempts Isolating poison messages without losing them Needs alerting, tooling and someone to own it
Drop after N attempts Data whose loss is acceptable, such as best-effort telemetry Anything you would need to recover

Common mistakes

  • Retrying forever, so a single poison message blocks a partition or burns capacity indefinitely.
  • Retrying immediately with no backoff, hammering a dependency that is already struggling.
  • Sending messages to a DLQ that has no alert and no owner, where they sit until someone notices missing orders.
  • Losing context: dead-lettering the payload without the error, attempt count and source.
  • Redriving the DLQ before deploying the fix, so everything fails again and returns to the DLQ.

In the wild

Dead letter queues are built into many brokers. Amazon SQS moves a message to a configured DLQ after a maximum receive count and can redrive it back to the source queue. RabbitMQ routes rejected or expired messages to a dead letter exchange. Kafka has no consumer DLQ of its own (it is a retained event stream rather than a queue, the difference explained in message queue vs pub/sub vs event streams ), so teams build retry and dead letter topics in their consumers, and frameworks such as Kafka Connect and Spring for Apache Kafka provide dead letter topic support.

Check yourself

  1. Which failure is worth retrying?

    Show answer

    B. A timeout from an overloaded downstream service Timeouts and throttling are transient and may succeed later. The others fail identically every time.

  2. Why use a separate retry queue rather than sleeping inside the consumer?

    Show answer

    B. The main flow keeps moving while the failed message waits elsewhere Sleeping holds the consumer (and the partition) hostage; a retry queue parks the message and frees the consumer.

  3. A DLQ has grown for three weeks and nobody noticed. What was missing?

    Show answer

    B. An alert on DLQ depth or oldest message age, and an owner A DLQ is only useful if someone is told when it fills and is responsible for fixing and redriving it.

  4. You redrive 500 messages from the DLQ. What must the consumer tolerate?

    Show answer

    B. Messages arriving late, out of order relative to newer ones, and possibly duplicates Redriven messages come after newer events and may already be partially applied, so consumers must be idempotent and order-aware.

Recap

  • Transient failures deserve retries; permanent ones (poison messages) do not.
  • Retry with exponential backoff and jitter, ideally via retry queues so the main flow keeps moving.
  • After a fixed number of attempts, move the message with its error context to a dead letter queue.
  • Alert on DLQ growth, fix the cause, then redrive.
  • Retries and redrives reorder and duplicate messages, so consumers must be idempotent and order-aware.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up