Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Asynchronous Processing and Messaging

HLD-M09

Asynchronous Processing and Messaging

Queues, pub/sub and event streams: how services hand off work, absorb spikes, survive failures and react to events, from Kafka internals to stream processing.

10 lessonsabout 125 min

Not every piece of work has to finish before you answer the user. Resizing images, sending emails, updating search indexes and feeding analytics can all happen a moment later, and doing them later makes systems faster, more resilient and easier to scale. The tools for this are queues, pub/sub topics and event streams.

This module starts with why queues help at all, then compares queues, pub/sub and logs, and takes Kafka apart to see how a distributed log works. You will learn what at-least-once and exactly-once really mean, how partition keys trade ordering for parallelism, how to handle poison messages with retries and dead letter queues, and how backpressure keeps a pipeline stable. It ends with the patterns built on top: the transactional outbox and change data capture, event-driven architecture and event sourcing, and stream processing with windows and watermarks.

Lessons

  1. HLD-09.01 Why async? Queues explained A queue lets a producer hand off work and move on, while consumers process it at their own pace. Beginner 10 min
  2. HLD-09.02 Message queues vs pub/sub vs event streams A message queue delivers each message to one consumer, which suits work distribution. Intermediate 12 min
  3. HLD-09.03 Kafka internals: the distributed log Kafka stores each topic as partitions, and each partition is an append-only log where messages get sequential offsets. Intermediate 18 min
  4. HLD-09.04 Delivery semantics At-most-once delivery may lose messages but never duplicates them; at-least-once never loses them but may deliver duplicates after retries; exactly-once means each message affects the result once. Intermediate 12 min
  5. HLD-09.05 Ordering, partition keys and parallelism To keep events for one entity in order, send them all to the same partition by using the entity id as the partition key. Advanced 10 min
  6. HLD-09.06 Retries, dead letter queues and poison messages Some messages fail repeatedly because of bugs or bad data. Intermediate 10 min
  7. HLD-09.07 Backpressure and flow control Backpressure is how a slow consumer tells fast producers to slow down. Advanced 10 min
  8. HLD-09.08 Transactional outbox and change data capture Writing to a database and publishing an event as two separate steps can fail halfway, leaving them inconsistent. Advanced 14 min
  9. HLD-09.09 Event-driven architecture and event sourcing In event-driven architecture, services announce facts ("order placed") and others react, instead of calling each other directly. Advanced 14 min
  10. HLD-09.10 Stream processing Stream processing computes results continuously over unbounded data instead of in periodic batches. Advanced 15 min

Start with lesson 1