Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Asynchronous Processing and Messaging

System Design HLD-09.02

Message queues vs pub/sub vs event streams

Message queue vs pub/sub vs event streaming: who gets each message, how long it is kept, when you can replay, and where SQS, SNS, RabbitMQ and Kafka fit.

Intermediate 12 min Last reviewed

The big idea

"Put it on a queue" hides a choice that shapes your whole design. When an order is placed, maybe one worker should charge the card. Maybe the email service, the analytics pipeline and the warehouse should each hear about it. Maybe next month a new fraud model should be able to read every order from the last week. Those are three different messaging models, and the tools built for each behave very differently.

Two questions separate them. Who gets each message? One consumer, or every interested party. How long does a message live? Until someone processes it, or for a fixed retention period regardless of who has read it.

Think of three ways to share a memo. A task in a shared inbox: one person picks it up and it is done. A mailing list: everyone subscribed gets their own copy as it is sent. A noticeboard where memos stay pinned for a week: anyone can walk up and read from wherever they left off, and someone new can start from the oldest memo still on the board.

How it works

A queue, a pub/sub topic and an event log side by side Three panels. In the queue panel, a producer sends messages to a queue, and each message goes to only one of two workers. In the pub/sub panel, a producer publishes to a topic, and every message is copied to both the email and the analytics subscribers. In the log panel, a producer appends to a log of numbered entries 0 to 5; a billing consumer group is at offset 5 and a fraud group at offset 2, each reading independently. Queue (point-to-point) Pub/sub (fan-out) Event log (retained) m1 m2 m1 m1 Producer Queue deleted on ack Worker A Worker B Producer Topic copies to all Email Analytics Producer Log 0 1 2 3 4 5 ... Billing offset 5 Fraud offset 2 Figure 1. Queue: each message goes to one worker. Pub/sub: every subscriber gets its own copy. Log: messages stay in order, and each consumer group reads at its own offset and can rewind.

Model 1: Message queue (point-to-point)

A point-to-point queueA messaging model where each message is delivered to exactly one of the consumers reading the queue, and is removed once that consumer acknowledges it. delivers each message to one consumer. Several workers can read the same queue, and they compete for messages, which is how work gets shared out ( Why async? Queues explained ). Once a worker acknowledges a message, it is deleted.

Use it for work distribution: jobs that must be done once by someone, such as charging a card, resizing an image or sending one email. RabbitMQ queues and Amazon SQS are the classic examples. The queue usually tracks each message individually: in flight, acknowledged, or to be redelivered.

Model 2: Publish/subscribe (fan-out)

In pub/subPublish/subscribe: a messaging model where publishers send messages to a topic and every subscriber of that topic receives its own copy., a publisher sends a message to a topic, and every subscriber receives its own copy. The publisher does not know or care who is listening. Add a new subscriber and it starts receiving messages with no change to the publisher.

Use it for broadcasting events: "order placed" goes to email, analytics and the warehouse, each doing its own thing. The catch in classic pub/sub is that messages are not kept for later: a subscriber that joins tomorrow does not see today's events, and a subscriber that is offline may miss messages unless its copy is held somewhere durable.

A common pattern combines the two: a topic fans out to one queue per subscribing service, and each service runs a pool of workers on its own queue. Amazon SNS topics delivering to SQS queues, and RabbitMQ fanout exchanges bound to several queues, both work this way. Every service gets every event, and within a service the work is shared.

Model 3: Event stream (the log)

An event streamA durable, append-only, ordered log of events that consumers read by position (offset); messages are kept for a retention period whether or not they have been read. is an append-only log. Producers append; each message gets a sequential position called an offsetThe position of a message in a log partition; a consumer's offset records how far it has read.. Reading does not delete anything. Each consumer, or consumer group, just remembers its own offset: "I have read up to 5".

That one change, keeping messages and letting readers track their own position, unlocks a lot:

  • Many independent readers: billing and fraud read the same events at their own speed. A slow reader does not slow the others.
  • Replay: rewind an offset to reprocess history after fixing a bug, or to backfill a brand-new service from the last week of events.
  • Ordering: within a log (or a partition of it), everyone sees events in the same order.

Kafka is the best-known example, and keeps messages for 7 days by default, configurable per topic, up to forever. Within one consumer group, partitions are divided among members, so a log also supports work sharing ( Kafka internals: the distributed log ).

See the difference in code

topologies.pyruns with python3 (standard library)
from collections import deque

events = ["order-1", "order-2", "order-3", "order-4"]

# 1. Queue: each message goes to ONE consumer, then it is gone.
queue = deque(events)
workers = {"w1": [], "w2": []}
while queue:
    for w in workers:
        if queue:
            workers[w].append(queue.popleft())
print("queue   :", workers, "| left in queue:", list(queue))

# 2. Pub/sub: each message is copied to EVERY subscriber present when it is published.
subscribers = {"email": [], "analytics": []}
for e in events:
    for inbox in subscribers.values():
        inbox.append(e)
subscribers["search"] = []                    # joins late: misses everything before it
print("pub/sub :", subscribers)

# 3. Log (stream): messages stay; each consumer group keeps its own offset.
log = list(events)
offsets = {"billing": 0, "fraud": 2}         # fraud has read 2 already
def poll(group, n=10):
    start = offsets[group]
    batch = log[start:start + n]
    offsets[group] = start + len(batch)
    return batch
print("log     : billing ->", poll("billing"), "| fraud ->", poll("fraud"))
offsets["billing"] = 0                        # replay: rewind the offset
print("replay  : billing ->", poll("billing", 2), "| log still has", len(log), "events")

Output

queue   : {'w1': ['order-1', 'order-3'], 'w2': ['order-2', 'order-4']} | left in queue: []
pub/sub : {'email': ['order-1', 'order-2', 'order-3', 'order-4'], 'analytics': ['order-1', 'order-2', 'order-3', 'order-4'], 'search': []}
log     : billing -> ['order-1', 'order-2', 'order-3', 'order-4'] | fraud -> ['order-3', 'order-4']
replay  : billing -> ['order-1', 'order-2'] | log still has 4 events

Same four events, three outcomes. The queue splits them between workers and ends empty. Pub/sub copies them to both subscribers, and the latecomer "search" gets nothing. The log keeps all four, lets each group read from its own offset, and replays from the start on request.

Going deeper

What the broker has to track

A queue broker tracks the state of every message: delivered, acknowledged or due for redelivery. That allows per-message features such as individual retries, delays and priorities, but costs bookkeeping per message. A log broker tracks one number per consumer group per partition. That is why logs scale to very high throughput, and also why per-message retries and priorities are awkward on a log: you cannot skip one message and come back to it without your own retry machinery ( Retries, dead letter queues and poison messages ).

Ordering guarantees differ

A queue with many competing workers does not preserve order in practice: message 2 may finish before message 1 because a different worker took it. A log preserves order within a partition, at the cost that one slow message holds up everything behind it in that partition ( Ordering, partition keys and parallelism ).

Trade-offs

OptionWins whenLoses when
Message queue (RabbitMQ, SQS) Distributing jobs to a worker pool; per-message retries, delays and priorities Replay, multiple independent readers, strict ordering with many workers
Pub/sub topic (SNS, fanout exchange) Broadcasting events to several services that each react Late joiners and offline subscribers miss messages unless each has a durable queue
Event log (Kafka, Kinesis, Pulsar) Many readers, replay, backfills, high throughput, per-partition ordering Per-message retries and priorities; more to operate; retention costs storage

Common mistakes

  • Using a plain queue when several services each need every event, so they steal messages from each other.
  • Assuming a pub/sub subscriber will see events published while it was offline or before it subscribed.
  • Picking a log, then trying to implement per-message priorities or delays on top of it.
  • Expecting strict ordering from a queue read by many parallel workers.
  • Keeping events in a log forever "for replay" without planning for schema changes or storage cost.

In the wild

You will usually find all three in one company. Background job systems such as Sidekiq, Celery and SQS-based workers use queues. Notifications and webhooks are classic pub/sub fan-out, often SNS to SQS on AWS or exchanges in RabbitMQ. Activity feeds, analytics pipelines and data integration between services increasingly run on logs such as Kafka, Amazon Kinesis or Apache Pulsar, which also offer consumer groups for sharing work.

Check yourself

  1. Three services must each react to every 'order placed' event. A plain queue with one worker from each service is used. What goes wrong?

    Show answer

    B. Each event goes to only one of the three, so the others miss it Queue consumers compete. Use pub/sub, or one queue per service fed by a topic.

  2. A new recommendation service launches and needs the last 7 days of click events. Which model makes this easy?

    Show answer

    C. An event log with 7+ days of retention A log keeps events after they are read; the new consumer simply starts from an older offset.

  3. Why can a log broker handle more throughput than a queue broker that tracks each message?

    Show answer

    B. It only tracks one offset per consumer group per partition instead of every message's state Sequential appends and one number per reader are far cheaper than per-message bookkeeping.

  4. You need delayed retries and priorities for individual jobs. What fits best?

    Show answer

    A. A message queue Queues track messages individually, which makes per-message delays, retries and priorities natural.

Recap

  • Queues deliver each message to one consumer and delete it on ack: work distribution.
  • Pub/sub copies each message to every subscriber: broadcasting events.
  • Event logs keep messages for a retention period; each group reads at its own offset and can replay.
  • Topic to per-service queues combines fan-out with work sharing.
  • Choose by need: work sharing, fan-out, or replay and many independent readers.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up