Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Asynchronous Processing and Messaging

System Design HLD-09.01

Why async? Queues explained

How a queue lets producers hand off work and move on: decoupling services, absorbing traffic spikes, background jobs and worker pools.

Beginner 10 min Last reviewed

The big idea

A user uploads a profile photo. Your server has to store it, make three resized versions, scan it for abuse and email a confirmation. Done one after another inside the request, that might take 8 seconds, and if the email provider is down, the upload fails even though the photo was fine.

Most of that work does not need to happen before you answer the user. A message queueA buffer that holds messages from producers until consumers are ready to process them, usually in first-in, first-out order. lets the web server write a small note, "resize photo 123", hand it off, and reply "uploaded" in milliseconds. Separate workers pick notes off the queue and do the slow parts at their own pace. The work happens asynchronouslyNot waiting for the result: the caller hands off the work and continues; the result arrives or is checked later.: later, but reliably.

Think of a busy restaurant. The waiter does not stand in the kitchen while your dish cooks. They clip the order onto the ticket rail and go take the next table's order. The cooks pull tickets from the rail as fast as they can. When ten tables order at once, the rail gets long for a few minutes, but no waiter is stuck and no order is lost.

How it works

Step 1: Producers and consumers

The program that puts messages on the queue is the producerA program or service that creates messages and sends them to a queue.. The program that takes them off and does the work is the consumerA program or service that reads messages from a queue and processes them., often called a worker. The queue sits between them and stores messages durably, so a message survives even if a worker crashes.

Producers, a queue and a worker pool Web servers act as producers and push messages into a queue during a traffic spike. The queue fills like a buffer, holding the extra messages. A pool of four workers pulls messages from the queue at a steady rate and writes results to storage and an email service. enqueue Users Web servers producers Queue messages waiting Worker 1 Worker 2 Worker 3 Worker 4 Figure 1. Web servers drop messages into the queue and reply to users immediately (for example 202 Accepted). During a spike the queue grows; the worker pool keeps pulling at its own steady pace and the backlog drains when traffic falls.

A worker's loop is simple: take a message, do the work, then acknowledge it so the queue deletes it. If the worker crashes before acknowledging, the queue gives the message to another worker. That is how no work gets lost, and also why a message can occasionally be processed twice, which you will deal with in Delivery semantics .

Step 2: Decoupling

Without a queue, the web server calls the image service directly. If the image service is slow, the upload is slow. If it is down, the upload fails. If you deploy it, uploads break for a minute.

With a queue in between, the producer only needs the queue to be up. The consumer can be slow, restarting, or being deployed, and messages simply wait. The two sides no longer need to be running at the same time, know each other's addresses, or scale together. That independence is called decouplingArranging components so that a change, slowdown or failure in one does not directly affect the other..

Step 3: Smoothing spikes

Traffic is bursty. A newsletter goes out and 50,000 people click in five minutes. Without a queue, you need enough workers for the peak, and they sit idle the rest of the day. With a queue, you size workers for the average, and the queue absorbs the peak as a temporary backlog.

The arithmetic is simple. Say 8 workers each handle 5 messages per second: 40 per second. Normal traffic is 20 per second. A spike sends 80 per second for 20 seconds. During the spike the queue grows by 80 - 40 = 40 messages a second, so after 20 seconds there are 800 waiting. After the spike, workers have 40 - 20 = 20 spare messages per second, so the backlog drains in 800 / 20 = 40 seconds. Nobody saw an error; some photos were resized a minute later.

Step 4: Background jobs and worker pools

The same pattern runs most "do it later" work: sending emails and notifications, generating reports and invoices, transcoding video, indexing documents for search, calling slow third-party APIs. A worker poolA group of identical consumer processes reading from the same queue, so work is shared among them and capacity grows by adding workers. is just several identical consumers on the same queue. Each message goes to exactly one of them, so adding workers adds capacity. Many teams scale the pool automatically based on queue depth.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. With the defaults (20 msg/s, 8 workers, spike), check the peak backlog and how long it takes to drain. Compare with the arithmetic in Step 3.
  2. Drop to 4 workers. Capacity (20 msg/s) now equals the average rate, so the backlog from the spike never really drains. What happens at 3 workers?
  3. Switch to the slow wave. Find the smallest worker count that keeps the peak wait under 10 seconds. That is the trade-off you make: fewer workers, longer waits at the peak.

Trade-offs

OptionWins whenLoses when
Synchronous call The user needs the result now (a search, a balance, a payment authorization) Slow or failing downstream services directly slow or fail the request
Queue + worker pool Work that can finish later: emails, thumbnails, reports, webhooks Results arrive later; one more system to run; duplicates and ordering need care
Fire and forget with no queue Truly unimportant work, like best-effort metrics Anything you would mind losing when a process restarts

Common mistakes

  • Putting work on a queue when the user actually needs the result to continue.
  • Letting the queue grow forever without alerting on its depth or the age of the oldest message.
  • Assuming each message is processed exactly once; retries mean workers must tolerate duplicates.
  • Putting large payloads (whole images) in messages instead of a reference to where they are stored.
  • Forgetting to tell users the work is still in progress, so a pending thumbnail looks like a bug.

In the wild

Queues are everywhere behind modern apps. Hosted queues such as Amazon SQS and Google Cloud Tasks, brokers such as RabbitMQ, and job libraries such as Sidekiq for Ruby or Celery for Python all implement this producer, queue and worker pattern. When more than one service needs each message, or you need to replay old ones, compare message queues vs pub/sub vs event streams . Whenever an app says "we will email you when your export is ready", a queue is almost certainly involved.

At the other extreme, a stock exchange's order matching engine feeds orders to its matching thread through an in-memory queue in strict sequence order, with no broker in between, because its whole budget is a few microseconds.

Check yourself

  1. What does a queue let a producer do?

    Show answer

    B. Hand off work and continue without waiting for it to be processed The producer only needs the queue to accept the message; processing happens later at the consumer's pace.

  2. Workers handle 40 msg/s in total. A spike sends 100 msg/s for 10 seconds. How big is the backlog at the end of the spike?

    Show answer

    C. 600 The queue grows by 100 - 40 = 60 messages each second, for 10 seconds: 600.

  3. The email provider is down for 5 minutes. With a queue, what happens to sign-ups?

    Show answer

    B. They succeed; confirmation emails wait in the queue and go out when the provider recovers Decoupling means the sign-up only depends on the queue, not on the email provider.

  4. Producers average 30 msg/s and workers handle 25 msg/s. What happens over time?

    Show answer

    B. The queue grows without limit A queue absorbs temporary bursts, not a permanent shortfall. Average capacity must exceed average arrival rate.

Recap

  • A queue lets producers hand off work and move on; consumers process it at their own pace.
  • Decoupling means the producer does not care if the consumer is slow, restarting or down.
  • Queues absorb spikes: size workers for the average and let the backlog drain afterwards.
  • Worker pools share a queue; add workers to add capacity.
  • If average arrivals exceed capacity, the queue grows forever, so watch its depth.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up