Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Computer and Network Foundations

System Design HLD-01.02

Processes, threads and concurrency

Processes, threads, blocking and non-blocking I/O, and why an event loop can hold thousands of idle connections that would sink a thread-per-request server.

Beginner 12 min Last reviewed

The big idea

A web server's main job is not computing. It is waiting. A typical request might use 1 ms of CPU and then spend 50 ms waiting for a database, a cache or another service to answer. If the server sits idle during every wait, it wastes almost all of its time. ConcurrencyMaking progress on many tasks during overlapping periods of time, for example by switching to another request while one waits for I/O. is how a server keeps working on other requests while some are waiting.

There are two classic ways to do it. Give every request its own worker that is allowed to sit and wait, or have one worker that never waits: it starts each slow operation, moves on, and comes back when the answer arrives. Both are everywhere, and choosing between them shapes how many users a machine can hold.

Picture a restaurant. In one, every table gets a dedicated waiter who stands by the kitchen door until that table's food is ready. In the other, one fast waiter takes an order, hands it to the kitchen, takes the next order, and delivers each dish when the kitchen rings the bell. The second restaurant serves many more tables per waiter, as long as nobody makes the waiter cook.

How it works

Step 1: Processes and threads

A processA running program with its own private memory, file handles and at least one thread. The operating system isolates processes from each other. is a running program with its own memory. Processes are isolated: one crashing does not corrupt another, but sharing data between them takes extra work.

A threadA sequence of instructions the CPU runs. Threads in the same process share its memory, so they are lighter than processes but can step on each other's data. is a line of execution inside a process. Threads share the process's memory, which makes them cheaper to create and easy to share data between, and also makes bugs like two threads updating the same value at once possible.

A CPU core runs one thread at a time. The operating system creates the illusion of many running together by switching between them thousands of times a second. Each context switchThe operating system pausing one thread and resuming another on a CPU core, saving and restoring their state. It costs a few microseconds of pure overhead. costs a few microseconds of overhead, small alone but large when there are thousands of threads.

Step 2: Blocking and non-blocking I/O

When code reads from a socket or a file in the usual way, the call blocksA call that does not return until the operation finishes, so the thread making it can do nothing else in the meantime.: the thread stops until the data arrives. With non-blocking I/O, the call returns immediately, and the program asks the operating system to notify it when the data is ready, using mechanisms such as epoll on Linux.

Step 3: Two server designs

Thread per request versus an event loop Left: five requests each own a thread; four threads are dim because they are blocked waiting for the database, one is active. Right: one event loop glowing at the center hands I/O to the operating system and receives completion events, serving the same five requests. Thread per request Event loop start I/O done Thread 1 waiting on DB Thread 2 waiting on DB Thread 3 running Thread 4 waiting on DB Thread 5 waiting on DB Database Event loop one thread OS: epoll watches sockets Database callbacks ready Figure 1. Left, each request holds a thread, and most threads sit blocked on I/O (dim). Right, one loop starts I/O, moves on, and runs each callback when its result arrives (glowing).

Thread per request. Each request gets a thread from a pool. The code is simple and reads top to bottom: query the database, wait, use the result. But every waiting thread still holds its memory (often around a megabyte of stack by default) and adds context switches. With 10,000 slow connections you need 10,000 threads, or the extra requests queue.

Event loop. A single thread runs a loop: take the next ready event, run its handler until the handler starts some I/O, then move on. Waiting requests cost only a small record of what to do next, often a few kilobytes. Node.js, Nginx and Redis work this way, and so does Python's asyncio. One loop per CPU core can hold tens of thousands of mostly idle connections.

Step 4: The catch with event loops

The loop only works if handlers return quickly. If one handler does 200 ms of CPU work, such as resizing an image or parsing a huge JSON file, every other request on that loop waits 200 ms too. Event-loop systems therefore move heavy CPU work to separate worker threads or processes, and use async versions of every I/O library.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Keep the defaults. The event loop serves about twice as many requests per second, because the 200 threads spend 99 percent of their time waiting, and it does so with a fraction of the memory.
  2. Raise the thread pool to 2,000. The gap narrows, but look at the memory line.
  3. Raise CPU per request to 20 ms. Now both designs are limited by the 4 cores and finish level: concurrency does not create CPU.

Going deeper

Little's Law in disguise

The thread pool's limit is a direct use of Little's LawAverage number of requests in a system equals arrival rate times average time each spends there: L = lambda x W.: with 200 threads and 101 ms per request, at most 200 / 0.101, about 1,980 requests per second, can be in progress. You will use it again in Latency vs throughput, and why percentiles matter .

Modern blends

The line is blurring. Go runs lightweight goroutines that look like blocking code but are scheduled onto a few OS threads, and Java's virtual threads do the same. You write simple sequential code and get event-loop efficiency. Many production servers also combine designs: one event loop per core, plus a worker pool for CPU-heavy tasks.

Trade-offs

OptionWins whenLoses when
Thread per request Simple sequential code, CPU-heavy work, moderate numbers of connections Tens of thousands of mostly idle connections: memory and context switches pile up
Event loop Huge numbers of I/O-bound connections such as proxies, chat and APIs Any blocking or CPU-heavy code: it stalls every request on the loop
Lightweight threads (goroutines, virtual threads) Simple code with high concurrency Runtimes without them, or code that holds OS threads through native calls

Common mistakes

  • Calling a blocking library inside an event loop. One synchronous database call can freeze the whole server.
  • Thinking more threads always mean more throughput. Past the number of cores, CPU-bound work only gets more switching.
  • Confusing concurrency with parallelism. Concurrency is juggling many tasks; parallelism is literally running them at the same instant on several cores.
  • Sharing data between threads without locks or other protection, causing rare, hard-to-reproduce bugs.

Check yourself

  1. What do threads in the same process share?

    Show answer

    B. The process's memory Threads share the process's memory (each has its own stack), which makes sharing data easy and races possible.

  2. A server's requests spend 1 ms on CPU and 100 ms waiting on a database. Which design holds the most concurrent connections per gigabyte of memory?

    Show answer

    B. Event loop Waiting requests cost the event loop only a small record each, while each blocked thread holds its stack.

  3. A Node.js handler spends 300 ms computing a report. What happens to other users?

    Show answer

    B. Their requests wait until the computation finishes The event loop runs one handler at a time. CPU work blocks everything; move it to a worker.

  4. A machine has 4 cores and every request needs 10 ms of CPU. What is the most it can serve, whatever the design?

    Show answer

    B. About 400 requests per second Each core does 100 requests per second of 10 ms CPU each, so 4 cores do about 400. Concurrency cannot create CPU.

Recap

  • A process has its own memory; threads inside it share memory and are cheaper.
  • Servers mostly wait on I/O, so concurrency is about using that waiting time.
  • Thread per request is simple but pays memory and context switches for every waiting request.
  • An event loop never waits, so it holds huge numbers of idle connections, but blocking code stalls it.
  • Concurrency cannot exceed the CPU: when work is CPU-bound, add cores or machines.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy