System Design HLD-01.02
Processes, threads and concurrency
Processes, threads, blocking and non-blocking I/O, and why an event loop can hold thousands of idle connections that would sink a thread-per-request server.
The big idea
A web server's main job is not computing. It is waiting. A typical request might use 1 ms of CPU and then spend 50 ms waiting for a database, a cache or another service to answer. If the server sits idle during every wait, it wastes almost all of its time. ConcurrencyMaking progress on many tasks during overlapping periods of time, for example by switching to another request while one waits for I/O. is how a server keeps working on other requests while some are waiting.
There are two classic ways to do it. Give every request its own worker that is allowed to sit and wait, or have one worker that never waits: it starts each slow operation, moves on, and comes back when the answer arrives. Both are everywhere, and choosing between them shapes how many users a machine can hold.
Picture a restaurant. In one, every table gets a dedicated waiter who stands by the kitchen door until that table's food is ready. In the other, one fast waiter takes an order, hands it to the kitchen, takes the next order, and delivers each dish when the kitchen rings the bell. The second restaurant serves many more tables per waiter, as long as nobody makes the waiter cook.
How it works
Step 1: Processes and threads
A processA running program with its own private memory, file handles and at least one thread. The operating system isolates processes from each other. is a running program with its own memory. Processes are isolated: one crashing does not corrupt another, but sharing data between them takes extra work.
A threadA sequence of instructions the CPU runs. Threads in the same process share its memory, so they are lighter than processes but can step on each other's data. is a line of execution inside a process. Threads share the process's memory, which makes them cheaper to create and easy to share data between, and also makes bugs like two threads updating the same value at once possible.
A CPU core runs one thread at a time. The operating system creates the illusion of many running together by switching between them thousands of times a second. Each context switchThe operating system pausing one thread and resuming another on a CPU core, saving and restoring their state. It costs a few microseconds of pure overhead. costs a few microseconds of overhead, small alone but large when there are thousands of threads.
Step 2: Blocking and non-blocking I/O
When code reads from a socket or a file in the usual way, the call blocksA call that does not return until the operation finishes, so the thread making it can do nothing else in the meantime.: the thread stops until the data arrives. With non-blocking I/O, the call returns immediately, and the program asks the operating system to notify it when the data is ready, using mechanisms such as epoll on Linux.
Step 3: Two server designs
Thread per request. Each request gets a thread from a pool. The code is simple and reads top to bottom: query the database, wait, use the result. But every waiting thread still holds its memory (often around a megabyte of stack by default) and adds context switches. With 10,000 slow connections you need 10,000 threads, or the extra requests queue.
Event loop. A single thread runs a loop: take the next ready event, run its handler until the handler starts some I/O, then move on. Waiting requests cost only a small record of what to do next, often a few kilobytes. Node.js, Nginx and Redis work this way, and so does Python's asyncio. One loop per CPU core can hold tens of thousands of mostly idle connections.
Step 4: The catch with event loops
The loop only works if handlers return quickly. If one handler does 200 ms of CPU work, such as resizing an image or parsing a huge JSON file, every other request on that loop waits 200 ms too. Event-loop systems therefore move heavy CPU work to separate worker threads or processes, and use async versions of every I/O library.
Try it
Try this:
- Keep the defaults. The event loop serves about twice as many requests per second, because the 200 threads spend 99 percent of their time waiting, and it does so with a fraction of the memory.
- Raise the thread pool to 2,000. The gap narrows, but look at the memory line.
- Raise CPU per request to 20 ms. Now both designs are limited by the 4 cores and finish level: concurrency does not create CPU.
Going deeper
Little's Law in disguise
The thread pool's limit is a direct use of Little's LawAverage number of requests in a system equals arrival rate times average time each spends there: L = lambda x W.: with 200 threads and 101 ms per request, at most 200 / 0.101, about 1,980 requests per second, can be in progress. You will use it again in Latency vs throughput, and why percentiles matter .
Modern blends
The line is blurring. Go runs lightweight goroutines that look like blocking code but are scheduled onto a few OS threads, and Java's virtual threads do the same. You write simple sequential code and get event-loop efficiency. Many production servers also combine designs: one event loop per core, plus a worker pool for CPU-heavy tasks.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Thread per request | Simple sequential code, CPU-heavy work, moderate numbers of connections | Tens of thousands of mostly idle connections: memory and context switches pile up |
| Event loop | Huge numbers of I/O-bound connections such as proxies, chat and APIs | Any blocking or CPU-heavy code: it stalls every request on the loop |
| Lightweight threads (goroutines, virtual threads) | Simple code with high concurrency | Runtimes without them, or code that holds OS threads through native calls |
Common mistakes
- Calling a blocking library inside an event loop. One synchronous database call can freeze the whole server.
- Thinking more threads always mean more throughput. Past the number of cores, CPU-bound work only gets more switching.
- Confusing concurrency with parallelism. Concurrency is juggling many tasks; parallelism is literally running them at the same instant on several cores.
- Sharing data between threads without locks or other protection, causing rare, hard-to-reproduce bugs.
Check yourself
-
What do threads in the same process share?
Show answer
B. The process's memory Threads share the process's memory (each has its own stack), which makes sharing data easy and races possible.
-
A server's requests spend 1 ms on CPU and 100 ms waiting on a database. Which design holds the most concurrent connections per gigabyte of memory?
Show answer
B. Event loop Waiting requests cost the event loop only a small record each, while each blocked thread holds its stack.
-
A Node.js handler spends 300 ms computing a report. What happens to other users?
Show answer
B. Their requests wait until the computation finishes The event loop runs one handler at a time. CPU work blocks everything; move it to a worker.
-
A machine has 4 cores and every request needs 10 ms of CPU. What is the most it can serve, whatever the design?
Show answer
B. About 400 requests per second Each core does 100 requests per second of 10 ms CPU each, so 4 cores do about 400. Concurrency cannot create CPU.
Recap
- A process has its own memory; threads inside it share memory and are cheaper.
- Servers mostly wait on I/O, so concurrency is about using that waiting time.
- Thread per request is simple but pays memory and context switches for every waiting request.
- An event loop never waits, so it holds huge numbers of idle connections, but blocking code stalls it.
- Concurrency cannot exceed the CPU: when work is CPU-bound, add cores or machines.