Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Caching

System Design HLD-05.01

Why caching works

Temporal and spatial locality, the 80/20 skew of real traffic, the hit ratio formula for average latency, and the layers where caches live.

Beginner 8 min Last reviewed

The big idea

A cacheA fast store, usually in memory, that keeps copies of data so repeated reads do not have to go back to a slower source. keeps a copy of data somewhere faster than where it really lives, so the next request for it does not pay the full price. That only helps if the same data is asked for again, and the remarkable fact about real systems is that it almost always is. Access is heavily skewed: a few products, a few posts, a few users' profiles receive most of the traffic. Keep those few in memory and most requests never touch the slow source.

The arithmetic is simple and worth knowing by heart. If a cache answers in 1 ms, the database in 20 ms, and 90 percent of reads hit the cache, the average read takes about 2.9 ms instead of 20, and the database sees only one read in ten. That is why nearly every system you will design has caches in several places at once.

Think of a kitchen. The salt and pepper sit on the counter, the flour in the cupboard, the bulk rice in the cellar. You do not walk to the cellar each time you season a dish, because you use the salt constantly. Keeping frequently used things close is all caching is.

How it works

Step 1: Locality

Caches work because of localityThe tendency of programs and users to access the same data, or nearby data, repeatedly over short periods.:

  • Temporal locality: something used recently is likely to be used again soon. A viral post read a thousand times this minute will be read a thousand more next minute.
  • Spatial locality: things near something used are likely to be used next. Reading row 100 suggests row 101; reading one product page suggests its images. Hardware caches and databases read whole blocks for this reason.

Step 2: The skew

Popularity is extremely uneven. A rough rule of thumb is that 20 percent of items get 80 percent of the requests, and for viral content the skew is far stronger. This is why a cache that holds a small fraction of the data can still serve most requests: it only needs to hold the hot set, the items in heavy use right now.

Step 3: The hit ratio formula

where is the hit ratioThe fraction of requests answered by the cache without going to the source.. With ms, ms and : ms. Notice which term dominates: the misses. Raising the hit ratio from 90 to 99 percent cuts the average from 2.9 ms to about 1.2 ms and cuts database reads tenfold.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Start at 90 percent and read the formula in the output line. Most of the 2.9 ms comes from the 10 percent of misses.
  2. Slide to 50 percent. The cache still helps, but the database still carries half the load.
  3. Slide to 99 percent. The database sees 200 reads per second instead of 20,000: often the difference between one database and a cluster.

Step 4: Where caches live

The layers of caching from browser to database A request passes through a stack of caches: the browser cache, the CDN edge, the application's in-memory cache, a shared cache such as Redis, and the database's own buffer pool, before reaching disk. Each layer glows when it can answer the request. Browser cache 0 ms CDN edge ~10 ms App memory ~0.1 ms Redis ~1 ms DB buffer pool ~1-5 ms Disk slowest Figure 1. Each arrow is a miss passing the request down a layer. Every layer is slower but larger than the one before, and the best request is one stopped at the first layer that has the answer.
  • Client: browser and app caches, controlled by HTTP headers.
  • CDN: shared edge caches for public content ( Content Delivery Networks ).
  • Application: an in-process map in each server, the fastest but per-server and lost on restart.
  • Distributed cache: Redis or Memcached shared by all servers ( Distributed caches: Redis and Memcached ).
  • Database: its buffer pool keeps hot pages in memory, so even "database reads" are often memory reads.

Going deeper

The price of a cache

Every cache is a second copy of data, and second copies can be wrong. A cache adds a component to run, memory to pay for, and above all the question of what happens when the source changes, which is the subject of Cache invalidation and consistency . A cache can also hide a capacity problem: if it restarts empty, all traffic falls on a database that can no longer handle it. Caching helps most when data is read far more often than it changes and slightly old data is acceptable.

Trade-offs

OptionWins whenLoses when
In-process cache The hottest, smallest data: config, reference data; fastest possible reads Consistency across servers and data that must survive restarts
Shared distributed cache Data read by many servers: sessions, product pages, computed results An extra network hop and a component to keep available
No cache Write-heavy or rarely repeated data, or data that must always be exact Read latency and database load at scale

Common mistakes

  • Adding a cache before measuring whether reads repeat. A cache with a 10 percent hit ratio mostly adds latency and complexity.
  • Caching data that changes on every request, so entries are always stale or always missing.
  • Planning database capacity assuming the cache is always warm.
  • Forgetting the caches you already have, such as the database buffer pool, before adding another layer.

Check yourself

  1. Cache 2 ms, database 30 ms, hit ratio 80 percent. What is the average read latency?

    Show answer

    B. About 7.6 ms 0.8 x 2 + 0.2 x 30 = 1.6 + 6 = 7.6 ms.

  2. Why can a cache holding 5 percent of the data serve most requests?

    Show answer

    B. Access is skewed: a small hot set receives most requests Popularity is highly uneven, so the hot set is small.

  3. Raising the hit ratio from 95 to 99 percent changes database load how?

    Show answer

    B. Cuts it by a factor of five Misses drop from 5 percent to 1 percent of reads.

  4. What is temporal locality?

    Show answer

    B. Recently used data is likely to be used again soon Recent use predicts near-future use; that is what most caches bet on.

Recap

  • A cache keeps a copy of data somewhere faster so repeated reads skip the slow source.
  • It works because of locality and the heavy skew of real access patterns.
  • Average latency = hit ratio x cache time + miss ratio x source time; the misses dominate.
  • Caches live in the browser, the CDN, app memory, shared caches and the database itself.
  • Every cache is a second copy that can go stale: cache read-heavy data that tolerates it.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up