- Learn
- High Level System Design
- Scaling Data
HLD-M07
Scaling Data
When one database is not enough: replication and its lag, multi-leader and quorums, sharding and consistent hashing, hot partitions, denormalization and safe migrations.
Sooner or later one database server cannot hold all the data, serve all the reads or survive its own failure. The answers are to copy data to more machines (replication) and to split it across machines (partitioning). Both are simple to describe and full of subtle consequences: replicas that lag behind, failovers that lose writes, conflicts between leaders, shard keys that create hot spots, and migrations that must happen while the system stays online.
This module walks through all of it, from leader-follower replication to Dynamo-style quorums, sharding strategies, the consistent hashing ring at the heart of many distributed stores, the celebrity problem, CQRS, and zero-downtime schema changes.
Lessons
- HLD-07.01 Replication: leader-follower In leader-follower replication, one leader accepts writes and streams its change log to followers that apply it in order. Intermediate 12 min
- HLD-07.02 Replication lag and read consistency guarantees With asynchronous replicas, a user may write something and then read from a replica that has not caught up, so their own change seems to vanish. Advanced 12 min
- HLD-07.03 Multi-leader and leaderless replication Multi-leader replication lets several nodes accept writes, often one per region, which improves write latency and availability but creates conflicts that must be resolved by rules like last-write-wins, by merging data types (CRDTs) or by application logic. Advanced 14 min
- HLD-07.04 Partitioning and sharding Sharding splits one large dataset across many databases so each holds a manageable piece. Intermediate 15 min
- HLD-07.05 Consistent hashing With modulo hashing, changing the number of servers moves almost every key. Intermediate 15 min
- HLD-07.06 Hot partitions and the celebrity problem Even with good sharding, some keys are far hotter than others, such as a celebrity's profile or a viral post. Advanced 10 min
- HLD-07.07 Denormalization, materialized views and CQRS Denormalization stores data in the shape reads need, even if that duplicates it, so a page loads with one query instead of many joins. Intermediate 12 min
- HLD-07.08 Zero-downtime schema and data migrations Changing a schema or moving data without downtime follows expand and contract. Advanced 14 min