- Learn
- High Level System Design
- Reliability and Resilience
HLD-M12
Reliability and Resilience
Keeping systems up when parts fail: timeouts and retries, circuit breakers, rate limiting, load shedding, redundancy, multi-region, cells, disaster recovery and chaos engineering.
Everything fails eventually: a dependency slows down, a disk dies, an availability zone loses power, a bad deploy ships, a customer sends ten times their usual traffic. Reliability is not about preventing every failure. It is about making sure one failure stays small instead of cascading into an outage.
This module works outward from a single remote call to the whole planet. You will learn how timeouts, retries with backoff and jitter, and circuit breakers stop slow dependencies from taking you down; how rate limiting and load shedding protect capacity; how redundancy across availability zones and regions survives losing a building or a whole region; how cells and shuffle sharding limit the blast radius of bad deploys and noisy customers; how RPO and RTO frame disaster recovery; and how chaos engineering tests all of it on purpose.
Lessons
- HLD-12.01 Timeouts, retries, backoff and jitter Every network call needs a timeout, or one slow dependency can tie up all your threads. Intermediate 12 min
- HLD-12.02 Circuit breakers and bulkheads A circuit breaker watches calls to a dependency. Intermediate 12 min
- HLD-12.03 Rate limiting algorithms Rate limiting protects services from overload and abuse by capping requests per client. Intermediate 15 min
- HLD-12.04 Load shedding and graceful degradation When demand exceeds capacity, it is better to serve some requests well than all requests badly. Advanced 10 min
- HLD-12.05 Redundancy, failover and multi-AZ Redundancy means running spare capacity so a failure does not become an outage. Intermediate 10 min
- HLD-12.06 Multi-region architecture Running in several regions cuts latency for global users, survives the loss of an entire region and helps meet data residency laws. Advanced 15 min
- HLD-12.07 Cell-based architecture and shuffle sharding Cell-based architecture splits a system into many independent copies, called cells, each serving a subset of customers, with a thin routing layer in front. Advanced 12 min
- HLD-12.08 Disaster recovery: RPO and RTO Recovery Point Objective is how much data you can afford to lose, measured in time; Recovery Time Objective is how long you can afford to be down. Intermediate 10 min
- HLD-12.09 Chaos engineering Chaos engineering deliberately injects failures, such as killed instances, added latency or blocked dependencies, to verify that a system behaves as expected under stress. Advanced 8 min