System Design HLD-12.08
Disaster recovery: RPO and RTO
How much data you can lose (RPO) and how long you can be down (RTO): backups, point-in-time recovery, pilot light, warm standby, multi-site and restore tests.
The big idea
Redundancy handles the failures you expect: a server, a disk, a zone. Disaster recovery is for the rest: a region-wide outage, a ransomware attack, an engineer running DELETE without a WHERE on production, a bug that silently corrupts data for a week. Replicas do not help with the last two, because they faithfully copy the deletion and the corruption within milliseconds.
Planning for disasters starts with two numbers the business must choose. The Recovery Point Objective (RPO)Recovery Point Objective: the maximum acceptable amount of data loss, measured as time before the failure, such as 5 minutes of writes. is how much data you can afford to lose, measured in time: "at most the last 5 minutes of writes". The Recovery Time Objective (RTO)Recovery Time Objective: the maximum acceptable time to restore service after a disaster. is how long you can afford to be down: "back within 1 hour". Smaller numbers cost more, and every strategy is a point on that curve.
Think of writing a long document. RPO is how often you press save: crash after saving every 5 minutes and you lose at most 5 minutes of work. RTO is how long it takes to get a working computer and reopen the file. A spare laptop ready on the desk shortens RTO; autosave shortens RPO.
How it works
Step 1: RPO and RTO on a timeline
RTO includes everything, not just the restore: noticing the problem, deciding to invoke recovery, provisioning infrastructure, restoring data, warming caches, verifying, and switching traffic. In real incidents, the human steps often take longest.
The right targets come from what loss and downtime cost. A rough calculation with example numbers:
ORDERS_PER_MIN = 200 # business write rate
REVENUE_PER_MIN = 9_000 # dollars of sales per minute
strategies = [ # (name, RPO minutes, RTO minutes)
("nightly backup, restore by hand", 24 * 60, 8 * 60),
("backups + point-in-time recovery", 5, 2 * 60),
("pilot light in a second region", 1, 45),
("warm standby (scaled down)", 1, 10),
("active multi-site", 0, 1),
]
print(f"{'strategy':<34}{'orders lost':>12}{'sales missed while down':>26}")
for name, rpo, rto in strategies:
print(f"{name:<34}{rpo * ORDERS_PER_MIN:>12,}{'$' + format(rto * REVENUE_PER_MIN, ','):>26}")Output
strategy orders lost sales missed while down
nightly backup, restore by hand 288,000 $4,320,000
backups + point-in-time recovery 1,000 $1,080,000
pilot light in a second region 200 $405,000
warm standby (scaled down) 200 $90,000
active multi-site 0 $9,000 For a shop taking 200 orders a minute, nightly backups mean losing up to a day of orders, while a warm standby loses about a minute. That difference in cost per incident is what justifies paying for the more expensive strategy, or not.
Step 2: Backups done right
- Full plus incremental backups, stored in a different account and region from production, so whoever or whatever destroys production cannot also destroy the backups. Immutable or write-once storage protects against ransomware.
- Point-in-time recovery (PITR): keep a base backup plus a continuous archive of the database's write-ahead log, so you can restore to any moment, such as one second before the bad
DELETE. This gives an RPO of minutes or less, and it is the only defense against logical corruption that replicas copy instantly. - Retention long enough to catch slow problems: corruption discovered after two weeks is unrecoverable with seven days of backups.
Step 3: Standby strategies
From cheapest and slowest to most expensive and fastest:
- Backup and restore: rebuild everything in a new region from backups and infrastructure-as-code. RTO in hours, RPO from the backup interval.
- Pilot light: data is continuously replicated to a second region, and core infrastructure is defined and ready, but application servers are off. On disaster, start and scale them. RTO in tens of minutes.
- Warm standby: a scaled-down but fully working copy runs in the second region and receives replicated data. On disaster, scale it up and switch traffic. RTO in minutes.
- Multi-site active-active: full capacity serves traffic in several regions all the time ( Multi-region architecture ). RTO near zero, RPO near zero for synchronously replicated data, at the highest cost.
Step 4: Test the recovery
A backup you have never restored is not a backup; it is a hope. Restores fail for mundane reasons: missing permissions, an expired encryption key, a forgotten configuration file, a restore that takes 14 hours instead of the assumed 2. Schedule regular restore tests that measure actual RPO and RTO, run full region-failover exercises, and keep the runbook current. Automated restore verification, such as restoring last night's backup into a scratch environment and running checks, catches problems early.
Going deeper
Replication is not backup
Replicas protect against hardware failure. They copy every change, good or bad, including deletions, corruption and ransomware encryption. You need both: replicas for availability and point-in-time backups, isolated from production credentials, for recovery from mistakes and attacks.
Dependencies set the real RTO
Your service can only be back when everything it needs is back: identity provider, DNS, secrets, container registry, payment provider. If your recovery region depends on a control plane in the failed region, the recovery fails. Map dependencies and give each critical one its own recovery plan.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Backup and restore | Internal and non-critical systems; lowest cost | RTO in hours, RPO up to the backup interval |
| Point-in-time recovery | Every production database; recovering from human error and corruption | Restore time grows with data size and log volume |
| Pilot light | Important systems that can tolerate tens of minutes of downtime | Scale-up during the disaster can be slow or capacity-constrained |
| Warm standby | Business-critical systems needing minutes of RTO | Running a second environment continuously |
| Multi-site active-active | Systems where any downtime is very expensive | Highest cost and complexity |
Common mistakes
- Treating replicas as backups and having nothing to restore after a bad delete or corruption.
- Keeping backups in the same account or region as production, where the same incident can destroy them.
- Never testing restores, then discovering a broken process during a real disaster.
- Choosing RPO and RTO without asking what loss and downtime actually cost the business.
- Forgetting dependencies such as DNS, identity and secrets in the recovery plan.
In the wild
Cloud providers document these same tiers (backup and restore, pilot light, warm standby and multi-site active-active) as standard disaster recovery strategies, and managed databases commonly offer automated backups with point-in-time recovery within a retention window. Financial and healthcare organizations are often required by regulation to define RPO and RTO for critical systems and to test recovery regularly.
Check yourself
-
Backups run every 6 hours and the database is destroyed 5 hours after the last one. How much data is lost?
Show answer
B. About 5 hours of writes Everything since the last recoverable point is gone; the worst case equals the backup interval.
-
An engineer accidentally deletes a table. Synchronous replicas exist in two zones. Do they help?
Show answer
B. No, the delete was replicated too; you need point-in-time recovery from backups Replicas copy every change, including mistakes; PITR restores to a moment before the delete.
-
Which strategy keeps a scaled-down but running copy of the system in a second region?
Show answer
C. Warm standby Warm standby runs everything at reduced size, ready to scale up and take traffic.
-
Why must restores be tested regularly?
Show answer
B. Restores often fail or take far longer than assumed, which you only learn by doing them Measured restores reveal the real RPO and RTO and the broken steps before a disaster does.
Recap
- RPO is how much data you can lose; RTO is how long you can be down.
- Set them from the cost of loss and downtime, then pick a strategy to match.
- Use isolated, immutable backups with point-in-time recovery; replicas are not backups.
- Backup and restore, pilot light, warm standby and multi-site trade cost for faster recovery.
- Test restores and failovers regularly and include every dependency in the plan.