Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Reliability and Resilience

System Design HLD-12.08

Disaster recovery: RPO and RTO

How much data you can lose (RPO) and how long you can be down (RTO): backups, point-in-time recovery, pilot light, warm standby, multi-site and restore tests.

Intermediate 10 min Last reviewed

The big idea

Redundancy handles the failures you expect: a server, a disk, a zone. Disaster recovery is for the rest: a region-wide outage, a ransomware attack, an engineer running DELETE without a WHERE on production, a bug that silently corrupts data for a week. Replicas do not help with the last two, because they faithfully copy the deletion and the corruption within milliseconds.

Planning for disasters starts with two numbers the business must choose. The Recovery Point Objective (RPO)Recovery Point Objective: the maximum acceptable amount of data loss, measured as time before the failure, such as 5 minutes of writes. is how much data you can afford to lose, measured in time: "at most the last 5 minutes of writes". The Recovery Time Objective (RTO)Recovery Time Objective: the maximum acceptable time to restore service after a disaster. is how long you can afford to be down: "back within 1 hour". Smaller numbers cost more, and every strategy is a point on that curve.

Think of writing a long document. RPO is how often you press save: crash after saving every 5 minutes and you lose at most 5 minutes of work. RTO is how long it takes to get a working computer and reopen the file. A spare laptop ready on the desk shortens RTO; autosave shortens RPO.

How it works

Step 1: RPO and RTO on a timeline

RPO and RTO on a timeline around a disaster A timeline from left to right. The last recoverable point, a backup or replicated position, comes first. Then the disaster strikes. Then service is restored. The span from the last recoverable point to the disaster, glowing red above the line, is data lost and must fit within the RPO. The span from the disaster to restoration, glowing amber below the line, is downtime and must fit within the RTO. last backup or replica position DISASTER service restored RPO: data lost writes since last recoverable point RTO: time down detect, decide, restore, verify Figure 1. Everything written between the last recoverable point and the disaster is lost: that gap must stay within the RPO. The time from the disaster until service is back is downtime: it must stay within the RTO.

RTO includes everything, not just the restore: noticing the problem, deciding to invoke recovery, provisioning infrastructure, restoring data, warming caches, verifying, and switching traffic. In real incidents, the human steps often take longest.

The right targets come from what loss and downtime cost. A rough calculation with example numbers:

rpo.pyruns with python3 (standard library)
ORDERS_PER_MIN = 200          # business write rate
REVENUE_PER_MIN = 9_000       # dollars of sales per minute

strategies = [                # (name, RPO minutes, RTO minutes)
    ("nightly backup, restore by hand", 24 * 60, 8 * 60),
    ("backups + point-in-time recovery", 5, 2 * 60),
    ("pilot light in a second region", 1, 45),
    ("warm standby (scaled down)", 1, 10),
    ("active multi-site", 0, 1),
]
print(f"{'strategy':<34}{'orders lost':>12}{'sales missed while down':>26}")
for name, rpo, rto in strategies:
    print(f"{name:<34}{rpo * ORDERS_PER_MIN:>12,}{'$' + format(rto * REVENUE_PER_MIN, ','):>26}")

Output

strategy                           orders lost   sales missed while down
nightly backup, restore by hand        288,000                $4,320,000
backups + point-in-time recovery         1,000                $1,080,000
pilot light in a second region             200                  $405,000
warm standby (scaled down)                 200                   $90,000
active multi-site                            0                    $9,000

For a shop taking 200 orders a minute, nightly backups mean losing up to a day of orders, while a warm standby loses about a minute. That difference in cost per incident is what justifies paying for the more expensive strategy, or not.

Step 2: Backups done right

  • Full plus incremental backups, stored in a different account and region from production, so whoever or whatever destroys production cannot also destroy the backups. Immutable or write-once storage protects against ransomware.
  • Point-in-time recovery (PITR): keep a base backup plus a continuous archive of the database's write-ahead log, so you can restore to any moment, such as one second before the bad DELETE. This gives an RPO of minutes or less, and it is the only defense against logical corruption that replicas copy instantly.
  • Retention long enough to catch slow problems: corruption discovered after two weeks is unrecoverable with seven days of backups.

Step 3: Standby strategies

From cheapest and slowest to most expensive and fastest:

  • Backup and restore: rebuild everything in a new region from backups and infrastructure-as-code. RTO in hours, RPO from the backup interval.
  • Pilot light: data is continuously replicated to a second region, and core infrastructure is defined and ready, but application servers are off. On disaster, start and scale them. RTO in tens of minutes.
  • Warm standby: a scaled-down but fully working copy runs in the second region and receives replicated data. On disaster, scale it up and switch traffic. RTO in minutes.
  • Multi-site active-active: full capacity serves traffic in several regions all the time ( Multi-region architecture ). RTO near zero, RPO near zero for synchronously replicated data, at the highest cost.

Step 4: Test the recovery

A backup you have never restored is not a backup; it is a hope. Restores fail for mundane reasons: missing permissions, an expired encryption key, a forgotten configuration file, a restore that takes 14 hours instead of the assumed 2. Schedule regular restore tests that measure actual RPO and RTO, run full region-failover exercises, and keep the runbook current. Automated restore verification, such as restoring last night's backup into a scratch environment and running checks, catches problems early.

Going deeper

Replication is not backup

Replicas protect against hardware failure. They copy every change, good or bad, including deletions, corruption and ransomware encryption. You need both: replicas for availability and point-in-time backups, isolated from production credentials, for recovery from mistakes and attacks.

Dependencies set the real RTO

Your service can only be back when everything it needs is back: identity provider, DNS, secrets, container registry, payment provider. If your recovery region depends on a control plane in the failed region, the recovery fails. Map dependencies and give each critical one its own recovery plan.

Trade-offs

OptionWins whenLoses when
Backup and restore Internal and non-critical systems; lowest cost RTO in hours, RPO up to the backup interval
Point-in-time recovery Every production database; recovering from human error and corruption Restore time grows with data size and log volume
Pilot light Important systems that can tolerate tens of minutes of downtime Scale-up during the disaster can be slow or capacity-constrained
Warm standby Business-critical systems needing minutes of RTO Running a second environment continuously
Multi-site active-active Systems where any downtime is very expensive Highest cost and complexity

Common mistakes

  • Treating replicas as backups and having nothing to restore after a bad delete or corruption.
  • Keeping backups in the same account or region as production, where the same incident can destroy them.
  • Never testing restores, then discovering a broken process during a real disaster.
  • Choosing RPO and RTO without asking what loss and downtime actually cost the business.
  • Forgetting dependencies such as DNS, identity and secrets in the recovery plan.

In the wild

Cloud providers document these same tiers (backup and restore, pilot light, warm standby and multi-site active-active) as standard disaster recovery strategies, and managed databases commonly offer automated backups with point-in-time recovery within a retention window. Financial and healthcare organizations are often required by regulation to define RPO and RTO for critical systems and to test recovery regularly.

Check yourself

  1. Backups run every 6 hours and the database is destroyed 5 hours after the last one. How much data is lost?

    Show answer

    B. About 5 hours of writes Everything since the last recoverable point is gone; the worst case equals the backup interval.

  2. An engineer accidentally deletes a table. Synchronous replicas exist in two zones. Do they help?

    Show answer

    B. No, the delete was replicated too; you need point-in-time recovery from backups Replicas copy every change, including mistakes; PITR restores to a moment before the delete.

  3. Which strategy keeps a scaled-down but running copy of the system in a second region?

    Show answer

    C. Warm standby Warm standby runs everything at reduced size, ready to scale up and take traffic.

  4. Why must restores be tested regularly?

    Show answer

    B. Restores often fail or take far longer than assumed, which you only learn by doing them Measured restores reveal the real RPO and RTO and the broken steps before a disaster does.

Recap

  • RPO is how much data you can lose; RTO is how long you can be down.
  • Set them from the cost of loss and downtime, then pick a strategy to match.
  • Use isolated, immutable backups with point-in-time recovery; replicas are not backups.
  • Backup and restore, pilot light, warm standby and multi-site trade cost for faster recovery.
  • Test restores and failovers regularly and include every dependency in the plan.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy