System Design HLD-02.03
SLIs, SLOs, SLAs and error budgets
Measure what users feel with SLIs, set SLO targets, keep SLAs looser, and use the error budget to decide when to ship and when to fix.
The big idea
"Make it reliable" is not a goal anyone can act on. How reliable? Measured how? What happens if we miss? Site reliability engineering answers with three precise terms. An SLIService level indicator: a measured number describing one aspect of service quality, such as the share of requests that succeed within 300 ms. is what you measure. An SLOService level objective: a target for an SLI over a time window, such as 99.9 percent of requests succeed within 300 ms over 30 days. is the target you aim for. An SLAService level agreement: a contract with customers that promises a level of service and states the penalty, often service credits, if it is missed. is the promise you make to customers, with consequences if you break it.
The clever part is what falls out of an SLO. If the target is 99.9 percent, then 0.1 percent of requests are allowed to fail. That allowance is the error budgetThe amount of unreliability an SLO permits in a window, such as 0.1 percent of requests over 30 days. Teams spend it on releases and experiments., and it turns an endless argument between "ship faster" and "be careful" into a simple rule: while there is budget left, ship; when it is spent, fix reliability first.
How it works
Step 1: Pick SLIs users would recognize
A good SLI measures something a user would complain about, as a ratio of good events to total events:
- Availability: successful requests divided by all valid requests. A 500 error counts against you; a 404 for a page that does not exist usually does not.
- Latency: the share of requests faster than a threshold, such as 300 ms. Written this way, latency becomes a ratio too.
- Freshness for pipelines: the share of data updated within the last N minutes.
- Correctness where possible: the share of responses with the right answer.
Measure as close to the user as you can. A server that is healthy while the load balancer in front of it is broken still produces angry users.
Step 2: Set an SLO, below 100 percent on purpose
An SLO pairs an SLI with a target and a window: "99.9 percent of checkout requests succeed and complete within 500 ms, measured over a rolling 30 days." Choose the target from what users actually need, not from what the system happens to do today. 100 percent is never the right target. Users cannot tell 99.99 from 100 because their own phone and network fail more often than that, and every extra nine slows down change.
Step 3: The SLA is looser than the SLO
The SLA is a business contract, so it should be set below the internal SLO. If the SLO is 99.9 percent, the SLA might promise 99.5 percent. Then the team gets an internal alarm long before customers are owed credits.
Step 4: Spend the error budget
At 99.9 percent over 30 days with 10 million requests, the budget is 10,000 failed requests, or about 43 minutes of full outage. The team tracks how fast it burns:
The policy is agreed in advance: while budget remains, the team can ship features, run experiments and take risks. When the budget is exhausted, releases slow down or stop, except for reliability fixes, until it recovers. Because both product and engineering agreed to the SLO, the decision is no longer a personal argument.
Step 5: Alert on burn rate
Instead of paging someone whenever errors appear, alert when the budget burns too fast. A burn rateHow fast the error budget is being consumed relative to a steady pace that would exactly use it up by the end of the window. A burn rate of 1 is on track; 14 means it would be gone in about 2 days of a 30-day window. of 1 uses the budget exactly over the window. A burn rate of 14 sustained for an hour means a serious incident and deserves a page; a burn rate of 2 for a day deserves a ticket. This catches real problems fast while ignoring noise.
Going deeper
Too reliable is also a problem
If a service is far more reliable than its SLO, other teams start depending on that and build systems that fail when it eventually has an outage. Google's SRE book describes deliberately taking a service down when it has stayed well above its target, so that hidden dependencies surface in a controlled way. Reliability you do not promise becomes reliability people assume.
SLOs for dependencies
Your SLO cannot exceed what your critical dependencies allow, because availability in series multiplies. If you depend on three services at 99.9 percent, a 99.99 percent SLO is fiction unless you add caching, fallbacks or redundancy. This is a common way SLOs drive architecture decisions.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Tight SLO (99.99%+) | Payments, authentication and core infrastructure that everything else depends on | Feature velocity and cost: tiny budgets mean slow, careful releases |
| Moderate SLO (99.5 to 99.9%) | Most user-facing features: room to ship weekly or daily | Systems where a few hours of downtime a year is unacceptable |
| No SLO | Prototypes and internal tools with forgiving users | Anything with real users: reliability arguments become opinions |
Common mistakes
- Measuring what is easy (CPU, server uptime) instead of what users feel (successful, fast requests).
- Setting the SLA equal to the SLO, leaving no warning before contractual penalties.
- Setting 100 percent targets, which make every change feel dangerous and every incident a crisis.
- Defining an error budget but never acting on it. The policy only works if a spent budget really changes priorities.
In the wild
This chapter defines SLIs, SLOs and SLAs the way this lesson uses them, and gives a sample objective: 99 percent of Get RPC calls will complete in less than 100 ms. It argues for percentiles over averages, noting a system where a typical request takes about 50 ms while 5 percent of requests are 20 times slower. It also tells the story of Chubby, Google's lock service, which was so reliable that other teams built in hidden dependencies on it; when real failures had not pushed its availability below the target, the team synthesized a controlled outage, taking it down on purpose to flush those dependencies out.
Check yourself
-
Which is the best SLI for a web API?
Show answer
B. Share of valid requests that succeed within 300 ms It measures what users experience, as a ratio of good events to total events.
-
An SLO is 99.9 percent over 30 days with 50 million requests. How many failed requests does the error budget allow?
Show answer
C. 50,000 0.1 percent of 50 million is 50,000.
-
Why should an SLA usually be looser than the SLO?
Show answer
B. So the team gets an internal warning before customers are owed penalties Missing the internal SLO is a signal to act; missing the SLA costs money and trust.
-
The error budget is spent with 10 days left in the window. What does a typical policy say?
Show answer
B. Pause risky launches and prioritize reliability work A spent budget shifts priority to reliability until the window recovers.
Recap
- SLI: what you measure; SLO: the target; SLA: the contract with penalties.
- Good SLIs are ratios of good events to all events, measured close to users.
- Never target 100 percent; set the SLA looser than the SLO.
- Error budget = 100 percent minus the SLO: ship while it lasts, fix when it is spent.
- Alert on burn rate, not on every error.
Next up
- Crack the system design round learning path