System Design HLD-17.04
Cloud cost engineering
Compute, storage and egress costs, reserved and spot capacity, right-sizing, and cost per request as a design metric next to latency and availability.
The big idea
In the cloud, every architectural decision has a price tag that arrives monthly. An extra replica, a cross-region call, a log level left at debug, an uncompressed image: each one is a line on the bill. Teams that only look at the total once a quarter are routinely surprised; teams that treat cost as a design metric, next to latency and availability, catch expensive choices while they are still cheap to change.
The most useful single number is cost per unitTotal infrastructure cost divided by the number of requests, users or transactions served, so efficiency can be compared over time and across services.: per request, per active user, per order. It shows whether the system is becoming more or less efficient as it grows, and it turns "the bill went up" into "this service's cost per request doubled after Tuesday's deploy".
Think of running a delivery business. You care less about the total fuel bill than about fuel cost per delivery; a route change that doubles it is a problem even in a quiet month.
How it works
Step 1: Where the money goes
- Compute: instances, containers and functions, priced by size and time. Usually the biggest line, and the easiest to waste with idle or oversized machines.
- Storage: priced per gigabyte per month, by tier: fast SSD volumes cost far more than object storage, which costs more than archive tiers.
- Network egress: data leaving the provider, to the internet or to another region, is charged per gigabyte, while data coming in is usually free. It is the classic surprise.
- Managed services: databases, queues, load balancers, often with per-request charges that grow with traffic.
# Monthly cost of an API serving 50 million requests a day, and where the money goes.
# Illustrative round-number prices; check your provider's current price list.
REQ_PER_DAY = 50e6
RESP_KB = 200
DAYS = 30
prices = {
"compute_instance_month": 140.0, # one mid-size VM
"egress_per_gb": 0.09, # data out to the internet
"cdn_per_gb": 0.02, # data out through a CDN at volume
"storage_per_gb_month": 0.023,
}
def monthly(instances, cdn_hit_ratio, storage_gb=2_000):
gb_out = REQ_PER_DAY * DAYS * RESP_KB / 1e6
origin_gb = gb_out * (1 - cdn_hit_ratio)
costs = {
"compute": instances * prices["compute_instance_month"],
"egress": origin_gb * prices["egress_per_gb"] + gb_out * cdn_hit_ratio * prices["cdn_per_gb"],
"storage": storage_gb * prices["storage_per_gb_month"],
}
total = sum(costs.values())
per_million = total / (REQ_PER_DAY * DAYS / 1e6)
return costs, total, per_million
for label, inst, cdn in [("no CDN, 12 instances", 12, 0.0), ("CDN 90% hits, 12 instances", 12, 0.9),
("CDN 90% hits, right-sized to 6", 6, 0.9)]:
costs, total, pm = monthly(inst, cdn)
parts = " ".join(f"{k} ${v:>8,.0f}" for k, v in costs.items())
print(f"{label:31} {parts} total ${total:>8,.0f} ${pm:.2f} per million requests")Output
no CDN, 12 instances compute $ 1,680 egress $ 27,000 storage $ 46 total $ 28,726 $19.15 per million requests
CDN 90% hits, 12 instances compute $ 1,680 egress $ 8,100 storage $ 46 total $ 9,826 $6.55 per million requests
CDN 90% hits, right-sized to 6 compute $ 840 egress $ 8,100 storage $ 46 total $ 8,986 $5.99 per million requests With these illustrative prices, an API serving 50 million 200 KB responses a day spends almost all of its money on egress, not compute. Putting a CDN in front with a 90 percent hit ratio cuts the bill by about two thirds; right-sizing the fleet then saves less, because compute was never the main cost. Measure before optimizing: the expensive part is often not the one you expect.
Step 2: The main levers
- Right-size: match instance sizes to measured use; many fleets run at 10 to 20 percent CPU.
- Commit for steady load: reserved or committed-use capacity costs much less than on-demand in exchange for a one- to three-year commitment.
- Spot or preemptible capacity for interruptible work such as batch jobs, CI and stateless workers, at a deep discount, with the risk of being reclaimed at short notice.
- Autoscale to avoid paying for the peak all day ( Autoscaling ).
- Cut egress: CDNs, compression, keeping chatty services in the same region and zone, and avoiding needless cross-region replication.
- Tier storage: lifecycle rules move old data to cheaper classes and delete what is no longer needed ( Object storage, block storage and file systems ).
Step 3: Make it visible
Tag every resource with its team and service, report cost per unit on the same dashboards as latency and errors, and review the cost impact of designs before building them. A cost estimate belongs in every design document, just like a capacity estimate.
Try it
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| On-demand | Variable or short-lived load; no commitment | Price: the most expensive way to buy steady capacity |
| Reserved or committed | Steady baseline load for a year or more | Flexibility if the architecture or scale changes |
| Spot or preemptible | Interruptible batch, CI and stateless workers | Reliability: machines can be reclaimed at short notice |
Common mistakes
- Optimizing compute when egress or a managed service dominates the bill.
- No tagging, so nobody knows which team or service is spending what.
- Cross-region traffic between chatty services, paid for on every call.
- Debug logging and unlimited retention quietly growing storage costs.
Check yourself
-
Why track cost per request rather than only the total bill?
Show answer
B. It shows efficiency as traffic grows and pinpoints regressions A rising total can be healthy growth; a rising cost per request is a problem.
-
Which cost is the classic surprise for media-heavy APIs?
Show answer
B. Network egress Data leaving the provider is charged per gigabyte and grows with every response.
-
Which workload suits spot instances?
Show answer
B. Retryable batch jobs Interruptible work survives reclaimed machines.
-
A fleet averages 12 percent CPU. What is the first lever to try?
Show answer
B. Right-size to smaller or fewer instances Most of that capacity is idle and being paid for.
Recap
- Every design choice has a monthly price; treat cost as a design metric.
- Compute, storage, egress and managed services are the main lines; egress is the usual surprise.
- Right-size, commit for steady load, use spot for interruptible work, autoscale the rest.
- Cut egress with CDNs, compression and keeping traffic in-region.
- Tag everything and track cost per request next to latency and errors.