Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Infrastructure Essentials

System Design HLD-17.04

Cloud cost engineering

Compute, storage and egress costs, reserved and spot capacity, right-sizing, and cost per request as a design metric next to latency and availability.

Intermediate 10 min Last reviewed

The big idea

In the cloud, every architectural decision has a price tag that arrives monthly. An extra replica, a cross-region call, a log level left at debug, an uncompressed image: each one is a line on the bill. Teams that only look at the total once a quarter are routinely surprised; teams that treat cost as a design metric, next to latency and availability, catch expensive choices while they are still cheap to change.

The most useful single number is cost per unitTotal infrastructure cost divided by the number of requests, users or transactions served, so efficiency can be compared over time and across services.: per request, per active user, per order. It shows whether the system is becoming more or less efficient as it grows, and it turns "the bill went up" into "this service's cost per request doubled after Tuesday's deploy".

Think of running a delivery business. You care less about the total fuel bill than about fuel cost per delivery; a route change that doubles it is a problem even in a quiet month.

How it works

Step 1: Where the money goes

  • Compute: instances, containers and functions, priced by size and time. Usually the biggest line, and the easiest to waste with idle or oversized machines.
  • Storage: priced per gigabyte per month, by tier: fast SSD volumes cost far more than object storage, which costs more than archive tiers.
  • Network egress: data leaving the provider, to the internet or to another region, is charged per gigabyte, while data coming in is usually free. It is the classic surprise.
  • Managed services: databases, queues, load balancers, often with per-request charges that grow with traffic.
cost-per-request.pyruns with python3 (standard library)
# Monthly cost of an API serving 50 million requests a day, and where the money goes.
# Illustrative round-number prices; check your provider's current price list.
REQ_PER_DAY = 50e6
RESP_KB = 200
DAYS = 30

prices = {
    "compute_instance_month": 140.0,    # one mid-size VM
    "egress_per_gb": 0.09,              # data out to the internet
    "cdn_per_gb": 0.02,                 # data out through a CDN at volume
    "storage_per_gb_month": 0.023,
}

def monthly(instances, cdn_hit_ratio, storage_gb=2_000):
    gb_out = REQ_PER_DAY * DAYS * RESP_KB / 1e6
    origin_gb = gb_out * (1 - cdn_hit_ratio)
    costs = {
        "compute": instances * prices["compute_instance_month"],
        "egress": origin_gb * prices["egress_per_gb"] + gb_out * cdn_hit_ratio * prices["cdn_per_gb"],
        "storage": storage_gb * prices["storage_per_gb_month"],
    }
    total = sum(costs.values())
    per_million = total / (REQ_PER_DAY * DAYS / 1e6)
    return costs, total, per_million

for label, inst, cdn in [("no CDN, 12 instances", 12, 0.0), ("CDN 90% hits, 12 instances", 12, 0.9),
                         ("CDN 90% hits, right-sized to 6", 6, 0.9)]:
    costs, total, pm = monthly(inst, cdn)
    parts = "  ".join(f"{k} ${v:>8,.0f}" for k, v in costs.items())
    print(f"{label:31} {parts}   total ${total:>8,.0f}   ${pm:.2f} per million requests")

Output

no CDN, 12 instances            compute $   1,680  egress $  27,000  storage $      46   total $  28,726   $19.15 per million requests
CDN 90% hits, 12 instances      compute $   1,680  egress $   8,100  storage $      46   total $   9,826   $6.55 per million requests
CDN 90% hits, right-sized to 6  compute $     840  egress $   8,100  storage $      46   total $   8,986   $5.99 per million requests

With these illustrative prices, an API serving 50 million 200 KB responses a day spends almost all of its money on egress, not compute. Putting a CDN in front with a 90 percent hit ratio cuts the bill by about two thirds; right-sizing the fleet then saves less, because compute was never the main cost. Measure before optimizing: the expensive part is often not the one you expect.

Monthly cost breakdown for the example API Bars for the example API: egress without a CDN costs 27,000 dollars a month, egress with a CDN at 90 percent hits 8,100 dollars, compute for 12 instances 1,680 dollars, and storage for 2 terabytes 46 dollars. monthly cost Egress (no CDN) $27,000 Egress (CDN, 90% hits) $8,100 Compute (12 instances) $1,680 Storage (2 TB) $46 Figure 1. For a response-heavy API, egress dwarfs compute and storage. Illustrative prices; check your provider's current price list.

Step 2: The main levers

  • Right-size: match instance sizes to measured use; many fleets run at 10 to 20 percent CPU.
  • Commit for steady load: reserved or committed-use capacity costs much less than on-demand in exchange for a one- to three-year commitment.
  • Spot or preemptible capacity for interruptible work such as batch jobs, CI and stateless workers, at a deep discount, with the risk of being reclaimed at short notice.
  • Autoscale to avoid paying for the peak all day ( Autoscaling ).
  • Cut egress: CDNs, compression, keeping chatty services in the same region and zone, and avoiding needless cross-region replication.
  • Tier storage: lifecycle rules move old data to cheaper classes and delete what is no longer needed ( Object storage, block storage and file systems ).

Step 3: Make it visible

Tag every resource with its team and service, report cost per unit on the same dashboards as latency and errors, and review the cost impact of designs before building them. A cost estimate belongs in every design document, just like a capacity estimate.

Try it

Which purchasing option fits each workload?

Show answers
  • The database primary that runs 24/7 for years: Reserved / committed. Steady, predictable, long-lived: commitment discounts fit.
  • Nightly batch jobs that can retry if a machine disappears: Spot / preemptible. Interruptible work tolerates reclaimed machines for a large discount.
  • Extra capacity for an unpredictable product launch next week: On-demand. Short-lived and uncertain: pay only for what you use, no commitment.
  • CI build runners: Spot / preemptible. A killed build can simply rerun.
  • The baseline web fleet that never drops below 10 instances: Reserved / committed. The always-on floor is steady load.
  • Autoscaled instances above the baseline during peaks: On-demand. Variable top-up capacity, often mixed with some spot.

Trade-offs

OptionWins whenLoses when
On-demand Variable or short-lived load; no commitment Price: the most expensive way to buy steady capacity
Reserved or committed Steady baseline load for a year or more Flexibility if the architecture or scale changes
Spot or preemptible Interruptible batch, CI and stateless workers Reliability: machines can be reclaimed at short notice

Common mistakes

  • Optimizing compute when egress or a managed service dominates the bill.
  • No tagging, so nobody knows which team or service is spending what.
  • Cross-region traffic between chatty services, paid for on every call.
  • Debug logging and unlimited retention quietly growing storage costs.

Check yourself

  1. Why track cost per request rather than only the total bill?

    Show answer

    B. It shows efficiency as traffic grows and pinpoints regressions A rising total can be healthy growth; a rising cost per request is a problem.

  2. Which cost is the classic surprise for media-heavy APIs?

    Show answer

    B. Network egress Data leaving the provider is charged per gigabyte and grows with every response.

  3. Which workload suits spot instances?

    Show answer

    B. Retryable batch jobs Interruptible work survives reclaimed machines.

  4. A fleet averages 12 percent CPU. What is the first lever to try?

    Show answer

    B. Right-size to smaller or fewer instances Most of that capacity is idle and being paid for.

Recap

  • Every design choice has a monthly price; treat cost as a design metric.
  • Compute, storage, egress and managed services are the main lines; egress is the usual surprise.
  • Right-size, commit for steady load, use spot for interruptible work, autoscale the rest.
  • Cut egress with CDNs, compression and keeping traffic in-region.
  • Tag everything and track cost per request next to latency and errors.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy