Latency and cost budgets

The big idea

Your assistant works. Then the numbers arrive: answers take 9 seconds, and at 100,000 requests a day the model bill is tens of thousands of dollars a month. Nobody planned for either, so nobody knows which part to fix.

A budgetA target limit for one resource, such as seconds of latency or dollars per request, split across the parts of the system so each part has its own share. fixes that before it happens. Like a household budget, you decide the total first ("answers start within 2 seconds, cost under 1 cent each"), split it across the parts of a request, measure what each part really uses, and cut where the overspend is. The arithmetic is simple. The discipline of doing it for every feature is what separates a demo from a product.

LLM features make budgeting unusually important for two reasons. Latency is measured in seconds, not milliseconds, because generating text is slow. And cost scales with every token, so a prompt that quietly doubles in size doubles the bill.

How it works

Step 1: break the request into parts

List every step on the request path, as in Figure 1, and get its real numbers from traces ( LLM observability and tracing ), not guesses. For each model call, record input tokens, output tokens, time to first token and output speed. Two numbers matter for latency: when the first words appear (everything before the main model's first token) and when the answer is complete. With streaming, users judge mostly the first.

Use percentiles, not averages. A p50 of 2 seconds with a p95 of 14 seconds means one user in twenty waits 14 seconds, often the ones with long conversations or slow tools ( Latency vs throughput, and why percentiles matter ).

Step 2: price each part

The cost of one model call is

where $T$ are token counts and $p$ the price per token. Output tokens usually cost several times more than input tokens, but prompts are usually much longer than answers, so both matter. Worked example with illustrative prices of 3 dollars per million input tokens and 15 per million output: 4,000 input tokens cost 0.012 dollars, 300 output tokens cost 0.0045 dollars, so 0.0165 dollars per request. At 1 request per second that is $0.0165\times 86,400\approx 1,426$ dollars a day, about 43,000 dollars a month. Small per-request numbers become large bills.

Remember every call: query rewrites, classifiers, judges, retries and each step of an agent loop each resend context and add cost.

Step 3: cut latency

• Parallelize independent work. A safety check, a query rewrite and retrieval for two sources do not depend on each other; run them at the same time and the request waits for the slowest, not the sum. Speculatively start the main call and cancel it if the safety check fails.
• Stream so the answer's length stops blocking the first words ( UX patterns for AI ).
• Shorten the prompt. Time to first token grows with input length. Retrieve fewer, better passages and trim history.
• Cap and shape the output. Generation time is output tokens divided by speed. "Answer in at most 3 sentences" and a max-tokens limit cut both time and cost.
• Use a smaller, faster model for steps that do not need the big one.

Step 4: cut cost

• Cache. Prompt cachingReusing the provider's stored computation for a prompt prefix that is identical to a recent request, billed at a large discount and with a faster first token. bills a repeated prefix (system prompt, tool definitions, a long document) at a fraction of the input price ( Prompt caching and cost control ). Keep stable content at the start of the prompt so it can be cached. Separately, an application cache can return a stored answer for repeated identical questions.
• Route to smaller models. A cheap classifier or rules send easy requests to a small model and hard ones to the big model. Small models often cost several times less per token, so routing most traffic to them cuts the bill sharply. Measure quality per route with evals ( Routing, cascades and cost optimization ).
• Batch what can wait. Offline jobs such as nightly summaries or classification of a backlog can use batch APIs at a discount.
• Limit the loop. Cap agent iterations and retries, the hidden multipliers of cost ( The agent loop (ReAct) ).

Try it

1. With the defaults, what does the feature cost per month at 1 request per second? Now move the traffic slider to 10 requests per second.
2. Set the safety check to "before the call", then to "in parallel". How much latency does running it in parallel save? Did the cost change?
3. Raise sequential model calls to 3, as in a small agent loop. What happened to cost, and why does input grow faster than output? Then set 75 percent of the input to come from cache.

The same levers applied one after another, with illustrative prices for a big and a small model:

# Cost (dollars) and latency of one feature under four designs. Prices and speeds are illustrative.
MODELS = {  # dollars per 1M input, per 1M output, time to first token (s), output tokens/s
    "big":   {"in": 3.00, "out": 15.00, "ttft": 0.8, "tps": 50},
    "small": {"in": 0.25, "out": 1.25,  "ttft": 0.3, "tps": 150},
}
CACHE_READ = 0.1          # cached input billed at 0.1x the input price
REQUESTS_PER_DAY = 100_000

def call(model, tokens_in, tokens_out, cached=0.0):
    m = MODELS[model]
    cost = (tokens_in * (1 - cached) * m["in"] + tokens_in * cached * m["in"] * CACHE_READ
            + tokens_out * m["out"]) / 1e6
    ttft = m["ttft"] * (1 - 0.5 * cached)       # assumption: cached input cuts TTFT
    return cost, ttft + tokens_out / m["tps"]

def design(name, share_small, cached, max_out):
    router = call("small", 300, 5)              # a tiny classifier picks the model
    small = call("small", 4000, max_out, cached)
    big = call("big", 4000, max_out, cached)
    cost = router[0] + share_small * small[0] + (1 - share_small) * big[0]
    show(name, cost, router[1] + small[1], router[1] + big[1])

def show(name, cost, fast, slow):
    print(f"{name:<28} {cost * 1000:5.2f} per 1k req {cost * REQUESTS_PER_DAY * 30:7,.0f} per month"
          f"  latency {fast:.1f} to {slow:.1f} s")

base_cost, base_lat = call("big", 4000, 400)
show("baseline: big model only", base_cost, base_lat, base_lat)
design("+ route 70% to small model", 0.7, 0.0, 400)
design("+ cache 75% of the input", 0.7, 0.75, 400)
design("+ cap answers at 200 tokens", 0.7, 0.75, 200)

baseline: big model only     18.00 per 1k req  54,000 per month  latency 8.8 to 8.8 s
+ route 70% to small model    6.53 per 1k req  19,594 per month  latency 3.3 to 9.1 s
+ cache 75% of the input      3.63 per 1k req  10,886 per month  latency 3.2 to 8.8 s
+ cap answers at 200 tokens   2.55 per 1k req   7,661 per month  latency 1.9 to 4.8 s

Routing 70 percent of traffic to the small model cuts the bill by almost two thirds, caching takes it lower, and capping answers at 200 tokens both saves money and halves the slowest path. Each lever is cheap to try, but each one can hurt quality, which is why every change goes through evals before it ships ( LLMOps: shipping and iterating ).

Going deeper

Budgets per feature, not per company

Set the budget where decisions are made: per feature and per request type. A support answer might get 2 seconds to first words and 1 cent; a nightly report can take a minute and 50 cents. Tag every model call with the feature name in traces, so dashboards show spend and latency per feature, and alert when a feature drifts beyond its budget.

Cost per successful outcome

The cheapest request is not always the cheapest outcome. If a small model resolves 60 percent of support conversations and the big model 80 percent, and each failure costs a human agent's time, the big model may be cheaper per resolved ticket. Budget on the business unit (cost per resolved ticket, per accepted code suggestion), not only on tokens.

Capacity, not just price

Providers enforce rate limits in requests and tokens per minute. Peak traffic, not average, decides whether you hit them, and long prompts eat token limits fast. Plan for peak load, queue or shed low-priority traffic when near the limit ( Load shedding and graceful degradation ), and keep a fallback model or provider behind your gateway.

Trade-offs

Common mistakes

• Budgeting from averages, so the slowest 5 percent of requests, usually the most complex, blow past every target.
• Counting only the main model call and forgetting rewrites, judges, retries and agent loops.
• Putting the user's message or a timestamp at the start of the prompt, which breaks prompt caching for everything after it.
• Running independent checks one after another instead of in parallel.
• Switching to a cheaper model to save money without an eval to show what quality was lost.

In the wild

Check yourself

Recap