Routing, cascades and cost optimization

The big idea

Your support assistant sends every message to the largest model you can buy. Most messages are "where is my order" or "how do I reset my password", which a model a tenth of the price answers just as well. A few are tangled billing disputes that really need the big model. Paying top price for every request is like sending every patient straight to the surgeon, including the ones who need a plaster.

Hospitals use triage: a nurse sees everyone first, handles what they can, and sends the hard cases on. LLM systems do the same in three ways. A routerA component that looks at a request before any answer is generated and picks which model should handle it, for example a small model for easy requests and a large one for hard requests. picks a model up front. A cascadeA setup in which a cheap model answers first and the request is escalated to a stronger, more expensive model only when a check says the cheap answer is not good enough. lets a cheap model try first and escalates when it is unsure. And caching and batching avoid paying full price for work that is repeated or not urgent.

Prices for the same request can differ by up to two orders of magnitude between models, so getting this right often matters more for the bill than any serving optimization.

How it works

Routing: pick a model before answering

A router looks at the request and chooses a model. It can be as simple as rules (requests from the "billing dispute" form go to the large model) or a small classifier trained to predict whether the cheap model will do well enough. Routing adds almost no latency, because only one model answers. Its weakness is that it must judge difficulty before seeing any answer, and a request that looks easy can still be hard.

Training a router needs labels saying which requests the small model handles well. You can get them by running both models on a sample of real traffic and grading the outputs with your eval tooling ( LLM-as-judge ).

Cascades: small model first, escalate when needed

A cascade lets the small model try, then checks its answer. The check is the whole game. Good checks are:

• Hard validators: the output must parse as JSON, the SQL must run, the code must pass its tests, the cited passage must contain the quoted text.
• Confidence signals: the model's own token probabilities, agreement between a few samples ( Reasoning models and test-time compute ), or an explicit "I am not sure" path.
• A small judge: a cheap grader model trained or prompted to spot the failure types you care about.

Escalated requests pay twice: the small call and the large one. So a cascade saves money only when most requests pass the check. If 20 percent escalate and the large model costs 10 times the small one, the average cost per request is $1+0.2\times 10=3$ small-model units, against 10 for always using the large model: a 70 percent saving. If half escalate, the cost is $1+0.5\times 10=6$ units and the saving drops to 40 percent, with added latency on every escalation.

A cascade can never be more accurate than its check. A check that waves through confident wrong answers turns cost savings into quality losses, which is why you measure the whole cascade on your eval set, not just each model.

Caching: do not pay twice for the same answer

• Exact caching: the same input returns the stored output. Safe and simple, and useful for repeated questions, retries and deterministic tool calls.
• Prompt caching at the provider: reuse of the KV cache for a shared prompt prefix, which lowers the price and latency of input tokens without reusing answers ( Prompt caching and cost control ).
• Semantic cachingReturning a stored answer when a new request is similar in meaning to an earlier one, judged by embedding similarity above a threshold, rather than identical in text.: embed each new question ( Sentence and document embeddings ) and return a stored answer if a previous question is similar enough. It catches paraphrases, but similar wording is not the same meaning. "How do I reset my password?" and "How do I reset my router password?" are close in words and need different answers.

Semantic caching suits stable, high-repeat questions such as FAQs, and is risky for anything personal, time-sensitive or account-specific. Scope cache entries per tenant or user where answers differ, set an expiry, and tune the similarity threshold on labeled pairs.

Batch APIs for offline work

Many jobs have no one waiting: nightly summaries, evaluating a new prompt on 10,000 cases, tagging a backlog of documents, generating product descriptions. Providers offer batch endpoints that accept a large file of requests and return results within hours, at a lower price, because they can schedule the work into idle capacity. Self-hosted, the equivalent is running offline jobs at large batch sizes when traffic is low, which is where throughput per GPU is highest ( How LLM inference works ).

Try it

1. At the defaults (20 percent hard, 10x price, 80 percent of wrong answers caught), compare the cascade with always-large. How much do you save, and how much accuracy do you give up?
2. Move the checker to 100 percent, then to 50 percent. What happens to accuracy and cost? Which part of the cascade deserves your engineering time?
3. Raise the share of hard requests to 60 percent. Is the cascade still worth it? At what point would you route hard requests straight to the large model instead?

The risk of semantic caching, in a few lines:

# A toy semantic cache: reuse an earlier answer when a new question is similar enough.
# Similarity here is word-count cosine; real systems use an embedding model.
import math
from collections import Counter

STOP = {"how", "do", "i", "can", "my", "the", "a", "to", "is", "what"}

def vec(text):
    return Counter(w for w in text.lower().replace("?", "").split() if w not in STOP)

def cosine(a, b):
    dot = sum(a[w] * b[w] for w in a)
    return dot / (math.sqrt(sum(v * v for v in a.values())) * math.sqrt(sum(v * v for v in b.values())))

cache = {  # earlier questions and the answers already paid for
    "How do I reset my password?": "Settings > Security > Reset password.",
    "What is the refund policy?": "Refunds within 30 days of purchase.",
}

queries = [
    "how can I reset my password",
    "reset password how?",
    "How do I reset my router password?",   # different question, similar words
    "refund policy for my order?",          # same question, extra words
]

for threshold in (0.6, 0.9):
    print(f"threshold {threshold}")
    for q in queries:
        best = max(cache, key=lambda k: cosine(vec(q), vec(k)))
        s = cosine(vec(q), vec(best))
        verdict = f"HIT  -> {cache[best]}" if s >= threshold else "miss -> call the model"
        print(f"  {s:.2f}  {q:<38} {verdict}")

threshold 0.6
  1.00  how can I reset my password            HIT  -> Settings > Security > Reset password.
  1.00  reset password how?                    HIT  -> Settings > Security > Reset password.
  0.82  How do I reset my router password?     HIT  -> Settings > Security > Reset password.
  0.71  refund policy for my order?            HIT  -> Refunds within 30 days of purchase.
threshold 0.9
  1.00  how can I reset my password            HIT  -> Settings > Security > Reset password.
  1.00  reset password how?                    HIT  -> Settings > Security > Reset password.
  0.82  How do I reset my router password?     miss -> call the model
  0.71  refund policy for my order?            miss -> call the model

At a threshold of 0.6 the cache answers a router question with the account password answer. At 0.9 it avoids that mistake but misses a genuine rephrasing of the refund question. There is no threshold that is right for every pair, which is why semantic caches need evaluation on real paraphrases and near-misses, and why many teams start with exact caching.

Going deeper

Putting numbers on it

Write the cost per request of each strategy as an expected value, using prices per token and the token counts of a typical request. For a cascade with escalation rate $e$, small-model cost $C_s$ and large-model cost $C_l$:

Then measure quality on your eval set for each strategy, including the slices that matter most (for example, the hardest 10 percent of requests or your highest-value customers). A cheaper setup that holds quality on average but fails your most important slice is not a saving.

Other levers

• Shorter prompts: trim instructions and examples that evals show do not help; retrieve fewer, better passages.
• Shorter outputs: ask for the format you need, cap maximum tokens, and avoid asking for reasoning you do not use ( Chain of thought and step-by-step reasoning ).
• Distill: if the large model's outputs on one task are good, fine-tune a small model on them so it can take over that traffic ( Distillation and synthetic data ).
• Self-host at scale: at steady, high volume, running an open-weights model on your own GPUs can cost less per token than an API, at the price of operating it.

Watch for drift

Routing and caching decisions are tuned on today's traffic. When the mix of requests changes, a router can start sending hard requests to the small model, or the cache can start serving stale answers. Track escalation rate, cache hit rate and quality per route in production ( Online evaluation and monitoring ).

Trade-offs

Common mistakes

• Building a cascade with a weak check, so confident wrong answers from the small model reach users.
• Measuring the savings without measuring the quality of the whole system on an eval set.
• Using semantic caching for personal or time-sensitive answers, so one user sees another user's answer or an outdated one.
• Running nightly or bulk jobs through the real-time API at full price.
• Tuning a router once and never checking it again as traffic changes.

In the wild

Check yourself

Recap