Prompt caching and cost control

The big idea

Before a model writes a single token, it processes the whole prompt, computing keys and values for every token in every layer ( Context windows and the KV cache ). In many applications most of that prompt is identical from call to call: the same system prompt, the same tool definitions, the same 30-page document the user keeps asking about. Processing it again every time wastes money and time.

Prompt cachingA provider feature that stores the processed form (the KV cache) of a prompt prefix so later requests starting with exactly the same prefix can reuse it, at a lower price and latency. stores the processed form of a prompt's beginning and reuses it when the next request starts with exactly the same tokens. Cached tokens are billed at a fraction of the normal input price and skip most of the processing time. The catch is in the word "exactly": the cache only matches an identical prefix, so the order of your prompt decides whether you get any benefit at all.

Think of a lecturer who prints the same 30-page handout for every class. Reprinting it each time costs the same as the first time. Keeping a stack ready means each class only needs today's one-page worksheet printed. But if the date is printed at the top of page one, every stack is out of date tomorrow.

How it works

Prefix matching

The cache works on prefixes: a request benefits only for the tokens, from the very start, that are identical to a cached prompt. One changed token early on and everything after it is a miss. Providers typically build the prefix from tool definitions, then the system prompt, then messages, and you mark where cacheable sections end, or the provider does it automatically.

So order the prompt from most stable to least stable:

1. Tool definitions and system prompt: identical for every user.
2. Long shared context: the document, the codebase summary, the few-shot examples.
3. Conversation history: grows, but earlier turns do not change.
4. The new question and anything request-specific, last.

Cache killers

• A timestamp, request ID or user name at the top of the system prompt.
• Tool definitions generated in a different order each time, or with a changing description.
• The question placed before the document.
• Editing earlier history, such as trimming old turns, which changes everything after the edit.

If you need today's date or the user's name, put it after the stable parts.

Lifetime and write cost

Caches expire after a short time without use, minutes by default, so caching pays when requests sharing a prefix arrive close together: a chat conversation, an agent loop, a batch of questions about one document. Some providers charge a premium to write a prefix into the cache. A prefix written once and never read again costs more than no caching at all.

Token accounting

You cannot control what you do not measure. API responses report tokens per request: uncached input, cache writes, cache reads and output. Log them with the feature, prompt version and user tier, price them, and look at the totals by feature. Common findings: one feature dominates the bill; output tokens cost several times more per token than input, so long answers matter; a cache that was meant to be hit is not.

Try it

Prices and multipliers are adjustable assumptions; set them to your provider's numbers.

Try this:

1. At the defaults (stable parts first, a 30,000-token document, 10 questions), 89 percent of input tokens are read from cache and input cost falls from about 1.11 to 0.25 dollars, 78 percent less.
2. Switch to "timestamp at the top". A 20-token line that changes every request breaks the prefix: nothing is read from cache, and because every request pays the write premium, caching now costs 25 percent more than no caching.
3. Go back to stable first and set questions to 1. A single request writes the cache and never reads it: caching costs more. Set it to 30 and savings rise to about 85 percent. Caching pays for repetition.

Going deeper

Accounting in code

# Token accounting: price every request from its usage numbers and see where the money goes.
# Usage records look like what APIs return; prices are made-up round numbers per 1M tokens.
from collections import defaultdict

PRICE = {"input": 3.00, "cache_write": 3.75, "cache_read": 0.30, "output": 15.00}

log = [  # feature, uncached input, written to cache, read from cache, output tokens
    ("doc_chat", 500, 36000, 0, 400),     # first question about a document: cache written
    ("doc_chat", 500, 0, 36200, 350),     # follow-ups read the document from the cache
    ("doc_chat", 600, 0, 36600, 420),
    ("doc_chat", 550, 0, 37000, 380),
    ("triage", 1800, 0, 0, 60),           # short prompt, no cacheable prefix
    ("triage", 1750, 0, 0, 55),
    ("summary", 9000, 0, 0, 2500),        # long outputs dominate here
]

def cost(inp, write, read, out):
    return (inp * PRICE["input"] + write * PRICE["cache_write"]
            + read * PRICE["cache_read"] + out * PRICE["output"]) / 1e6

by_feature = defaultdict(lambda: [0, 0.0, 0.0])   # calls, cost, cost if nothing were cached
for feature, inp, write, read, out in log:
    f = by_feature[feature]
    f[0] += 1
    f[1] += cost(inp, write, read, out)
    f[2] += cost(inp + write + read, 0, 0, out)

print(f"{'feature':10} {'calls':>5} {'cost':>9} {'uncached':>9}  saved")
for feature, (calls, c, u) in by_feature.items():
    print(f"{feature:10} {calls:5} {c:9.4f} {u:9.4f}  {100 * (1 - c / u):4.0f}%")
total = sum(f[1] for f in by_feature.values())
out_share = sum(o * PRICE["output"] / 1e6 for *_, o in log) / total
print(f"total {total:.4f} dollars; output tokens are {100 * out_share:.0f}% of it")

feature    calls      cost  uncached  saved
doc_chat       4    0.1976    0.4671    58%
triage         2    0.0124    0.0124     0%
summary        1    0.0645    0.0645     0%
total 0.2745 dollars; output tokens are 23% of it

The document chat saves 58 percent, even counting the first request's cache write. Triage and summaries save nothing: triage has no long shared prefix, and the summary's cost is mostly its long output. Caching cannot help output tokens; shorter answers can ( Latency and cost budgets ).

Other cost levers

Caching is one lever among several: route easy requests to cheaper models ( Routing, cascades and cost optimization ), trim context you do not need ( Context engineering ), cap output length, and use batch APIs for work that can wait, which providers often discount. Measure first, then pull the lever that matches where the money actually goes.

Trade-offs

Common mistakes

• A timestamp or request ID at the top of the system prompt, silently disabling every cache hit.
• Putting the user's question before the shared document.
• Assuming caching works without checking the cache-read numbers in the usage data.
• Caching prompts that are never repeated within the lifetime, paying write premiums for nothing.
• Optimising input tokens while long outputs make up most of the bill.

In the wild

Check yourself

Recap