LLMOps: shipping and iterating

The big idea

Someone edits one sentence in the support assistant's system prompt to make it friendlier, and ships it on a Friday afternoon. On Monday the refund complaints arrive: the new wording quietly dropped the 30-day rule, and the assistant has been promising refunds it should not. No test failed, because there were no tests. Nobody can say exactly which prompt was live when, because the prompt lived in a string in the code and a copy in someone's notes.

In an AI product, a prompt edit, a new retrieval setting or a switch to a different model changes behavior as much as a code change does, and usually less predictably. So treat them like code. Version every prompt and model setting, run the eval suite in CIContinuous integration: an automated pipeline that builds and tests every proposed change before it can be merged. on every change, release to a small share of users first, and keep a one-click way back.

Think of how a restaurant changes its menu. A new recipe is written down with a date, the kitchen cooks it for staff tasting before it goes on the menu, it is offered as a special to a few tables first, and if diners send it back, yesterday's recipe is still in the book. Nobody improvises a new dish for the whole dining room on a busy night.

How it works

Version prompts and model settings

Pull prompts out of the code into a prompt registryA store of prompt templates where each version is kept under an id, and configuration decides which version is live.: files in the repository or a small service, each version with an id, never edited in place. The same goes for everything else that shapes behavior: model id, temperature, max tokens, tool definitions, retrieval settings such as top k. Bundle them as one versioned configuration per feature.

Three things follow. Every trace can record which version produced it ( LLM observability and tracing ), so "what changed?" has an answer. Rolling back means pointing configuration at the previous id, which takes seconds. And code review covers prompt changes, so a reviewer sees the diff that dropped the 30-day rule.

Pin the model version

Model providers release new versions and retire old ones. If your configuration names a moving alias, the model under your prompt can change without any commit on your side, and behavior can shift: formatting, refusals, verbosity, tool use. Where the provider offers dated or pinned version ids, use them in production and upgrade on purpose, through the same gate as any other change. Retirement dates still force upgrades, so plan for them instead of being surprised.

Eval gates in CI

An eval gateAn automated check in CI that runs the eval suite on a proposed change and blocks the merge if quality drops below a set threshold. runs your eval set ( Building an eval set ) on every change to prompts, models or retrieval, compares with the current version, and fails the pipeline on a regression. Three rules make gates useful:

• Compare per category, not just overall. A 2-point overall drop can hide a 20-point collapse in refund questions. Gate on each important slice.
• Allow for noise. Model outputs vary between runs, and LLM judges ( LLM-as-judge ) vary too. Set a tolerance, run more cases for the slices that matter, or run each case several times. A gate that fails randomly gets ignored.
• Keep it fast and cheap enough to run every time. A few hundred well-chosen cases run on every pull request; the large suite can run nightly.

Deterministic checks (valid JSON, required fields, forbidden phrases) belong in the gate too. They are cheap, exact and catch the most embarrassing failures.

Canary rollout

Evals only test what you thought to include. Real users ask things your eval set never imagined, and some quality problems only show in behavior, such as users rephrasing, abandoning or asking for a human. So release the change to a small share of traffic first, a canaryA release where a small share of users gets the new version while monitoring compares it with the old one, before the rest of users get it., and compare the two versions on online signals ( Online evaluation and monitoring ): thumbs-down rate, escalations, retries, output validation failures, cost per request and latency.

A feature flagA runtime switch, read from configuration, that decides which users get which version of a feature, without a new deploy. makes this easy: it assigns users (not single requests, so a conversation does not switch versions halfway) to the old or new version. Raise the share in steps, such as 1, 5, 25 and 100 percent, holding each step long enough to see a difference. Before any real traffic, some teams also run a shadowSending a copy of real requests to the new version without showing its output to users, to compare outputs, cost and latency safely. test: the new version answers real requests in the background, and its outputs are compared offline.

Roll back fast

Keep the previous version deployed and live-switchable. Rolling back should be a config change that takes effect in seconds, not a new build. Decide the rollback conditions before the canary starts ("thumbs-down rate more than 1 point above control for 2 hours"), so nobody argues about it at 2 a.m.

Try it

1. Step through change 2 (reworded policy). Where is it stopped, and how many users saw it?
2. Change 3 (always clarify) passes the gate. Step to the canary with 5 percent for 4 hours. Can you tell the versions apart? Raise the duration until you can. Why did the evals miss this one?
3. Change 4 involves no commit at all. What has to exist in your system for this change to be caught?

The gate itself is a short script. Here the candidate prompt lost the refund window, and two eval cases catch it:

# An eval gate as CI would run it: versioned prompts, a pinned model, a fixed eval set,
# and a non-zero exit code when the candidate regresses. The "model" is a fake so it runs offline.
import hashlib, re

PROMPTS = {  # prompt registry: every version is kept, the live one is chosen by config
    "refunds-v14": "Refunds: items can be returned within 30 days. Be brief.",
    "refunds-v15": "Refunds: customers may return items. Be brief and friendly.",
}
CONFIG = {"live": "refunds-v14", "candidate": "refunds-v15", "model": "small-model-2026-06-01"}

EVAL_SET = [  # (question, check on the answer)
    ("Can I return shoes after 3 weeks?", lambda a: "30 days" in a),
    ("What is the refund window?", lambda a: "30 days" in a),
    ("Do you refund gift cards?", lambda a: len(a.split()) < 40),
    ("Return policy please", lambda a: "return" in a.lower()),
]

def fake_model(model, system, question):  # answers only from what the prompt says
    days = re.search(r"\d+ days", system)
    return f"You can return items{' within ' + days.group(0) if days else ''}. Thanks for asking!"

def run(version):
    system = PROMPTS[version]
    passed = [check(fake_model(CONFIG["model"], system, q)) for q, check in EVAL_SET]
    digest = hashlib.sha256(system.encode()).hexdigest()[:8]  # content hash, logged in traces
    return sum(passed) / len(passed), passed, digest

base, base_cases, h1 = run(CONFIG["live"])
cand, cand_cases, h2 = run(CONFIG["candidate"])
print(f"model {CONFIG['model']}")
print(f"{CONFIG['live']} ({h1}): {base:.0%}   {CONFIG['candidate']} ({h2}): {cand:.0%}")
for (q, _), b, c in zip(EVAL_SET, base_cases, cand_cases):
    if b and not c:
        print(f"  REGRESSION: {q}")
ok = cand >= base - 0.02   # tolerance: allow at most a 2-point drop
print("gate:", "PASS, safe to merge" if ok else "FAIL, merge blocked")
exit_code = 0 if ok else 1   # in CI: sys.exit(exit_code) fails the pipeline
print("exit code", exit_code)

model small-model-2026-06-01
refunds-v14 (d70efe15): 100%   refunds-v15 (b7ca63f4): 50%
  REGRESSION: Can I return shoes after 3 weeks?
  REGRESSION: What is the refund window?
gate: FAIL, merge blocked
exit code 1

In CI the script would end with a non-zero exit code and the merge button would stay disabled. The content hash next to each version id is what traces record, so a trace can always be matched to the exact prompt text.

Going deeper

Small effects need big canaries

A canary can only detect differences larger than its noise. With a 6 percent baseline thumbs-down rate, 400 canary conversations give a 95 percent interval of roughly plus or minus 2.3 points, so a 2-point regression is invisible. You need thousands of conversations, which means a larger share or a longer hold. This is why big regressions should be caught by evals before the canary, and why the canary is the net for what evals cannot see, not a replacement for them.

Everything that is a change

Prompt edits are obvious. Less obvious changes that deserve the same path: tool descriptions (the model reads them), retrieval settings and re-indexed documents, a new embedding model, guardrail thresholds, and the provider's model version. If a change can alter what the model sees or does, it gets a version, an eval run and a gradual rollout.

From production back to evals

The loop closes when production failures become eval cases. Each rolled-back canary, user complaint or flagged trace is a case the eval set did not have. Add it, with the expected behavior, and the gate will catch the same failure next time. Over months this is how an eval set grows from a few dozen hand-written cases to one that reflects how users really behave ( Error analysis ).

Trade-offs

Common mistakes

• Keeping prompts as strings scattered through the code, so nobody knows which version is live or what changed.
• Gating on a single overall score, so a collapse in one important category hides inside the average.
• Pointing production at a moving model alias and treating the provider's update as "no change".
• Declaring a canary healthy after an hour at 1 percent, when the sample is far too small to show anything.
• Rolling back by rebuilding and redeploying, instead of switching configuration to a version that is still running.

In the wild

Check yourself

Recap