Making agents reliable

The big idea

An agent demo works. The same agent in production, on real requests, works most of the time. The gap is arithmetic: an agent run is a chain of steps, and the run succeeds only if every step does. If each step is right 97 percent of the time, a 20-step task succeeds $0.97^{20}\approx 0.54$, about 54 percent of the time. Nobody would ship a function that fails every other call.

You cannot make the model perfect, so you engineer around it, the way aviation does with pilots who are excellent but human: clear instruments, checklists, a co-pilot who cross-checks, and the ability to recover from a mistake rather than assuming none will happen. For agents the equivalents are tools that are hard to misuse, checks that catch failures, retries for the ones that can be fixed, checkpoints so a crash does not lose an hour of work, and evaluation of whole runs, not just final answers.

How it works

Tool design: the cheapest reliability you can buy

Most agent failures are tool failures: the wrong tool chosen, the right tool called with the wrong arguments, or a result the model misreads. Anthropic's guidance on writing tools for agents boils down to a few rules:

• Few, purposeful tools. Build tools for workflows, not one per API endpoint. A search_contacts tool beats list_contacts, which makes the model read every contact; a schedule_event tool that finds a free slot and books it beats three separate tools for users, events and creation.
• Clear, unambiguous names. Name parameters user_id, not user. When many tools are loaded, namespacing by service and resource, such as asana_projects_search, helps the model pick the right one.
• Meaningful, concise results. Return names rather than opaque UUIDs where you can, offer a concise and a detailed response format, and paginate or truncate large results with sensible defaults.
• Errors that teach. "Invalid input" gives the model nothing. "date must be YYYY-MM-DD, for example 2026-05-12" gets the next call right, as you saw in the lab in The agent loop (ReAct) .
• Descriptions written for a new hire. Spell out formats, terms and relationships you would otherwise assume. Small edits to descriptions can change success rates a lot, so test them.

Retries with limits

Some failures are worth retrying, others are not. A timeout, a rate limit or a server error is transientA failure caused by a temporary condition, such as a timeout or overloaded server, that may succeed if the same request is tried again later.: retry the same call automatically, with exponential backoff and jitter, and a cap of a few attempts ( Timeouts, retries, backoff and jitter ). A validation error or "not found" is permanent: repeating the identical call cannot work, so return the error to the model and let it change its approach.

Two safety rules. Retried writes must be idempotentAn operation that has the same effect whether it runs once or several times, such as setting a value or a payment request with a unique key., usually with an idempotency key, or a retry can book the flight twice ( Idempotency and safe retries ). And the model's own retries need limits too: the repeated-action check and iteration cap from The agent loop (ReAct) stop an agent that keeps trying the same failing thing.

Checkpoints

A long agent run is a long-running process, and processes crash, time out and get redeployed. If the only state is the in-memory conversation, every failure restarts the task from zero. Instead, persist the run after each step: the messages, the plan, tool results and any files written. Then a failure resumes from the last good step. Anthropic's multi-agent research system does exactly this, because restarting a long run is expensive and frustrating for users ( Multi-agent systems ). Checkpoints also give you something to inspect, replay and test against.

Guardrails

GuardrailsAutomated checks around a model that block or repair unsafe or invalid inputs, outputs and actions according to explicit rules. are checks that run outside the model, so they hold even when the model is confused or manipulated:

• Input guardrails: reject off-topic requests or obvious injection attempts before the agent starts.
• Validation: check tool arguments against their schema before execution, and final outputs against a schema or business rules ( Structured outputs ).
• Action guardrails: the permission checks and approval gates from Coding agents and computer-use agents , plus hard limits such as a maximum refund amount.
• Budget guardrails: caps on steps, tokens, time and money per run.

Cheap guardrails can run in parallel with the main model call, so they add little latency ( Workflows vs agents: the patterns ). More on their design in Guardrails .

Evaluating trajectories

A trajectoryThe full record of an agent run: every model message, tool call, tool result and the final outcome. is everything the agent did. Judging only the final answer misses runs that got lucky, runs that took 40 steps for a 5-step job, and runs that did something forbidden along the way. Evaluate at two levels:

• Outcome: is the end state right? For agents that change things, compare the final state of the world (the database rows, the files, the tickets) with the expected state, rather than checking every intermediate step, because there are often several valid paths.
• Process: did it use sensible tools, stay within budget, follow policy, ask for approval when it should? These are checks over the trace, by code where possible and by an LLM judge where not ( LLM-as-judge ).

And measure consistency, not just average success. Run the same task several times: an agent that succeeds on 60 percent of tasks might succeed every time on some and almost never on others, or succeed 60 percent of the time on all of them. Users feel the second kind as flakiness.

Try it

1. Start at 97 percent per step and 20 steps with no detection and no retries. What fraction of runs succeed? How many steps can you afford before success drops below 90 percent?
2. Add 2 retries but leave detection at 0 percent. Why does nothing change? Now raise detection to 80 percent.
3. With your best settings, compare pass^1 with pass^8. If a customer runs this task every day, which number matters to them?

pass@k and pass^k measured from simulated runs, the way benchmarks estimate them from repeated trials:

# Agent reliability over repeated runs: pass@k vs pass^k (tau-bench).
import random
from math import comb

def run_task(p_step, steps, detect, retries, rng):
    """One run: every step must succeed; detected failures retry."""
    for _ in range(steps):
        for attempt in range(retries + 1):
            if rng.random() < p_step:
                break                         # step worked
            if attempt == retries or rng.random() >= detect:
                return False                  # out of retries, or silent
    return True

def pass_at_k(n, c, k):                       # at least one of k succeeds
    return 1 - comb(n - c, k) / comb(n, k)

def pass_hat_k(n, c, k):                      # all k succeed
    return comb(c, k) / comb(n, k)

rng = random.Random(7)
tasks = [(0.99, 10), (0.97, 20), (0.95, 25), (0.98, 30)]  # (p, steps)
setups = [("no checks", 0.0, 0), ("checks + 2 retries", 0.9, 2)]
for label, detect, retries in setups:
    rows = []
    for p, steps in tasks:
        n = 40                                # trials per task
        c = sum(run_task(p, steps, detect, retries, rng)
                for _ in range(n))
        rows.append((n, c))
    for k in (1, 4, 8):
        at = sum(pass_at_k(n, c, k) for n, c in rows) / len(rows)
        hat = sum(pass_hat_k(n, c, k) for n, c in rows) / len(rows)
        print(f"{label:<19} k={k}: pass@k {at:6.1%}  pass^k {hat:6.1%}")

no checks           k=1: pass@k  63.7%  pass^k  63.7%
no checks           k=4: pass@k  96.1%  pass^k  26.6%
no checks           k=8: pass@k  99.8%  pass^k  16.4%
checks + 2 retries  k=1: pass@k  93.8%  pass^k  93.8%
checks + 2 retries  k=4: pass@k 100.0%  pass^k  77.2%
checks + 2 retries  k=8: pass@k 100.0%  pass^k  59.9%

pass@k rises with k, because one success out of k is enough; pass^k falls, because all k must succeed. With no checks, pass@8 is near 100 percent while pass^8 is about 16 percent. Checks and retries raise every number, and the gap between pass^1 and pass^8 is the flakiness your users will see. The lab above treats every run as equally likely to succeed, so its pass^k is just the run success raised to the power k. Real task sets mix easy tasks that almost always pass with hard ones that rarely do, which is why the simulation's pass^8 stays well above its average pass^1 raised to the 8th power.

Going deeper

The math of compounding

If each step succeeds with probability $p$, a detected failure is retried up to $r$ times, and a fraction $d$ of failures is detected, then each step succeeds with probability

and a run of $n$ steps succeeds with probability $s^n$. With $p=0.97$, $d=0.8$ and $r=2$: $(1-p)d=0.024$, so $s=0.97\times (1+0.024+0.000576)\approx 0.9938$, and 20 steps succeed with probability $0.9938^{20}\approx 0.88$, up from 0.54. Notice the terms: retries only help in proportion to $d$. Detection, through clear errors, validation and tests, is what makes retries worth anything.

pass^k

The tau-bench paper argues that for agents serving users, the right reliability metric is not pass@k (at least one of $k$ tries succeeds), which suits tasks with a verifier such as code with unit tests, but pass^k: the chance that all $k$ independent trials succeed, averaged across tasks. If a task is run $n$ times with $c$ successes, an unbiased estimate is $\left(\genfrac{}{}{0ex}{}{c}{k}\right)/\left(\genfrac{}{}{0ex}{}{n}{k}\right)$. In its simulated retail and airline customer-service domains, the best function-calling agent at publication succeeded on fewer than 50 percent of tasks, and pass^8 in retail was below 25 percent.

What to fix first

Read failed trajectories, label why each failed, and count ( Error analysis ). In practice the top categories are usually tool problems (wrong tool, bad arguments, unhelpful results) and missing checks, both of which are cheaper to fix than switching models. Anthropic reports that when it launched its web search tool, the model was needlessly appending the year to search queries, which hurt results; a better tool description fixed it.

Trade-offs

Common mistakes

• Wrapping every API endpoint as a tool, with vague names and opaque errors, then blaming the model.
• Retrying everything, including validation errors and non-idempotent writes, or retrying forever.
• Keeping run state only in memory, so any crash starts a long task over.
• Reporting average success from one run per task, which hides flaky behaviour.
• Grading only final answers, so lucky runs and policy violations along the way go unnoticed.

In the wild

Check yourself

Recap