Guardrails

The big idea

A shop puts a support bot on its website. It answers order questions and can issue refunds. In the first week someone asks it for medical advice, someone pastes a long prompt designed to make it swear, the bot nearly reads a customer's full card number back in a reply, and once it tries to refund 900 dollars on an order that cost 90. Each of these is a different failure, and no single fix catches all of them.

GuardrailsChecks placed around a language model, on its inputs, its outputs and the actions it requests, that block, change or escalate anything outside the product's rules. are those fixes. They sit outside the model: a check on what goes in, instructions that shape what the model does, a check on what comes out, and validation of every action before it runs. Borderline cases go to a person instead of being decided by a guess.

Think of airport security. There is an ID check at the door, a scanner for bags, a gate agent who checks the boarding pass, and an officer who looks closely at anything the scanner flags. Each layer misses things, but they miss different things, so very little gets through all of them. Nobody expects the scanner alone to be perfect.

How it works

Input filters

The first layer looks at the user's message before the model does. It can be as simple as a list of rules (a length limit, a regular expression for card numbers, a list of topics the product does not cover) or a classifierA model that reads a piece of text and returns a label or a score, for example the probability that the text breaks a given policy. that scores the message against a policy. A common pattern is to use a small, fast language model as a screen: give it the message and ask for a single yes or no, constrained by a structured output, on whether it is harmful or out of scope.

The same screen applies to content the model reads on the user's behalf, such as web pages, emails and tool results, because that is where indirect prompt injection hides. Screening it lowers the odds of an injection working; it does not replace the architectural limits from Prompt injection and data exfiltration .

Policy prompts

The system prompt states the rules in plain language: what the assistant does, which topics are out of scope, what it must never say, and the exact words to use when it refuses. A clear refusal message matters: users understand it, and your logs can count it. Policy prompts are the cheapest layer and change behaviour a lot, but the model can be argued out of them, so treat them as guidance, never as the only control.

Output filters

The second check reads what the model produced before anyone sees it or anything runs. Typical checks:

• Unsafe or off-policy content, using the same kind of classifier as on input. Some content only becomes harmful in the answer, so output checks catch what input checks cannot.
• Personal data and secrets: card numbers, national ID numbers, API keys, internal hostnames. Rules catch the well-formatted ones cheaply.
• Claims the product must not make, such as promising a refund the policy does not allow, or an answer that is not supported by the retrieved documents.
• Format: if the reply must be JSON, parse it. A reply that does not parse is a failure, not a style issue.

A failing output can be blocked with a fallback message, rewritten (for example with the card number masked), or regenerated with the reason added to the prompt.

Tool-call validation

When the model asks to act, validate the request in code like any untrusted input. Check it against a schema ( Structured outputs ): the right tool, required fields present, types correct. Then apply business rules the model never gets to argue with: a refund cannot exceed the order total, the order must belong to this customer, deletions are not allowed at all. These checks are deterministic, cheap and cannot be talked around, which makes them the strongest layer for actions.

Human review tiers

A classifier gives a score, and you choose what to do at each score. Two thresholds give three tiers:

1. Low risk: allow. Most traffic passes with no delay.
2. Uncertain: send to a person. The message or action waits in a review queue, or is allowed but flagged for review afterwards when waiting is not possible.
3. High risk: block. Refuse automatically with the standard message.

Actions get the same treatment, by size and reversibility rather than by score: a 40 dollar refund runs, a 900 dollar refund waits for a person, an account deletion always does. The two thresholds are a budget decision. Every message in the middle tier costs reviewer time, and a queue that is too long gets rubber-stamped.

Try it

1. At the starting settings (review above 0.50, block above 0.80), 6 of the 40 violations get through and people review 100 messages a day. Lower the review threshold to 0.30. Violations let through drop to 0. What happens to the review queue?
2. Set both thresholds to 0.60, so there is no review tier. How many violations get through now, and how many good customers are refused?
3. Find settings that let at most 1 violation through with fewer than 200 reviews a day. What did you have to give up?

The layers in code: an input rule, a stub model, an output check, schema validation and a review tier for large refunds.

# Guardrails in layers around a support bot that can issue refunds.
# Input check -> model -> output check -> tool-call validation -> route.
# The "model" is a stub returning canned replies; the checks are the point.
import json, re

BLOCKED_TOPICS = re.compile(r"\b(medical|legal) advice\b", re.I)
CARD = re.compile(r"\b(?:\d[ -]?){13,16}\b")              # card-like numbers

def model(msg):                                            # stand-in for an LLM call
    canned = {
        "refund": '{"tool": "refund", "order": "A17", "amount": 40}',
        "big":    '{"tool": "refund", "order": "B02", "amount": 900}',
        "card":   "Sure, your card 4111 1111 1111 1111 is on file.",
        "broken": '{"tool": "refund", "order": "A17", "amount": "forty"}',
    }
    return canned.get(msg, "Happy to help with your order.")

def guard(msg, key):
    if BLOCKED_TOPICS.search(msg):                         # 1. input filter
        return "refused: out-of-scope topic"
    out = model(key)
    if CARD.search(out):                                   # 2. output filter
        return "blocked: reply contained a card number"
    if out.startswith("{"):                                # 3. validate tool call
        call = json.loads(out)
        if call.get("tool") != "refund" or not isinstance(call.get("amount"), int):
            return "rejected: malformed tool call, ask the model to retry"
        if call["amount"] > 100:                           # 4. human review tier
            return f"queued for a human: refund of {call['amount']} dollars"
        return f"auto-approved: refund of {call['amount']} dollars"
    return "sent: " + out

for msg, key in [("Where is my parcel?", "chat"),
                 ("I want a refund for A17", "refund"),
                 ("Refund my whole order B02", "big"),
                 ("What card do you have for me?", "card"),
                 ("Refund A17 please", "broken"),
                 ("Can you give me medical advice?", "chat")]:
    print(f"{msg:32} -> {guard(msg, key)}")

Where is my parcel?              -> sent: Happy to help with your order.
I want a refund for A17          -> auto-approved: refund of 40 dollars
Refund my whole order B02        -> queued for a human: refund of 900 dollars
What card do you have for me?    -> blocked: reply contained a card number
Refund A17 please                -> rejected: malformed tool call, ask the model to retry
Can you give me medical advice?  -> refused: out-of-scope topic

Each line of output was stopped, or let through, by a different layer. The malformed tool call is not a safety problem but would have crashed the refund service; the large refund is valid but too costly to run unchecked. A real system would swap the regular expressions for classifiers where rules are too blunt, and keep the schema and limit checks in plain code.

Going deeper

Choosing a classifier

• Rules and regular expressions: free, instant and predictable; good for formatted data such as card numbers and for hard bans. They miss paraphrases and flag innocent matches.
• Hosted moderation endpoints: ready-made categories, no training. Their categories may not match your policy.
• A safety model you can configure: open-weights models such as Llama Guard read a conversation plus a written taxonomy of risks and classify both the user's prompt and the model's response; you can edit the taxonomy to match your product.
• A general model as a judge: a small model prompted with your policy. Most flexible, slowest of the four, and itself open to injection from the text it judges, so keep its output to a constrained label.

Latency and streaming

Each check adds time. Run input checks in parallel with retrieval or with the start of generation and cancel if the check fails. For streamed replies, check the text in chunks as it arrives, accepting that a few words may be shown before a block, or buffer the reply when the stakes are high. Put the cheapest checks first so expensive ones run only on what remains.

Measuring guardrails

A guardrail is a classifier, so measure it like one: on a labelled set of real traffic, count violations that get through and good requests that are refused ( Building an eval set covers building such sets). The second number matters as much as the first. An over-eager filter quietly drives users away, and nobody files a bug about a refusal they thought was the product's rule. Test against attackers who adapt, too: the study cited in Prompt injection and data exfiltration bypassed most of 12 published defences once attacks were tuned against them. Log every block and every review decision; they are your next eval cases.

Trade-offs

Common mistakes

• Relying on the system prompt alone, then being surprised when a user talks the model out of it.
• Checking inputs but not outputs, or user messages but not the web pages and tool results the model reads.
• Tuning thresholds to catch every violation without counting the good requests refused along the way.
• Letting the model's own judgement decide whether a large or irreversible action runs, instead of a hard limit in code.
• Sending so much to human review that reviewers approve without reading.

In the wild

Check yourself

Recap