Structured outputs

The big idea

A person can read "Sure! Here is the result: area is billing, severity high" and understand it. A program cannot. As soon as model output feeds code, a database or another tool, it has to arrive in an exact shape: valid JSON, the right field names, values from an allowed list. One malformed reply in a thousand is enough to crash a pipeline at scale.

There are three layers of defence, and serious applications use all of them. Ask clearly: describe the exact schema in the prompt. Enforce: use the provider's structured output feature, or constrained decodingRestricting which tokens a model may generate at each step, by masking tokens that cannot lead to output matching a grammar or schema, so the output is guaranteed to have the required structure., so the model physically cannot produce the wrong structure. Verify: validate every reply in code, and retry with the error message when it fails.

Think of a paper form versus a blank sheet. Ask people to "write down your details" and you get every format imaginable. Give them a form with labelled boxes and tick-lists and the answers line up. Check each form at the desk before accepting it, and you catch the rest.

How it works

Describe the schema

Put the exact structure in the prompt: field names, types, which fields are required, and the allowed values for each. A JSON Schema, a typed example or a short table all work. Name the allowed values ("severity: one of S1, S2, S3, S4") rather than describing them ("a severity level"). Tell the model to reply with the JSON only, no prose around it.

JSON mode and schemas

Most providers offer a JSON modeA provider setting that guarantees the model's reply is syntactically valid JSON, without necessarily matching any particular schema., which guarantees parseable JSON, and a stricter structured output option where you pass a JSON Schema and the reply is guaranteed to match it. Tool or function calling ( Function calling and tool use ) is another route: you define a tool whose input schema is your desired output, and the model's tool call arrives as structured arguments.

Validate every reply

Even with guarantees, validate in code. A schema can say a field is a string but not that it is a real customer ID; a date can be well-formed and still in the wrong year. Parse the JSON, check it against the schema with a validation library or a typed model class, then apply your business rules. Validation is cheap and turns silent bad data into a visible, handleable error.

Retry with the error

When validation fails, do not just resend the same request. Send the model its own reply plus the specific error: "Field 'area' is 'billing'; it must be one of Auth, Billing, Performance, Data." Models are good at fixing a mistake once it is named. Cap the retries, often at 2 or 3, and send what still fails to a fallback: a default value, a queue for a person, or an error the caller can handle.

Constrained decoding

At each step, a model produces a probability for every token in its vocabulary ( Decoding and sampling ). Constrained decoding tracks where the output is in a grammar, such as a JSON Schema or a regular expression, and sets the probability of every token that could not continue a valid output to zero before sampling. If the schema says the value must be one of four strings, only tokens that begin one of those strings remain. The structure is then guaranteed by construction, whatever the model would have preferred.

This is how structured output modes typically work under the hood, and libraries offer it for open-weights models you run yourself.

Try it

The candidate tokens and probabilities are illustrative, chosen to show typical failure modes.

Try this:

1. With the constraint off, press Run to end. The model opens with "Sure, here is the JSON:", writes "billing" in lower case and "high" instead of a severity code. The whole output is not parseable JSON, and even the JSON part fails the schema.
2. Switch the constraint on, Reset, and press Step a few times. "Sure" is masked at the first step even though it was the most likely token, and at the area step the lower-case "billing" is masked, so "Billing" wins.
3. Run to end with the constraint on. The output parses and matches the schema. Notice that at the severity step the model's favourite, "high", was masked: the constraint guaranteed the structure, but the value "S2" is only as good as the model's second choice. Structure is guaranteed; correctness is not.

Going deeper

Validation and retry in code

A stand-in model gets the format wrong twice, then fixes it after each specific error message.

# Validate every model reply against the schema; on failure, retry with the error message.
# `fake_model` stands in for an API call: it gets the format wrong once, then fixes it.
import json

SCHEMA = {"area": {"Auth", "Billing", "Performance", "Data"},
          "severity": {"S1", "S2", "S3", "S4"}}

def validate(text):
    """Return (object, None) if valid, else (None, a message the model can act on)."""
    try:
        obj = json.loads(text)
    except json.JSONDecodeError as e:
        return None, f"Reply was not valid JSON ({e.msg} at char {e.pos}). Reply with JSON only."
    if not isinstance(obj, dict):
        return None, "Reply must be a JSON object."
    for key, allowed in SCHEMA.items():
        if key not in obj:
            return None, f"Missing required field '{key}'."
        if obj[key] not in allowed:
            return None, f"Field '{key}' is {obj[key]!r}; it must be one of {sorted(allowed)}."
    return obj, None

REPLIES = iter(['Sure! {"area": "Billing", "severity": "S2"}',      # chatty prefix
                '{"area": "billing", "severity": "S2"}',            # wrong case
                '{"area": "Billing", "severity": "S2"}'])
def fake_model(messages):
    return next(REPLIES)

messages = [{"role": "user", "content": "Classify: 'Charged twice for March.' Reply as JSON."}]
for attempt in range(1, 4):
    reply = fake_model(messages)
    obj, error = validate(reply)
    print(f"attempt {attempt}: {reply}")
    if obj:
        print("  valid ->", obj)
        break
    print("  error ->", error)
    messages += [{"role": "assistant", "content": reply}, {"role": "user", "content": error}]
else:
    print("giving up: route to a fallback or a human")

attempt 1: Sure! {"area": "Billing", "severity": "S2"}
  error -> Reply was not valid JSON (Expecting value at char 0). Reply with JSON only.
attempt 2: {"area": "billing", "severity": "S2"}
  error -> Field 'area' is 'billing'; it must be one of ['Auth', 'Billing', 'Data', 'Performance'].
attempt 3: {"area": "Billing", "severity": "S2"}
  valid -> {'area': 'Billing', 'severity': 'S2'}

The error messages are written for the model to act on: they say what was wrong and what is allowed. Each retry carries the previous reply and the error, so the model sees its mistake.

Constraints can hurt quality

Forcing a rigid format can make the model worse at the task itself, particularly if it would normally reason before answering. Two fixes: include a field for reasoning before the answer fields in the schema (fields are generated in order), or let the model reason freely in a first step and extract the structured result in a second.

Keep schemas simple

Flat objects with clear names, enums for categorical values, and descriptions on fields work best. Deeply nested, optional-heavy schemas raise the error rate and may not be supported by a provider's strict mode. If the output is large, consider splitting it into several smaller calls.

Trade-offs

Common mistakes

• Parsing model output with string tricks instead of a JSON parser and a schema validator.
• Trusting a structured output guarantee and skipping validation of the values.
• Retrying with the same request instead of sending the specific error back.
• Unlimited retries, so one bad input loops forever and burns money.
• Putting the answer fields before a reasoning field, so the answer is chosen before any reasoning.

In the wild

Check yourself

Recap