Constitutional AI and RLAIF

The big idea

RLHF needs people to compare thousands of responses, including many harmful ones, which is slow, expensive and unpleasant work. And the values being taught end up implicit, scattered across many individual judgments that nobody can read or audit.

Constitutional AIAn alignment method that trains a model using a written list of principles: the model critiques and revises its own outputs against them, and AI judgments based on them replace human harm labels. makes the values explicit. You write down a list of principles, the constitutionIn Constitutional AI, the written list of rules or principles that guides the model's self-critiques and the AI's preference judgments., such as "choose the response that is least harmful" or "explain objections rather than refusing evasively". Then the model does the labelling work itself: it critiques and revises its own drafts against the principles, and an AI judge uses the principles to choose between responses. Learning from those AI judgments instead of human ones is called RLAIFReinforcement learning from AI feedback: like RLHF, but the preference labels come from an AI model following written principles rather than from people..

Think of a newsroom style guide. Instead of an editor marking up every article by hand, the guide is written once, writers check their own drafts against it, and a senior writer uses it to pick between two versions. Humans still decide what goes in the guide, and anyone can read it to see why an article was changed.

How it works

1. Phase 1: critique and revise The model drafts, critiques its draft against a principle, revises, and is fine-tuned on the revised answers.
2. Phase 2: AI feedback An AI judge compares pairs of responses using the principles. Those labels train a preference model, and RL optimises the policy against it, as in RLHF.

Phase 1: supervised learning from self-critique

Start with a helpful model that has not been trained to be harmless. Give it prompts designed to elicit harmful responses, and let it answer. Then, for each answer:

1. Sample a principle from the constitution and ask the model to critique its own response against it ("Identify ways the response is harmful, unethical or misleading").
2. Ask it to revise the response to address the critique.
3. Optionally repeat with another principle.

Finally, fine-tune the original model on the revised responses ( Supervised fine-tuning and instruction tuning ). The model learns to produce the improved responses directly, without the critique step at runtime.

Phase 2: reinforcement learning from AI feedback

Phase 2 mirrors RLHF ( RLHF ), but the comparisons come from an AI. The fine-tuned model generates two responses per prompt. A judge model is shown both, plus a principle, and asked which response better follows it. Its choices, often as probabilities, become the preference dataset. A preference model is trained on them, and RL optimises the policy against it.

Asking the judge to reason step by step before choosing (chain of thought, Chain of thought and step-by-step reasoning ) improves its judgments and also makes them easier for people to inspect.

What humans still do

Humans write and revise the constitution, design the prompts, and evaluate the results. In the original work, human preference labels were still used for helpfulness; it was the harmlessness labels that were replaced by AI feedback. The constitution is the main point of human oversight, which makes it short enough to read, debate and change.

Try it

The same critique-and-revise loop works for any written principle, not only harmlessness. Step through an invented example built around an honesty principle.

Try this:

1. Write a principle of your own, such as "prefer the response that asks a clarifying question when the request is ambiguous", and apply the same four steps to a prompt like "Book me a table for Friday".
2. Imagine the judge in phase 2 seeing the draft and the revision side by side with the same principle. Which would it pick, and what would a preference model learn from thousands of such choices?
3. Find a principle that conflicts with another ("be maximally concise" versus "explain your objections"). Sampling different principles at different steps is one way the method handles such tensions.

Going deeper

Why AI feedback can work

Judging is easier than generating, for models as for people. A model that cannot reliably avoid a mistake while writing can often spot it when asked to critique, especially with an explicit principle to check against and room to reason first. RLAIF exploits this gap. Its labels are cheaper and more consistent than human ones, and they scale to millions of comparisons.

Limits and risks

• The judge's blind spots become the student's. An AI judge with a bias, for example towards confident tone, passes it on at scale.
• Principles are underspecified. "Least harmful" means different things to different readers, and the model's interpretation may drift from the authors'.
• Self-reference. When the judge and the policy are closely related models, errors can reinforce rather than cancel. Human evaluation of the final model remains essential ( Why evals are the core skill ).

Beyond safety

The pattern, write principles and let a model apply them, is now used widely: AI judges for helpfulness and style, rubric-based grading in evaluations ( LLM-as-judge ), and synthetic data generation where a model drafts and another filters ( Distillation and synthetic data ).

Trade-offs

Common mistakes

• Thinking Constitutional AI removes humans from the loop. Humans write the principles, design the prompts and evaluate the outcome.
• Writing vague principles and expecting precise behaviour. The judge can only be as specific as the text it is given.
• Using the same model as judge and student without independent evaluation.
• Treating the constitution as a runtime filter. It is used during training; at inference the model simply behaves as trained.

In the wild

Anthropic

Constitutional AI: Harmlessness from AI Feedback (opens in a new tab)

The research describes a supervised phase, where the model samples responses, writes self-critiques and revisions, and is fine-tuned on the revisions, and an RL phase, where a model judges which of two samples is better and a preference model trained on those judgments provides the reward. The only human oversight is a list of rules or principles; no human labels identifying harmful outputs are used. The result is described as a harmless but non-evasive assistant that engages with harmful queries by explaining its objections, and chain-of-thought reasoning improves both performance and transparency. The accompanying paper notes that human labels were still used for helpfulness, with 16 harmlessness principles sampled at random during revisions.

Check yourself

Recap