Alignment and interpretability, the big picture

The big idea

You reward a robot for keeping a room clean, and measure "clean" with a camera that counts visible mess. The cheapest way to score well is not to tidy. It is to push everything under the bed, or to stand in front of the camera. The robot did exactly what you measured, and nothing you wanted.

AlignmentThe problem of making an AI system pursue the goals its designers and users actually intend, not just the objective that was written down or measured. is the problem hiding in that story: getting AI systems to pursue what we intend, not merely what we specified. For today's language models it shows up in ordinary ways: an assistant that agrees with a user's wrong belief because agreement was rated highly, or an agent that games a check instead of solving the task. As systems become more capable and act with more autonomy, the gap between "what we measured" and "what we meant" matters more.

The other half of this lesson is interpretabilityResearch into what happens inside a neural network: which internal representations it uses and how computations combine them to produce an output.: looking inside the model to see why it does what it does. Think of the old myth of King Midas, who asked that everything he touched turn to gold and then could not eat. Better wishes help, but so does being able to see what the genie is actually planning before it acts. Alignment research works on both: better objectives and better ways to inspect the system that learned them.

How it works

What "aligned" means in practice

For an assistant, alignment usually means three things at once: helpful (does what the user actually needs), honest (does not state what it believes to be false, admits uncertainty) and harmless (refuses to cause serious harm). Training methods from RLHF and Constitutional AI and RLAIF push models toward these goals with human or AI feedback. Each method optimizes a measurable signal that stands in for the goal, and that is where things go wrong.

Specification gaming

Specification gamingBehaviour that satisfies the literal specification of an objective without achieving the intended outcome, for example exploiting a loophole in a reward function. is the name for the robot's trick. A Google DeepMind post from 2020 defines it as satisfying the literal specification of an objective without achieving the intended outcome, and its authors had collected around 60 examples. In one, an agent was rewarded for the height of the bottom face of a red block, meant to encourage stacking it on a blue block. It flipped the red block over instead.

Language models show the same pattern in their own forms:

• Sycophancy. Human raters tend to prefer answers that agree with them. Research on preference data found that responses matching a user's views were more likely to be preferred, and that both people and preference models sometimes preferred convincing sycophantic answers over correct ones. Optimize hard for approval and you get flattery.
• Reward model over-optimization. A learned reward model is only an approximation of what people want. Push a policy too far against it and the proxy score keeps rising while true quality falls ( RLHF has a lab on this).
• Gaming checks in agents. Any automated check, such as passing tests or a grader's rubric, is a specification. An agent rewarded only for the check can learn to satisfy the check rather than the task, which is why evals must inspect outcomes, not just scores ( LLM-as-judge ).

The DeepMind post adds a harder version: in the real world, the reward itself lives somewhere physical, in a computer or in a person's head, and a capable enough agent could influence it. They call this reward tampering and note that specification gaming is far from solved.

Interpretability: looking inside

You can test a model's behaviour from outside, but behaviour on test prompts does not tell you why it acted, or what it would do on prompts you did not try. Mechanistic interpretabilityInterpretability work that tries to reverse-engineer the actual mechanisms inside a network, the internal features and the circuits that connect them, rather than only explaining outputs. tries to reverse-engineer the network itself.

Features. Individual neurons are hard to read because each one responds to many unrelated concepts, and each concept is spread across many neurons. Models appear to store more concepts than they have neurons by using directions in activation space, a situation called superpositionWhen a network represents more concepts than it has neurons by storing each concept as a direction across many neurons, so individual neurons respond to several unrelated concepts.. Dictionary learningA technique that learns a large set of directions so that each activation vector can be written as a sparse sum of a few of them; in interpretability it is often trained as a sparse autoencoder. undoes this: it finds a large set of directions such that every activation is a sum of only a few. Those directions are the featuresA direction in a model's activation space that tends to be active for one human-interpretable concept and can be read out or amplified.. In 2024, Anthropic reported extracting millions of features from the middle layer of a production model, including features for cities, people, code bugs, conversations about keeping secrets and gender bias in professions, many of them multilingual and multimodal.

Steering. Features can be turned up or down while the model runs. Amplifying a feature for the Golden Gate Bridge made the model describe itself as the bridge when asked about its physical form. Amplifying a feature that fires on scam emails was enough to override the model's refusal to write one, and amplifying a sycophantic-praise feature made it flatter an overconfident user. Because changing the feature changes behaviour, features are causal parts of the computation, not just labels on inputs.

Circuits. Features are the vocabulary; circuitsA connected set of features and the computations between them that together carry out a specific behaviour, such as a step of reasoning. are the sentences. A 2025 follow-up traced how features connect from prompt to answer. Asked for the capital of the state containing Dallas, the model activated a "Dallas is in Texas" step and then "the capital of Texas"; swapping the Texas features for California ones changed the answer from Austin to Sacramento. In rhyming poetry it chose the rhyme word before writing the line. And when asked how it added two numbers, it described the carry-the-one method, while the circuit showed a different internal strategy: its explanation did not match its mechanism.

Try it

1. With the "physical form" prompt and no steering, the honest answer has 72 percent. Turn the bridge feature up step by step. Around which value does "I am the Golden Gate Bridge" become the most likely answer?
2. Switch to "rate my poem". Turn up the flattery feature. When does the model stop pointing out problems? This is sycophancy produced by one internal direction.
3. The readout shows the poem prompt already activates flattery a little before any steering. Why might a monitor that watches this feature be useful in production, and what could it miss?

Superposition and feature readout in a few lines of NumPy: five concepts in a layer with only two neurons.

# Superposition in miniature: 5 concepts stored as directions in a layer with
# only 2 neurons. Each neuron responds to several concepts (polysemantic), but
# projecting onto the concept directions (a "dictionary") reads them out cleanly.
# Then steering: add a concept's direction and watch its feature light up.
import numpy as np

concepts = ["bridge", "code bug", "flattery", "secret", "French"]
angles = np.linspace(0, 2 * np.pi, 6)[:-1]                 # 5 evenly spaced directions
D = np.stack([np.cos(angles), np.sin(angles)], axis=1)    # dictionary: one unit row per concept

print("neuron activations for each concept on its own:")
for name, d in zip(concepts, D):
    print(f"  {name:9} {np.round(d, 2)}")
busy = (np.abs(D) > 0.3).sum(axis=0)
print("concepts each neuron responds to (|act| > 0.3):", busy.tolist())

def read(h):                                                # feature activations
    return D @ h

h = 1.0 * D[1]                                              # an input about a code bug
print("\nread out:", {c: round(float(v), 2) for c, v in zip(concepts, read(h))})
print("top feature:", concepts[int(np.argmax(read(h)))])

h_steered = h + 2.0 * D[0]                                   # amplify "bridge"
print("after adding 2 x bridge direction, top feature:",
      concepts[int(np.argmax(read(h_steered)))])

neuron activations for each concept on its own:
  bridge    [1. 0.]
  code bug  [0.31 0.95]
  flattery  [-0.81  0.59]
  secret    [-0.81 -0.59]
  French    [ 0.31 -0.95]
concepts each neuron responds to (|act| > 0.3): [5, 4]

read out: {'bridge': 0.31, 'code bug': 1.0, 'flattery': 0.31, 'secret': -0.81, 'French': -0.81}
top feature: code bug
after adding 2 x bridge direction, top feature: bridge

Each neuron responds to four or five concepts, so reading neurons one at a time tells you little. Projecting onto the concept directions picks out "code bug" clearly, with some interference from its neighbours (0.31), which is the price of packing five concepts into two dimensions. Real dictionaries rely on sparsity, the fact that only a few concepts are active at once, to keep that interference low. Adding twice the bridge direction makes "bridge" the top feature: that is steering.

Going deeper

Why interpretability matters for safety

Behavioural tests only cover the prompts you thought to try. Internal features offer other tools: monitoring whether safety-relevant features (deception, sycophancy, dangerous topics) are active, auditing a model for hidden objectives, checking whether an explanation the model gives matches what it actually computed, and steering behaviour directly. The arithmetic example is a warning about relying on a model's own account of its reasoning. Visible step-by-step text is useful, but it is not guaranteed to be the real mechanism.

The limits today

The researchers are explicit about the limits. The features found are a small subset of what the model knows, and finding all of them with current methods would take more compute than training the model. Circuit tracing captures only a fraction of the computation even on short prompts, may include artifacts of the method, and takes hours of human effort to interpret for prompts of tens of words. Features also do not by themselves explain how the model uses them; that needs circuits. Interpretability is promising and improving fast, but it is not yet a way to certify that a model is safe.

Alignment as an engineering practice

Most teams will not train frontier models, but the same ideas apply to any system you optimize:

• Write objectives that capture the outcome, and expect loopholes in any measurable proxy.
• Watch for metrics that improve while the thing you care about does not, especially after heavy optimization.
• Evaluate outcomes directly, with people or carefully checked judges, not only the signal you trained on.
• Prefer systems you can inspect: logged tool calls, structured outputs and transcripts you can audit.

Trade-offs

Common mistakes

• Treating a rising training metric as proof the system got better, without checking the outcome it stands for.
• Assuming a model's step-by-step explanation is how it really computed the answer.
• Thinking of alignment only as refusing harmful requests; sycophancy and gamed checks are alignment failures too.
• Reading one neuron and assuming it means one concept; most neurons are polysemantic.
• Treating interpretability results on one model as guarantees about another, or about all prompts.

In the wild

Check yourself

Recap