AI AI-07.06

Context engineering

Treat the context window as a scarce budget: what to include, compress or drop, and how long-running agents use summaries, notes and sub-agents.

Intermediate 14 min Last reviewed 2026-10-04

The big idea

Prompt engineering asks "how do I word the instructions?". Context engineering asks a wider question: of everything that could go into this model call, what should? A real application fills the window with far more than a prompt: the system prompt, definitions of every tool, the conversation so far, documents fetched for this question, and in an agent, the result of every tool call it has made.

The window is a budget, and it is smaller than it looks. Every token costs money and latency on every call. And more is not better: as the context grows, models get worse at finding and using what matters in it. Irrelevant material is not harmless padding; it competes for the model's attention with the material that matters.

Think of briefing a surgeon mid-operation. You do not hand over the patient's entire file, every past scan and the hospital handbook. You give the three facts that matter right now, and keep the rest one request away.

How it works

The budget

Every model call has a fixed window, and everything shares it. A typical agent call might hold a few thousand tokens of system prompt, a few hundred per tool definition, several retrieved chunks of hundreds of tokens each, the history, and space reserved for the reply. History is the part that grows: every turn and every tool result adds to it, and tool results such as file contents or search results can be large.

Even when everything fits, quality drops as the context fills with material the current step does not need. Models attend less reliably to the middle of long contexts ( Context windows and the KV cache ), and similar-looking but irrelevant passages can mislead them. Cost and latency also grow with every token ( Prompt caching and cost control ).

What to include

Ask of every piece: does this call need it to do this step well?

System prompt: clear and complete, but at the right level of detail: specific behaviour and heuristics, not a pile of brittle if-then rules.
Tools: only the tools relevant to this task, each with a clear, unambiguous description. Twenty overlapping tools confuse the model and cost tokens on every call.
Examples: a few diverse, canonical ones, not every edge case you have seen ( Few-shot prompting and examples ).
Retrieved documents: the few most relevant chunks, not everything that matched ( Hybrid search and reranking ).
History: the turns that still matter.

Compress or drop

Trim tool results: return a summary, the relevant lines or the first rows, not a whole file or a 10,000-row table. Old tool results that the agent has already acted on can often be cleared entirely.
Summarise history: replace older turns with a summary of decisions, open questions and facts learned, and keep recent turns word for word.
Load just in time: keep lightweight references, such as file paths, record IDs or URLs, and let the agent fetch content with a tool when it needs it, instead of loading everything up front.

Long-running agents

An agent working for hours, on a codebase or a research question, will outgrow any window. Three techniques keep it going:

Compaction: when the context nears its limit, summarise it and continue in a fresh window that starts from the summary plus recent turns. The art is choosing what the summary must keep: goals, decisions, unresolved problems, key facts.
Structured notes: the agent writes progress notes, to-do lists or findings to a file outside the window and reads them back when needed. Notes survive compaction intact.
Sub-agents: the main agent hands a focused subtask, such as "search the codebase for every caller of this function", to a sub-agent with its own clean window. The sub-agent may read tens of thousands of tokens; it returns a short summary, so the main agent's context stays small ( Multi-agent systems ).

Try it

Each agent step adds a tool call and result of an assumed 1,500 tokens; tool definitions are 300 tokens each and retrieved chunks 800. Change the assumptions in your head to match your system.

Try this:

At the defaults (128k window, keep everything), step 20 uses about 44k tokens, a third of the window, and history is already the largest segment. Now pick a 32k window: keeping everything overflows at step 13.
Still at 32k, switch to compaction. History is summarised 8 times over the 60 steps (magenta dots) and the run never overflows. Each dot is a point where detail is thrown away, which is why what the summary keeps matters.
Switch to sub-agents. The tool work happens in the sub-agents' own windows and only 200-token summaries come back, so even step 60 needs about 26k tokens. Then set tool definitions to 40: 12k tokens of tools on every call, before any work is done.

Going deeper

Compaction in code

The summariser here is a stand-in for a model call. The pattern is the real one: measure, and when history passes the budget, replace the old turns with a summary and keep the last few verbatim.

compact.pyruns with python3 + numpy

# Compaction for a long-running agent: when history nears the budget, replace old turns with
# a summary and keep the most recent turns verbatim. `summarise` stands in for a model call.
BUDGET = 3000          # tokens we allow for history (the rest of the window is for other parts)
KEEP_RECENT = 4        # turns kept word for word after compaction

def tokens(text):      # rough estimate: about 4 characters per token for English
    return max(1, len(text) // 4)

def summarise(turns):
    files = sorted({t.split("read ")[1].split(":")[0] for t in turns if "read " in t})
    return f"[summary of {len(turns)} earlier turns] Goal: fix the failing date test. Files read: {', '.join(files)}."

def compact(history):
    if sum(tokens(t) for t in history) <= BUDGET:
        return history, False
    old, recent = history[:-KEEP_RECENT], history[-KEEP_RECENT:]
    return [summarise(old)] + recent, True

history = []
for step in range(1, 31):
    result = f"tool: read src/mod{step % 7}.py: " + "x = parse(date)  # line of code\n" * 25
    history.append(result)
    history, did = compact(history)
    if did or step % 10 == 0:
        used = sum(tokens(t) for t in history)
        print(f"step {step:2}: {len(history)} items, {used:5} tokens" + ("  <- compacted" if did else ""))
print("first item now:", history[0][:90])

Output

step 10: 10 items,  2060 tokens
step 15: 5 items,   865 tokens  <- compacted
step 20: 10 items,  1895 tokens
step 26: 5 items,   865 tokens  <- compacted
step 30: 9 items,  1689 tokens
first item now: [summary of 12 earlier turns] Goal: fix the failing date test. Files read: src/mod0.py, sr

History stays bounded at a few thousand tokens however long the run gets. Notice what the second compaction does: it summarises the first summary along with newer turns, so the count of earlier turns is already lost. Each compaction loses detail, which is why important state belongs in notes, not only in the conversation.

What a good summary keeps

A compaction prompt should ask for the goal, the decisions made and why, the current state, unresolved errors and next steps, and pointers (paths, IDs) to anything that can be reloaded. It should drop raw tool output that has already been used. Test compaction like any prompt: does the agent still finish the task after it?

Context for multi-turn chat

Chat assistants face the same problem more slowly. Common patterns are a sliding window of recent turns plus a running summary, and retrieval over older conversation for facts the user mentioned long ago. Memory across sessions is covered in Memory for agents .

Trade-offs

Option	Wins when	Loses when
Load everything up front	Simple; nothing is missed	Cost and latency on every call; distraction; overflows
Just-in-time loading with tools	Small context; always current data	Extra tool calls and latency; agent must know where to look
Compaction	Runs of any length in one thread	Lossy; a bad summary drops something the agent needed
Sub-agents	Main context stays clean; work in parallel	More calls and tokens overall; coordination overhead

Common mistakes

Treating a large window as permission to fill it, and paying for the distraction in quality and cost.
Registering every available tool on every call, with overlapping, vague descriptions.
Returning whole files or full API responses as tool results.
Compacting with a generic "summarise this" prompt that drops decisions and open problems.
Keeping important state only in conversation history, where the next compaction can erase it.

In the wild

Anthropic

Effective context engineering for AI agents (opens in a new tab)

The post defines context engineering as curating and maintaining the optimal set of tokens during inference, beyond the prompt alone, and notes that recall degrades as context grows, so models have a limited attention budget. It recommends system prompts at the right altitude between brittle hard-coded logic and vague guidance, clear self-contained tools, diverse canonical examples, and just-in-time loading through lightweight references such as file paths. For long tasks it describes compaction, structured note-taking outside the window and sub-agents with clean context windows that return condensed results.

Check yourself

Your agent's tool returns a 40,000-token log file every time it checks status. What is the best fix?
Show answer

B. Return only the relevant lines or a short summary, with a way to fetch more if needed Tool results are context too. Trim them to what the next step needs and load more just in time.
Why can adding more context make answers worse even when it fits in the window?
Show answer

B. Irrelevant material competes for attention and models use long contexts less reliably Recall degrades as context grows, and distracting passages can mislead the model.
An agent loses track of decisions after compaction. What helps most?
Show answer

B. A compaction prompt that keeps goals, decisions and open problems, plus notes written outside the window Compaction is lossy. Say what the summary must keep, and store important state in notes that survive it.
What is the main context benefit of a sub-agent?
Show answer

B. It does token-heavy work in its own window and returns a short summary, keeping the main context small The main agent sees only the condensed result, not every file the sub-agent read.

Recap

Context engineering decides what goes into the window at each call: instructions, tools, history, documents, results.
The window is a budget of money, latency and attention; irrelevant context makes answers worse, not just pricier.
Include only what the step needs, trim tool results, and load data just in time through references.
Long-running agents use compaction, notes outside the window and sub-agents with their own clean context.
Compaction is lossy: specify what summaries keep, and put durable state in notes.