Context Engineering: What It Is and How I Use It to Build Better AI Agents

Table of contents

Why “prompt engineering” became the wrong frame

“Prompt engineering” implies that the key lever is the text you write in the system prompt or user message. Spend enough time crafting the right instructions, the right wording, the right format, and the model will do what you need.

That’s true up to a point. A well-written system prompt is necessary. But the model’s behavior is determined by everything in the context window — not just your system prompt. It’s shaped by:

• The conversation history (what happened in prior turns)
• The documents or data you’ve retrieved and injected
• The tool call results the model has seen so far
• The token count and position of each piece of information

If you’re thinking only about prompt wording and ignoring the rest of what fills the context window, you’re optimizing one input while leaving the others unmanaged. That’s why “context engineering” is the more accurate frame for serious agent work.

What context engineering actually is

Context engineering is the discipline of deciding what information goes into the model’s context window, in what order, at what point in the conversation.

The context window is the model’s working memory. It’s finite. Every token you put in is a token that displaces something else — or increases cost. And unlike human working memory, the model has no way to go “look something up” outside of what’s in the window (unless you give it tools to do so). What it sees is all it has.

Context engineering is the practice of treating that window as a resource to be managed deliberately:

• What does the model need to know to complete this step?
• What did it need to know in a prior step but no longer needs?
• What’s stable across runs vs. what’s dynamic per request?
• Where in the window should each piece of information appear?

These aren’t prompt-wording questions. They’re information-architecture questions. And the answers drive agent reliability as much as model selection.

The four layers I architect

Every agent I build has four distinct context layers. I think about each of them separately.

Layer 1: The system prompt

This is the stable, turn-independent foundation. It defines who the agent is, what it can do, what it can’t do, and how it should handle edge cases.

The mistake most people make here is writing the system prompt once and treating it as finished. In practice, the system prompt needs to answer three questions explicitly:

1. What is this agent for? (The model needs a tight scope, not a vague mission.)
2. What should it do when the input is ambiguous or incomplete?
3. What should it never do? (Negative constraints matter — see why agents fail in production.)

Keep the system prompt minimal. Every unnecessary sentence is overhead that competes with the dynamic content where reasoning actually happens. I aim for system prompts that are specific and short rather than comprehensive and long.

One practical tip: if you’re using Claude with the API, use cache_control on your system prompt. A large, stable system prompt that’s cached costs roughly 10% of what an uncached one costs per turn — and you get better latency too. I covered the mechanics in prompt caching with the Claude API.

Layer 2: Conversation history

In a multi-turn agent, the conversation history is dynamic and grows with every turn. Left unmanaged, it becomes the biggest driver of context bloat — and the sneakiest source of agent degradation.

The problem: early turns contain information the model no longer needs (a clarification from the user three steps ago, a failed tool call that’s been resolved). Keeping all of it wastes tokens and can actually confuse the model by giving it stale context to reason from.

What I do:

• Truncate or summarize old turns when the history exceeds a threshold. Summarizing is better than chopping — a short summary of “the user asked for X, we retrieved Y, the user approved” is more useful than raw turn text.
• Only keep tool call results that are still relevant. If a tool call was used to fetch data that’s now been processed and consumed, its raw output doesn’t need to stay in the window.
• Never let the history grow unbounded in a long-running agent. Set a max token budget for history and enforce it.

Layer 3: Retrieved content

This is the layer that separates mediocre agents from good ones. Most agents need to pull in external data at runtime — documents, database records, search results, API responses. How you handle that injection matters enormously.

Two principles I apply:

Retrieve only what’s relevant to the current step. Don’t inject a 50-page document when the current step only needs one section of it. A retrieval step that pulls the right chunk and discards the rest is doing context engineering — a retrieval step that stuffs the whole document in is not.

Position matters. Research on long-context models is clear: information at the start and end of the context is weighted more heavily than information in the middle. If there’s a piece of retrieved content the model absolutely must use, don’t bury it in the middle of a long injection. Put it near the relevant instruction, not in the middle of the conversation history.

Layer 4: Tool outputs

In an agentic loop, the model calls tools and gets results back. Those results go into the context window as tool-use blocks. They accumulate. And unlike conversation history, people rarely think about managing them.

The fix is the same: after a tool result has served its purpose, you don’t need to keep it in the window. In a multi-step agent, I carry forward a structured summary of “what we’ve established so far” rather than the raw tool output of every prior step. The model gets the conclusion, not the intermediate evidence.

The context budget: what to include and what to cut

I use a simple mental model: the context window is a budget, and every token is a spend. Before each agent turn, I ask:

• What does the model need to know right now to do this step?
• What can I leave out or summarize without losing anything important?
• What’s duplicated across layers (same information in both the system prompt and the retrieved content)?

The goal is to pack the window with the highest-signal information possible at each step, not to be comprehensive. Comprehensive context sounds safe but it’s not — it dilutes the signal-to-noise ratio and can lead the model to fixate on irrelevant information.

This is the context engineering mindset: not “what should I include?” but “what can I cut without losing reliability?”

Three context engineering mistakes I made in production

1. Floating timestamps in the stable prefix. I was putting Current date: {{date}} at the top of my system prompt. That string changes every day, which silently invalidated my prompt cache every 24 hours. The model never saw a cached hit, and I was paying full input price on every request for months before I caught it. Move volatile information — timestamps, user IDs, request-specific context — to the end of the context, after the stable prefix.

2. Treating tool outputs as append-only. I was running agentic loops where every tool call result stayed in the context. By turn 8, the model was reasoning from a context that was 80% stale tool outputs it didn’t need. Agent reliability dropped noticeably. The fix was carrying forward a running summary object instead of the raw outputs.

3. Skipping the eval on context changes. When I changed what information went into the context window, I thought it was a “non-functional” change that wouldn’t affect outputs. It absolutely affected outputs. Context changes are model behavior changes. I now run the same eval harness on context changes that I run on prompt changes.

My context engineering workflow in practice

Before I write a single line of agent code, I sketch out the context layers:

System Prompt:    ~500 tokens, stable, cached
History Budget:   ~2000 tokens max, summarized after each step
Retrieved Context: ~1000-3000 tokens per step, relevant chunks only
Tool Output Budget: current step only, summarized forward

Then I run a test at 1x, 5x, and 10x the expected input volume to see where the window fills up and what degrades. The eval harness catches reliability drops before they hit production.

The model selection question only comes after this. Once I know what context engineering I need to do, I pick the cheapest model that holds the reliability bar under the right context setup. In my experience, a cheaper model with well-engineered context usually beats a more expensive model with bloated context — because signal quality matters more than raw capability for most production tasks.

FAQ

What’s the difference between prompt engineering and context engineering?

Prompt engineering focuses on the wording of your system prompt and user messages. Context engineering is the broader discipline: deciding what information goes into the full context window — including conversation history, retrieved data, and tool outputs — in what order, and at what token cost. Prompt engineering is a subset of context engineering.

How big should my system prompt be?

As small as possible while being specific. I aim for under 800 tokens for most agents. A system prompt that tries to anticipate every scenario ends up being too long to be read reliably by the model, and the token cost compounds across every request. Write the minimum that gives the model a clear scope and explicit edge-case handling.

Does context engineering matter more for some models than others?

It matters for all of them, but the stakes are higher with smaller models. A large frontier model can sometimes recover from a poorly structured context; a smaller model running on a tighter budget can’t. Context engineering is how you make the economics of smaller, cheaper models work reliably — which is why it’s the core skill for running efficient agent fleets.

How do I know if my context engineering is working?

Track the same metrics you’d track for any reliability change: success rate on your eval set, cost per successful outcome, and the distribution of errors by step. If you’re seeing errors cluster in steps with large retrieved injections, that’s a context quality problem. The eval harness is the tool I use to catch these before they reach production.

Should I always compress or summarize history?

For short, transactional agents: no, the overhead isn’t worth it. For multi-turn agents that run more than 5-6 exchanges or agents that run long agentic loops: yes, always. The rule of thumb I use — once the history budget exceeds 30% of my total context budget, I start summarizing.