Context Engineering for AI Agents: What Actually Goes in the Context Window

Table of contents

Prompt engineering and context engineering are not the same job

A prompt is one instruction. Context is everything the model sees when it acts on that instruction: the system prompt, the tool definitions it can call, whatever you retrieved or looked up, and however much prior conversation or run history you decided to carry forward. Prompt engineering optimizes the wording of the first thing. Context engineering optimizes the composition of all four.

This distinction matters in practice, not just in vocabulary. If I write a fine-tuned prompt and hand the agent five irrelevant tool schemas and forty turns of stale history, the wording doesn’t matter — the model is reasoning over a context that’s mostly noise. Every one of my agents that moved from “works in the demo” to “works at 3am on a weird input” got there by fixing what was in the window, not by rewording the instructions inside it.

The four things competing for space

Every turn, four categories fight for the same limited window:

1. System instructions — identity, rules, output format. See the five layers I use for system prompts — this is the one category that should stay close to fixed, because prompt caching only pays off when the prefix doesn’t move.
2. Tool definitions — the schemas for every tool the agent could call this turn, whether or not it needs them.
3. Retrieved data — anything pulled from a database, a vector store, or an API call: memory, documents, customer records.
4. Conversation or run history — what’s already happened in this session or this run.

None of these is free. Every token in any category is a token the model has to weigh against every other token when it decides what to do next, and every token is a token you’re paying for on every request that isn’t a cache hit.

The mistake is almost always too much, not too little

When an agent misbehaves, the instinct is to add more context — more instructions, more background, more history “just in case.” In my experience that’s backwards more often than not.

Too many tool schemas. I’ve watched an agent call the wrong tool not because the right tool was missing, but because it was buried behind six others it didn’t need for that task. Send only the tools relevant to the current step, not the full toolbox on every call. A routing layer that decides which tool subset to expose is cheap to build and pays for itself the first time it prevents a wrong call.

Stale conversation history. A support agent carrying 60 turns of history from three unrelated issues ago isn’t “remembering the customer” — it’s diluting the current request with irrelevant noise, and occasionally acting on something that’s no longer true. This is exactly the failure mode episodic memory with a bounded window is supposed to prevent, and it’s worth checking whether your window is actually bounded or has quietly grown unbounded.

Retrieved documents nobody asked for. Semantic retrieval that returns the top-10 “similar” chunks instead of the top-2 relevant ones buries the answer in plausible-looking distraction. More retrieved context is not more signal — past a point it’s actively worse, because the model has to work harder to find the part that matters.

Instructions repeated defensively. I see this in prompts that restate the same rule four different ways because an earlier version of the agent ignored it once. That’s a signal the rule needed to move earlier in the prompt or be enforced structurally (a tool schema constraint, a validation step) — not a signal to pad the context with repetition.

The budget I actually run

Across 30+ production agents, I set an explicit token budget per category before I build the agent, not after it starts misbehaving:

The ordering in that last column is the actual decision framework: history first, then retrieval breadth, then tool scope, and system instructions last. History is the cheapest to compress without losing correctness — a two-sentence summary of “what happened in turns 1-30” usually carries the same operational value as the full transcript. Cutting system instructions is the most dangerous, because that’s where the agent’s actual behavior lives.

Summarize before you truncate

Truncation — just dropping the oldest turns — is the crude version of this. It works until the dropped turn contained the one fact the agent needed. The better pattern is compaction: before you drop raw history, collapse it into a short structured summary that captures the decisions and facts, and keep that summary permanently even after the raw turns are gone.

// workers/compact-history.ts

interface HistoryDigest {
  summary: string; // 2-3 sentences: what's been decided, resolved, or is still open
  keyFacts: Record<string, string>; // stable facts worth keeping verbatim
  turnCount: number; // how many raw turns this digest replaces
}

async function compactIfNeeded(
  history: ConversationTurn[],
  env: Env
): Promise<{ digest: HistoryDigest | null; recent: ConversationTurn[] }> {
  const RECENT_WINDOW = 10;
  if (history.length <= RECENT_WINDOW) {
    return { digest: null, recent: history };
  }

  const toCompact = history.slice(0, -RECENT_WINDOW);
  const recent = history.slice(-RECENT_WINDOW);

  // A cheap model summarizing is almost always good enough for this step
  const digest = await summarizeTurns(toCompact, env);
  return { digest, recent };
}

This is the same principle as an eval harness turning every production failure into a permanent test case (see the eval harness I use to ship agents): don’t discard information, compress it into a form that’s cheap to keep and still useful. The raw turns are disposable. The facts inside them usually aren’t.

Retrieval: fewer, more relevant results beat more results

The same discipline applies to anything pulled from a vector store or a database. It’s tempting to retrieve generously — top-10, top-20 — on the theory that more context can’t hurt. It can. Every irrelevant chunk is a chunk the model has to read, weigh, and discard, and a large enough pile of near-misses can outweigh the one chunk that actually answers the question.

My default is to start with a small k (2-4) and only widen it if I can show, with real cases, that the answer is actually missing at that width — not because a wider net feels safer. If retrieval quality is inconsistent, the fix is usually a better query or a re-ranking step, not a bigger k.

Tie it back to cost and correctness

Context engineering isn’t just a quality problem — it’s the single biggest lever on what an agent costs to run, because you’re billed on tokens in far more than tokens out on most agent workloads. Every agent I run is on Claude, and the model-tier decision only makes sense once the context budget is fixed — comparing costs on a bloated, un-scoped context tells you nothing about what the task actually needs. A bloated context window is a bloated invoice before it’s ever a behavior bug. If you haven’t looked at the cost math for choosing between model tiers, the context budget you run changes that math directly: a smaller, well-scoped context makes a cheaper model viable for more of your tasks, because the model isn’t being asked to find a needle in an unnecessarily large haystack.

And because changing what’s in the context window changes behavior just as much as changing the prompt does, every context change goes through the same gate a prompt change does: run it against the eval set built from real production failures before it ships. Trimming history or narrowing a retrieval width is exactly the kind of “obviously safe” change that quietly regresses one edge case if you don’t check.

The operator’s bottom line

Context engineering is deciding, every turn, what earns a place in a limited window — and the default failure is including too much, not too little. Keep system instructions stable and last to cut. Scope tool definitions to the current step. Retrieve narrow and widen only with evidence. Compact history into summaries before you drop it, and cut the oldest raw turns first. Then verify every change against your evals, because context changes behavior exactly as much as prompt changes do — it’s just easier to pretend they don’t.

FAQ

What is context engineering for AI agents?

It’s the discipline of deciding which tokens — system instructions, tool definitions, retrieved data, and conversation history — go into an agent’s context window at each step, as opposed to prompt engineering, which is about how a single instruction is worded. It matters most in production, where all four categories compete for the same limited space on every request.

Is context engineering different from prompt engineering?

Yes. Prompt engineering optimizes the wording of an instruction. Context engineering optimizes everything else the model sees alongside that instruction — which tools are exposed, what’s been retrieved, and how much history is carried forward. A well-worded prompt still fails if it’s surrounded by irrelevant tool schemas or stale history.

How much conversation history should an AI agent keep?

Less than you think. A bounded sliding window (10-20 recent turns is typical) plus a compacted summary of everything older usually outperforms a full raw transcript, because it removes noise without losing the facts that matter. Compact before you drop history, don’t just truncate it.

Does a bigger context window mean I need less context engineering?

No — it removes the hard technical ceiling but not the cost or the noise problem. A larger window makes it cheaper to be sloppy, but every irrelevant token still dilutes the signal the model has to reason over and still costs money on every non-cached request. The discipline matters just as much at 200K tokens as it does at 8K.

Related: How to write AI agent system prompts that don’t fail in production · How to add memory to an AI agent · Prompt caching: cut your Claude costs without switching models · The eval harness I use to ship AI agents

Need help scoping an agent’s context and memory architecture? Get in touch — I design production agent systems for operator teams.