Context Engineering: The Skill That Replaced Prompt Engineering

"Prompt engineering" made sense when the whole interaction was one instruction and one answer. Word the request cleverly, get a better result. But agents don't work that way. An agent runs for many turns, calls tools, reads files, and accumulates history, and on every single inference call the model sees a fresh pile of tokens assembled from all of it. The quality of that pile — not the cleverness of one sentence — is what determines whether the agent succeeds. Designing that pile is context engineering, and in 2026 it is the skill that separates demos from systems.

Anthropic's applied team defines it as "curating and maintaining the optimal set of tokens (information) during LLM inference." Put more plainly: context engineering is the practice of deliberately designing what the model sees on every call — the system prompt, user input, retrieved documents, conversation history, tool definitions, tool results, and long-term memory. If you're deciding what the agent retrieves, in what order, with what formatting, and what gets evicted when the window fills, you're doing it. This guide walks the five parts of the job.

From wording to token budgets

The mental shift is from authorship to budgeting. Prompt engineering asks "how do I phrase this?" Context engineering asks "of everything I could put in the window, which tokens earn their place?" That reframing matters because context is not free in two senses: you pay for every token, and — the part people miss — every token you add competes for the model's attention with every other token. A window packed with marginally relevant material doesn't just cost more; it makes the model worse. The craft is finding the minimum set of high-signal tokens that maximizes the odds of the outcome you want.

None of the old skill is wasted — clear instructions still matter, and our guide on prompt engineering is still worth reading. It's now a component of a bigger job, the way good sentence-writing is a component of good editing.

Retrieval design

Retrieval is how external data enters the window, and it's the highest-leverage decision you make. It takes several forms: classical retrieval-augmented generation (RAG) over a vector database, structured queries against a SQL store, plain file reads from a filesystem, and — increasingly the default — just-in-time retrieval, where the system pulls content in only at the moment it's needed rather than front-loading everything.

The just-in-time pattern is worth internalizing because it inverts the old instinct. Instead of retrieving 20 documents up front and hoping the answer is in there, you give the agent the ability to fetch and let it pull the two or three things the current step actually requires. Anthropic's own coding agent is a clean example: it drops a lightweight guide file into context up front, but then uses ordinary tools like glob and grep to navigate the codebase and read specific files only when a task calls for them — a hybrid of a little upfront context and a lot of on-demand retrieval. If you're weighing retrieval against retraining, our comparison of RAG vs. fine-tuning covers when each earns its keep.

The retrieval question is not "how do I get more into the window?" It's "how do I get the right small thing into the window at the right moment?" Just-in-time beats just-in-case.

Context budgets and context rot

Here is the fact that forces the discipline: model quality is not constant across the window. Independent testing by Chroma across 18 frontier models found that every one degrades as input length grows — the effect now widely called context rot. And it's not new: an earlier Stanford study found that with around 20 retrieved documents in context, accuracy on a retrieval task can fall from roughly 70–75% down to 55–60%, with the model losing information stranded in the middle of a long input.

So you set a context budget and spend it deliberately: rank and re-rank retrieved chunks by relevance, drop the low-signal ones, and put the most important material where the model attends best (the beginning and end, not buried in the middle). This is also where cost and quality align — the smaller, better-curated window is usually both cheaper and more accurate, which is why context engineering and context economics are two sides of one coin.

Structured tool results

When an agent calls a tool, whatever the tool returns lands in the context window — and a lot of tool output is noise. A raw API response might be 4,000 tokens of nested JSON when the agent needs three fields. Good context engineering shapes tool results before they hit the window: return the minimal structured payload, not the full dump; summarize or truncate large outputs and hand back a reference the agent can follow if it needs more; and format results consistently so the model can parse them reliably. The Model Context Protocol (MCP) has made this easier by standardizing how tools expose capabilities and return results — our MCP servers list is a map of the ecosystem — but the responsibility to keep tool output lean is yours.

A concrete example: a "search the database" tool that returns 50 full rows will bloat every subsequent turn with data the agent glanced at once. Return the 5 most relevant rows with just the columns that matter, plus a note that more are available. Same information value, a fraction of the tokens, and far less to distract the model on the next call.

Memory patterns

For agents that run across many turns — or many sessions — you cannot keep everything in the active window; it would overflow and rot. Two patterns solve this. Structured note-taking: the agent writes durable notes to external storage (a file, a memory store) and reads them back when relevant, so it can track progress on complex work without holding every detail in context. Compaction: when the conversation approaches the window limit, older turns are summarized into a compact form that preserves the important state and discards the transcript bulk. A related move is simply clearing stale tool results and completed reasoning that no longer inform the next step.

The pattern that makes long-horizon agents work is essentially externalized memory plus a lean working set: keep the active window small and high-signal, and lean on notes, summaries, and just-in-time reads for everything else. An agent that writes a clear artifact for its future self at the end of each session — what it did, what's left, what it learned — can make steady progress across a task far larger than any single context window. This is also exactly the behavior your agent evals should be checking, since good memory hygiene is invisible until it fails.

Anti-patterns

• Dumping everything "just in case." The biggest one. Stuffing the whole knowledge base, full chat history, and every tool schema into the window pays for tokens you don't need and triggers context rot. Retrieve the minimum instead.
• Too many tools. A long list of tool definitions is context you pay for on every call, and it can confuse the model's selection. Keep the tool set focused, or load tools on demand.
• Never evicting. Letting stale tool results and old reasoning accumulate turn after turn slowly poisons the window. Clear or compact what's no longer load-bearing.
• Burying the lede. Placing the most important instruction or fact in the middle of a long input, where models attend least. Put critical content at the edges.
• Raw tool output. Passing back full, unshaped API responses. Structure and trim them first.

The through-line of every one of these is the same principle stated at the top: fewer, better tokens beat more tokens. Context engineering is not a bag of tricks — it's the habit of asking, on every call, "does this token earn its place?" and evicting the ones that don't. Master that, and you'll build agents that stay sharp across long tasks while the ones that dump everything into a giant window quietly get slower, pricier, and worse.

Sources: Anthropic, "Effective context engineering for AI agents" and "Effective harnesses for long-running agents"; Chroma "context rot" evaluation across 18 frontier models; Liu et al., "Lost in the Middle" (Stanford); Sourcegraph's 2026 context-engineering guide. Product behaviors change — confirm current capabilities in each tool's documentation.

Common questions

What is context engineering? The practice of deliberately designing everything a model sees on each inference call — system prompt, retrieved documents, conversation history, tool definitions, tool results, and long-term memory — rather than only wording the instruction. Anthropic frames it as curating and maintaining the optimal set of tokens during inference.

How is it different from prompt engineering? Prompt engineering is about phrasing a single instruction. Context engineering owns the full token lifecycle across a long-running agent: what to retrieve, in what order, how to format tool results, what to remember, and what to evict when the window fills. For agents, wording is a small part of the job.

Does a bigger context window remove the need for it? No. Accuracy degrades as the window fills (context rot), and you pay for every token. The goal is the smallest set of high-signal tokens that gets the job done — exactly what context engineering optimizes.