Context Engineering: Fill the Window With Signal

Prompt engineering asks “what words do I write?” Context engineering asks the bigger question: what configuration of tokens is most likely to produce the behavior I want? Anthropic frames it as “the set of strategies for curating and maintaining the optimal set of tokens during inference” in its guide Effective context engineering for AI agents.

Context is everything the model sees on a call: the system prompt, tool definitions, the conversation history, retrieved documents, and any memory you inject. Two constraints make this hard. First, the window is finite. Second — and less obvious — a model's ability to use information degrades as the context grows longer and noisier, a pattern researchers call “context rot.” More tokens is not more intelligence.

So the job is not “stuff the window.” It's the opposite: assemble the smallest set of high-signal tokens that reliably gets the outcome. Every token you add competes with every other token for the model's attention. Treat context as a scarce, curated resource and most agent-reliability problems get easier.

Give your context a budget the way you'd give a page a word count. A rough split for an agent turn: a stable system prompt and tools up front, then just-in-time retrieved content, then compacted history, with generous headroom left for the model's own output. If any zone balloons, something else has to give.

Measure with real token counts, not vibes. Do not estimate Claude usage with OpenAI's tiktoken — it under-counts on other models and badly on code. Use the provider's own counter:

from anthropic import Anthropic
client = Anthropic()

n = client.messages.count_tokens(
    model="claude-opus-4-8",
    system=SYSTEM_PROMPT,
    messages=history,
).input_tokens
print("prompt tokens:", n)      # decide what to trim BEFORE you send

Long-context windows (many models now reach 1M tokens) do not make budgeting optional. You still pay per token, latency rises with input size, and attention still thins out. Budget first; fill deliberately. Want to see how much fits? Our context window visualizer makes the scale concrete.

Retrieval-augmented generation pulls relevant chunks from an external store and drops them into context. It's the right tool when your knowledge is large, changing, or private — product docs, a codebase, a customer's records — things the model was never trained on and that update faster than any fine-tune.

But RAG hurts when it's used reflexively. Retrieving ten mediocre chunks to answer a question already covered by the conversation just adds noise and invites the model to latch onto the wrong passage. Over-retrieval is a leading cause of confident-but-wrong answers.

• Chunk with meaning, not fixed byte counts — split on sections, not mid-sentence.
• Retrieve, then rerank. Pull a wide candidate set, then use a reranker to keep only the few chunks that truly match.
• Prefer just-in-time. Let an agent fetch on demand (a search tool) rather than pre-loading everything it might need.

candidates = vector_store.search(query, k=20)     # wide net
top = reranker.rerank(query, candidates, k=4)      # keep the best few
context = "\n\n".join(c.text for c in top)          # inject only these

Not sure whether to retrieve or fine-tune in the first place? Work through RAG vs fine-tune before you build.

Caching is a prefix match: the provider caches the front of your prompt so repeated calls skip re-processing it, cutting cost and latency dramatically. The one rule that governs everything — any byte change anywhere in the prefix invalidates the cache from that point on. So put stable content first (a frozen system prompt, a deterministic tool list) and volatile content (timestamps, the user's new question) last.

client.messages.create(
    model="claude-opus-4-8",
    system=[{"type": "text", "text": BIG_STABLE_PROMPT,
             "cache_control": {"type": "ephemeral"}}],   # cache the prefix
    messages=[{"role": "user", "content": todays_question}],  # volatile, last
)

Verify it's working by reading usage.cache_read_input_tokens. If it's zero across identical-prefix calls, a silent invalidator is at work — a datetime.now() in the system prompt, an unsorted JSON dump, or a tool set that varies per user. See Anthropic's prompt caching docs for placement patterns; OpenAI and Google offer similar prefix/context caching. The mental model is identical across all three: freeze the front, vary the back.

When a tool returns data to the model, that data becomes context — and most tools return far too much of it. A tool that dumps a 5,000-row query result, or a full HTML page, buries the three fields the model actually needed under thousands of tokens of noise.

Design tool outputs like an API you'd hand a junior engineer: return a compact, structured shape with only the fields the next step needs.

// Noisy: the whole record, nested and verbose
{"user": {"id": 42, "name": "...", "address": {...}, "orders": [ ...200 rows... ]}}

// Curated: just what the model needs to decide the next action
{"user_id": 42, "plan": "pro", "open_tickets": 1, "last_order_days_ago": 3}

For genuinely large results, return a handle instead of the payload — a file path, a row count, a summary — and let the model ask for detail only if it needs it. Keep the schema consistent across calls so the model learns the shape, and truncate long text fields with an explicit marker so the model knows data was omitted.

An agent that runs for many turns will overflow any window if it just keeps appending. Three patterns, used together, keep it coherent:

• In-window history — the recent turns, kept verbatim. Cheapest, but bounded.
• Compaction — summarize older turns into a short digest when the window fills, preserving decisions and discarding chatter.
• External memory — write durable notes to a file or store outside the window, then retrieve them just-in-time. Anthropic ships a memory tool and context editing that clears stale tool results for exactly this.

# Write once, recall on demand -- don't keep it all in the window
memory.write("user_prefs.md", "Prefers Python. Timezone US/Central.")
# ...many turns later...
notes = memory.read("user_prefs.md")   # pulled back in only when relevant

The principle: the window holds what's active; everything else lives outside and is fetched when needed.

Almost every context bug is one of these. Learn the smell of each:

• Context rot / lost-in-the-middle — the model ignores facts buried in a long context. Fix: trim, put the most important content at the edges, retrieve just-in-time.
• Distraction — irrelevant retrieved chunks pull the answer off course. Fix: rerank and cut top-k.
• Tool-result bloat — a verbose tool output crowds out everything else. Fix: structured, truncated results (Lesson 5).
• Silent cache misses — a moving value in the prefix costs you every cache hit. Fix: freeze the prefix; check cache_read tokens.
• Window overflow — the agent silently drops early turns. Fix: compaction plus external memory.