Context Engineering for Production AI Agents

Prompt engineering was always a workaround. You couldn't control what the model knew, so you controlled how you asked. But when you move from a single-turn chatbot to a production agent that accumulates tool outputs, retrieves documents, maintains conversation history, and hands state to downstream subagents, the words in your system prompt become the least important variable in the equation. What the model sees — the full assembled context at every inference call — is what determines output quality, latency, and cost.

Context engineering is the discipline of deliberately constructing that assembled context: what gets included, in what order, at what priority, and what gets evicted when the window fills. In document-heavy agent systems, restructuring the context assembly layer can change groundedness more than another prompt rewrite because the prompt is not the evidence. The architecture decides which evidence reaches the model at all.

Why Prompt Engineering Fails at Agent Scale

Prompt engineering assumes a relatively static input: a user query plus a fixed system instruction. In an agentic system, the input is dynamic, accumulating, and often adversarial to your token budget. By the third tool call in a multi-step research agent, the "context" reaching the model might include a system prompt, three prior conversation turns, four tool outputs of variable length, and two retrieved document chunks — assembled in whatever order your scaffolding appended them. If you haven't made deliberate choices about that assembly, you've made accidental ones.

The failure modes that prompt engineering cannot fix:

• Context poisoning — a low-relevance retrieved chunk that contradicts the high-relevance chunk, causing the model to hedge or hallucinate a synthesis
• Silent truncation — when the context window fills, most frameworks truncate from the front, silently dropping system instructions
• State bleed — tool outputs from prior agent steps leaking into the context of a specialist subagent that has no business seeing them
• Positional degradation — the documented "lost in the middle" phenomenon where models under-attend to information placed in the middle of long contexts (Nelson et al., "Lost in the Middle", 2023)

These are structural problems. A better-worded instruction cannot fix a structurally corrupted context window. For a deeper look at how stateful agent architectures create these pressures, our LangGraph state management guide covers the checkpoint layer that makes context scoping tractable.

Context window overflow is not a token budget problem — it's an architecture problem. Agents without explicit eviction policies silently drop system instructions when windows fill, producing unpredictable outputs with no error signal.

The Four Context Layers Every Production Agent Needs

A production agent's context is not a single string. It's composed of four structurally distinct layers that must be managed independently, with explicit priority ordering that governs what gets evicted under token pressure. Treating them as a flat concatenation is the most common context engineering mistake we encounter in inherited codebases.

Diagram 1: The four context layers assembled at each agent node — instruction, episodic memory, retrieved knowledge, and tool state — with explicit priority ordering for eviction.

Our production RAG pipeline checklist covers the reranking and deduplication implementations in detail. The key integration point for context engineering: the reranker score becomes your relevance gate threshold, and that threshold is a tunable parameter per agent role — a validation agent needs higher confidence than an exploratory research agent.

from sentence_transformers import CrossEncoder
from dataclasses import dataclass


RERANKER = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

@dataclass
class RetrievedChunk:
    content: str
    source: str
    similarity_score: float
    rerank_score: float = 0.0


def gate_retrieval_context(
    query: str,
    raw_chunks: list[RetrievedChunk],
    relevance_threshold: float = 0.3,
    max_chunks: int = 5,
    dedup_threshold: float = 0.92,
) -> list[RetrievedChunk]:
    """
    Apply reranking, relevance gating, and deduplication to raw retrieval results.
    relevance_threshold: minimum reranker score to pass the gate (0.3 = moderate confidence)
    dedup_threshold: cosine similarity above which two chunks are considered duplicates
    """
    if not raw_chunks:
        return []

    # Rerank
    pairs = [(query, chunk.content) for chunk in raw_chunks]
    scores = RERANKER.predict(pairs)
    for chunk, score in zip(raw_chunks, scores):
        chunk.rerank_score = float(score)

    # Gate by relevance
    gated = [c for c in raw_chunks if c.rerank_score >= relevance_threshold]
    gated.sort(key=lambda c: c.rerank_score, reverse=True)

    # Deduplicate: keep highest-scoring, skip near-duplicates
    # Simplified: in production use embedding cosine similarity for dedup
    seen_sources: set[str] = set()
    deduplicated: list[RetrievedChunk] = []
    for chunk in gated:
        # Source-level dedup as minimum viable implementation
        if chunk.source not in seen_sources:
            deduplicated.append(chunk)
            seen_sources.add(chunk.source)
        if len(deduplicated) >= max_chunks:
            break

    return deduplicated

Retrieval context gating — reranking, relevance thresholding, and deduplication applied before context assembly — addresses the structural root cause of hallucination in RAG agents more directly than any instruction-level prompt modification.

Tool Schema Management and Sparse Tool Activation

Tool definitions consume context before the agent retrieves evidence or calls a tool. Registering the full catalog on every call spends that budget on capabilities the current node may never use. Scope tool schemas alongside the retrieved knowledge and tool state layers.

For a LangGraph workflow with explicit research, execution, and verification phases, bind a tool set to each phase. A research node receives search and document retrieval tools; an execution node receives the operations needed for its task; a verification node receives read-only inspection tools. The executor must still enforce permissions independently of which schemas the model sees.

When task phases cannot determine the tool set, retrieve tool definitions by semantic similarity or use a classifier before the reasoning call. Measure missed-tool selections and added routing latency before adopting either approach. A smaller catalog is useful only if the agent can still select the operation it needs.

Compress descriptions by removing repeated prose and redundant examples. Retain purpose, parameter constraints, output shape, and distinctions between similar tools. Log active tool names and schema token counts with each node call so a registry change cannot silently consume the retrieval budget.

Model Routing by Context Density

Once you treat context as a structured artifact, model selection becomes a context-aware routing decision rather than a global configuration. Not every agent node requires the same model, and the cost difference is not marginal. A fast lower-cost model is usually a better fit for short, structured retrievals and classification steps where context is dense but decision complexity is low. A higher-reasoning model belongs on multi-document synthesis steps where the assembled context is large, contradictory, or requires multi-hop reasoning.

The routing key is context density score: the ratio of retrieved knowledge tokens to total context tokens. When a node's context is dominated by retrieved documents requiring synthesis, route to the higher-reasoning model. When it is primarily structured state and a short query, route to the fast model. This heuristic requires no ML model — it is computable from token counts before the inference call.

Expert Insight: Set your context budget as a fraction of model maximum, not an absolute number When you swap models, the context budget should auto-scale as a fraction of the new model's window, not remain as an absolute token count. Resolve the budget from model_max_tokens at routing time and reserve explicit headroom. This prevents the budget from silently becoming a hard ceiling when you route to a smaller model mid-deployment.

Prompt Caching for Stable Context Prefixes

Model routing changes which model processes a request. Prompt caching addresses repeated processing of an unchanged prompt prefix. Keep stable system instructions before conversation history, current task state, and variable tool results so changes late in the request do not invalidate the reusable prefix.

For Anthropic integrations that use explicit cache controls, place cache boundaries around stable content. OpenAI's automatic prefix caching likewise depends on a matching prefix. Keep tool definitions in a deterministic order within each active tool set. A change to the selected set can change the prefix, so measure caching alongside sparse tool activation rather than assuming their cost effects add together.

Only include retrieved documents in a reusable prefix when the task needs them and their content and ordering remain stable. Do not reorder relevance-ranked evidence or retain stale documents just to improve cache reuse. Minimum eligible lengths, retention, and billing rules depend on the provider and model; use the applicable API contract rather than a universal threshold.

Record cache reads, cache writes where exposed, uncached input, and request cost in the same trace. Compare repeated workloads before claiming savings. Caching does not reduce the context window occupied by the prompt, so the existing budget and eviction rules still apply.

What Breaks at Scale

The patterns above are sound at moderate concurrency. At scale — hundreds of concurrent agent sessions, long-running tasks measured in hours, or multi-agent graphs with 8+ specialist nodes — several failure modes emerge that the architecture must anticipate.

Checkpoint store latency under concurrent load. LangGraph's Redis-backed checkpoint store can add visible latency when many agent sessions hydrate state at once. The fix: pre-warm the checkpoint store during pod startup, and use read replicas for state hydration in supervisor nodes. Do not rely on the default single-node Redis configuration in production.

Eviction policy drift across agent versions. When you deploy a new version of an agent node with a different context schema, in-flight sessions in the checkpoint store may have messages that the new eviction policy misclassifies. We maintain a schema version field in AgentState and apply migration logic in the state loader when the version doesn't match. Skipping this causes the eviction policy to silently remove the wrong layer during the rollout window.

Reranker latency at high chunk volume. Cross-encoder rerankers can become synchronous bottlenecks when every agent call waits on large chunk batches. In production, run the reranker as a dedicated async service with its own latency budget, separate from the agent's main inference path. Agents that fail to get a reranker response within that budget should fall back to raw-similarity retrieval with a warning log — not a hard failure.

Context injection by adversarial tool outputs. If your agent calls external APIs and injects those outputs into Layer 4, a malicious or misconfigured external service can inject content designed to override Layer 1 instructions — the classic prompt injection via retrieved content. For agents operating in high-trust environments, Layer 4 tool outputs must be sanitized before context assembly. This is not a theoretical risk; for a deeper treatment of the threat model, our self-correcting agent architecture guide covers validation loops that catch this class of failure.

Multi-agent systems with 8+ specialist nodes require dedicated async reranker infrastructure, checkpoint store pre-warming, and schema-versioned state migration — or context engineering degrades silently as concurrency increases.

Production Anti-Patterns and How to Detect Them

Concurrency failures are only part of the operational picture. Context can also degrade within a session or across routine prompt edits. Instrument the assembly path before the model call so the trace shows what changed and which layer consumed the budget.

Unbounded scratchpad growth. Track tool-state tokens by node and turn in LangSmith or OpenTelemetry spans. If completed tool calls keep increasing the next request's input, retain their decision-relevant results and source references, and store raw responses outside the prompt. Give database and API output formatters explicit limits before their results enter agent state.

System prompt drift. Count instruction tokens when reviewing prompt changes and fail a local or CI check when the configured instruction budget is exceeded. This catches incremental additions that consume space reserved for retrieved evidence. Review the new instruction's purpose before increasing the allocation.

Per-turn cost blindness. Emit instruction, tool-schema, retrieved-context, history, and tool-output token counts separately, together with agent version, node, and eviction events. Aggregate by workflow as well as session. A rising retrieval layer calls for a different repair from a growing scratchpad, even when both produce a larger request. Keep the measurements in traces without copying sensitive document or tool-response bodies into metrics.

The decision rule

Treat context as an architecture artifact when the agent's failures come from missing, stale, contradictory, or overlong evidence. If the same prompt behaves differently as retrieved context changes, the next fix belongs in assembly, gating, routing, or eviction policy, not in another instruction rewrite.