Context Engineering for Production AI Agents

Prompt engineering was always a workaround. You couldn’t control what the model knew, so you controlled how you asked. But when you move from a single-turn chatbot to a production agent that accumulates tool outputs, retrieves documents, maintains conversation history, and hands state to downstream subagents, the words in your system prompt become the least important variable in the equation. What the model sees — the full assembled context at every inference call — is what determines output quality, latency, and cost.

Context engineering is the discipline of deliberately constructing that assembled context: what gets included, in what order, at what priority, and what gets evicted when the window fills. In a financial document processing pipeline we deployed processing roughly 4,000 pages per day, restructuring the context assembly layer — without changing the system prompt — reduced hallucinated citations from approximately 11% to under 2%. The prompt didn’t change. The architecture did.

Why Prompt Engineering Fails at Agent Scale

Prompt engineering assumes a relatively static input: a user query plus a fixed system instruction. In an agentic system, the input is dynamic, accumulating, and often adversarial to your token budget. By the third tool call in a multi-step research agent, the “context” reaching the model might include a system prompt, three prior conversation turns, four tool outputs of variable length, and two retrieved document chunks — assembled in whatever order your scaffolding appended them. If you haven’t made deliberate choices about that assembly, you’ve made accidental ones.

The failure modes that prompt engineering cannot fix:

• Context poisoning — a low-relevance retrieved chunk that contradicts the high-relevance chunk, causing the model to hedge or hallucinate a synthesis
• Silent truncation — when the context window fills, most frameworks truncate from the front, silently dropping system instructions
• State bleed — tool outputs from prior agent steps leaking into the context of a specialist subagent that has no business seeing them
• Positional degradation — the documented “lost in the middle” phenomenon where models under-attend to information placed in the middle of long contexts (Nelson et al., “Lost in the Middle”, 2023)

These are structural problems. A better-worded instruction cannot fix a structurally corrupted context window. For a deeper look at how stateful agent architectures create these pressures, our LangGraph state management guide covers the checkpoint layer that makes context scoping tractable.

Context window overflow is not a token budget problem — it’s an architecture problem. Agents without explicit eviction policies silently drop system instructions when windows fill, producing unpredictable outputs with no error signal.

The Four Context Layers Every Production Agent Needs

A production agent’s context is not a single string. It’s composed of four structurally distinct layers that must be managed independently, with explicit priority ordering that governs what gets evicted under token pressure. Treating them as a flat concatenation is the most common context engineering mistake we encounter in inherited codebases.

Diagram 1: The four context layers assembled at each agent node — instruction, episodic memory, retrieved knowledge, and tool state — with explicit priority ordering for eviction.

Our production RAG pipeline checklist covers the reranking and deduplication implementations in detail. The key integration point for context engineering: the reranker score becomes your relevance gate threshold, and that threshold is a tunable parameter per agent role — a validation agent needs higher confidence than an exploratory research agent.

from sentence_transformers import CrossEncoder
from dataclasses import dataclass


RERANKER = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

@dataclass
class RetrievedChunk:
    content: str
    source: str
    similarity_score: float
    rerank_score: float = 0.0


def gate_retrieval_context(
    query: str,
    raw_chunks: list[RetrievedChunk],
    relevance_threshold: float = 0.3,
    max_chunks: int = 5,
    dedup_threshold: float = 0.92,
) -> list[RetrievedChunk]:
    """
    Apply reranking, relevance gating, and deduplication to raw retrieval results.
    relevance_threshold: minimum reranker score to pass the gate (0.3 = moderate confidence)
    dedup_threshold: cosine similarity above which two chunks are considered duplicates
    """
    if not raw_chunks:
        return []

    # Rerank
    pairs = [(query, chunk.content) for chunk in raw_chunks]
    scores = RERANKER.predict(pairs)
    for chunk, score in zip(raw_chunks, scores):
        chunk.rerank_score = float(score)

    # Gate by relevance
    gated = [c for c in raw_chunks if c.rerank_score >= relevance_threshold]
    gated.sort(key=lambda c: c.rerank_score, reverse=True)

    # Deduplicate: keep highest-scoring, skip near-duplicates
    # Simplified: in production use embedding cosine similarity for dedup
    seen_sources: set[str] = set()
    deduplicated: list[RetrievedChunk] = []
    for chunk in gated:
        # Source-level dedup as minimum viable implementation
        if chunk.source not in seen_sources:
            deduplicated.append(chunk)
            seen_sources.add(chunk.source)
        if len(deduplicated) >= max_chunks:
            break

    return deduplicated

Retrieval context gating — reranking, relevance thresholding, and deduplication applied before context assembly — addresses the structural root cause of hallucination in RAG agents more directly than any instruction-level prompt modification.

Model Routing by Context Density

Once you treat context as a structured artifact, model selection becomes a context-aware routing decision rather than a global configuration. Not every agent node requires the same model, and the cost difference is not marginal. In our deployments, a fast lower-cost model handles short, structured retrievals and classification steps where context is dense but decision complexity is low. A higher-reasoning model handles multi-document synthesis steps where the assembled context is large, contradictory, or requires multi-hop reasoning.

The routing key is context density score: the ratio of retrieved knowledge tokens to total context tokens. When a node’s context is more than 60% retrieved documents requiring synthesis, route to the higher-reasoning model. When it’s primarily structured state and a short query, route to the fast model. This heuristic requires no ML model — it’s computable from token counts before the inference call.

Expert Insight: Set your context budget as a fraction of model maximum, not an absolute number When you swap models, the context budget should auto-scale as a fraction of the new model’s window, not remain as an absolute token count. Build your CONTEXT_BUDGET_TOKENS constant as model_max_tokens * 0.70 resolved at routing time. This prevents the budget from silently becoming a hard ceiling when you route to a smaller model mid-deployment.

What Breaks at Scale

The patterns above are sound at moderate concurrency. At scale — hundreds of concurrent agent sessions, long-running tasks measured in hours, or multi-agent graphs with 8+ specialist nodes — several failure modes emerge that the architecture must anticipate.

Checkpoint store latency under concurrent load. LangGraph’s Redis-backed checkpoint store begins to add measurable latency at high concurrency. In a 12-agent pipeline we measured p99 state hydration of 4 seconds when the checkpoint store wasn’t pre-warmed, with cold-start reads dominating. The fix: pre-warm the checkpoint store during pod startup, and use read replicas for state hydration in supervisor nodes. Do not rely on the default single-node Redis configuration in production.

Eviction policy drift across agent versions. When you deploy a new version of an agent node with a different context schema, in-flight sessions in the checkpoint store may have messages that the new eviction policy misclassifies. We maintain a schema version field in AgentState and apply migration logic in the state loader when the version doesn’t match. Skipping this causes the eviction policy to silently remove the wrong layer during the rollout window.

Reranker latency at high chunk volume. The cross-encoder/ms-marco-MiniLM-L-6-v2 reranker we reference above takes roughly 40-80ms per batch of 20 pairs on CPU. At high request rates this becomes a synchronous bottleneck. In production we run the reranker as a dedicated async microservice with a 200ms SLA, separate from the agent’s main inference path. Agents that fail to get a reranker response within the SLA fall back to top-5 by raw similarity with a warning log — not a hard failure.

Context injection by adversarial tool outputs. If your agent calls external APIs and injects those outputs into Layer 4, a malicious or misconfigured external service can inject content designed to override Layer 1 instructions — the classic prompt injection via retrieved content. For agents operating in high-trust environments, Layer 4 tool outputs must be sanitized before context assembly. This is not a theoretical risk; for a deeper treatment of the threat model, our self-correcting agent architecture guide covers validation loops that catch this class of failure.

Multi-agent systems with 8+ specialist nodes require dedicated async reranker infrastructure, checkpoint store pre-warming, and schema-versioned state migration — or context engineering degrades silently as concurrency increases.

Engineer Intelligence with ActiveWizards

Building production AI agents where hallucination rates, context overflow, or inference costs are blocking deployment? Our team has shipped context-engineered multi-agent systems across document processing, financial analytics, and enterprise search — and we can architect yours.

Contact Us for a Consultation