LLM Context Engineering: Design Prompts That Use Every Token

Context engineering is the discipline of treating the context window as a fixed budget you allocate deliberately — deciding what information enters, in what order, at what compression level, and for how long. It sits one level above prompt engineering: instead of asking "how should I phrase this instruction?", it asks "what should the model see at all, and what should it never see?" A well-engineered context improves output quality by 20-40% on knowledge tasks, cuts input token costs by 30-60%, and reduces hallucination by keeping only relevant information in view. This guide covers budget allocation, information hierarchy, compression, and dynamic assembly — with production code.

The Context Budget: Know Your Numbers

Every model has a context window — 128K for GPT-5, 200K for Claude 4, 2M for Gemini 2.5 Pro — but the useful budget is smaller than the window for three reasons: output tokens consume the same window, long contexts degrade middle recall, and every redundant token costs money. Treat the window as a budget with lines:

def budget_for(model_window=128000, max_tokens=1000):
    output = max_tokens
    safety = int(model_window * 0.05)
    system = int(model_window * 0.10)
    knowledge = int(model_window * 0.55)
    history = model_window - output - safety - system - knowledge
    return {"output": output, "safety": safety, "system": system,
            "knowledge": knowledge, "history": history}
# 128K window: ~13K system, ~70K knowledge, ~37K history, 6.4K safety

The explicit budget turns "just stuff everything in" into a deliberate allocation — and gives you a target for compression when you're over.

Information Hierarchy: What Goes Where

Order matters more than most teams realize. Models attend to the start and end of context most reliably ("lost in the middle" effect), so place information by priority:

1. Start (highest recall): system instructions, critical constraints, task definition. This is the last thing the model "forgets" in long contexts.
2. Middle (weakest recall): bulk knowledge, retrieved documents, reference material. Retrieval quality here depends on the model's long-context handling — Claude's 200K handles middle placement measurably better than most.
3. End (strong recall): the current question, recent conversation turns, immediate task data. The model reads these with full attention.

def assemble_context(system, knowledge, history, question):
    return [
        {"role": "system", "content": system},           # START: rules
        {"role": "user", "content": "CONTEXT:\n" + knowledge},  # MIDDLE: bulk
        *history[-6:],                                    # END: recent turns
        {"role": "user", "content": question},            # END: the ask
    ]

Compression Strategies: Making More Fit

1. Instruction compression

Verbose system prompts are the easiest win. A 2,000-token instruction set often compresses to 600 tokens with identical behavior — remove examples that duplicate each other, merge redundant constraints, and use terse imperative phrasing.

2. Conversation summarization

Old turns become a rolling summary instead of raw text:

def compress_history(history, budget=30000, mini_client=None):
    """Keep recent turns raw; summarize everything older."""
    recent, old = history[-8:], history[:-8]
    if not old:
        return history
    summary = mini_client.chat.completions.create(
        model="gpt-5-mini",  # cheap model does the compression
        messages=[{"role": "user", "content":
            "Summarize key facts, decisions, and user preferences "
            "from this conversation in under 400 tokens: " + str(old)}])
    return [{"role": "system", "content": "Earlier context: " +
             summary.choices[0].message.content}] + recent

This is the single highest-leverage context technique: a 100-turn conversation collapses from 60K tokens to ~1.5K while preserving everything that matters.

3. Knowledge chunking and retrieval

Don't stuff a whole document into context — retrieve the relevant slice. The RAG guide covers chunking and retrieval in depth; the context-engineering takeaway is: retrieved context should be the answer-relevant 500-2,000 tokens, not the whole corpus.

4. Deduplication

Multi-turn apps that re-send full document context each turn burn 5-10x input tokens. Send the document once, then reference it ("per the document in CONTEXT, section 3...").

Dynamic Assembly: Context as a Function of the Request

Static prompts waste budget on irrelevant knowledge. Dynamic assembly decides what enters context per request:

def build_dynamic_context(request, user_profile, kb_retriever):
    parts = []
    # 1. Always: core instructions (compressed, ~400 tokens)
    parts.append(SYSTEM_CORE)
    # 2. Always: user profile facts (only durable facts, ~200 tokens)
    if user_profile:
        parts.append(f"USER FACTS: {format_facts(user_profile)}")
    # 3. Conditional: knowledge — only if the request needs it
    if needs_knowledge(request):
        chunks = kb_retriever.top_k(request.query, k=4)
        parts.append(f"CONTEXT:\n{join(chunks)}")
    # 4. Always: recent turns (compressed) + the question
    parts.append(compress_history(user_profile.history))
    parts.append(f"QUESTION: {request.text}")
    return {"role": "user", "content": "\n\n".join(parts)}

The conditional step is where most teams overspend: a "summarize my calendar" request does not need your entire product knowledge base.

Prompt Caching: Engineering for the 50-90% Discount

Prefix caching rewards stable context structure. Providers and gateways cache repeated prompt prefixes at 50-90% input discount — but only if the prefix is byte-identical across requests. Engineering implications:

• Put everything stable (system prompt, static knowledge, user facts) at the START — it becomes the cached prefix
• Put everything variable (question, new turns) at the END — cache hits survive
• Never interleave variable content into the stable prefix
• If your knowledge base is static per user, keep it in the prefix and update it rarely (bump a version token when it changes)

# CACHE-FRIENDLY order (stable prefix first):
messages = [
    {"role": "system", "content": SYSTEM_CORE},      # stable
    {"role": "system", "content": USER_KB},          # stable per user
    {"role": "user", "content": question},           # variable — cache still hits
]
# CACHE-BREAKING order (variable first):
messages = [
    {"role": "user", "content": question},           # changes every call
    {"role": "system", "content": SYSTEM_CORE},      # prefix now never matches
]

On a 30K-token system prefix with 10K requests/day, the cache-friendly layout saves roughly 60% of input spend versus the naive order — see the prompt caching guide for the full math.

Measuring Context Quality

You can't improve what you don't measure. Instrument three numbers per request:

• Context utilization — tokens actually in context vs window (target: 60-85% of the budget used; under 50% means you're either wasting window or could afford richer knowledge)
• Relevant-context ratio — estimated share of context tokens that answer the query (sample with an LLM judge: "what fraction of this context is relevant?")
• Input token trend — per-request input tokens over time; a rising trend means context bloat (summarization backlog, growing knowledge)

# Log per request — pair with the observability guide
log_metric("context_utilization", input_tokens / window)
log_metric("relevant_ratio", judge_relevance(messages))  # sampled
log_metric("input_tokens", input_tokens)

Context Engineering Checklist

1. Set explicit budget lines (system/knowledge/history/output/margin)
2. Place critical instructions at the start; current question at the end
3. Compress system prompts to essentials; summarize old conversation turns
4. Retrieve knowledge conditionally — never dump the corpus
5. Structure prompts for prefix caching (stable first, variable last)
6. Deduplicate re-sent context across turns
7. Measure utilization, relevance ratio, and token trends

Context engineering compounds: every token you remove from input is cost saved on every request, every well-placed instruction is quality gained on every response. For the adjacent disciplines, read the context window guide (which window fits which job), the prompt optimization guide (phrasing), and the response caching guide (eliminating repeat calls entirely). Build and measure your context pipeline with a free DrAI key at ai.dr-ai.top/signin — pricing at dr-ai.top/pricing.