LLM Prompt Caching: Cut Input Costs by 50% with Prefix Caching

If your application sends the same system prompt, tool definitions, or few-shot examples with every request, you are paying full price to reprocess those tokens every single time. LLM prompt caching — specifically prefix caching — lets providers reuse the expensive computation for the static part of your prompt and charge you a fraction of the input rate. Teams that structure their prompts for cacheability routinely cut input costs by 50% or more, and in multi-turn conversations the savings reach 90%. This guide explains how prefix caching works under the hood, the exact conditions that produce a cache hit, how to design prompts so the static prefix stays intact, and where prompt caching ends and response caching begins.

How LLM Prefix Caching Works

When a model processes your prompt, it does not read it as text — it converts every token through the model's layers and caches the intermediate state (the key-value cache) so that generation can continue token by token. Prefix caching exploits a simple observation: if a new request starts with the exact same sequence of tokens as a previous request, the expensive part — the forward pass over those tokens — is already done. The provider stores the KV cache for recently seen prefixes, and when your next request begins with a matching prefix, it reuses that computation and only processes the new suffix.

# Request 1 (full price for everything)
[system prompt] + [few-shot examples] + [user: "What is RAG?"]
# Request 2 (user 2)  -- prefix matches, only the suffix is computed
[system prompt] + [few-shot examples] + [user: "What is fine-tuning?"]
#                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#                        cached: billed at the discounted cache-read rate

Two properties matter. First, the match is on exact tokens — not semantic similarity. Adding a space, reordering a sentence, or inserting a dynamic timestamp in the middle of the static block breaks the match at that point, and everything before it (if it still matches) is cached, but the block after the change is recomputed. Second, the cache is scoped: providers typically cache per account, per organization, or per project, so two different users of your application do not share cached prefixes with each other — each user's session builds its own cache entries as it goes.

Cache Hit Conditions: What Actually Produces a Hit

Every provider implements prompt caching slightly differently, but the hit conditions converge on a few rules that are worth internalizing:

The minimum-length rule is the one most teams miss. If your static prefix (system prompt + tools + examples) is shorter than the provider's threshold, you never get a hit — the entire prompt is billed at the normal input rate. This is common with minimal prompts, and it explains why some teams see zero savings from prompt caching until they move to a more detailed system prompt or move their few-shot examples into the shared prefix.

Designing Prompts for Cacheability

Cacheability is a prompt-engineering constraint: put everything static before everything dynamic. The canonical prompt layout for a cached application:

1. SYSTEM: full system prompt (static)      <-- cached
2. SYSTEM: tool/function definitions (static) <-- cached
3. USER: few-shot examples (static)          <-- cached
4. USER: current user message (dynamic)      <-- computed
5. ASSISTANT: prior turns (session-scoped)   <-- cache-friendly

In practice, that means:

• Never interpolate dynamic data into the system prompt. Things like the current date, the user's name, or a request ID look harmless, but they force a cache miss at exactly that position. If you must include a date, put it at the end of the static block or in the user turn.
• Keep tool definitions and few-shot examples byte-identical across requests. Same order, same whitespace, same formatting. Sort your examples by a stable key so they do not reshuffle between calls.
• Move rarely-changing context (brand voice, safety rules, formatting instructions) into the system prompt rather than injecting it per request.
• Version the prefix deliberately. When you do change the system prompt, change it wholesale and accept one full-price request to rebuild the cache; then all subsequent requests hit again.

# BAD - dynamic value inside the prefix, cache breaks on every user
messages = [
  {"role": "system", "content": f"You are a support agent for {company}. Today is {date}."},
  {"role": "user", "content": user_text},
]
# GOOD - static prefix intact, dynamic content in the user turn
messages = [
  {"role": "system", "content": "You are a support agent. Follow the style guide below."},
  {"role": "user", "content": f"Company: {company}
Today's date: {date}

Question: {user_text}"},
]

A useful mental model: the prefix is a compilation artifact. Treat it the way you treat a build output — deterministic, versioned, and changed only by explicit rebuild.

The Math: What 50–90% Savings Look Like

Providers bill cache reads at roughly 10–25% of the normal input rate (the exact discount varies by provider and model). Because input tokens dominate most LLM workloads — a 2,000-token system prompt with 200 output tokens is 90% input — the savings compound quickly. A concrete example on a typical mid-range model pricing of $1.50 per 1M input tokens and $6 per 1M output tokens:

The multi-turn row is where caching shines: the conversation history grows with every turn, and all of it except the newest message is a cacheable prefix. At 1M requests per month, the chat-app scenario above goes from about $5,100 to $2,500 — a 51% cut. The agent scenario saves 57%, and a heavily multi-turn workload lands closer to 70–90% depending on how much of the history is stable between calls. Remember the write side too: the first request after a cache miss (or TTL expiry) is billed at the full input rate, so savings are realized per user session, not from request one.

Prompt Caching vs Response Caching

Prefix caching and response caching solve different problems, and teams frequently confuse them:

Use prefix caching for anything with a long stable prefix — assistants, agents, RAG pipelines with a fixed instruction block. Use response caching for exact-duplicate requests: FAQ-style endpoints, documentation queries, and any question where "the same answer every time" is the correct behavior. The two compose: response cache first, then prompt caching for the misses. Our response caching guide covers the second layer in depth.

Caching in Multi-Turn Conversations

Conversation history is the highest-value caching surface because it is long, mostly stable, and expensive to reprocess. The pattern: keep the conversation as a single growing prefix and always append the newest user message and assistant reply at the end. Each new turn re-sends the full history, matches the cached prefix, and pays only for the new suffix:

# Turn 5: the first 4 turns are a cached prefix
messages = [
  system_prompt,            # static, cached
  t1_user, t1_assistant,    # turn 1, cached
  t2_user, t2_assistant,    # turn 2, cached
  t3_user, t3_assistant,    # turn 3, cached
  t4_user, t4_assistant,    # turn 4, cached
  t5_user,                  # new - the only full-price tokens
]

Two rules keep the pattern intact. First, never rewrite old turns — if you edit turn 2's text, everything from turn 2 onward becomes a cache miss. This includes trimming whitespace or normalizing quotes; normalize at write time, not read time. Second, think about what your summarization does to the cache (see our chat history guide for the full design): when you compress old turns into a summary, the prefix changes and you lose the cache for the summary block — which is exactly why many teams keep the last N raw turns as a cached tail and summarize only what falls outside it.

There is a nuance when history crosses the context limit: naive truncation — dropping the oldest turns — preserves the prefix from the system prompt onward, which is good for the cache, but it also silently deletes decisions the summary never captured. Cache-aware truncation instead keeps the stable prefix intact and moves the boundary by summarizing the dropped turns into the summary block at the end of the cached region, so subsequent requests still hit. The trade-off is engineering complexity; for most products, dropping old turns and accepting the smaller context is the right call, because the cache hit still covers the large static portion of the prompt.

Gotchas and Failure Modes

• Dynamic content in the prefix. Timestamps, user names, or request IDs injected into the system prompt silently defeat caching. Profile your requests and confirm the prefix is byte-stable.
• Cache rebuild spikes. After every TTL expiry or prefix change, the next request pays full price. At high volume, schedule prefix changes (prompt deploys) during low traffic.
• Per-user cache scope. Savings scale with session length, so short-lived anonymous sessions benefit less. Long-lived logged-in sessions are where the wins are.
• Cache writes vs reads. Some providers price a cache "write" at full input rate when a new prefix is stored; the discount applies to subsequent reads. Budget for the first request of every session.
• Provider differences. Minimum prefix length, TTL, and discount rates differ across providers and models. Read the provider docs and verify with usage logs — do not assume your gateway applies the same rules.
• Prompt injection via cached content. Caching does not change what the model reads; it only changes billing. Treat the cached prefix with the same trust boundaries as any other prompt content.

Measuring Cache Hit Rate

You cannot optimize what you do not measure. Every major provider exposes cache usage in the response's usage object — typically cached_tokens and the discount applied — and OpenAI-compatible gateways such as DrAI surface the same fields. Log them per request and track three numbers: the cache hit rate (cached tokens / total input tokens), the effective input price per 1M tokens (which should sit well below the list price), and the dollar savings versus no caching. Alert when the hit rate drops — a sudden drop usually means a prompt change, a session-handling bug, or a provider TTL change. A healthy chat workload runs a 60–90% hit rate; if yours is below 30%, the prompt structure or session handling is the problem, not the provider.

The Prompt Caching Checklist

1. Confirm your provider supports prefix caching on your model and account tier
2. Move all static content (system prompt, tools, examples) ahead of dynamic content
3. Verify the static prefix is byte-identical across requests — test it in your test suite
4. Check your prefix length clears the provider's minimum (typically 1k–2k tokens)
5. Never interpolate dates, names, or IDs into the cached region
6. Use a stable, versioned prompt template; change it wholesale, not incrementally
7. Keep multi-turn histories append-only; normalize at write time
8. Log cached_tokens per request and track hit rate and effective input price
9. Pair prefix caching with response caching for exact-duplicate queries
10. Budget for cache-write requests at full price per session

Prompt caching is the cheapest optimization most LLM applications are missing. It requires no model changes, no prompt rewrite of your product logic — just discipline about where static content lives in your request. Combined with response caching and cost-aware model routing, it routinely takes input spend down by half or more. DrAI's OpenAI-compatible API supports prompt caching across its model catalog with usage telemetry on every response, so you can see the hit rate immediately — sign in to start, or compare plans at /pricing.html.