AI API Response Caching: Save 30% Costs with Smart Caching

AI API response caching eliminates duplicate model calls entirely — a cached answer costs ~60ms and $0 instead of 300-2000ms and full token price. Production Q&A and support workloads hit 20-30% cache rates with exact-match caching alone, and semantic caching (embedding-similarity lookups) pushes that higher while catching paraphrased duplicates. This guide covers the full caching stack with measured savings.

Why Caching Works for LLMs

LLM calls are embarrassingly cacheable: the same prompt at temperature 0 returns the same answer, and real workloads have heavy duplication. Support bots get the same 30 questions; Q&A systems see the same docs queried repeatedly; classification pipelines process recurring patterns. Every duplicate is 100% wasted spend.

Exact-Match Caching (Start Here)

import hashlib, json, redis

r = redis.Redis(host="cache", decode_responses=True)

def cached_chat(messages, model="gpt-5-mini", ttl=86400):
    # Canonical key: model + messages + temperature=0
    key = "llm:" + hashlib.sha256(
        json.dumps({"m": messages, "model": model, "t": 0},
                   sort_keys=True).encode()).hexdigest()
    
    cached = r.get(key)
    if cached:
        return json.loads(cached)          # ~60ms, $0
    
    resp = client.chat.completions.create(
        model=model, messages=messages, temperature=0)
    r.setex(key, ttl, json.dumps(resp.model_dump()))
    return resp.model_dump()               # normal latency, full cost

Design notes: include temperature: 0 in the key (non-deterministic temps shouldn't cache), include model in the key (same prompt on different models is a different answer), and set TTL deliberately — a support answer from yesterday is fine; a financial quote from yesterday may not be.

Semantic Caching: Catch the Paraphrase

Exact matching misses "How do I reset my password?" vs "I forgot my password, help!" — same intent, different tokens. Semantic caching embeds the query and looks up nearest neighbors above a similarity threshold:

import numpy as np

def semantic_cached_chat(messages, threshold=0.95):
    query = messages[-1]["content"]
    q_emb = embed(query)   # text-embedding-3-small, 1536d
    
    # Nearest neighbor search (pgvector / Redisearch / Qdrant)
    hits = vector_store.search(q_emb, top_k=1, score_threshold=threshold)
    if hits:
        return hits[0].answer          # cached answer for near-identical query
    
    answer = client.chat.completions.create(model="gpt-5-mini", messages=messages)
    vector_store.insert(embed(query), {"answer": answer, "ts": now()})
    return answer

Threshold tuning matters: too high (0.99) catches nothing beyond exact; too low (0.85) serves wrong answers. Start at 0.95, measure answer acceptance, adjust ±0.02. Semantic caching typically adds 5-15 points of cache rate over exact matching for Q&A workloads.

TTL and Invalidation Strategy

Use namespaced keys (docs:v2:<hash>) so a content update bumps the namespace and invalidates a whole class atomically — far simpler than tracking individual keys.

Where Caching Sits in the Stack

Request → [Edge cache (CDN/CF)] → [Gateway cache (Redis)]
        → [Provider prefix cache (prompt cache)] → [Model]

Three layers, different granularity:
1. Edge/HTTP cache: identical HTTP requests (rare for APIs)
2. Gateway semantic cache: near-identical prompts (this guide)
3. Provider prefix cache: repeated prompt prefixes get 50-90% input discount
   (automatic on many providers; keep stable content at prompt start)

DrAI applies layer 2 automatically at the gateway — cached responses serve in ~60ms and cost nothing, with the 23% average hit rate observed across production traffic. You don't build or operate the cache; your dashboard shows hits. Layer 3 (prefix caching) is also active for long-context workloads. See the cost optimization deep-dive for the full savings stack and latency guide for the 60ms story. Try it free at ai.dr-ai.top.