Token Optimization Techniques: Cut LLM Costs Without Losing Quality

Every token sent to an LLM costs money. As applications scale from hundreds to millions of requests, token consumption becomes the dominant expense. A poorly optimized application can cost ten times more than a well-optimized one delivering identical results. This guide covers proven, production-tested techniques to reduce token usage by 50-90% while maintaining—or even improving—output quality.

Understanding Token Economics

Before optimizing, you need to understand how tokens work. A token is roughly 4 characters or 0.75 words of English text. You pay for both input tokens (your prompt) and output tokens (the model's response). Output tokens typically cost 3-5x more than input tokens.

Consider a RAG application that retrieves 5 documents of 1000 tokens each, adds a 500-token system prompt, and generates a 300-token response. That is 5500 input + 300 output tokens per request. At GPT-5 pricing of $5/$15 per million tokens, each request costs about $0.073. At 10,000 requests per day, that is $730 daily or $22,000 monthly—for a single feature.

With optimization, the same application could run for under $3,000 monthly. The techniques below show how.

Technique 1: Prompt Compression and Simplification

The most immediate savings come from reducing prompt size. Many applications send bloated prompts full of redundant instructions, verbose examples, and unnecessary context. Every unnecessary token is wasted budget.

Trim Verbose Instructions

# BAD: 89 tokens
system_prompt = """You are a very helpful and knowledgeable assistant
whose primary goal is to provide accurate and comprehensive answers
to all user questions. You should always strive to give detailed
explanations and include relevant examples where appropriate.
Please make sure your responses are well-structured and easy
to understand."""

# GOOD: 14 tokens (same behavior)
system_prompt = "Answer user questions accurately and concisely."

Models in 2026 are smart enough that they do not need verbose instructions. "Answer accurately and concisely" produces the same behavior as a paragraph of explanation. Audit every system prompt for unnecessary words.

Remove Redundant Examples

Few-shot examples are powerful but token-expensive. Each example adds hundreds of tokens to every request. Audit whether each example is necessary. Often, a well-written instruction replaces multiple examples. When you do use examples, use the minimum number—often one or two suffice.

Use Delimiters Instead of Explanatory Text

# BAD: 45 tokens
prompt = """The following text is the user's document that you
should analyze. Please read it carefully and then answer
the question below.

Document: {document}

Question: {question}"""

# GOOD: 12 tokens
prompt = """Analyze:
<doc>{document}</doc>
Q: {question}"""

Technique 2: Context Window Management

The context window is where most waste occurs. Applications cram full documents, entire conversation histories, and large tool definitions into every request. Smart context management is the highest-impact optimization.

Conversation History Truncation

In multi-turn conversations, sending the full history becomes exponentially expensive. Implement intelligent truncation:

def manage_conversation_history(messages, max_tokens=4000):
    """Keep conversation within token budget."""
    # Always keep system prompt and last few messages
    system_msgs = [m for m in messages if m["role"] == "system"]
    conversation = [m for m in messages if m["role"] != "system"]

    # Keep most recent messages within budget
    kept = []
    token_count = count_tokens(system_msgs)

    for msg in reversed(conversation):
        msg_tokens = count_tokens([msg])
        if token_count + msg_tokens > max_tokens:
            break
        kept.insert(0, msg)
        token_count += msg_tokens

    # Optionally: summarize dropped messages
    if len(kept) < len(conversation):
        summary = summarize_old_messages(
            conversation[:-len(kept)]
        )
        kept.insert(0, {
            "role": "system",
            "content": f"Previous conversation summary: {summary}"
        })

    return system_msgs + kept

Sliding Window with Summarization

For long conversations, use a sliding window approach: keep the last N messages verbatim, summarize older messages into a compact summary. This preserves context while dramatically reducing tokens:

def sliding_window_with_summary(messages, window_size=6):
    """Keep recent messages, summarize older ones."""
    if len(messages) <= window_size + 1:
        return messages

    system = messages[0]
    old_messages = messages[1:-window_size]
    recent_messages = messages[-window_size:]

    # Summarize old messages (cache this!)
    summary = llm_summarize(old_messages)

    return [system,
            {"role": "system",
             "content": f"Earlier in this conversation: {summary}"},
            *recent_messages]

RAG: Retrieve Less, Retrieve Better

In RAG applications, the instinct is to retrieve many documents "just in case." This is wasteful. Better retrieval quality means you need fewer chunks. Optimize your retrieval pipeline to return fewer, more relevant results:

# BAD: retrieve 10 chunks, send all
results = vector_db.similarity_search(query, k=10)
context = "\n".join([r.content for r in results])

# GOOD: retrieve 5, rerank, send top 3
results = vector_db.similarity_search(query, k=5)
reranked = cross_encoder_rerank(query, results)[:3]
context = "\n".join([r.content for r in reranked])

Cross-encoder reranking adds latency but dramatically improves relevance, allowing you to use 60% fewer tokens for the same answer quality.

Technique 3: Prompt Caching

GPT-5, Claude, and other major models now support prompt caching. If your application sends the same prefix repeatedly (system prompt, tool definitions, retrieved documents), caching can cut input token costs by 50-90%.

# OpenAI-compatible prompt caching (automatic on DrAI)
response = client.chat.completions.create(
    model="gpt-5",
    messages=[
        # Static prefix - will be cached after first call
        {"role": "system", "content": LARGE_SYSTEM_PROMPT},
        {"role": "system", "content": TOOL_DEFINITIONS},
        {"role": "system", "content": COMPANY_KNOWLEDGE_BASE},
        # Dynamic suffix - changes each request
        {"role": "user", "content": user_question}
    ]
)
# Subsequent requests with same prefix get 50% discount
# on cached tokens (GPT-5 pricing)

Caching is most effective when the static prefix is large (1000+ tokens) and reused frequently. The model provider caches the prefix for a period (typically 5-60 minutes) and charges a reduced rate for cached portions.

Technique 4: Model Routing and Cascading

Not every request needs GPT-5. Simple queries can be handled by cheaper models. Model routing directs each request to the cheapest model that can handle it:

def route_model(query: str) -> str:
    """Route to appropriate model based on complexity."""
    query_lower = query.lower()

    # Simple tasks - use cheap model
    simple_patterns = ["translate", "summarize", "grammar",
                       "spell", "rephrase"]
    if any(p in query_lower for p in simple_patterns):
        return "gpt-5-mini"  # 10x cheaper

    # Medium complexity
    if len(query) < 200:
        return "gpt-5-mini"

    # Default to powerful model
    return "gpt-5"

# Cascading: try cheap model, upgrade if needed
def cascade_response(query: str) -> str:
    """Try cheap model first, escalate if quality is low."""
    try:
        response = call_model("gpt-5-mini", query)

        # Quality check
        if is_low_quality(response):
            response = call_model("gpt-5", query)

        return response
    except Exception:
        return call_model("gpt-5", query)  # Fallback

For a detailed implementation guide, see our AI Model Routing Strategy article. In production, model routing typically saves 40-70% with negligible quality impact.

Technique 5: Output Length Control

Output tokens cost 3-5x more than input tokens. Controlling output length is one of the highest-impact optimizations:

# BAD: unlimited output
response = client.chat.completions.create(
    model="gpt-5",
    messages=messages
)

# GOOD: explicit limits
response = client.chat.completions.create(
    model="gpt-5",
    messages=messages,
    max_tokens=150,  # Hard limit
    temperature=0.3   # Lower temperature = more concise
)

# Instruct for brevity
system_prompt = "Respond in 2-3 sentences. Be direct."

Setting max_tokens prevents runaway responses. But instruction-based control is better—the model naturally produces shorter responses when told to be concise, rather than being cut off mid-sentence.

Technique 6: Semantic Deduplication

In RAG systems, retrieved documents often contain overlapping information. Deduplicate semantically before sending to the model:

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

def deduplicate_chunks(chunks, threshold=0.92):
    """Remove semantically similar chunks."""
    if len(chunks) <= 1:
        return chunks

    embeddings = embed_chunks(chunks)
    sim_matrix = cosine_similarity(embeddings)

    keep = [0]  # Always keep first (highest relevance)
    for i in range(1, len(chunks)):
        # Check if too similar to any kept chunk
        if all(sim_matrix[i][j] < threshold for j in keep):
            keep.append(i)

    return [chunks[i] for i in keep]

# Reduces 5 chunks to 3 unique ones = 40% fewer tokens

Technique 7: Batch Processing

If you process multiple independent requests, batch them instead of sending one at a time. Batch APIs offer 50% discounts:

import asyncio

async def batch_process(queries: list[str]) -> list[str]:
    """Process multiple queries concurrently."""
    tasks = [call_model_async("gpt-5", q) for q in queries]
    results = await asyncio.gather(*tasks)
    return results

# Or use batch API for 50% discount (async, results in hours)
batch = client.batches.create(
    input_file_id=file_id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    metadata={"description": "Daily batch processing"}
)
# Batch API: 50% discount, 24-hour turnaround

Technique 8: Caching Responses

For applications where users ask similar questions, cache responses. This eliminates redundant API calls entirely:

import hashlib
import redis
import json

redis_client = redis.Redis()

def cached_llm_call(messages, model="gpt-5", ttl=3600):
    """Cache identical requests."""
    # Create cache key from request
    key_data = {"model": model, "messages": messages}
    cache_key = "llm:" + hashlib.sha256(
        json.dumps(key_data, sort_keys=True).encode()
    ).hexdigest()

    # Check cache
    cached = redis_client.get(cache_key)
    if cached:
        return json.loads(cached)

    # Make API call
    response = client.chat.completions.create(
        model=model, messages=messages
    )
    result = response.choices[0].message.content

    # Cache result
    redis_client.setex(cache_key, ttl, json.dumps(result))

    return result

For FAQ bots, documentation assistants, and similar applications, caching can eliminate 40-60% of API calls. For more on rate limiting and caching layers, see our API Rate Limiting Guide.

Measuring and Monitoring Token Usage

You cannot optimize what you do not measure. Implement comprehensive token tracking:

from dataclasses import dataclass
from collections import defaultdict

@dataclass
class TokenUsage:
    input_tokens: int = 0
    output_tokens: int = 0
    cached_tokens: int = 0
    cost: float = 0.0

class TokenTracker:
    def __init__(self):
        self.usage_by_feature = defaultdict(TokenUsage)

    def track(self, feature: str, response):
        u = response.usage
        tracker = self.usage_by_feature[feature]
        tracker.input_tokens += u.prompt_tokens
        tracker.output_tokens += u.completion_tokens
        tracker.cached_tokens += getattr(
            u, 'prompt_tokens_details', {}
        ).get('cached_tokens', 0)

        # Calculate cost
        cost = (u.prompt_tokens * 5 / 1_000_000
                + u.completion_tokens * 15 / 1_000_000)
        tracker.cost += cost

    def report(self):
        for feature, u in self.usage_by_feature.items():
            print(f"{feature}: "
                  f"{u.input_tokens:,} in / "
                  f"{u.output_tokens:,} out / "
                  f"${u.cost:.2f}")

Optimization Impact Summary

Combined, these techniques can reduce total token costs by 70-90% while maintaining quality. Start with the low-effort, high-impact techniques: prompt compression, output control, and prompt caching.

Real-World Case Study

A SaaS company running a customer support chatbot was spending $15,000/month on GPT-5 API calls. After applying these techniques, they reduced costs to $2,800/month—a 81% reduction—with no measurable change in customer satisfaction scores. Here is what they did:

First, they compressed their system prompt from 2,400 tokens to 600 tokens by removing redundant instructions. Second, they implemented conversation history truncation with summarization, cutting average input tokens by 45%. Third, they added prompt caching for their knowledge base context, saving 60% on cached tokens. Fourth, they routed 65% of simple queries to GPT-5-mini. Fifth, they implemented response caching for FAQ-type questions, eliminating 35% of API calls entirely.

For more case studies and strategies, see our AI Cost Optimization Guide and AI Cost Calculator Guide.

Common Mistakes in Token Optimization

Over-compressing to the point of quality loss: Aggressive prompt compression can remove essential context. Always A/B test optimized prompts against originals to verify quality is maintained.

Caching too aggressively: Response caching can return stale or context-inappropriate answers. Use semantic similarity matching for cache keys, not just exact matching, and set appropriate TTLs.

Routing to wrong models: Model routing that sends complex queries to cheap models degrades quality. Implement quality checks and fallback mechanisms. Test your routing logic against a golden dataset.

Ignoring security implications: Some optimizations (like response caching) can introduce security issues if cache keys include sensitive data. For security best practices, see our LLM Security Guide.

Future-Proofing Your Token Strategy

Token costs are declining as models become more efficient and competition increases. However, usage is growing faster than prices are falling. The net effect is that most organizations will spend more on LLM APIs in absolute terms even as per-token prices drop. Optimization remains essential.

Emerging trends that will impact token economics: longer context windows reduce the need for complex retrieval (but increase per-request costs), multimodal models add image and audio token costs, agent frameworks multiply token usage across multiple model calls, and speculative decoding and model quantization reduce inference costs at the provider level.

Build your token optimization strategy to be model-agnostic. The techniques in this guide work across GPT-5, Claude, DeepSeek, and any future model. The DrAI platform provides a unified API where you can apply these optimizations across 40+ models. Check our pricing for the most competitive rates.

Conclusion

Token optimization is not a one-time task—it is an ongoing discipline. Start by measuring your current usage, then apply the high-impact, low-effort techniques first: prompt compression, output control, and prompt caching. Layer in context management, model routing, and caching as your application matures.

The combined effect of these techniques is transformative. Applications that seemed prohibitively expensive become viable. Features that were cut due to cost become feasible. And your margin on AI-powered features improves dramatically.

For more on building efficient AI applications, explore our guides on GPT-5 Function Calling, AI Agent Frameworks, and Model Routing Strategy.

Optimize Your AI Costs with DrAI →

Sources & Further Reading

• OpenAI prompt caching documentation (50-90% input savings)
• Anthropic Claude prompt caching documentation
• OpenAI API pricing (input vs output token rates)
• DrAI: AI API Cost Optimization Guide — 10 techniques with real savings
• DrAI: AI Model Routing Strategy