AI API Latency Optimization: From 3 Seconds to 300ms

AI API latency optimization reduces time-to-first-token (TTFT) and total response time through five levers: model selection, prompt engineering, streaming, caching, and connection reuse. Production systems routinely cut p50 latency from 3,000ms to under 300ms by applying all five — GPT-5-mini first token arrives in 180ms, response caching serves 23% of traffic in under 60ms, and persistent connections eliminate 100-200ms of TLS overhead per request.

Anatomy of LLM Latency

Every API call spends time in five stages. Optimizing the right stage matters:

Lever 1: Model Selection — The 3x Swing

Model choice is the largest single latency factor:

Routing latency-sensitive paths to mini-class models is a 2-3x TTFT improvement with zero engineering. DeepSeek R1's 900ms reflects deliberate reasoning-token generation — never use it on interactive paths.

Lever 2: Prompt Length — Every 1K Input Tokens Costs 10-40ms

Input processing is linear in tokens. Three techniques cut prompt cost:

System prompt compression

Replace verbose instructions with terse ones: a 2,000-token system prompt costs 20-80ms on every single call. Compress to 500 tokens of dense instruction and save 15-60ms per request across your entire volume.

Conversation windowing

# Keep only what the model needs: recent turns + summary
def build_messages(history):
    if len(history) > 10:
        summary = summarize(history[:-6])  # cheap model does this
        return [{"role": "system", "content": f"Earlier: {summary}"}] + history[-6:]
    return history

Structural deduplication

Don't repeat document context in multi-turn conversations — send it once, reference it after. Tools that re-send full context per turn burn 5-10x necessary input tokens.

Lever 3: Streaming — Perceived Latency's Biggest Lever

Streaming doesn't reduce total generation time, but it moves first-visible-content from end-of-response to TTFT:

stream = client.chat.completions.create(
    model="gpt-5-mini", messages=msgs, stream=True,
    max_tokens=500
)
for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        send_to_ui(content)  # user sees tokens as they arrive

A 2-second total generation feels like 180ms with streaming — a 10x perceived improvement. Always stream interactive UIs. See the full Python streaming guide for SSE + AsyncIO patterns.

Lever 4: Response Caching — 23% of Traffic Under 60ms

Deterministic requests (same prompt, temperature=0) return identical outputs. Caching them eliminates the model call entirely:

import hashlib, json, redis

r = redis.Redis()

def cached_chat(messages, ttl=3600):
    key = "ai:" + hashlib.sha256(
        json.dumps(messages, sort_keys=True).encode()
    ).hexdigest()
    
    cached = r.get(key)
    if cached:
        return json.loads(cached)  # ~60ms total
    
    result = client.chat.completions.create(
        model="gpt-5-mini", messages=messages, temperature=0
    )
    r.setex(key, ttl, json.dumps(result.model_dump()))
    return result.model_dump()

Production cache hit rates reach 23% for Q&A and support workloads — that's 23% of requests at 60ms and 23% less spend. DrAI applies this server-side automatically; a caching strategy deep-dive covers semantic caching extensions.

Lever 5: Connection Reuse — The Free 100-300ms

TLS handshake to a new host costs 100-300ms. SDK clients with connection pooling amortize it to near zero after the first request:

# WRONG: new client per request (in serverless especially)
def handler(request):
    client = OpenAI()  # full TLS handshake every call
    return client.chat.completions.create(...)

# RIGHT: module-level client, pooled connections
client = OpenAI(base_url="https://api.dr-ai.top/v1")

def handler(request):
    return client.chat.completions.create(...)

Keep-alive also matters at the gateway: DrAI maintains warm provider connections, one reason its p50 overhead is under 40ms versus direct calls.

Bonus: max_tokens Capping

Uncapped generations are a latency bomb — a 4,000-token response at 40ms/token runs 160 seconds. Cap outputs to what your UI actually displays: 512 for chat, 2,000 for documents. Combined with streaming, users see completion within budget.

Case Study: 3,000ms → 280ms

A customer support bot migrated from unoptimized GPT-5 to the full stack:

Total: 91% latency reduction, 58% cost reduction. Every technique here works with DrAI's OpenAI-compatible endpoint — get a free key at ai.dr-ai.top/signin and benchmark it yourself. For model-level latency data across 18+ models, see the latency benchmarks table and DrAI pricing.