LLM Inference Optimization: Speed Up Generation Without New Hardware

Published 2026-08-16 · 2,141 words · 9 min read

Latency is the silent tax on every LLM application. A model that answers in two seconds feels instant; the same model answering in eight seconds feels broken, and users churn. The frustrating part is that most teams reach for more GPUs before they exhaust the software levers that cost nothing: KV cache management, speculative decoding, quantization, batching, prefix caching, and parallelism. This guide explains how modern LLM inference actually spends its time, then walks through each optimization from the perspective of an engineer who wants faster generation without buying new hardware.

Where the Time Actually Goes: Prefill vs. Decode

Every LLM request has two phases with radically different economics. Prefill processes the input prompt in parallel — the GPU crunches all prompt tokens at once — and it is compute-bound. Decode generates output tokens one at a time, each token depending on all previous tokens, and it is memory-bandwidth-bound: for every output token the model must read the entire set of weights from memory, which takes roughly the same time whether the token is easy or hard. On a typical setup, a 1,000-token prompt prefills in tens of milliseconds while a single output token takes 10-40 milliseconds, so a 500-token answer costs 5-20 seconds of decode alone.

This asymmetry drives almost every optimization decision. If your workload is short prompts with long outputs, decode dominates and the winning moves are KV cache reuse, speculative decoding, and quantization. If your workload is huge prompts with short outputs, prefill dominates and the winning moves are prefix caching and prompt compression. Measure the split before you optimize — one time_to_first_token and inter_token_latency metric each tells you which half of the pipeline deserves your attention.

KV Cache: The Memory Bottleneck You Can Reuse

When a transformer attends over a sequence, it computes Key and Value vectors for every token and stores them so later tokens can attend to them without recomputation. That store is the KV cache, and it is why decoding is memory-bound: with a 4,096-token context on a 7B-parameter model, the KV cache alone can exceed 1 GB of memory per request. The cache grows with context length and batch size, which is why long-context requests run out of memory even when the model weights fit easily.

Four practical moves tame the KV cache. First, cap context: a model advertised at 128K tokens does not mean every request should use 128K — trim inputs to what the task needs. Second, use cache reuse for multi-turn conversations: the prefix of a follow-up request is identical to the previous request, and providers that support prefix caching (either automatic or via explicit prompt caching) skip recomputing it, cutting both prefill time and cost on chat workloads by 50-90%. Third, where you control serving, enable KV cache quantization (FP8 or INT8 keys and values), which cuts cache memory roughly in half for a small quality cost on long contexts. Fourth, in self-hosted serving, watch cache fragmentation: vLLM and similar engines manage paged KV blocks, and a fragmented cache silently reduces effective batch size.

Speculative Decoding: Draft Fast, Verify Once

Decode is serial and memory-bound, so the cleverest optimization attacks it by generating several tokens per step instead of one. Speculative decoding runs a small, fast draft model (or a shallow head of the same model) to propose, say, four tokens; the big model then verifies all four in a single forward pass, which is nearly as cheap as verifying one, and accepts the proposal if the probabilities agree. When the draft is right, throughput jumps 2-3x; when it is wrong, you waste the draft work but lose nothing in quality, because verification guarantees the output distribution is unchanged.

The draft model does not have to be a separate download. Many serving frameworks support self-speculation, where the model itself predicts its next few tokens through a lightweight head — the same weights, no extra deployment. The win depends on how predictable your traffic is: code, JSON, and repetitive business text speculate beautifully; freeform creative writing less so. If you self-host and your workload is structured, speculative decoding is the cheapest 2x on the shelf. If you use an API, ask your provider whether they enable it — several now do transparently, and it shows up as a lower inter-token latency.

Quantization: INT8, FP8, and the Quality/Speed Bargain

Model weights are stored as FP16 (16-bit) by default. Quantization stores them in 8-bit or 4-bit formats, roughly halving or quartering the memory bandwidth per token — and since decode is bandwidth-bound, that translates almost linearly into speed. The formats you will meet in 2026, in rough order of aggressiveness:

Two details matter more than the format name. First, activation quantization (W8A8 means weights AND activations in 8-bit) is what unlocks fast kernels on tensor cores; weight-only quantization still helps memory but gains less speed on modern accelerators. Second, quality impact is task-dependent — a model that scores identically on benchmarks can still regress on structured output, so always run your eval set through the quantized model before promoting it. For most self-hosted teams, INT8 or FP8 with an eval gate is the correct default, and 4-bit is the fallback when the model barely fits in memory.

Batching: The Utilization Multiplier

A GPU is only fast when it is full. Processing one request at a time leaves most of the hardware idle during memory-bound decode, so serving systems batch multiple requests together, sharing one forward pass across many sequences. Naive static batching waits for a full batch before starting, which adds latency; modern continuous (in-flight) batching admits new requests as slots free up and evicts finished sequences immediately, keeping utilization high without a fixed batch barrier. This single technique is responsible for much of the throughput difference between naive inference code and production serving engines, and it explains why "my GPU is only at 30% utilization" is almost always a batching problem, not a hardware problem.

For API users, batching appears as two levers: the provider's server-side batching (choose providers with continuous batching; you can detect it by comparing single-stream vs. parallel-request throughput) and your client-side batching for offline workloads. If you generate summaries for 10,000 documents, send requests concurrently up to the rate limit instead of sequentially — with continuous batching on the server, throughput scales nearly linearly until you saturate it, and your wall-clock time drops by 5-10x for free. The rate limit is the only constraint, and most providers' RPM limits are far above what sequential clients ever reach.

Prefix Caching: Never Recompute the Same Prompt Twice

Real workloads repeat themselves. Chat conversations repeat the entire history on every turn; agent loops repeat system prompts, tool definitions, and long instructions; RAG pipelines repeat the same context template with a different retrieved passage. Prefix caching stores the KV cache of prompt prefixes so that when a new request shares a prefix with a cached one, the shared portion is loaded from cache instead of recomputed. On multi-turn chat, this can cut time-to-first-token by 70-90% and, because cache reads are billed far below compute, cut cost by 50-80% as well.

The engineering rule for cache-friendly prompts: put stable content first. System prompt, tool schemas, instructions, then variable content last — a cache hit is broken the moment the prefix diverges, so a prompt template that interpolates user data into the middle of the system prompt quietly disables caching for everyone. If your provider exposes a cache API (or manages it automatically), measure the cache hit rate as a first-class metric; a cache hit rate under 30% on a chat product usually means your prompt template is ordered wrong.

Parallelism: Tensor, Pipeline, and the Data Center

When a model is too big for one GPU — or when one GPU is too slow — you split the work across devices. Tensor parallelism shards the weight matrices across GPUs so every layer runs on all of them simultaneously; it scales throughput but demands fast interconnect (NVLink or InfiniBand) because every layer needs an all-reduce. Pipeline parallelism splits the model by layers, each GPU owning a contiguous slice; it uses cheap interconnect but has idle bubbles between stages and helps more with memory than with latency. Data parallelism runs full replicas of the model on separate GPUs and simply serves more requests — often the right answer for serving, because throughput, not single-request latency, is what most products need.

The practical guidance for a team without a GPU cluster: do not build this yourself. Serving engines like vLLM, TensorRT-LLM, and SGLang have already solved tensor/pipeline scheduling, paged KV caches, continuous batching, and speculative decoding, and they outperform naive frameworks by an order of magnitude. Your job is to pick the engine, configure the knobs that matter (max batch size, KV cache allocation, quantization, speculative draft), and run a latency regression suite so a configuration change never silently doubles p95. If your workload is bursty, remember that elasticity beats optimization: a provider that scales replicas on demand may be cheaper than squeezing the last 20% out of one fixed GPU.

The API Developer's Playbook: What You Can Control

If you consume models through an API rather than hosting them, most of the levers above are inside the provider — but several are yours. First, stream everything: SSE streaming moves time-to-first-token perception from "wait for the whole answer" to "first word in 300ms," which users experience as dramatically faster even when total generation time is unchanged. Second, set max_tokens honestly: every reserved but unused token in the budget can change batching behavior, and a tight cap prevents runaway generations. Third, choose the model deliberately — a small or distilled model on your task is the single biggest latency lever, often 3-5x faster than the flagship with acceptable quality, and model routing lets you send easy queries to the fast model and hard ones to the strong model. Fourth, shorten prompts: input tokens are not free — they prefill serially into time-to-first-token, so trimming context (or moving it to a cache-friendly prefix) directly cuts perceived latency. Fifth, add client-side timeout and retry policies that distinguish "slow" from "down," and use parallel sub-requests for fan-out workloads so the total wall clock is the slowest branch, not the sum.

Finally, instrument the right metrics. Time-to-first-token measures prefill and networking; inter-token latency measures decode speed; end-to-end latency is what users feel; and p95 across all three is where the painful reality lives. Track them per model and per prompt shape, because a latency regression that appears only on long-context requests points at KV cache or prefix-cache behavior, while a regression across all shapes points at the provider or your own networking. Every optimization in this guide should be justified by these numbers, not by vibes.

The Bottom Line

Faster LLM inference is mostly a software problem. Understand prefill versus decode, reuse the KV cache, let a draft model speculate, quantize to INT8 or FP8 with an eval gate, batch continuously, order prompts for prefix cache hits, and let serving engines handle parallelism. Each lever is worth 1.5-3x on the right workload, and they compound: a chat product that combines streaming, prefix caching, and a routed fast model often feels 10x faster without a single new GPU. DrAI sits on the API side of this equation: 40+ models across speed tiers, OpenAI-compatible streaming, and per-key dashboards so you can measure latency and cost per model before you commit. Sign up free at sign in, compare pricing, and read the cost optimization playbook for the other half of the ledger.