LLM Quantization Guide: INT8, FP8, and 4-Bit Models Explained

Published 2026-08-17 · 2,187 words · 8 min read

The story of running capable LLMs outside the cloud — for cost, latency, privacy, or on-premise policy — is written in quantization. A 70B model that takes 280 GB in FP16 takes 35 GB at 4-bit; that single idea turns "impossible to host" into "fit on a single A100" and "fit on an M2 laptop" on the smaller inferences. But the term "quantization" is overloaded onto a half-dozen different methods — weight-only round-to-nearest, scale-only INT8, Activations-aware INT8, FP8 on Hopper, GPTQ group-size, AWQ activation-aware, and the Llama.cpp GGUF families each — and the developers making quantize-vs-api choices often get less-performant-than-guaranteed setups. This guide is the developer view of quantization: what each numeric format actually trades away, what each method does in practice, what the quality loss really looks like by category of work, and a decision tree on when you should quantize locally vs. call an API.

If you want the high-level serving/compression packing context, see our AI model deployment guide. This article is the deep dive into the numeric formats and methods that enable serving large models on limited hardware, with the cost decisions explicit.

What Quantization Actually Is, In One Sentence

Quantization is the process of representing model weights, activations, or both, in a lower-precision numeric type than they were trained in, accepting a controlled trade-off against model accuracy in exchange for dramatic drops in memory and latency. Modern LLMs ship at FP16 or BF16. Quantization down to FP8, INT8, or 4-bit reduces the bits-per-weight by factors of 2×, 4×, or 4×, tapering off logarithmically as precision pressure increases on the model. Two families of techniques follow different rules:

• Weight-only quantization: weights stored compressed; activations upsampled to FP16 at compute time. Quality loss is low; the speedup is moderate (memory bandwidth reduction dominates).
• Weight-and-activation quantization: activations also stored in low precision and the matmuls happen in INT8 or FP8 directly. Speedup is large (dedicated INT8/FP8 silicon); quality loss depends more on signal calibration.

Most modern local-inference setups use weight-only quantization targeting 4-bit weights. The sweet spot is the matrix of "best quality at lowest memory" that we walk through below.

The Three Numeric Formats You Need To Know

The progressive trade is clear: FP8 is "free" on H100, INT8 is "cheap, minimal loss" with calibrated activation scales, 4-bit is "default if memory is your bottleneck." Below 4-bit crosses a brittle diagonal where most general-purpose models break visibly on reasoning and coding benchmarks. Treat 4-bit as the practical floor unless you are running a domain specifically tuned for sub-4-bit quantization.

Weight-Only Methods Compared: Where The Quality Loss Comes From

GPTQ — Group-Quantized Round-To-Nearest With Compensation

GPTQ is the workhorse weight-only 4-bit method. It quantizes weights column-by-column using a small calibration dataset to estimate the scale per group (typically group size 128), then "compensates" for the error introduced by each quantized weight by updating the not-yet-quantized columns. Quality at 4-bit is robust across chat, classification, and common extracted tasks; 3-bit is usable but clearly degraded coding math.

Best suited to: GPU inference at 4-bit using group_size 128 (default), 7B-70B models, single-tool output without exotic hardware.

AWQ — Activation-Aware Weighing

AWQ is weight-only 4-bit with activation-aware calibration: it identifies which weight channels are activated by the calibration data, scales those high-impact channels so they preserve precision post-quantize, and quantizes the rest more aggressively. AWQ achieves slightly better-than-GPTQ throughput at 4-bit and better quality on instruction-tuned models, especially when the model produces calibrated activations with realistic distribution.

Best suited to: Instruction-tuned models at 4-bit on consumer GPUs. Production pick when accuracy matters slightly more than top speed.

GGUF (Llama.cpp) — Single-File Hosting For Variants

GGUF and the Llama.cpp ecosystem wrap the model in a single file with selectable quantization subtype at load time: Q4_K_M, Q4_K_S, Q5_K_M, Q8_0, plus higher variants Q2_K, Q3_K_M, Q6_K. "M" variants mix quantization accuracy across layer types — sensitive layers get Q6, others get Q4 — capturing the most quality given the size budget. Q4_K_M is the most widely-deployed 4-bit for Q-spec local inference on CPU and Apple Silicon.

Best suited to: CPU / Apple Silicon local inference; mixing-tier Q4_K_M and Q5_K_M; single-file deployments on a laptop.

Quality Impact By Category — Where You Lose Accuracy

Aggregate benchmark quality loss obscures the more important per-category story. Tested across a stable run of 7B and 13B models with calibrated GPTQ at group_size 128, the empirical loss per task category.

The pattern: short conversation is mostly free; extractionary and multi-step reasoning lose the most. The corollary is that quantization is task-specific. A 4-bit model serves a chat-bin feature well and breaks an extraction agent earlier than FP16 — measuring per-task is the only valid measurement.

Speedup in Practice — The Memory Floor and the Throughput Ceiling

The textbooks say "lower bit → faster". In practice, two distinct kinds of speedup hit differently:

• Bandwidth-bound regime (memory dominated): larger models become feasible because they fit. A 70B model at FP16 needs 140 GB and an A100-80GB + far swap; at 4-bit it fits to a single A100-80GB with headroom to spare and runs at 4-6× raw forward-pass speedup over FP16 swap-throttled running.
• Compute-bound regime (large batch): when you saturate the large-batch use case, INT8/FP8 silicon drives have hardware features (Tensor Cores @ INT8, FP8) that add a further 2-3× on top of memory savings. Without Hopper for FP8 or Tensor for INT8, the compute speedup is smaller than the books claim.

The fastest path to "fast LLM serving at small scale" is often picking the format that fits with the most resting memory overhead, then topping up with native silicon acceleration. On Hopper, FP8 + Large-scale INT8 cover both throughput and quality gracefully.

Quantization-Aware Training vs. Post-Training Quantization

Post-training quantization (PTQ) is the dominant production route: take a pretrained checkpoint, apply a calibration pass with a small (~128 example) cached set, emit the quantized weights in the chosen format. The model is never re-finetuned. PTQ is fast and works for most gains.

Quantization-aware training (QAT) inserts fake-quantization nodes into the forward pass during fine-tuning, then re-runs the optimizer. QAT recovers a chunk of quality loss extra relative to PTQ at 4-bit and below. It is more expensive than PTQ and rarely justified above 9-bit formats, but it is the right choice if your quality target is FP16-equivalent at 4-bit.

A pragmatic rule: PTQ for 6-8 bit, PTQ for general 4-bit if quality tolerance matches measured loss, QAT only when 4-bit is unacceptable domain-application loss demonstrably needs to heal.

When to Quantize Locally vs. Calling an API

Quantization unlocks local deployment — but local is not free. The decision tree:

The economics shift hard as you scale up: an API charges a $/token rate that compounds lakh-shape use; self-quantized serving charges per-hour-over-time plus ops overhead plus quality loss. At the cross-over, the swing favors a quantized deployment but nevermore scrape-from-zero: keep the API default path available for burst-y traffic and edge-load. DrAI's gateway gives you both: one OpenAI-compatible key routing to the API by default with quantization-friendly side-paths off-routed when traffic volume hits the policy gate.

The Developer Workflow — How To Actually Quantize A Model

The kit you ring-tier with — same general idiomatic workflow across the formats:

1. Acquire the model in FP16/BF16 safetensors or original repo. The hosted HF Hubauthorization of quantize-from pretrained weights is your base.
2. Pick a calibration sample. ~128 examples typical, representative of your deployment distribution — chat-style task, code-completion, etc.
3. Calibrate and quantize. AutoGPTQ for 4-bit GPTQ, llm-awq for AWQ, llama.cpp convert + quantize for GGUF, llama.cpp factory for GGUF Q4_K_M.
4. Emit the quantized artifact. Output as safetensors-format or GGUF, version-name with method, bits, group_size, and a calibration spec.
5. Run benchmark рублей. Benchmark against your own task-eval suite, per category — not aggregate benchmark numbers.
6. Serve with a quantized-native engine. vLLM, llama.cpp, TensorRT-LLM, exllamav2, or mlx-lm on Apple Silicon.
7. Measure per-category quality in production. Sample production traffic and grade; re-evaluate the 1% loss across tasks before scaling to the full surface.

If you skip step 7 — the production sample grade — you will release the 4-bit model into traffic and discover in a month that your extraction feature quietly lost 6% accuracy. The aggregate benchmark looked fine; the production eval was the one that mattered.

The Common Pitfalls

• Quantizing the wrong model. Quantization preserves quality with limited headroom; a model that was already weak at FP16 collapses faster than one that shipped with strong FP16 baseline. Always start from the strongest possible base, not the smallest one you can fit.
• Wrong calibration data. Calibrate with a sample that matches deployment distribution. Code-completion calibrated with conversational samples underperforms on actual code passes. This is a single cheap step that can change your end-state smile.
• Defaulting to 4-bit "because smaller." If you have INT8 silicon native, INT8 preserves more quality per step and is routinely faster per-prompt than 4-bit. Reach for 4-bit only when memory-bound.
• Forgetting KV cache memory. The model loads compressed, but at inference time the KV cache living at higher precision than the weights grows with token-length. A 70B model at 4-bit weights still slides a single A100 into swap with a sufficiently long context and concurrency. Budget for the KV cache.
• Cache key contamination. A serving cluster with mixed precision tiers will cache equivalent-but-different responses per tier. Quantize-only-one-tier per canonical version for cache coherence.

The Quantization Checklist

1. Pick the format matched to the deploy hardware — FP8 on Hopper, INT8 on Tensor-equipped or VNNI, 4-bit elsewhere; GGUF on edge/CPU
2. Use activation-aware weighing (AWQ) over round-to-nearest (GPTQ) on instruction-tuned vended improvements
3. Calibrate on distribution-matching data; ~128 representative samples, never less
4. Measure quality per task category, not by aggregate benchmark numbers
5. Benchmark memory footprint including KV cache, not just model weights
6. QAT only when PTQ-derived quality is demonstrably unacceptable for the deployment
7. Default to API at small scale (small QPS and per-task tolerances) — quantized local is most economic at ~20+ QPS or stamped with privacy/constraint gate
8. Re-grade per-category quality monthly on production traffic — token-level drift catches silent quality regressions
9. Keep one canonical version per quant tier per model — minimum cache-comparison-and-versioning combined
10. Per-category grade regression-result-revert: assign tier, benchmark, ship, monitor

Quantization is a precise tool, not a magic shrink-ray. The teams that use it well start from a strong base, pick the format matched to their hardware, calibrate against real deployment distribution, and verify per-task quality rather than aggregate benchmark numbers. DrAI's gateway gives you one OpenAI-compatible endpoint that routes between hosted API (high-quality, pay-per-token) defaulf and self-hosted quantized tiers for capacity and privacy routing — so as your volume traffic crosses the quantize-vs-API economic cross-over, you swap providers behind one client with no code change. Start with a free account at sign in, or check pricing for usage-based plans that scale across hosted and quantized tiers.