8-bit KV cache is free, 4-bit silently kills high-precision quants: a 35-quant KLD benchmark of Qwen3.8-27B
A 35-quant KLD benchmark across llama.cpp, ExLlamaV3, and vLLM: 8-bit KV is free, 4-bit adds a flat penalty that explodes on precise weights and grows with context.
A practitioner benchmark of 35 Qwen3.8-27B weight quantizations across llama.cpp, ExLlamaV3, and vLLM finds that an 8-bit KV cache is essentially free in two of the three engines. A 4-bit cache adds a roughly fixed absolute KL penalty that lands hardest on high-precision weights and grows with context depth. The results, published by Eric, the independent blogger behind erli.xyz, give working engineers a concrete answer to a question the community has only argued about anecdotally.
What shipped
The author tested 35 weight quants of the open-weight Qwen3.8-27B — GGUF quants from Unsloth, ISTA-DASLab, and Bartowski; ExLlamaV3 bpw quants; and vLLM checkpoints (AutoRound INT4, AWQ INT4, NVIDIA NVFP4, Unsloth NVFP4, Qwen FP8) — each run with a 16-bit, 8-bit, and, where supported, 4-bit KV cache. Every config is scored token-by-token against an unquantized bf16 reference using KL divergence (KLD) and top-1 agreement. It is a follow-up to the author's earlier Qwen3.6-27B weight-only benchmark.
Headline results, all author-reported:
- 8-bit KV is essentially free in llama.cpp and ExLlamaV3. For every quant,
q8_0and ExLlamaV3's 8-bit cache move KL by less than 0.00012 in either direction — some quants even measure slightly better with it, which the author reads as numeric noise. The cache halves from 64 KiB/token to 34 KiB/token (4.00 GiB → 2.13 GiB at 64k context). - vLLM's FP8 cache costs a small but systematic amount. It adds +0.0004 to +0.0006 KL on every checkpoint. The author speculates why: enabling the FP8 cache switches vLLM's attention backend from FlashAttention to FlashInfer. That is a hypothesis, not a confirmed cause.
- A 4-bit cache adds a roughly fixed absolute penalty. llama.cpp's
q4_0adds +0.0016 to +0.0027 KL across the GGUF quants. Because it is roughly constant in absolute terms, its relative damage grows sharply on precise weights: about +6–10% on a Q3-class quant, ~+39% on UD-Q4_K_XL, and ~+390% on Q8_0 (KL going from 0.0007 to 0.0034 — worse than UD-Q5_K_XL with a 16-bit cache). - The 4-bit penalty grows with context. Across 12 long documents scored at every position, the added KL at 16–32k tokens runs about 15–30% higher than in the first 4k. The same quants with a 16-bit cache are flat across context length, so the growth is attributable to the 4-bit cache. The author expects 128k+ contexts to be worse.
- vLLM checkpoints still lose to smaller GGUF/EXL3 quants, and activation quantization is a major contributor. Re-running NVIDIA's NVFP4 and Qwen's FP8 through vLLM's Marlin kernels with activations kept in bf16 (weight-only) more than halves NVFP4's KL (0.0256 → 0.0117) and cuts FP8's by ~30% (0.0039 → 0.0028). Even then FP8 only ties UD-Q5_K_XL, which is 8.7 GiB smaller. vLLM has no 4-bit KV cache option at all.
Why it matters
The takeaway is a rule of thumb the community's "never quantize" vs. "4-bit is fine" debate missed: 8-bit KV is the obvious default, and 4-bit KV silently destroys exactly the quants you'd buy it for.
The mechanism: a 4-bit cache's penalty is roughly a flat absolute KL offset, so it lands hardest on the high-precision weights with the least headroom. On Q8_0 — a quant chosen precisely because it is near-lossless — a 4-bit cache multiplies the error ~5x and pushes it past a smaller quant with a full-precision cache. If you are memory-constrained, you are usually better off dropping a weight-precision tier and keeping a 16-bit cache than keeping the precise weights and compressing the cache to 4-bit.
The context dependence compounds that: the penalty grows as the sequence lengthens, so the long-context workloads that most need the memory savings are exactly the ones that pay the most.
The cross-engine comparison is also useful because the author established a calibration baseline: unquantized llama.cpp and vLLM differ by KL 0.0004, which he attributes to two correct implementations doing floating-point math in a different order. That makes KLD numbers comparable across engines, and it is why the vLLM FP8 cache's +0.0004–0.0006 is meaningful — it is on the order of the entire cross-engine numerical difference.
Caveats
These are all self-reported numbers from a single practitioner, and I could not find any independent reproduction of this specific benchmark. Treat them as strong directional evidence, not ground truth.
- The post does not link to any code or dataset, so the benchmark is not directly reproducible as published.
- KLD is a lower bound. The author computes it from the top-200 logits at each position; the reference put on average 0.0036 of its probability outside its top-200 tokens.
- EXL3 quants were scored against the llama.cpp bf16 reference rather than an ExLlamaV3-native reference. The author asserts there should be "essentially no difference," but it is an assumption.
- The "FlashAttention → FlashInfer" explanation for vLLM's FP8 cache penalty is the author's speculation, not a measured cause.
- Weight sizes exclude the KV cache and activations, so the "memory use" figures are not deployment totals.
The results do line up directionally with two independent sources: a community comment on the Qwen3.8-27B HuggingFace discussion page noting that Qwen models are "quite resilient to KV cache quantization... but don't overdo it" on long contexts, and a separate HuggingFace MRCR needle-retrieval benchmark that found FP8 KV degrading on very long prompts. That agreement corroborates the direction, not the specific KLD numbers.