BitDecoding-FP4: native FP4 KV caching on Blackwell — 2.4–3.0× serving throughput, and the softmax fixes that make 4-bit KV accurate

Researchers at the University of Edinburgh and Tencent Hunyuan have released BitDecoding-FP4, an open-source algorithm–system co-design that serves long-context LLMs with the KV cache stored natively in FP4 microscaling formats. On an NVIDIA B300 it reports 2.4–3.0× higher peak serving throughput than a BF16 KV cache, within 0.5 points of BF16 accuracy across a seven-benchmark average.

What shipped

BitDecoding-FP4 (arXiv:2611.09825, code on GitHub under MIT) targets a known bottleneck: unlike weights, the KV cache grows linearly with sequence length and concurrency and is re-read on every decode step, so it dominates memory capacity and bandwidth in long-context serving.

The paper's premise is that existing 4-bit KV approaches fail at one of two ends. Vanilla NVFP4/MXFP4 quantization costs accuracy — TensorRT-LLM's NVFP4 KV loses 8.5 average points against BF16 on Qwen3-4B — while accuracy-oriented vector-codebook methods pay lookup overhead that erases the decoding gain. Native FP4 has a third benefit beyond density: Blackwell's fifth-generation Tensor Cores multiply block-scaled NVFP4/MXFP4 operands directly, so a decoder skips runtime dequantization entirely (dequantization takes 7–21% of a FlashInfer-XQA decode kernel's time at 128K context, per the paper's measurements).

The contribution is making that native path accurate, with three named error mitigations:

Dual-Residual Query Quantization. The query is stored as the sum of two FP4 tensors: a first approximation plus an FP4 approximation of the residual. The authors report this cuts the query's share of logit error from 60% to ~1% (measured 1.2% on Qwen3-4B). Both terms stack into one narrow FP4 matrix multiply and recombine in FP32 before softmax.

Underflow-Aware Probability Scaling. The most distinctive fix. NVFP4 uses an 8-bit E4M3 per-block scale, and the paper finds the standard max/6 block-scaling rule zeros out long-context softmax probabilities: nearly 90% of probability blocks fall in a tiny-maximum range and round to zero on long reasoning traces. The fix rescales probabilities by a constant factor of 8 inside the online-softmax exponent (it cancels in normalization), caps the scale at 2 for small blocks and 6 above a calibrated threshold, and lowers the underflow floor 24×. MXFP4's E8M0 power-of-two scales don't underflow this way, so the MXFP4 recipe omits probability scaling entirely.

Outlier-Aware Key Quantization. FP4 Tensor Core MMA lays blocks out token-wise, but K's outliers run along the head/channel dimension, so a channel spike contaminates every token-wise block it touches. The fix combines offline-calibrated per-head mean subtraction (static smoothing), a normalized Hadamard rotation that spreads residual spikes while preserving dot products, and a search over candidate block scales to minimize reconstruction error.

At the system level, a transposed softmax spreads reduction work across keys (warp-level butterfly reductions) and fuses probability quantization into that pass; a "stacked query MMA" runs both query terms and integrates with paged KV caches. The cache keeps the first 16 attention-sink tokens and the in-flight page in BF16, storing completed pages in FP4.

All numbers are self-reported. Across seven benchmarks (LongBench Pro, MRCR, RULER, AIME 2024/2025, LiveCodeBench v6, GPQA-Diamond), Qwen3-4B-Thinking-2507 averages 67.27% with BitDecoding-NVFP4 against 67.69% for BF16 KV — a 0.42-point gap — and both Qwen3-4B and Qwen3-30B-A3B stay within the claimed 0.5-point bound. On a B300, end-to-end serving at 64K–128K context delivers 3.00× (Qwen3-4B) and 2.62× (Qwen3-30B-A3B) the BF16-KV throughput at concurrency 1–64, which the abstract rounds to 2.4–3.0×.

The ablations show where the time goes. At 128K context on B300, Qwen3-4B decode latency falls from 3,994 µs with a conventional softmax to 1,638 µs with the transposed layout, then 1,598 µs with query stacking; the MoE model goes 1,988 → 1,040 → 866 µs. Reference BF16 KV: 4,784 µs and 2,356 µs, respectively.

Why it matters

Read the paper this way: once attention runs on FP4 Tensor Cores, it becomes a four-operand precision problem — queries, keys, softmax probabilities, values — and the wall is the softmax, not K/V quantization. The underflow finding generalizes: an E4M3 block scale was never designed for probability distributions with tiny block maxima, and any FP4 KV cache on Blackwell will hit it (MXFP4's E8M0 scales sidestep it, the paper says).

The token-wise layout problem generalizes to anyone writing FP4 kernels: block scales are per-token, so channel-wise key outliers silently inflate an entire block's dynamic range. The static-smoothing + Hadamard + scale-search recipe works, and the paper is unusually specific about it (the eval config exposes every knob: key_smoothing, hadamard_rotation, adaptive_block_scale_search, candidate scales {4, 6}, probability rescale constant 8).

For teams already serving on B200/B300, this is a drop-in open-source vLLM candidate backend, not a proposal: MIT-licensed, TP=1 at 64K, TP=2 at 128K, with kernel and end-to-end benchmark harnesses in the repo. Part of the throughput story is capacity arithmetic — at concurrency 64 with 64–128K contexts, BF16 and FP8 caches run out of memory while the FP4 cache fits every request. That is why 4-bit pays off in the KV cache, not the weights.

Caveats

Everything above is self-reported. This is a preprint, not peer-reviewed, and we found no independent reproduction or third-party analysis.

A few specifics to hold onto:

• The 60%→1% figure is a share of logit error, not end-to-end accuracy. It measures query quantization's contribution to attention-score error; the accuracy claim is the 0.42-point gap on Qwen3-4B.
• The peak-throughput win is partly capacity-driven. BF16/FP8 caches OOM at high concurrency, so part of the 2.4–3.0× is fitting requests into memory, not faster per-request math. Per-request latency is claimed to drop up to 1.81×, not 3×.
• The MXFP4 accuracy claim may be simulated. The repo reuses the NVFP4 cache dtype with MXFP4-style scale search, and the shipped kernels are NVFP4-only — the "1 point with MXFP4" result may not exercise native MXFP4 hardware execution.
• Scope is narrow. Kernels are decode-only — no prefill results appear in the paper — end-to-end serving was measured only on B300, kernel benchmarks only on B300 and RTX PRO 5000, and there's no multi-node. Hardware requirement is Blackwell-class (B200/B300, SM100/SM103) with CUDA 13+ and driver 580+.
• Two claims lack published detail: the RL-training-rollout result gives no model, algorithm, or metrics, and the generalization "up to 770B" doesn't name the model in the accessible text; the largest named models are GLM-4.7 (GQA) and Hy4-preview (MLA), tested with FP8 weights.

Sources

• BitDecoding-FP4 abstract — arXiv
• BitDecoding-FP4 full text — arXiv HTML
• BitDecoding-FP4 repository — GitHub
• Repository README
• Benchmark guide — repository
• Evaluation configuration (method parameters) — repository
• Introducing NVFP4 — NVIDIA developer blog
• AlphaXiv summary with page-level citations (secondary, machine-generated)