capy-flash runs a 125B MoE in full NVFP4 on two 16 GB GPUs at 100 tok/s — and documents the TP=2 partial-sum rounding trap
Apache-2.0 engine offloads routed experts to host RAM; the author shows FP16 transport payloads restore single-GPU numerical parity at no speed cost
capy-flash, an Apache-2.0 inference engine released October 16, serves Qwen3.8-Flash-Next — a 125B-parameter MoE with 6B active — in NVIDIA's official NVFP4 quantization on two 16 GB RTX 5070 Ti, decoding at 100.8 tok/s with the routed experts that don't fit in VRAM computed on the CPU. The same release documents a numerical trap for every tensor-parallel engine: at TP=2, partial sums rounded to BF16 before the all-reduce are rounded twice, and where the two halves cancel, large errors occur ~5× more often than on a single GPU.
What shipped
capy-flash v0.5.4 is the two-GPU tensor-parallel fork of Neroued/ninfer (renamed from ninfer-tp2 at v0.5.0), built by ValerioDolci (FeydRowan on Reddit). One binary serves two models: the dense Qwen3.8-27B and the Qwen3.8-Flash-Next, a 48-layer MoE with 512 experts (10 routed + 1 shared per layer) and 262,144 tokens of native context. The weights are NVIDIA's official NVFP4 checkpoint converted with the repo's own recipe — "not re-quantized" — into a 126.6 GB artifact (E2M1 codes, UE4M3 block scales, FP32 per-expert scale), under the NVIDIA Open Model License plus the Qwen Community License 1.0.
How it fits on consumer hardware: tensor parallelism over PCIe without peer-to-peer (a pinned-host mailbox), a VRAM cache of routed experts partitioned by expert id, and the rest computed on the CPU — a 64.6 GiB page-locked host allocation fed to an AVX-512 NVFP4 kernel with the GPU's W4A4 activation quantization, coordinated inside the captured CUDA graph via a handshake (cuStreamWriteValue32/WaitValue32, no GPU spin). Around that: MTP speculative decoding (K=3, acceptance 0.533, 2.59 tokens per window), an INT8 KV cache with an elastic KV/expert budget up to 262,144 tokens, prefix caching across turns, and an OpenAI- and Anthropic-compatible server.
Requirements are specific: Linux, CUDA 13.1, two sm_120 (RTX 50-series) 16 GB GPUs, an AVX-512 CPU, ~70 GiB of free RAM, ~135 GB of disk. Cold start takes 26–29 s, most of it reading the 64.6 GiB of experts from disk.
All numbers below are author-measured on one machine — Ryzen 9 9900X, 96 GB DDR5-5600, 2× RTX 5070 Ti 16 GB on PCIe 5.0 x8+x8, no P2P — against Strata on the same box (0.1.44.1; the agent-task IQ3_S comparison used 0.1.43, which the README notes did not change its IQ3_S path):
- Prefill: 3,778 tok/s on a 15k-token prompt (Strata NVFP4: 2,703; Strata 3-bit IQ3_S: 2,906); 3,977 on 55k (3,821 / 4,037).
- Agentic replay (53 turns, prompts to ~81k tokens, ~90% prefix reuse): TTFT median 1.82 s vs 2.42 / 1.86; decode 100.8 tok/s vs 78.3 / 124.1.
- Agent tasks (5 multi-step coding tasks × 4 runs): 20/20 solved, 0 malformed tool calls in 1,124; 42.9k generated tokens per run vs 52.7k for Strata IQ3_S and 26.5k for the GB300 reference — Strata's 3-bit build generates 23% more tokens, so both agents finish in about the same time (4.28 h vs 4.32 h for 20 runs).
- Quality vs the datacenter reference (NVIDIA's NVFP4 checkpoint served by vLLM on a GB300, same prompts): GSM8K 0.9765 vs 0.9810 (McNemar p 0.070); MMLU-Pro 0.8452 vs 0.8452 (non-inferior within a 2.5 pt margin).
Why it matters
The headline is the port; the durable part is the side finding. In any tensor-parallel engine, each rank rounds its partial sum to a 2-byte format before the all-reduce. At TP=2 the value is rounded twice where a single GPU rounds once — and when two large halves nearly cancel, the relative error blows up. The author reports large errors ~5× as often on Qwen3 at TP=2 as on one GPU. The fix is nearly free: carry the exchange payload in FP16 instead of BF16 — same 2 bytes, 8× finer mantissa — with a power-of-two shift (v·2^-k, saturated to the FP16 range) and an exact rescale on receipt to rule out overflow. capy-flash made FP16 the default for the prefill MoE exchange (tag T127; BF16 was the earlier default), keeps decode MoE exchanges in FP32, and accumulates partials in FP32 throughout. The author is explicit that the effect on benchmarks is small — "free numerical parity rather than a quality boost" — but it is a checkable, transferable fix: any TP engine shipping BF16 all-reduce payloads can test the same change.
The second pattern worth stealing is the CPU expert tier. The design choices — a single page-locked host allocation, the AVX-512 VNNI kernel running inside the captured CUDA graph with a no-spin handshake, and activation quantization matched to the GPU path (W4A4) so output doesn't depend on which experts happen to be resident — are the difference between a working offload and a PCIe bottleneck. This is what makes a 125B MoE run on two 16 GB consumer GPUs.
Third: NVFP4 as published, not re-quantized. The quality gate against the GB300 reference is the strongest part of the report — statistically non-inferior on GSM8K and MMLU-Pro — and the conversion is reproducible from NVIDIA's checkpoint at a pinned revision hash, so the artifact is auditable rather than a black-box re-quant.
Caveats
- Everything above is author-measured on one machine. There is no independent reproduction and no third-party coverage of capy-flash yet; the ~5× statistic is a single-model observation (Qwen3 at TP=2) reported in the announcement.
- The "two 16 GB GPUs" headline hides a ~70 GiB free-RAM requirement (64.6 GiB page-locked); commenters on the announcement called this out.
- Serving limits: max-concurrency 1; output is not bit-reproducible run to run (which experts sit in VRAM depends on timing); decode is bound by host-RAM bandwidth (~270–300 tok/s per MTP window); generated tokens run ~1.6× the GB300 reference — the author attributes the gap to reasoning length, not correctness.
- The HuggingFace card's quality numbers (0.9780 / 0.8482) are from the previous router build; the README's v0.5.3 numbers above are newer.
- Hardware lock-in: sm_120 only, Linux, CUDA 13.1.