capy-flash v0.7.1: a 125B MoE on 64 GB of RAM — GSQ-RCO i-quant experts at 234 tok/s on two 16 GB cards, 27B 3-bit on one
ISTA-DASLab's ~3-bit GSQ-RCO experts shrink Flash-Next's off-device footprint from 65 to 48 GiB, so the 125B MoE runs on a 64 GB machine with one or two RTX 50-series cards — and a 3-bit Qwen3.8-27B fits a single 16 GB card.
capy-flash v0.7.1, released October 22, runs the 125B-parameter MoE Qwen3.8-Flash-Next on 64 GB of RAM and one or two 16 GB RTX 50-series cards: 234 tok/s on two cards in the developer's own measurements. It also fits a 3-bit Qwen3.8-27B on a single 16 GB card at 174 tok/s. The load-bearing change is the expert weights: instead of the 65 GiB of routed parameters the NVFP4 artifact needs in host RAM, capy-flash now loads ISTA-DASLab's ~3-bit GSQ-RCO i-quants, which fit in 48 GiB and run on the CPU when not cached in VRAM.
What shipped
capy-flash is an Apache-2.0 inference engine for Qwen3.8-Flash-Next and Qwen3.8-27B on one or two 16 GB RTX 50-series (sm_120) GPUs, "[built for agents]" (repo). It descends from ninfer: the author renamed his two-GPU tensor-parallel fork at v0.5.0, and v0.7 added GGUF i-quant expert formats with new AVX-512/AVX2 CPU and GPU kernels. v0.7.0 "Start small" was released October 22 at 08:27, with a v0.7.1 fix the same day at 12:59.
The headline configuration is the "64 GB edition" of Flash-Next. Its routed experts — 48 layers × 512 experts — come from ISTA-DASLab's GSQ-RCO GGUF release, copied verbatim rather than re-quantized: a mixed ~3-bit recipe (gate/up weights in IQ2_S, IQ3_XXS, IQ3_S or IQ4_XS per layer, down projections in IQ4_NL/Q2_0) that needs 48 GiB of host RAM instead of the 65 GiB of the NVFP4 experts. "Hot" experts sit in a VRAM cache partitioned by expert ID across the two GPUs; the rest are computed on the CPU (README). Author-reported peaks: 55.2 GiB of RSS inside a 64 GiB memory cgroup without swap, and about 57-58 GiB in the one-GPU profile (release notes). The model card for the resulting weights is also the source for the layer-mix description above.
Speeds per the announcement's table ("LocalMaxxing reasoning-v1, one request, MTP, verified runs"; RTX 5070 Ti 16 GB, Ryzen 9 9900X):
| Model, 3-bit | 1× 16 GB GPU | 2× 16 GB GPUs |
|---|---|---|
| Qwen3.8-Flash-Next (125B MoE) | 121 tok/s | 234 tok/s |
| Qwen3.8-27B | 174 tok/s | 288 tok/s |
The other new configuration is Qwen3.8-27B on a single 16 GB card: ISTA-DASLab's 3-bit GSQ weights (q3_g128_fp16 codebooks, imported verbatim) plus the official MTP speculative-decoding head, INT8 KV, 65,536 tokens of context, vision, at 13.9 GB of VRAM. v0.7 also adds AVX2 CPU kernels (Zen 2/3, Intel client), so Flash-Next no longer requires AVX-512. A Windows/WSL2 fix lets the server start even when the system grants less page-locked memory than the experts need; the overflow layers stay in pageable RAM and transfer through a pinned staging ring (changelog). Two empty-reply guards are on by default: a terminal token drawn right after thinking is redrawn, and a reply that ends inside its reasoning is closed and continued once.
v0.7.1 fixes the one-GPU Flash-Next profile, which held too few conversation checkpoints in RAM. The fix keeps 8 checkpoints in RAM (+0.66 GiB): on a 53-turn agentic replay, prompt tokens served from cache rose from 34% to 83%, median time-to-first-token fell from 13.3 s to 1.9 s, and whole replay time from 858 s to 276 s (v0.7.1 notes).
Why it matters
Most MoE-offload work treats PCIe as the bottleneck and schedules transfers. capy-flash inverts that: routed experts live in host RAM and are computed on the CPU with AVX-512 VNNI/AVX2 kernels the author says are bit-identical to the GPU's W4A4 path inside the captured CUDA graph; tensor parallelism runs over PCIe without peer-to-peer (pinned-host mailbox, FP32 partial sums); and a VRAM expert cache partitioned by expert ID decides what stays resident on the GPUs (README). The developer's bet is that the constraint is RAM bandwidth, not PCIe width. That bet is what makes a 125B MoE at 121-234 tok/s plausible on a 5070 Ti-class card with 64 GB: the quant cuts the off-device footprint by a quarter, and CPU compute is cheap enough that cache misses stop dominating.
The quantitative win is the hardware floor. The default Flash-Next artifact needs two 16 GB cards and ~96 GB of RAM; the 64 GB edition needs "[one or two] cards and a 64 GB machine (~56 GiB free)", with the same engine and model files either way — "start small, grow later" (README). The dense 27B at 13.9 GB of VRAM is the first capy-flash configuration that fits a single consumer card, with 64k context and MTP intact.
On quality, the author's paired runs (3-bit capy-flash vs NVFP4 served by vLLM on a GB300, same prompts) put the quantized Flash-Next experts "on par" with NVFP4: GSM8K 0.982 vs 0.985 (330 questions), MMLU-Pro 0.837 vs 0.849 (252 questions) (model card). The dense 27B is where 3 bits cost something: MMLU-Pro drops to about 0.72 vs 0.78, and the author says the NVFP4 27B remains the better choice if you have two cards (announcement).
What makes the two-card numbers easy to believe is the author's concession against his own interest: "[If you have a single 16 GB card], Strata is probably already the better choice for Flash-Next today. capy-flash is at its best with two cards, where it adds tensor parallelism and a VRAM expert cache shared by both GPUs" (announcement).
Caveats
- Every benchmark above is the developer's own, measured on a single machine: AMD Ryzen 9 9900X, 96 GB DDR5-5600, 2× RTX 5070 Ti 16 GB on PCIe 5.0 x8 + x8 without P2P, Ubuntu, CUDA 13.1 (README). No third party has re-benchmarked the two-GPU numbers; the only independent confirmation is one commenter's ~170 tok/s on the single-card 27B.
- The announcement table and the GitHub measured tables disagree. The README/release notes report 239 tok/s for two-GPU Flash-Next on the same LocalMaxxing prompt, and 139/230 tok/s for the 27B with MTP on an 8,000-token prompt, versus 234 and 174/288 in the announcement (release notes, README). The author does not reconcile the gaps; treat all figures as single-machine, condition-dependent.
- The quality numbers are author-run paired single-checks, and the author notes the engine is not bit-reproducible run to run because which experts sit in VRAM depends on timing (README).
- The quant naming is loose. A commenter pointed out that ISTA-DASLab does not publish IQ4_XS/IQ4_NL formats for this model; the author acknowledged the model card was "maybe ambiguous" and said he would fix it (announcement thread). Treat the layer-mix description as the author's summary of the weights, not ISTA-DASLab's spec.
- The "i-quants" short form is used without an official expansion in these sources; the underlying formats are the GGUF IQ family (IQ2_S through IQ4_NL, Q2_0). GSQ and RCO expand to Gumbel-Softmax Quantization and Riemannian Constrained Optimization in ISTA-DASLab's own README.
- RTX 50-series (sm_120) only, no Ada/Ampere support, and a mixed pair runs at the pace of the slower card (README).