A 125B MoE at ~200 tok/s on a 16GB RTX 5080: Basalt, Unsloth's UD-IQ3_XXS, and a further-trained drafter
A reproducible guide runs Qwen3.8-Flash-Next on a single 16GB card by keeping experts in system RAM, computing them on the CPU in parallel, and drafting with the model's own MTP head.
A 125B-parameter MoE with 6B active runs at ~200 tok/s decode on a single 16GB RTX 5080, per a reproducible guide. The card never holds the model: a purpose-built engine keeps the experts in system RAM, computes them on the CPU in parallel with the GPU, and drafts with the model's own MTP head.
What shipped
A LocalLLaMA user (u/yehyakar, GitHub xyehya) published a reproducible guide for running Qwen3.8-Flash-Next on one RTX 5080 16GB with 64GB of system RAM. It combines three independent open-source pieces: jesdga95's Basalt inference engine, Unsloth's new UD-IQ3_XXS quant (Dynamic V3.1, uploaded Nov 4), and eddoursul's further-trained Lamina MTP drafter. It ships as a pinned, SHA-256-checked setup script on a branch of the Basalt repo.
Qwen3.8-Flash-Next is a 125B-parameter MoE with 6B active, plus a 51B n-gram embedding and a 4B MTP layer — 512 experts (10 routed + 1 shared), 262,144-token native context, and Qwen's experimental preview of the architecture that will underpin Qwen4 (Gated DeltaNet + Qwen Sparse Attention). Qwen released it Aug 26, 2026 under the Qwen Community License 1.0.
Basalt is the engine. It's an MIT-licensed fork of Strata, Blackwell-only (sm_120, CUDA 13), by jesdga95. Its trick: experts that don't fit in VRAM live in system RAM and are computed on the CPU in parallel with the GPU, with an optional second GPU as a peer expert tier, and it uses the model's own MTP layer for speculative decoding. The guide builds on upstream v0.2.1 plus the author's own patches — host KV prefault, host KV pool on normal pages, spin-then-yield in the decode loop, an expert-start profile, a streamed conversation disk cache, and --adapt-margin for expert-swap hysteresis — two of which are credited in the official 0.2.1 changelog.
The headline numbers, per the author on the same box and same model file:
- ~3.5x llama.cpp decode without drafting, ~5x with drafting, ~7x faster prompt reads.
- A real opencode agent session (151 requests, up to 202K context): 195.6 tok/s average decode, 263 tok/s best request, 156 tok/s worst, ~5,700 tok/s prompt reads.
The new quant. Unsloth's UD-IQ3_XXS (Dynamic V3.1) is 82.0 GB — about the same footprint as the GSQ-RCO IQ3_S that's the usual choice for 16GB cards — with top-1 accuracy 85.414 and mean KLD 0.165120 per Unsloth's own table. The author reports it costs 1–3.5% decode speed over IQ3_S.
The drafter. Lamina's MTP drafter (a separate artifact, eddoursul/Lamina-Qwen3.8-Flash-Next-MTP) starts from Basalt's self-distilled MTP weights and is trained further on Lamina's own output (987 requests, 2.4M positions). On its held-out data it carries 2.489 tokens per verify pass on IQ3_S vs 2.402 for Basalt's stock dense part. The author measured draft acceptance rising from 83.1% to 85.8% on a replay of a recorded session.
Why it matters
A $1,000-class 16GB consumer GPU runs a 125B MoE at usable agent speeds. That's not a GPU-memory story — the card never holds the model. It's an engine story: keep the experts in system RAM, compute them on the CPU in parallel with the GPU, and use the model's own MTP head to hide latency. The guide is a working, reproducible recipe that combines three independent open-source efforts, and the Basalt repo's own community benchmark table independently corroborates the ballpark: a single RTX 5080 with the older IQ3_S quant decodes at ~138 tok/s.
For engineers, it's a concrete, pinned configuration you can reproduce this week — not a vendor slide. It also points at where local inference is heading: purpose-built MoE engines and model-native speculative decoding are closing the gap between what fits in VRAM and what runs at agent speed.
Caveats
- The llama.cpp multipliers (3.5x/5x/7x) are the author's own measurements. No controlled llama.cpp baseline is published, and community data points are mixed, so treat them as directional until someone runs a head-to-head.
- The author's UD-IQ3_XXS wikitext perplexity of 3.247 is a red flag: it sits below the BF16 reference of 3.273 on the same text, and a 3-bit quant shouldn't beat full precision on identical rows. Basalt's own quality table lists IQ3_S at 3.383 and BF16 at 3.273. Treat the 3.247 figure as unverified; the directionally plausible claim is that UD quants are more accurate at equal size.
- The agent-session numbers are a single real workload, not a benchmark; a commenter asked about prompt-cache hit rate and it wasn't fully answered.
- Basalt is Blackwell-only (sm_120 + CUDA 13). The speedups don't transfer to RTX 30/40-series without community ports that lose Blackwell-specific features.
- Naming: "Lamina" is technically the Strata-fork engine; the drafter used here is the separate Lamina-Qwen3.8-Flash-Next-MTP artifact. And "same size as IQ3_S" is roughly true (82.0 GB vs 81.0 GiB) but the packs differ in contents — Basalt packs add the MTP drafter, vision projector, and int8 hyper-connection table.