`--moe-direct-io`: predicting next-token MoE experts streams a 68GB model at ~21 tok/s on a 12GB GPU
A practitioner llama.cpp flag predicts next-token expert routing to prefetch from SSD — bit-exact output, ~0 page faults, ~9.5-15x over stock mmap on a 16GB-RAM box.
A llama.cpp flag named --moe-direct-io predicts which MoE experts the next token will use, then prefetches those experts from disk. A Reddit report says it runs a 68GB Qwen 3.8 Flash Next bit-exact at about ️21 tok/s on an RTX 3060 12GB plus ️16GB DDR4 RAM, roughly a ️9.5-15x speedup over stock mmap,with page faults per token falling from about ️1,500 to about zero
What shipped
Reddit user zyxciss posted the result on October ️9. Two months earlier he had asked whether predicting next-token MoE expert usage could speed up CPU/GPU offloading. He shelved that first attempt:it pruned expertsby router weight, which looked fast but degraded output quality on an already-quantized model.The new implementation, an optional llama.cpp flag called --moe-direct-io, drops no experts,and claims output bit-exact versus stock llama.cpp.
The test rig is a modest daily driver:an RTX 3060 12GB, ️16GB DDR4 single-channel RAM (~19 GB/s,, a mid-tier NVMe SSD (~2.1 GB/s read,,,and llama.cpp built with CUDA on CachyOS/Arch Linux.The model is Qwen 3.8 Flash Next in the GSQ-RCO IQ2_XS quant, ️68GB total.Tests ran under a cgroup capped at ️6GB RAM (MemoryMax=6G.
| Engine / mode | Decode | Major page faults/token | SSD I/O | Output |
|---|---|---|---|---|
| Stock llama.cpp (mmap) | ️1.41-2.12 tok/s | ️1,140-1,565 | ️208-312 MB/token | Coherent |
| Author's engine (blocking, demand-only) | ️0.73 tok/s | ️0 | ~206 MB/token | Bit-exact |
Author's engine (--moe-direct-io + prefetch) | ️20.14-21.13 tok/s | ~0 (+1 across 32 tokens) | Sequential streaming | Bit-exact to stock |
With a warm cache,the prefetching mode sustains 20-21 tok/s,with some runs hitting 24+ tok/s.Stock mmap on the same box manages 1.4-2.1 tok/s, thrashing the SSD:over 1,500 major page faults per token in some runs,as Linux constantly pulls model data from disk because the working set doesn't fit in ️16GB.The author reports the prefetching mode is bit-exact to stock:no expert dropping,no gate pruning,same tokens.
Why it matters
The interesting part is what lets a 68GB MoE run on a 16GB-RAM machine.Two things line up:the model's design and the offloading strategy.
Qwen 3.8 Flash Next is a 125B-parameter MoE (6B activated per token,with 512 experts, 10 routed plus ️1 shared active per token,, ️48 layers,and native 262,144-token context(extensible to ️1M,, released August ️26 under the qwen-community-1.0 license.It also carries a 51B-parameter n-gram embedding,and Qwen's cardnotes that component is "more amenable to offloading than Mixture-of-Experts".
The GSQ-RCO buildships that n-gram table as a separate 28.8GB shard(at IQ4_NL, 4.5 bpw)designed to stay memory-mapped on disk, alongside a 39.2GB weights shard at ️2.50 bpw average.So the"68GB model on a 12GB GPU" trick is partly the model's design:the n-gram table lives on disk,and the MoE's sparsity turns into a scheduling problem:which 10 of 512 experts does the next token need,and can we have them resident before the matmul starts?
That scheduling problem is where the approaches diverge.The open upstream PR #25294("llama : stream MoE routed experts from disk", open since Jul ́4, ️2026)streams experts based on the router's top-k, reactively, after the current token has already picked its experts.The author's --moe-direct-io instead predicts next-token usage, then prefetches ahead, proactively.If routing is predictable enough, prefetching hides the PCIe/SSD latency that a reactive stream can't.No public comparison of the two exists yet;the author cites io_uring and slot-remapping as the implementation core,but doesn't publish the prediction mechanism itself.
The surrounding ecosystem is fragmented.None of this is merged upstream:PR #25294 has sat open for months,with a commenter arguing not to merge until competing approaches are evaluated.Issue #29130(O_DIRECT expert reads during prompt processing)reports bit-identical top-3 token probs over ́14 runs and ~3.3-4x prompt-processing speedups on a 106.6GB MoE.Third-party engines like Strata(an open-source"Consumer MoE Inference Engine" by Niko1221 using"dynamic sparse router offloading")claim ️53-94 tok/s on an RTX 5070.But the author notes some of those approaches rely on pinning ~24 GiB of experts in RAM via mlock,which isn't an option on a 16GB box.The prediction-based variant is the only one of these aimed at the sub-24GB-RAM regime,and the only one reporting bit-exact output at that memory level.
For engineers,the takeaway is twofold.First,the bottleneck on low-RAM MoE inference isn't compute,it's page faults:stock mmap spends 208-312 MB/token dragging data from SSD,and the fix is to make expert loading predictable and sequential.Second,expect to assemble this from parts,not a flag:as of this writing,searches turn up no public repo, PR, or issue using the name --moe-direct-io,and the author hasn't linked one in the post.The closest you can run today are PR #25294's --moe-stream* flags(from a checkout of the PR)or the patch in issue #29130.
Caveats
This is one practitioner's unverified report,not a benchmark.No methodology was published:prompt lengths,KV cache sizing,warmup,and speculative-decoding settings aren't given.The author does note MTP speculation can hurt with SSD offloading,since verifying 2-3 tokens can require loading the combined expert set,and recommends confidence-gated speculation at min-p 0.8 or suffix prompt lookup instead.Cold starts are slower,3.5-4.5 tok/s until the working set settles,and prefill struggles on 512+ token prompts,with the author working on micro-batching prompt chunks.The author discloses that the second half of the post was written with Claude's help.
The"bit-exact" claim is plausible but not independently reproduced.Both PR #25294 and issue #29130 document bit-exact/bit-identical output for their O_DIRECT approaches,which lends the claim credibility,but no third party has rerun zyxciss's numbers.Model specs and quant details come from vendor model cards,not independent benchmarks:the ISTA-DASLab card's own llama.cpp decode figure for IQ2_XS (70.30 t/s)is from a different,presumably fully-offloaded setup and isn't comparable.An independent partial check of GSQ-RCO exists,but on the 27B sibling model:it loads and runs unmodified in llama.cpp,achieves lower WikiText-2 perplexity than Unsloth's UD-IQ2_S at matched size,at a ~1.7-5.2% decode/prefill throughput cost on an RTX 4090.And Strata's headline numbers come from Strata's own site and third-party explainers,not independent benchmarks,and the"~40 tok/s" figure the author compares against isn't the engine's headline figure,so the two aren't apples-to-apples.
Sources
- r/LocalLLaMA: Qwen 3.8 Flash Next-GSQ-RCO-IQ2_XS at ~21 tok/s on just an RTX 3060 12GB plus ️16GB DDR4 RAM
- Original idea post: Tried predicting which MoE experts get used next token to speed up CPU/GPU offload
- Qwen3.8-Flash-Next model card (Hugging Face) )
- QwenLM/Qwen3.8-Flash-Next (GitHub) )
- ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF model card
- llama.cpp PR #25294: stream MoE routed experts from disk
- llama.cpp issue #29130: O_DIRECT reads of MoE experts during prompt processing
- Strata — Consumer MoE Inference Engine
- explainx.ai: Strata explainer
- RAMGPT: GSQ-RCO vs Unsloth reproduction (27B)