Lean-QwenFlash streams a 125B MoE's experts straight from NVMe: ~20 tok/s on a 16 GB card with 8 GB of RAM
A port of sybil-solutions' GLM offload engine replaces 48–64 GB of pinned RAM with a disk-backed third memory tier for Qwen3.8-Flash-Next — quality and SSD endurance remain unproven.
Lean-QwenFlash, a new open-source runtime, runs Alibaba's Qwen3.8-Flash-Next — a 125B-parameter MoE with a 51B-parameter n-gram table — at 20.0–20.8 tok/s decode and 1,536–1,627 tok/s prefill on an RTX 4090 capped at 16 GB VRAM and ~8 GB of system RAM. It streams the model's rarely-activated experts straight off an NVMe SSD instead of pinning them in main memory.
What shipped
Lean-QwenFlash (https://github.com/MaxDevv/lean-qwenflash), released to r/LocalLLaMA on 25 October (https://www.reddit.com/r/LocalLLaMA/comments/1xev6bo/), is a Python runtime built on a vendored copy of ExLlamaV3 1.6.0 plus native offload extensions. Per the README it is adapted from sybil-solutions' glm53-flash-offload engine (https://github.com/sybil-solutions/glm-flash-lite), the project that runs GLM-5.3-Flash from a single 24 GB RTX 3090 using RAM and NVMe tiers instead of an all-RAM working set.
The trick works because of architecture, not just plumbing. Per Alibaba's model card (https://huggingface.co/Qwen/Qwen3.8-Flash-Next/raw/main/README.md), Flash-Next is a hybrid Gated DeltaNet + Qwen Sparse Attention (QSA) model: 48 layers, each routing through 512 experts with just 10 routed plus 1 shared expert active per token — ~2% of expert weights do the forward pass at any instant.
Alibaba also built the 51B-parameter n-gram embedding table to be offloadable; the QwenLM repo (https://github.com/QwenLM/Qwen3.8-Flash-Next) says it "can be offloaded to host memory and overlapped with model computation through asynchronous prefetching." Lean-QwenFlash takes that design affordance all the way to disk.
The runtime works as three memory tiers:
- GPU (16 GB VRAM): dense attention mixers, DeltaNet recurrent state, routers, shared experts, the output head, a 4-bit KV cache, and an adaptive expert cache.
- Host RAM: a configurable 4–16 GB cache holding the most recently activated experts.
- NVMe SSD: a ~47 GB packed expert store plus the 32.6 GB n-gram table. Misses are read with O_DIRECT (bypassing the page cache) through async ring buffers and staged ahead of the compute stream. All expert math stays on the GPU — misses cross PCIe rather than falling back to CPU execution.
All throughput figures are the author's, measured on the turboderp/Qwen3.8-Flash-Next-exl3 checkpoint (https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3) at 3.05bpw_h5_ng5 with a 4-bit KV cache:
| Hardware | Context | Host cache | Prefill | Decode |
|---|---|---|---|---|
| RTX 4090 (16 GB VRAM, ~8 GB RAM) | 8k | — | 1,080 tok/s | 28–33 tok/s prose, 19–22 tok/s code |
| RTX 4090 (16 GB VRAM, ~8 GB RAM) | 100k | — | 1,536 tok/s | 20.8 tok/s |
| RTX 4090 (16 GB VRAM, ~8 GB RAM) | 262k (full window) | — | 1,627 tok/s | 20.0 tok/s |
| RTX 3080 Laptop (16 GB VRAM) | 100k | 8 GB | 647 tok/s | 17.6 tok/s |
| RTX 3080 Laptop (16 GB VRAM) | 100k | 16 GB | 693 tok/s | 20.9 tok/s |
Turboderp's own README labels this branch "3.05 bits per weight / H5, NG5"; NG5 means a 5-bit n-gram table, but the H5 label's meaning is not defined in the documentation. The branch totals ~52.3 GB of weight shards plus a 32.6 GB n-gram table, so the download is the ~85 GB the README quotes.
The server layer is OpenAI-compatible, with streaming, reasoning-effort control (off/low/medium/xhigh), and function calling; the README points it at Pi, OpenCode, or Claude Code with the full 262,144-token window. Recurrent state checkpoints are cached alongside the KV cache so cached follow-up turns start decoding at a reported 0.8–1.4 s TTFT.
The comparison the author draws is against Strata (https://github.com/Niko1221/Strata), the current community engine for this model. Strata's design docs (https://github.com/Niko1221/Strata/blob/main/docs/HOW_IT_WORKS.md) describe RAM holding all 24,576 experts with the CPU computing any expert the GPU doesn't have, and its model picker (https://www.stratat.com/en/models) lists Flash-Next quants needing 32 GB+ of RAM (Coder), 48 GB+ (Q2_0), 64 GB+ (IQ2_XS, IQ3_XXS, IQ3_S), or 80 GB+ (UD-IQ4_XS).
That framing is slightly simplified: Strata also ships mmap and low-RAM resident modes and already reads its n-gram table from SSD, so the real difference is which tier serves a miss, not RAM-only versus SSD.
Why it matters
Lean-QwenFlash converts NVMe bandwidth into RAM capacity for a model people already run, and it changes what a reasonable local setup looks like. The requirements are Linux, an NVIDIA GPU with ≥16 GB VRAM, and a PCIe 3.0/4.0 NVMe drive with ~3+ GB/s reads — the hardware class of a recent gaming laptop. That is why a 47 GB expert store whose weights are ~2% hot can stream instead of load.
It also validates "disk as a third memory tier" as a recognizable technique rather than a one-off hack. The lineage is explicit: sybil-solutions' GLM work (https://github.com/sybil-solutions/glm-flash-lite) — which itself credits FreeToken/FlashML for the host-memory expert-tier idea — documented O_DIRECT record stores and layer-ahead prefetch. Lean-QwenFlash ports it to a different model; the vendored ExLlamaV3 copy (https://raw.githubusercontent.com/MaxDevv/lean-qwenflash/master/exllamav3/README.md) still advertises CPU offloading but no NVMe tier.
A 125B-class MoE with agents attached now runs on a laptop at 100k context; the full 262k window was demonstrated on the RTX 4090.
Caveats
- Everything measurable is author-reported. Independent coverage explicitly notes no reproduction was established, and the numbers come from a single hardware set (one 4090, one 3080 laptop).
- No quality evaluation. There is no perplexity or task-score comparison of the disk-streamed 3.05 bpw setup against the base model or an all-RAM baseline. 3.05 bpw with a 5-bit n-gram table is aggressive quantization, and nothing in the repo shows what it costs in output quality.
- Licensing is unresolved. The repo ships no license file (the vendored ExLlamaV3 and the sybil engine it derives from are MIT; the repo itself is not). The model is under the qwen-community-1.0 license, not an OSI-approved open one.
- SSD endurance questions are unanswered. Commenters on the announcement thread asked about drive health under sustained streaming; the README publishes no endurance or write-amplification data.
- "8 GB of RAM" describes only the 4090 config. The 3080 laptop runs used 19.0–23.9 GB of system RAM depending on the host cache size.
- Untested territory. 12 GB and 8 GB GPUs, and 24/32 GB GPUs, are on the author's own to-do list; only the 16 GB class has numbers.
Sources
- https://github.com/MaxDevv/lean-qwenflash
- https://www.reddit.com/r/LocalLLaMA/comments/1xev6bo/
- https://huggingface.co/Qwen/Qwen3.8-Flash-Next/raw/main/README.md
- https://github.com/QwenLM/Qwen3.8-Flash-Next
- https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3
- https://github.com/sybil-solutions/glm-flash-lite
- https://github.com/Niko1221/Strata
- https://github.com/Niko1221/Strata/blob/main/docs/HOW_IT_WORKS.md
- https://www.stratat.com/en/models
- https://groundtruth.day/news/lean-qwenflash-streams-125b-model-from-ssd.html
- https://raw.githubusercontent.com/MaxDevv/lean-qwenflash/master/exllamav3/README.md