Measured: MoE offload speed hacks undo a quant's precision — Q8_0 with shipped flags lands next to Q2_0 on KLD

Two MoE offload "speed hack" flags — expert routing bias and tail skip — can undo several quant levels of precision on Qwen 3.8 Flash Next. With the flags set to values some engines ship as defaults, a 178 GiB Q8_0 model lands next to a 65 GiB Q2_0 on KL divergence (0.401 vs 0.436 KLD).

What shipped

Jesús Gulfo, developer of the Basalt inference engine, posted the ablation to r/LocalLLaMA on October 31. He added the two flags to Basalt as opt-in switches so he could measure their cost with everything else fixed, then ran the model on an RTX 5090 + RTX 5060 Ti with 128 GB of DDR5. Qwen 3.8 Flash Next is a 125B-parameter MoE with 512 experts, 10 routed plus 1 shared active per token — the kind of model that only fits on a consumer GPU with offloading.

The two flags:

• Routing bias (route prior): pushes the router toward experts already resident on the GPU by adding a constant to their router logits, so the engine sometimes computes a convenient expert instead of the one the router wanted.
• Tail skip: when a low-ranked expert pick isn't on the GPU, skip it instead of fetching it.

Settings tested: "shipped" = bias 1.5, skip rank ≥ 7 (values other engines ship, or have shipped, as defaults on smaller cards); "worst" = bias 16, skip rank ≥ 1 plus relaxed draft acceptance.

KLD, 100 wikitext chunks, decode path, vs a llama.cpp Q8_0 reference:

Pack + setting	File	PPL	KLD mean	Same top-1
Q8_0 exact	178 GiB	3.276	0.031	94.1%
NVFP4 exact	122 GiB	3.304	0.068	91.4%
IQ3_S exact	81 GiB	3.385	0.155	87.3%
IQ3_XXS exact	74 GiB	3.470	0.243	84.0%
IQ3_S + shipped	81 GiB	3.583	0.244	84.3%
NVFP4 + shipped	122 GiB	3.931	0.348	81.8%
Q8_0 + shipped	178 GiB	4.221	0.401	79.4%
Q2_0 exact	65 GiB	3.949	0.436	78.2%
NVFP4 + worst	122 GiB	5.917	0.823	71.3%
Q8_0 + worst	178 GiB	12.858	1.616	56.8%

With the shipped values, every pack lands between IQ3_XXS and Q2_0 on KLD. A Q8_0 ends up next to a Q2_0 — the flags undo several quant levels of precision, and the damage is roughly the same whether the model is quantized or not.

KLD is only a proxy, so the author also ran task evals: Ladders (300 chained steps, scored by step) and Facts (1,200 closed-book questions across 5 popularity bands, 3 seeds, thinking off). Q8_0 exact scores 45.4 / 38.1 at 117 tok/s; Q8_0 + shipped drops to 43.8 / 36.6 (Δ −1.6 [−3.4, +0.2] and −1.5 [−2.5, −0.5]).

On the big card the flags buy little: nothing on NVFP4 (272 → 271 tok/s) and more on Q8_0 (117 → 202 tok/s), for ~1.5 points and ~4% of known facts in the NVFP4 case. On a small card (the 5090 alone, capped to 16 GB with a 5 GB expert cache, to mimic a 16 GB card) the trade looks different: IQ3_S exact scores 44.1 / 36.7 at 192 tok/s, IQ3_S + shipped 38.9 / 34.3 at 299 tok/s — +56% speed for roughly three times the damage. At the worst setting, Facts collapses from 38.1 to 2.7 and the model "says things like 'The Great Composer' with total confidence."

Two things make this measurement more credible than the usual vendor bench:

• The accused party measured itself. Basalt ships both flags off by default, never auto-picks them, and prints a lossy flags enabled warning at startup. The numbers in the post match Basalt's own published quality gate in the repo's docs/precision/routing.md, and the flags are in the public repo, so the whole thing is rerunnable.
• The "shipped" values are real defaults — Strata's. The post names no engine, but bias 1.5 / skip rank ≥ 7 match the defaults of Strata, the engine Basalt forks and the biggest in this niche: STRATA_ROUTE_PRIOR=1.5 was the default on smaller cards (≤13 GiB) and ROUTE_TAIL_SKIP=7 has been the default since Strata 0.1.42. Strata disabled its route-prior default on 2026-10-29 — two days before this post — over repetition loops in long agent sessions, not over quality metrics.

Why it matters

The practical takeaway is a checklist item for anyone running a MoE model that doesn't fit in VRAM: if your engine is faster than it should be on a model that doesn't fit, check whether it's still running the model you downloaded. The flags are invisible in the GGUF — they live in engine config and change which experts get computed, not just how fast.

The cost profile is the non-obvious part. On a big card, routing bias and tail skip are nearly pure downside: NVFP4 loses ~4% of known facts for 1 tok/s. On a small card they buy real speed (+56%) at a real quality cost — and small cards are exactly where engines turn these flags on by default. The task evals show the damage concentrates in chained reasoning and recalled facts, which is what agent workloads depend on, while easy tests barely notice. That asymmetry is why KLD tables alone understate the problem.

There is also a model-side twist: Qwen 3.8 Flash Next itself can lock into an endless run of "0" tokens in its reasoning on some prompts. The author reproduced it 4/4, 4/4, 3/4, 3/4 across quants and engines, and 6/6 on the official API (OpenRouter pinned to Alibaba), which silently cuts the run at ~500 zeros. Basalt patched it with a thinking-loop cutoff that closes the thinking block after 256 identical tokens. Not every bad output is the engine's fault.

Caveats

• Single rig, single model, and the author is the engine's developer (disclosed; the post notes it was "polished by AI").
• KLD measures divergence from a reference, not task performance; the author's own task evals show the damage is task-dependent.
• The post names no engine; the "shipped defaults" attribution to Strata is inferred from Strata's public configuration and the thread's commenters, not stated in the post.
• The KLD and task numbers are the author's own measurements from a single rig. The only independent data point we found is helix.ml's speed benchmark — 229 tok/s on 2× RTX PRO 6000 — which explicitly did not independently evaluate the pack's quality figures.
• Qwen's own benchmark claims for the model are vendor-reported.

Sources

• Reddit post: "Is your engine lobotomizing your model? Kinda, and I measured it"
• Basalt repository
• Basalt docs: routing flags and the quality gate
• Basalt docs: configuration reference
• Basalt issue #153: Strata turned its route-prior default off
• Strata repository
• Strata issue #4402: route prior default on small cards
• Ken Ashe: what expert-skip defaults cost on a single 4090
• Qwen 3.8 Flash Next model card
• Qwen 3.8 Flash Next tech report
• GSQ paper (arXiv:2604.18556)
• RCO paper (arXiv:2605.00649)
• helix.ml: independent Basalt speed benchmark