tandem.cpp pins MoE experts to a GPU cache and offloads the rest to CPU — 3.3× decode on a 4090 for Qwen3.8-Flash-Next
A general-purpose llama.cpp patch set gets within ~6% of a purpose-built MoE offload engine on a single 4090 — with honest caveats about where it loses.
A llama.cpp patch set called tandem.cpp claims to roughly triple single-user decode speed for Mixture-of-Experts models that don't fit in VRAM. On a consumer PC it reports 97.3 tok/s for Qwen3.8-Flash-Next against 29.8 for stock llama.cpp — a 3.3× gain that lands just behind Strata, the purpose-built MoE offload engine, in exact mode. It keeps the most-used experts in a GPU cache and splits the rest between CPU compute and cache uploads.
What tandem.cpp does
tandem.cpp is an independent patch set for llama.cpp by GitHub user ewhacc, based on llama.cpp commit f5222f25c (2026-10-28). It is not affiliated with ggml-org. For any MoE model llama.cpp supports, it keeps the experts in host RAM and uses the GPU memory left over after the model, KV cache, and compute buffers as a cache of recently used experts. When a token routes to experts that are not in the cache, the miss is split between the CPU — which computes its share while the GPU runs the cached experts — and an upload into the cache. The split follows the measured CPU and PCIe speeds, using the same rule as Strata.
The headline numbers are from the author's own benchmark on a consumer PC (i5-12400F, 64 GB DDR4, RTX 4090), GSM8K decode tok/s:
| Model | llama.cpp | tandem.cpp | Strata |
|---|---|---|---|
| Qwen3.8-Flash-Next IQ3_S + MTP, greedy | 29.8 | 97.3 (3.3×) | 103.6 |
| same, Qwen's default sampling | 29.4 | 96.1 (3.3×) | 98.1 |
| gpt-oss-120b, greedy | 27.8 | 51.5 (1.85×) | — |
| gpt-oss-120b, recommended sampling | 27.7 | 48.0 (1.73×) | — |
The author tuned for Qwen3.8 only, but reports that other MoE models that don't fit in VRAM gain too — typically 1.4–2.1× for gpt-oss-120b, GLM-4.5-Air, Qwen3.6-35B-A3B, and DeepSeek-V4.1, with less gain when the model nearly fits. Prompt processing is also faster, 1.4–1.8× on a 64k-token prompt.
Strata is a dedicated MoE offload engine by Niko Veit (Niko1221/Strata, MIT-licensed, open source). Its default mode skips low-ranked experts (STRATA_ROUTE_TAIL_SKIP), which is faster but approximate — 127.1 tok/s on the same Qwen3.8 setup. The tables above show Strata in exact mode (STRATA_ROUTE_TAIL_SKIP=0), where every routed expert is computed, as llama.cpp and tandem.cpp do.
In that comparison, Strata still wins: 103.6 vs 97.3 tok/s, about 6% ahead. tandem.cpp's own approximate mode (LLAMA_MOE_TAIL_SKIP=7, off by default) lands at 123.6/124.3 tok/s — also behind Strata's 127.1 default. So the honest headline is that a general-purpose llama.cpp patch set gets within ~6% of a purpose-built engine on its home turf, not that it beats it.
The target and the build
The practical target is the engineer with one consumer GPU who wants to run a model whose routed experts exceed VRAM. Stock llama.cpp's default placement puts the dense layers on the GPU and then as many layers' experts as fit, leaving the rest to run from system RAM — exactly the slow path tandem.cpp attacks. The mechanism is a scheduling change, not a quantization or accuracy trade: the README states accuracy is within run-to-run noise between the engines in every row, and the default settings are exact or differ only in floating-point rounding from kernel choice.
The build is a drop-in: build.sh clones llama.cpp, checks out the base commit, applies tandem.patch, and builds llama-server. Running it is a matter of flags — -cmoe keeps every expert in host RAM, --moe-cache-mib auto hands the leftover GPU memory to the expert cache, and -lm none reads weights into RAM instead of mmapping. The author also ships a Qwen3.8 MTP head whose output and expert tensors are quantized to Q4_K (1.8 GiB of VRAM instead of 3.6) to free more memory for the cache.
Qwen3.8-Flash-Next is Alibaba Qwen's open-weight model, described as an early preview of the architecture that will be used in Qwen4, with roughly 6B active parameters per token across 512 experts (10 routed plus one shared) and 48 layers mixing Gated DeltaNet and Qwen Sparse Attention. Its ~4B MTP (multi-token prediction) head is used for speculative decoding, which is how the benchmark reaches 97 tok/s on a single 4090.
Caveats
Every performance number here is self-reported by the author in the GitHub README. There is no independent verification, and the Reddit announcement thread is the only other coverage — it returned HTTP 403 when fetched, so community reaction is unknown. Treat the 3.3× figure as a claim to reproduce, not a settled result.
The README is unusually honest about where it fails. The clearest counterexample is DeepSeek-V4.1-Flash-1017 IQ2_XS, whose 139 GiB of routed experts is three times the consumer PC's RAM and has to be read from SSD. There, tandem.cpp is slower than stock llama.cpp (5.99 vs 6.86 tok/s, 0.87×), because its CPU path does not take misses that must come from the SSD — the author flags this as "to be fixed." Strata, by contrast, keeps a RAM store of the most-used experts, which is why it wins that case (8.41 tok/s).
Two more limits: the gains are tuned for Qwen3.8 and shrink as the model approaches fitting in VRAM, and the CPU path only supports SwiGLU experts (fused or separate gate/up, optional clamp) and gpt-oss's SwiGLU-OAI with biases. For other expert types the server logs a warning and uploads every miss instead — correct, but slower. On a 6-core CPU, speculative-draft heads like EAGLE3 can slow both engines down, because verifying several tokens multiplies the experts the CPU must run.