Betelgeuse runs a 68GB MoE bit-exact at 22 tok/s on a 12GB RTX 3060 — one CUDA-graph split, and a README that corrects its own hype
An open-source llama.cpp fork pages experts between VRAM, RAM, and NVMe behind a single graph split — but the headline number is a warm-cache figure from a different run, and the author says so.
Betelgeuse, an open-source llama.cpp fork, runs the 68 GB Qwen 3.8 Flash Next MoE at 22.03 tok/s warm on an RTX 3060 12 GB with 16 GB of system RAM — byte-for-byte identical output to stock llama.cpp — by paging experts between VRAM, a host-RAM victim cache, and NVMe behind a single CUDA-graph split. The author's own README corrects the headline the community ran with: 22.03 is a warm-cache figure from a different run than the "cold" 16.80 tok/s, and the upstream PR is an unmerged RFC the automated reviewer already flagged.
What shipped
Betelgeuse (bel-server / bel-cli) is a fork of llama.cpp that treats a MoE's experts as a paged resource rather than a resident tensor. On the tested box — an RTX 3060 12 GB, 16 GB single-channel DDR4, a consumer NVMe drive, all inside a MemoryMax=12G cgroup — the 68 GB GSQ-RCO IQ2_XS build of Qwen 3.8 Flash Next (48 layers, 512 experts per layer, top-10 routing) keeps only the dense layers, GDN, QSA attention, and LM head resident in VRAM, about 3.2 GiB. The remaining VRAM holds ~5,000 expert slots, with more capacity on early layers 0–4, where there is no compute slack to hide a miss (PR #31469).
The key trick is the miss path. Stock llama.cpp splits the compute graph once per layer and synchronizes with the CPU to learn which experts are resident — the author reports ~360 ms/token burned in cudaStreamSynchronize on this workload (Reddit post). Betelgeuse instead resolves topk_indices → slot_ids on-device (GGML_OP_MOE_SLOT_RESOLVE_KICK / _WAIT) and rings a mapped host doorbell only when an expert is actually missing, so the entire decode pass runs as a single CUDA-graph replay. Evicted experts move into a host-RAM victim cache kept exclusive of VRAM (MOE_DROP_MMAP drops the redundant GGUF page-cache backing), and cold misses are read from NVMe in 4 KiB-aligned single-read blobs via io_uring (PR #31469, README).
The repackaged bundle splits the model into a ~3.77 GB dense GGUF, a 35.48 GB experts.bin, and a 26.82 GiB PLE n-gram shard (README). The author's benchmark table reports, on that box:
| Engine | Decode (cold / warm) | Physical disk reads | Graph splits | Parity |
|---|---|---|---|---|
| Stock llama.cpp (mmap) | 1.47 / 1.63 tok/s | 208.3 / 235.0 MiB/tok | 2 | reference |
Betelgeuse --moe-direct-io --moe-slots 5000 | 16.29 / 22.03 tok/s | 35.37 / 12.43 MiB/tok | 1 (CUDA graph) | 100% byte-exact |
Betelgeuse + --auto-warm | 16.80 / 21.85 tok/s | 35.37 / 12.43 MiB/tok | 1 | 100% byte-exact |
The bit-exactness claim is backed by an unusually thorough self-check: 0.000e+00 maximum absolute difference in slot-remap tests, 28,080/28,080 byte-identical router records, 576/576 identical greedy tokens across 17 test configurations, and all 24,576 expert tensors verified byte-for-byte against the source GGUF (PR #31469). No expert is dropped or gate-pruned — which is the point of the author's retraction: in an earlier post about expert prediction on Qwen 3.6 35B, he says he scrapped a gate-pruning prototype because it "quietly degraded output quality and reasoning" on an already low-bit quant (Reddit post). For comparison, the author reports Strata — the other engine built for this model — at 8.11 tok/s on the same box under --low-ram mmap (Reddit post).
The engine is MIT-licensed on GitHub; the pre-packed Alpha-Orionis-XS bundle is on Hugging Face (explicitly not compatible with stock llama.cpp, Ollama, or LM Studio). The author also reports the same box runs the ~190 GB Q8_K_XL quant at 8 tok/s cold / 12 tok/s warm, with bursts to 21 (Reddit post). The upstream PR #31469 is open and unmerged.
Why it matters
The single-split CUDA graph is the real contribution. For out-of-core MoE, the miss-handling path — not the expert compute — decides whether offload is viable: stock llama.cpp pays a per-layer CPU/GPU sync on every token, which is why a 68 GB model on 12 GB of VRAM crawls at ~1.5 tok/s. Moving the resident check on-device and interrupting the host only on a miss turns that 48-way split into one graph replay, and the exclusive RAM victim cache stops hot experts from being double-cached in VRAM and the OS page cache. That pattern — slot pool, victim cache, pager, single-split replay — is portable to any out-of-core MoE workload, not just this model.
It is also a useful case study in reading local-LLM performance claims. The community headline — "22 tok/s sustained, spiking to 48" — is a warm repeated-workload number, and the author's own measurement record says that 16.80 and 22.03 "come from different historical runs" and "are not a paired result," and that "20+ tok/s is a warm repeated-workload result, not a cold-start guarantee" (fluxaeros.md). The project also carries two different stock baselines: 0.9 tok/s in the repo's own docs and 1.47/1.63 tok/s in the PR's controlled table (README, PR #31469). An independent tester on an RTX 5080 measured a regression — 20.24 tok/s baseline vs 14.00 with the flags on — and the author's reply was to raise the slot count until OOM, which is another way of saying the paging parameters are hardware-specific (PR #31469).
Caveats
- Every performance and correctness number above is vendor self-reported; no third party has reproduced the bit-exactness or the throughput.
- The tested envelope is a 512-token context; the model's native context is 262,144. A long prompt at these speeds would take hours to ingest.
- The PR is unmerged and bot-flagged (multiple backends in one PR, AI-written description, large diff), overlaps with the already-merged
llama-moe-cachework, and a maintainer asked how it differs (PR #31469). - The repo admits no MTP configuration is certified for exact-output paging (README).
- Licensing is mixed: the engine is MIT, but the weights are Qwen Community License 1.0 — the quant repo declares Apache-2.0, and the author flags the discrepancy himself (Hugging Face).