Lookahead Expert Scheduling: OpenBMB runs a 35B MoE at 20.5 tok/s on an iPhone — routing untouched, no experts skipped

OpenBMB has published the engineering details behind MiniCPM-V 4.7's on-device MoE speed: a Lookahead Expert Scheduling (LES) pipeline that decodes a 35.2B-parameter mixture-of-experts model at 20.51 tok/s on an iPhone 18 Pro Max — 6.07× over its plain SSD-loading baseline. It does so without capping the expert pool, skipping experts, or touching the model's routing.

What shipped

MiniCPM-V 4.7 is the first mixture-of-experts model in the MiniCPM-V line, announced on October 12 with two Apache-2.0 checkpoints: the 35B-A3B flagship and a 1B variant (model card). The flagship is a 35,212,875,824-parameter MoE with roughly 3 billion parameters active per token (the "A3B" in the name): a 40-layer text stack typed qwen3_5_moe_text, 256 experts with 8 active per token, and a 262,144-token context (config.json as summarized by RITS/NYU Shanghai). The BF16 weights alone run ~70.6 GB — the model is designed to stream most weights from storage, not hold them in memory.

What's new is the LES technical note in llama.cpp-omni, OpenBMB's MIT-licensed llama.cpp fork, which reports a staged speedup table measured on Apple silicon. Decode speed is generated tokens over total decode time; the baseline is expert weights loaded from SSD with no optimizations enabled. On iPhone 18 Pro Max (12 GB):

• Baseline: 3.38 tok/s
• • Lookahead Expert Scheduling: 6.57 tok/s (1.94×)
• • load optimization (weight layout, fewer read calls, direct writes into GPU-visible memory): 11.34 tok/s (3.36×)
• • precise layer-wise mixed-precision quantization: 17.16 tok/s (5.08×)
• • adapted DSpark speculative sampling: 20.51 tok/s (6.07×) — which the note describes as "lossless inference"

On an iPad Pro 11-inch M4 (16 GB) the same progression — with the scheduling and load-optimization steps bundled — runs 2.97 → 9.81 (3.30×) → 14.92 (5.02×) → 17.39 tok/s (5.86×), with quality checks across MMMU, MathVista, CharXiv_reasoning, OCRBench, HumanEval, MBPP, AIME2025/26 and Math500. The model card separately claims 13+ tok/s in about 3 GB of memory on the previous-generation iPhone 17 Pro Max. The staging isolates each component's contribution: load optimization and quantization deliver the biggest absolute jumps; speculative sampling adds the final ~19% on top of the quantized iPhone pipeline.

How LES works. The one-paragraph version from the technical note: a cross-layer routing predictor preloads the experts the next layer is likely to need while the current layer computes. Among the experts the real router selects, the already-resident ones execute first; the rest load in parallel. Prediction drives prefetch and scheduling only — computation still follows the model's own token-level routing, and any required expert the prediction missed is loaded and evaluated as usual. LES requires no additional training. The press release adds that in most scenarios more than 80% of required experts are prepared in advance; because the original router still makes the final selection, outputs are identical with LES enabled or disabled.

Two more components sit under the scheduler. First, weight access: precise layer-wise mixed-precision quantization, expert weights repacked into contiguous storage to cut scattered reads and improve sequential read efficiency on flash, and direct writes into GPU-visible memory. Second, an adapted DSpark speculative-sampling stage that runs on top of the scheduling. The vision stack got its own speed lever: mixed 4×/16× visual-token compression, where OpenBMB reports 90.9 on MMBench and 90.4 on OCRBench at 16× compression (press release).

Everything is public: model weights under Apache-2.0, engine under MIT, with llama.cpp-omni carrying the LES + DSpark implementation, GGUF/FP8/AWQ/GPTQ quant variants, day-0 chip support from Apple, Arm, AMD, Intel, NVIDIA, Rockchip and Houmo (plus a FlagOS path for ten more chip vendors), and SGLang, vLLM, llama.cpp and Ollama serving paths.

Why it matters

On-device MoE was never blocked by FLOPs. With ~3B active parameters per token, compute is phone-class; the problem is that data-dependent routing means nobody knows which ~8 of 256 experts fire at the next layer until the current layer finishes. Wait for the routing decision and every expert transfer sits on the critical path — a 35B MoE becomes a storage-bound slideshow. LES targets exactly that: it bets on cross-layer routing predictability, prefetches during live compute, and overlaps load with execution. Because the model itself never changes, you get the speed without the capability tax that fixed-expert-subset schemes pay.

The practical recipe for anyone shipping quantized MoEs on edge hardware is in the staged table: scheduling plus the load-optimization pass buys 3.36× before quantization is applied, quantization adds another 1.5×, and speculation supplies the final stretch. The technique is model-agnostic — no training, built on the router's own behavior — so it should transfer to other MoEs. OpenBMB also claims a better quality-per-latency point than the comparable Qwen3.6 35B-A3B (higher overall score in less inference time in its own Instruct-mode evaluations), positioning the release as the same model class made portable.

Caveats

• All speed and memory numbers are vendor-measured on OpenBMB's own stack (Metal on Apple devices); no independent reproduction was found as of this writing. Treat 20.51 tok/s as a well-staged vendor claim, not a verified third-party result.
• "Lossless" and "outputs identical" need scoping: they refer to the routing and expert set being untouched by the scheduler. The pipeline still stacks lossy quantization and speculative decoding on top, and no bit-exact comparison between on-device and reference outputs has been published (a point ExplainX.ai also makes).
• The "~3 GB" figure is a resident-memory claim, not a download size — the Q4_K_M GGUF is 21.2 GB on disk and the whole point of LES is that most weights stream from flash. OpenBMB does not define exactly what the 3 GB measures.
• Quality numbers attach to the iPad staging; the headline iPhone figure is speed-only. Prefill latency for images/video, thermals and battery draw are not quantified.
• Benchmark comparisons against Qwen/Gemma are OpenBMB's own evaluations of official checkpoints, not peer-reviewed. Upstream llama.cpp support merged on October 10 but, per a third-party review, an image-coordinate bug in returned bounding boxes was still open on October 20; the LES numbers come from the fork, not from mainline llama.cpp.

Sources

• LES: Lookahead Expert Scheduling (llama.cpp-omni docs)
• llama.cpp-omni repository
• MiniCPM-V 4.7 35B-A3B model card (Hugging Face)
• ModelBest and OpenBMB launch MiniCPM-V 4.7 (PR Newswire)
• RITS/NYU Shanghai write-up on the release
• ExplainX.ai independent write-up
• OrcaRouter: checking MiniCPM-V 4.7's on-device speed claims
• MiniCPM-o repository (licensing)
• Original r/LocalLLaMA thread