Specter previews expert routing with an INT4 copy of the model itself — 1.50–1.88× TPOT for single-GPU MoE offloading

Specter, an ASPLOS 2027 paper from Shanghai Jiao Tong University, turns an INT4-quantized copy of a Mixture-of-Experts model into a speculative-decoding draft: its token predictions double as expert-routing forecasts, letting the runtime prefetch offloaded experts over PCIe before verification needs them. On a single A100 40 GB at batch size one, the authors report an average 1.50× and up to 1.88× speedup in time-per-output-token (TPOT) over SpecMoEOff, the prior state of the art, with code and a reproduction artifact.

What shipped

Specter targets the PCIe bottleneck in single-GPU MoE inference. When a model's experts don't fit in VRAM, they are offloaded to host memory and swapped in on demand. Because routing is data-dependent — you only know which experts a token needs after the router runs — expert transfers sit on the critical path, and each one costs milliseconds for gigabyte-sized experts.

Prior offloading and prefetching systems predict expert use within a single token's execution (intra-token). Specter's authors argue that is structurally insufficient: the prediction window is microseconds, orders of magnitude shorter than the millisecond-level PCIe transfer latency (HTML full text, Section 1). The two diagrams below contrast that alternating transfer/execute schedule with Specter's overlapped one.

Specter uses the model's own quantized self as the lookahead oracle. The draft is a GPTQ INT4 (group size 128) copy of the target's routed experts with the identical routing architecture — routers and shared experts literally reuse the target's tensors. Because draft and target share a router, the draft's speculative token sequence implies which experts the target will touch K tokens ahead. That is the "zero-training-cost" claim: no separate draft model is trained, and routing correspondence holds by construction.

The runtime has three named components: a Speculative Inference Controller (SIC) that drives the draft/verify loop and profiles speculation depth; a Predictive I/O Orchestrator (PIO) holding the prefetch planner and an Expert Lookahead Buffer that stages experts before they are needed; and a Streamlined Execution Engine (SEE) with fused quantized MoE kernels — Marlin W4A16 GEMMs that dequantize INT4 weight tiles while computing in FP16 — plus expert-centric reordering for target-side verification (paper, design doc). The draft runs entirely in GPU VRAM, gathering routing counts on-device and forming a compact prefetch plan at chunk boundaries.

Verification stays lossless: the target holds the committed KV, the draft appends temporary KV, and on acceptance the target's KV is cropped to the accepted boundary while the draft rebases onto it. The authors state that this draft-then-verify mechanism "mathematically guarantees zero degradation in generation quality and output distribution" — so the speedup does not come with a quality tradeoff.

The headline number is an average 1.50× and up to 1.88× TPOT over SpecMoEOff under single-request inference (abstract). SpecMoEOff is itself a 2025 speculative-offloading system that claims up to 2.5× over earlier MoE offloading, so Specter sits atop a stack of overlapping baseline-relative claims.

The artifact pins down the setup: models are DeepSeek-V2-Lite, Qwen1.5-MoE-A2.7B-Chat, and Phi-3.5-MoE-instruct; target FP16, draft INT4; single A100 40 GB, batch size one, greedy decoding, 128 output tokens. TPOT is defined as (last commit − first commit)/(N−1) and speedup as baseline-mean-TPOT ÷ Specter-mean-TPOT (protocol, baselines README).

Datasets are GSM8K, Wikitext-2, HumanEval, GPQA (password-protected distribution), and C4 prefixes, 5 inputs each with 3 repeats (config). Per-model settings in the artifact range from 16 resident experts per layer (DeepSeek-V2-Lite's high setting, 48 offloaded) down to 1 of 16 for Phi-3.5-MoE's low setting, with speculation depths of 8–16 tokens.

The repo is a full runnable system: 192 source files in the Zenodo artifact (DOI 10.5281/zenodo.22937478), with baseline implementations of SpecMoEOff and Mixtral-Offloading, and an artifact experiment that regenerates the reported results. DeepSeek-V2-Lite is supported out of the box, with prepared checkpoints fetched from the project's GitHub release (~68 GB of disk); Qwen and Phi checkpoints are not bundled. One flag: the repository carries no license file, so treat the code as read-and-ask.

Why it matters

Three things worth taking from Specter:

1. The draft-model trick transfers to any MoE you can quantize. Instead of training a small sibling draft — which needs routing correspondence and its own memory footprint — Specter quantizes the target itself and reuses its routers and shared experts. Because the INT4 copy shares the target's router, correspondence holds by construction at little memory cost. Fidelity still has to be measured (the paper ships a route-fidelity probe for that), but there is no trained draft to keep in sync.
2. It turns I/O stalls into scheduled transfers. The speedup comes from overlapping PCIe movement with execution, made possible because the draft foresees expert usage several tokens out. The paper's sharpest point: intra-token prediction gives a microsecond horizon against millisecond transfers; only multi-step, inter-token lookahead closes the gap.
3. It is quality-neutral by construction. Per the paper's draft-then-verify guarantee, accepted tokens are exactly what the target would have produced — there is no accuracy-versus-latency dial to tune. That makes it a clean comparison point for anyone currently hiding offload latency another way.

It is runnable today for DeepSeek-V2-Lite on a single A100, with the artifact set up to regenerate the speedup table.

Caveats

• Tightly scoped regime. Single request, batch size one, greedy, 128 output tokens, A100 40 GB. There are no batched-serving numbers. A commenter on the Hugging Face Papers discussion asks the obvious follow-ups: how often does the INT4 draft mispredict expert routes, what does a miss cost in TPOT versus plain on-demand offloading, and does the win survive on a desktop with ordinary DDR4 host memory?
• Baseline-relative numbers. 1.50×/1.88× is measured against SpecMoEOff, which itself claims up to 2.5× over earlier work. Stack the claims carefully before quoting a figure in a deck.
• Headline drift across revisions. The same arXiv ID launched in November 2025 as "MoE-SpeQ" with a headline up to 2.34× on Phi-MoE; the current v3 (October 2026) is "Specter" with 1.50×/1.88× against SpecMoEOff (version history). Version and baseline matter more than the number.
• Predictor fidelity is not quantified in the accessible text. The paper reports expert-ID match-rate distributions for the draft's routing predictions (figure below), but the full evaluation tables and per-model miss rates could not be extracted from the public text, and the artifact flags that not every original-paper configuration was fully reconstructed — ds_low8 is marked unconfirmed, and the reference-memory budget is not enforced as a process cap (protocol).
• Self-reported. These are the authors' measurements on their hardware. The artifact can force exact greedy output agreement, but the speedups are not yet independently replicated.
• No license file on the main repository.

Sources

• Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading — arXiv abstract
• Full text (arXiv HTML, v3)
• SpecMoEOff baseline paper (arXiv 2508.21706)
• Specter GitHub repository
• Specter design document
• Benchmark protocol.json
• Baselines README
• Example configuration
• Zenodo artifact (DOI 10.5281/zenodo.22937478)
• Hugging Face Papers discussion