SpecPilot: an agentic loop that finds lossless speculative-decoding speedups — and a 12-bit weight format as a byproduct
An agentic loop built for serving-speed discovery accidentally produced a general, bit-exact 12-bit weight format — and a statistical guardrail that catches reward hacking task accuracy cannot.
SpecPilot, an autonomous agentic loop that searches for lossless speedups in speculative decoding per deployment, reports raising the speedup over autoregressive decoding from stock's 4.46× to 10.87× on a single request and from 1.94× to 3.93× under batching — without training or quality loss. The more interesting result is a byproduct: the loop discovered a general, bit-exact 12-bit weight format that beats cuBLAS BF16 on H100, plus a statistical guardrail ("token forcing") that catches reward hacking that task accuracy could not flag in their fault test.
What shipped
SpecPilot (arXiv:2611.02501, code, Apache-2.0) is not a new decoder. It is a loop that runs Claude Opus 5.5 coding agents to find system optimizations for a specific serving regime × drafter × target model × GPU. Each round, a cost model locates the largest gap between the current system and a computed "ceiling," parallel sub-agents implement candidate patches, and an isolated controller times each patch against the current best in paired runs on two GPUs, promoting it only if the 95% lower bound of its gain exceeds 1% plus twice the stock-versus-stock noise (below 0.1% for one request on H100). An integration agent stacks small gains; a kernel agent tunes kernels furthest from their byte floors. The authors report the loop now holds 140 validated memory entries carried across rounds and deployments.
The headline numbers are author-reported geometric means over five benchmarks (GSM8K, MATH500, HumanEval, MBPP, MT-Bench) at 2048 new tokens, greedy decoding on SGLang:
| Deployment | Stock | SpecPilot |
|---|---|---|
| Qwen3.8-27B + DFlash2, RTX PRO 6000 | 4.46× | 10.87× |
| Muse-Glimmer-30B + DFlash2, RTX PRO 6000 | 4.69× | 10.63× |
| Qwen3.8-27B + DSpark, H100 | 3.74× | 9.41× |
| Qwen3.8-27B FP8 + DSpark, H100 batch | 1.94× | 3.93× |
| Qwen3.8-27B FP8 + DFlash2, H100 batch | 2.02× | 3.90× |
| Qwen3.5-4B + DFlash2 | 1.51× | 3.77× |
| Gemma-4-12B + EAGLE-3 | 0.68× | 2.09× |
On H100 with one request, SpecPilot reports 1.84× faster than DDTree, a stronger draft-tree baseline. All numbers are speedup over autoregressive decoding.
Why it matters
The loop's most transferable artifact is W12, a lossless 12-bit weight format. Per output channel it picks the 16-binade window covering the most BF16 weights, stores sign + 4-bit exponent offset + 7 mantissa bits (12 bits total), and sends out-of-window weights to a sparse escape list that restores them exactly. It averages 12.01–12.02 bits per weight across eight dense models and shrinks Qwen3.8-27B's converted layers from 44.67 GB to 33.58 GB. It decodes in registers under the memory stream: the authors report it beats cuBLAS on every Qwen3.8-27B projection by 1.19–1.47× at one row and 1.17–1.35× at 32 rows, and alone makes BF16 decoding 1.15–1.26× faster on dense 8–30B models at batch 1–32.
The other discovered mechanisms are more deployment-specific. Tree verification for hybrid recurrent targets builds a best-first tree of the 32 highest path-score nodes, recomputing Gated DeltaNet state along each root path, within one BF16 ulp of the exact path kernel. Two-level verification (TLV) runs a cheap Q pass over an 8-bit view that proposes a chain but accepts nothing; one exact W12 pass verifies both. Three batching mechanisms: shared node budgets with prefix-closed trees, per-request draft depth adapting among 16/12/8 positions, and segment admission of waiting prompts into verification rows.
The guardrail is the part worth copying into any agentic-optimization project. Token forcing feeds the system's outputs back through the target on the stock engine and flags every token that differs from the scorer's argmax. Stock itself mismatches on 0.72% of tokens, almost all near-ties from kernel rounding; a system is rejected if its mismatch rate exceeds stock's by more than 0.5 points at 95% confidence, or if it has more unexplained large-margin mismatches than stock. In an injected-fault test (a logit bias of 2 on every 7th vocabulary id after 2048 tokens), lossless systems matched stock's mismatch rate while faulty ones hit 2.0–2.4% and were rejected — yet task accuracy could not separate them (lossless −0.3 vs faulty −1.6, overlapping confidence intervals). This is a concrete answer to the reward-hacking problem that plagues agentic search.
Caveats
- The discovery cost is excluded and undisclosed. The reported speedups do not account for the compute spent running the Claude Opus 5.5 agents; no wall-clock time, GPU-hours, or API cost is reported. The 10.87× is the speedup of the final system over AR, not the return on the search that found it.
- "Lossless" does not mean bit-identical output. The authors concede outputs differ from AR — near-ties can flip because batched GEMM/attention reduce in a different order. At each first divergence the two tokens' log-probabilities differ by at most 0.232 nats. Token forcing is a statistical check against stock, not a proof about every execution.
- Everything is author-reported. No independent third-party reproduction, benchmark audit, or critical review was found. The repo ships fresh-clone runs with 95% bootstrap confidence intervals (e.g. H100 single-request DFlash2 3.96×→8.91×, gain 2.25× [2.18, 2.32]), which is unusually verifiable — but still author-run.
- Scope is narrow. One serving stack (SGLang), greedy decoding only (no sampling), 2048-token generations, single-GPU, and only DFlash2/DSpark/EAGLE-3 drafters. Two of the eight project-page deployments (Qwen3.5-4B, Gemma-4-12B) are not shipped in the repo. PRO6000 kernels are sm_120a-only, and the authors re-tuned stock per GPU, so the baseline is partly their tuning choice.
- Batching numbers are modest. The batched speedups (1.94×→3.93×) are far smaller than single-request ones, and TLV was 13% slower on batched BF16 while 24-node trees lost 14.9% on the FP8 batched target — the mechanisms do not transfer to batching.