Fusing the whole speculative-decoding cycle into one CUDA kernel: Qwen3.8-27B at 140 tok/s on a single 3090
OpenJet's megakernel runs the entire decode cycle in one persistent cooperative kernel launch — 1.4–1.9x over llama.cpp with MTP on identical hardware
A developer has open-sourced a CUDA inference engine that runs Qwen3.8-27B on a single RTX 3090 at up to 140 tok/s on code — 1.4–1.9x faster than llama.cpp with MTP speculative decoding on identical hardware, per the author's benchmarks. The speedup comes from one structural change: the entire speculative-decoding cycle runs in a single persistent cooperative kernel launch, so verifying 4–5 drafted tokens costs about the same as verifying one.
What shipped
OpenJet (L-Forster/open-jet, AGPL-3.0; PyPI open-jet ≥ 0.4.32) is an OpenAI-compatible server for Qwen3.8-27B. Its megakernel engine targets unsloth's Q4_K_M GGUF of the model on RTX 30/40/50-series GPUs with 24 GB+ (sm_86/sm_89/sm_120), tuned and benchmarked on a 3090. The author reports these numbers, same hardware, megakernel vs. llama-server with MTP:
- Writing code: 140 vs 73 tok/s (1.9x)
- Coding with a 4K-token file in context: 95 vs 56 tok/s (1.7x)
- Prompt processing (prefill): ~1,600 vs ~1,100 tok/s (1.4x)
- Greedy mean over four chat prompts: 63.1 → 106.7 tok/s (1.69x)
- Reasoning mode (thinking on): 50.9 → 78.0 tok/s (1.53x)
- With speculation disabled: 40.4 vs 42.9 tok/s — both pinned near the memory-bandwidth limit
In llama.cpp-style speculative decoding, the main-model forward pass, the verification of drafted tokens, and the MTP drafting are separate kernel launches with host round-trips between them. OpenJet's megakernel runs the entire cycle — main model over T = 1 + n_draft tokens, lm_head + verification on device, MTP catch-up rows for accepted tokens, then n_draft MTP draft steps — in one persistent cooperative kernel launch, with the ~340 phases of the cycle separated by a software grid barrier. The host only reads back the accepted tokens (megakernel README).
The model makes this cheap. Qwen3.8-27B is a dense 27B model with 64 layers, of which 16 run full gated attention and 48 run linear attention (Gated DeltaNet) with a constant recurrent state, plus a built-in MTP draft head (vLLM Ascend docs). The recurrent state makes rollback cheap: it is committed only through the surely-accepted token, and drafts are replayed from a small (k, v, g, beta) ring.
The kernel uses dp4a GEMV with shared activations, so T = 4 costs almost the same memory time as T = 1. On-device MTP drafting is restricted to the 32,768 most frequent tokens (covering ~99.4% of tokens). It also uses split-K flash decoding with tensor-core MMAs and double-buffered K/V tiles.
The author also published a correctness analysis that is unusually rigorous for a hobby project: a token-by-token comparison against llama.cpp on the same GGUF (13,299 positions, 1024-token prompts) shows 99.08% top-token agreement with KL 0.0009 and perplexity 2.4274 vs. the reference's 2.4263 — "as close to llama.cpp as llama.cpp's own configurations are to each other" (llama.cpp's own batched config scores 98.68% agreement, KL 0.0019). Speculative verification is exact: greedy output is the main model's greedy choice. Outputs are not byte-identical to llama.cpp — near-tie rounding can pick the other token, which the author confirmed in the Reddit thread.
The KV cache is fp16 up to ~80K context and q8_0 above that (e.g. the 100K setting), with q8 costing roughly 1–2% on decode and 2–7% on prefill.
Why it matters
The point is not the model or the GPU — it is what the numbers say about where single-stream decode time goes. With speculation disabled, the megakernel does 42.9 tok/s vs. llama.cpp's 40.4: both near the memory-bandwidth limit. The entire 1.4–1.9x comes from making verification of 4–5 drafted tokens nearly free, which is only possible because the fused kernel eliminates per-phase launch overhead and host round-trips (repo README). For anyone serving a single user on a single GPU, that is a transferable lesson: at batch size 1, kernel-launch and host-sync overhead — not FLOPs — is the bottleneck, and fusing the whole decode loop into one persistent kernel is a general way to attack it.
It is also a reminder that MTP speculative decoding is now a first-class feature of the Qwen3.8 generation: the model ships with a built-in draft head, and llama.cpp reads the proposer from the same GGUF with --spec-type draft-mtp (Jetson AI Lab). The megakernel shows the remaining gap is engineering overhead, not algorithm.
Caveats
- Baseline asymmetry. The llama.cpp baseline ran with
--spec-draft-n-max 2(two MTP drafts) while the megakernel was benchmarked at 3–4 drafts. A llama.cpp run at four drafts has not been published; a commenter in the thread called the baseline "weak" for this reason. The no-speculation comparison (40.4 vs 42.9 tok/s) shows raw decode is nearly equal, so the gap is speculation depth plus fused verification — but the fairer head-to-head is unmeasured. - Single model, single GPU. Only unsloth's Qwen3.8-27B Q4_K_M GGUF (the loader refuses other quant mixes), one request at a time, no batching, single-GPU only as documented, top_k capped at 32, presence/repetition penalties ignored.
- The advantage shrinks with context. Decode speedup falls from 1.9x at short prompts to ~1.4x at 29.5K context; prefill from 1.5x to ~1.16x. No published decode/prefill numbers at 100K+.
- Self-measured, unreplicated. All throughput and correctness numbers are the author's own measurements from public benchmark scripts; the post is days old and there is no independent reproduction yet (one tester reports "a solid improvement" without numbers).
- The author says the kernel was written with Claude Opus 5.5's help — plausible but not independently verifiable.