OpenJet's CUDA megakernel runs 4 coding agents on one RTX 3090 — 256–288 tok/s total by pushing every agent's decode rows through a single weight read
The v0.5.3 release of the Qwen3.8-27B engine fuses the whole speculative cycle into one persistent kernel. All numbers are self-measured against a 2-draft llama.cpp baseline.
OpenJet v0.5.3, released October 27, 2026, adds multi-agent serving to its custom CUDA megakernel engine for Qwen3.8-27B Q4_K_M. Four coding agents on one RTX 3090 decode at 256–288 tok/s total, against 72–80 tok/s for llama-server --parallel 4 — a 3–4x gap the author attributes to batching every agent's decode rows through a single weight read per step instead of taking turns.
What shipped
OpenJet is a local coding agent (AGPL-3.0-only, repo L-Forster/open-jet) that runs Qwen3.8-27B Q4_K_M through a hand-written CUDA "megakernel" rather than llama.cpp. The engine previously served one conversation at a time; v0.5.3 ("parallel coding sessions on one GPU") adds multi-agent serving. The author's numbers, all measured on an RTX 3090 with Qwen3.8-27B Q4_K_M:
| per agent | llama-server --parallel 4 | OpenJet megakernel |
|---|---|---|
| decode, short prompts | 18–20 tok/s | 64–72 tok/s |
| decode, 23K context each | 15–16 tok/s | 52–55 tok/s |
| first token of the next request on those contexts | 13.4 s | 0.2–0.5 s |
| another agent decoding while a 20K prompt is processed | 0.6 tok/s | 9.5 tok/s |
The main README rounds the headline differently: ~75 tok/s per agent short-context, ~290 tok/s total; ~40 tok/s per agent at 10K context each (~160 total); and 40–75 tok/s per agent for four real coding agents that pause to run tools and tests (150–200 tok/s total). One agent alone still gets 117 tok/s.
The baseline is llama-server with -ngl 99 -fa on -c 131072 --parallel 4 -ctk q8_0 -ctv q8_0 --jinja; the harness is bench/parallel_bench.py (coding tasks, 512 tokens, temperature 0.6, thinking off), per the engine docs.
Supporting machinery: a shared 132K-token paged KV cache (q8_0) lets one agent use most of the pool while the others stay short. The system prompt and tool list common to all agents are processed once; follow-up requests return to the slot already holding that conversation, so only new tokens get processed. mk_top.py shows live per-agent tok/s, context use, and current task.
Why it matters
The mechanism is the story, and it is the author's own description of the design. Decode on a GPU is memory-bandwidth-bound: every step reads all ~17 GB of weights from VRAM, about 18 ms at the 3090's 936 GB/s bandwidth alone, and the arithmetic on those weights is comparatively cheap. So the engine doesn't run agents one after another. Each step stacks every active agent's work into one batch of rows — each agent's next token plus its 2 MTP/prompt-lookup draft tokens, 12 rows for 4 agents — streams the weights through once, and multiplies each loaded weight against all 12 rows on the tensor cores while it's on chip. Four agents cost about one weight read per step, not four.
That's why per-agent throughput drops from 117 to 64–72 tok/s rather than to ~30: the shared weight read is most of a step's cost. What grows with agent count is attention (each agent reads its own KV cache) and the extra rows' compute.
The whole speculative cycle runs as one persistent cooperative kernel: main model over T = 1 + n_draft tokens, on-device lm_head and verification, MTP catch-up rows for accepted tokens, then new drafts — roughly 340 phases separated only by a software grid barrier, with the host reading back just the accepted tokens. No per-layer launches, and drafts are verified in the same pass, so every accepted draft is an extra token from the same weight read. New prompts don't stall the other agents either: each step carries the next chunk of the new prompt alongside their rows, and the author reports the result is bit-identical to processing the prompt on its own.
Two things make this tractable, and both are specific to the model. Qwen3.8-27B is a dense 27B model with 262,144-token native context and a trained MTP head; of its 64 layers, 48 are Gated DeltaNet linear-attention layers and 16 are full attention (model card). Rolling back and batching drafted rows is far easier when most layers keep a fixed-size recurrent state instead of a growing KV cache. This is a model-specific engine: it serves Qwen3.8-27B Q4_K_M (plus a separate mixed-quant engine for UD-Q4_K_XL), and every other model falls back to llama-server.
For local multi-agent setups, the practical change is economics: four agents on one 24 GB card at 64–72 tok/s each behind an OpenAI-compatible API on 127.0.0.1:18080/v1, instead of llama-server's serialized degradation to 18–20 tok/s per agent. Whether the gap survives a fairer baseline is the open question.
Caveats
Every throughput and correctness number above is self-reported by the author; we found no independent reproduction. Two things temper the headline:
- Asymmetric baseline. The published comparison gives llama.cpp at most two MTP drafts while the megakernel uses three to four. The author's own no-spec comparison — 40.4 tok/s (llama.cpp) vs 42.9 tok/s (megakernel) for single-stream decode — shows both engines sit near the memory-bandwidth limit; the single-conversation speedup is speculative-verification engineering, not faster arithmetic.
- No run manifest. The benchmark page publishes no GPU clocks, power limits, driver version, or exact llama.cpp commit, no confidence intervals, and aggregates greedy runs best-of-two.
Correctness: output is not byte-identical to llama.cpp because the arithmetic is ordered differently. Against llama.cpp's token-by-token distribution (13,299 scored positions, q8_0 KV reference), the megakernel reports 99.08% top-token agreement with mean KL 0.0009, in the same range as llama.cpp batched vs itself (98.68% / 0.0019); long-context checks at 30K and 96K report 99.6%+ agreement.
Limits: the Q4_K_M engine only (the mixed-quant build still serves one conversation at a time), at most 8 agents, presence/repetition penalties ignored, top_k capped at 32, text only, and Linux with an RTX 30/40/50 GPU with 24 GB+ and CUDA 12+ (12.8+ for RTX 50). Turing support is experimental.