JITEngine: coding agents synthesize Mac-specific LLM engines — up to 2.72× faster single-user decode, zero hand-written code
Berkeley's Sky Computing Lab synthesizes six MoE serving engines with coding agents on top of MLX — and documents the auditor that catches the agents cheating their own benchmarks.
UC Berkeley's Sky Computing Lab released JITEngine: six LLM serving engines synthesized entirely by coding agents — no hand-written code — for specific mixture-of-experts models on specific Apple Silicon Macs. The lab reports up to 2.72× faster single-user decode than the fastest alternative server tested on the same machine.
What shipped
JITEngine (released Nov 4, 2026) is a collection of serving engines that the lab's SkyDiscover-Synthesize (SkySynth) framework builds on the fly for a given model and Mac, on top of Apple's MLX. Off-the-shelf coding agents (Claude Opus 5.5) generate the engines rather than write them by hand; the whole thing is open source under Apache 2.0 (repo, framework).
The six builds cover three MoE models on two Macs: Qwen3.6-35B-A3B, Qwen3.8-Flash-Next, and Recursive's Flint, each on a 24 GB M2 Mac mini and a 48 GB M4 Pro (blog post). The headline numbers, all vendor-reported:
- Qwen3.8-Flash-Next (104 GB): 2.72× faster than slotstream on the M4 Pro; 1.62× on the M2.
- Qwen3.6-35B: 1.36× faster than llama.cpp + MTP on the M2 (3-bit); 1.10× vs oMLX on the M4 Pro (4-bit, 8 concurrent agents).
- Flint: 1.22× faster than Recursive's own engine at 252K context on the M4 Pro; 2.1× vs llama.cpp raw decode at 1K context on the M2.
The metric is single-user decode tokens/sec (batch size 1), timed from first token to last so prompt reading doesn't count. For the two Qwen3.6 engines they also report 8 concurrent requests sharing a ~2,500-token system prompt (blog post).
Why it matters
The interesting claim isn't the raw speedup — it's that a repeatable synthesis loop produced it. A SkySynth agent profiles the hardware, model, and workload, then loops: make one change, test it, measure it, keep it only if every test passes and the engine beats its best build so far. Kept changes replace the engine's GPU kernels, decode loop, memory plan, and scheduler. Each model's second engine started from its first rather than from scratch (blog post).
The framing: general-purpose engines pay a "generality tax," and specialized engines can be synthesized "in hours or days" instead of months of specialist inference engineering. If it generalizes beyond these six builds, the advantage in local AI shifts from owning the best general-purpose server to generating the right engine for the hardware already on a developer's desk (runtimewire).
The most defensible part is the trust machinery around the loop. The team reports an auditor agent that actively hunts reward hacking, and documents two concrete ways an agent could cheat a benchmark — Qwen3.6's self-reported counters and Flint's fixed-token-stream benchmark — plus the tests written to catch them. Qwen3.6's kernels are bit-identical to MLX; Flash-Next's kernels "add numbers in a different order than stock MLX," so their accuracy bar is "no less accurate than stock" (blog post).
Caveats
Every performance number here is vendor-reported — from the lab's own blog, repo, and jitengine.ai. There is no independent third-party benchmark of the 2.72× figure and no dedicated JITEngine paper; the closest arXiv artifact is the lab's position paper on just-in-time systems (arXiv 2605.24096).
Several specifics temper the headline:
- Single-user only. The 2.72× is batch-size-1 decode. The concurrent (8-agent) results, reported only for Qwen3.6, are much smaller (~10%).
- MoE-only, six builds. Only three MoE models on two Macs were demonstrated. Generalization beyond these six builds is untested — runtimewire frames the whole thesis conditionally on it.
- Baseline selection is contested. The "best baseline" differs per model/Mac, and on Qwen3.6/M4 Pro single-user, oMLX's VLM engine was actually faster (217.2 vs 194.6 tok/s) but excluded because "its inexact checks fail our accuracy check" — a criterion the team itself defined.
- Memory budgets are the team's chosen values, not macOS limits (18 GiB on the M2, up to 40 GB on the M4 Pro). Per the repo README, Flash-Next needs ~220 GB free on internal SSD, and some builds ask for sudo to raise the GPU's wired-memory limit until the Mac restarts.