llmopen-sourceinferenceserving

jitllm relocates a running model's layers between GPUs mid-reply: 8 ms pause, KV cache moves with the layers

A Go engine treats layers like OS processes: JIT kernels, a paged KV cache that travels with them, and an 8 ms handoff on T4s.

Richland AI Desk·15 October 2026·6 min read

jitllm relocates a running model's layers between GPUs mid-reply: 8 ms pause, KV cache moves with the layers

jitllm, a new Apache-2.0 inference engine announced on r/LocalLLaMA this week, moved a Llama-3.2-3B model from one T4 to another mid-reply with a measured 8.1 ms pause and zero tokens re-prefilled. The paged KV cache travels with the layers, so the conversation survives the move; the author reports the output matches a run that never moved.

What shipped

jitllm is a Go inference engine from a single contributor (repo, announcement) that positions itself as "an operating system for LLM inference." One binary detects the machine's CPUs and GPUs, JIT-emits kernels at load time for each target it finds — x86 (SSE, AVX2, AVX-512, VNNI), Arm (NEON, dot-product), PTX for CUDA (tensor cores from Volta on), SPIR-V for Vulkan, Metal, plus an AMDGPU/ROCm backend the README says is untested on AMD hardware. It decides at runtime where each layer lives.

The jitllmd server exposes OpenAI/Anthropic-compatible endpoints with streaming, tool calling, and structured output; continuous batching on CPU and GPU; KV-aware admission; and a control plane (models, sessions, place, stats, Prometheus /metrics). A TUI lets you move a model's blocks between CPU and GPU with the arrow keys while the conversation keeps its history across the move. Model coverage is broad for a young engine: 57 graphs spanning 65 GGUF architecture names and 30 Hugging Face classes, plus 17 vision projectors.

jitllm JIT-emits kernels at load time — the inner loop of a Q4_K matrix-vector product emitted for an AVX2+VNNI CPU and then for CUDA.

The announcement's headline numbers, all measured by the author:

Hardware	Model	Move	Pause
2× T4	Llama-3.2-3B Q4_K_M	GPU 0 → GPU 1	8.1 ms
2× T4	Llama-3.2-3B Q4_K_M	GPU 1 → GPU 0	8.4 ms
2× V100	Llama-3.2-3B, 1.8k-token chat	GPU 0 → GPU 1	20–21 ms
RTX 3050 Ti laptop	Llama-3.2-1B Q4_K_M	CUDA → Vulkan	91 ms

The 8 ms is not the weights moving: from the author's comment on the thread, on the T4s "it's P2P over plain PCIe: no NVLink there, and the KV goes card to card with cuMemcpyPeer. The 8 ms is just that KV handoff: the target GPU gets the weights and kernels ready while the reply keeps streaming." The V100 pair does the same 1.8k-token move over NVLink in 20–21 ms, and the laptop's CUDA→Vulkan move cannot copy device-to-device across backends, so it goes through a host buffer — hence 91 ms.

The KV paging design doc describes per-layer KV pages sized for the destination device — powers of two from 64 to 256 positions, K stored transposed — so a relocation is "a load at the destination's P": history copied position-by-position into destination-owned pages at the destination's page size, with the source pages returned to the pool. Hybrid state-space models carry their recurrent state the same way; the runtime principles state that "any block, its KV pages and its recurrent state can move host to device and between devices mid-sequence, with the same answer."

Placement simulation: model blocks page from the .jlm file onto a GPU and the CPU and relocate, KV cache with them, as a GPU is added, the placement changes, and the GPU is removed.

Paging rounds the design out: disk, RAM and VRAM are one paged hierarchy, with weights read on demand via O_DIRECT (never mmap), each MoE expert on its own page so a token touches only the experts it routes to, and KV history paging through the card. The author's placement history puts the cost model at ~36 µs per host/device crossing, with a ~2 MB break-even for moving weights.

Why it matters

This turns placement into a runtime scheduling problem instead of a load-time setting. llama.cpp's -ngl offload count is fixed when the model loads; vLLM pages the KV cache but keeps each worker's placement static; and the closest academic analogue, ServerlessLLM (OSDI '24), migrates LLM inference between servers by shipping the prompt tokens (tens to hundreds of KB) rather than the KV cache, to keep the transfer small — the opposite optimization. jitllm moves the KV with the layers, so the conversation survives while the hardware changes.

The repo ships the machinery to exploit that. -gpu-grow starts a model on the CPU and moves blocks onto the GPU while serving; -relocate hands the last block back to the host when the device can't grow the context; sessions can be parked to the host and resumed to time-slice a card (-fairness 0-100); and jitllmd place moves blocks between devices on demand. For someone building serving infrastructure, that means models bigger than the card, mismatched GPU pools, and conversations whose working set shifts over time can be scheduled rather than re-provisioned.

The benchmark claims are secondary and the author's own measurements: up to 4.62x llama.cpp's prefill on Qwen3-Next-80B across four V100s (ahead on 34 of 35 models at prefill, one at parity, and faster decode on all 35, up to 1.35x), up to 5.8x sooner to the first token than vLLM 0.18.1 with 1.26x decode and up to 5.53x batched decode for Llama-3.1-8B on a single V100, up to 1.67x prefill on a 12-core Xeon, and up to 6x mistral.rs decode — all from a paired, documented harness with engine versions pinned in the scoreboard. The same docs list where it is still behind: two sessions decoding together at 0.94x, the first token of a ~48-token prompt 3–14% behind on one card, and decision models on a V100 well behind llama.cpp.

Benchmark chart: jitllm's ratios against llama.cpp and vLLM — up to 4.62x prefill on Qwen3-Next-80B across four V100s, faster decode on all 35 models, and up to 5.53x batched decode versus vLLM.

Caveats

Everything above is the author's own measurement. The methodology is unusually honest for a project this size — paired A/B passes with A/A self-controls and versions pinned — but no independent replication was found, and the Reddit thread is the only coverage we could find. The move demo is on 3B and 1B models; commenters immediately asked for a 30B+ run where the KV cache is the real cost, and none is public yet (the author argues the benefit grows with model size, so hot layers can be pinned on accelerators). The T4 demo's context length is not stated, so the transfer rate implied by "8 ms" cannot be checked; and the author's own V100 figures disagree between the post's table (20–21 ms) and his comment ("20–35 ms") for the same 1.8k-token move. "Same answer" is asserted and consistent with the design (the docs show identical token-id sequences for paged and unpaged runs), but it is not a formal bit-level guarantee — kernels are regenerated per device, and float rounding across backends is not promised.

This is a very young project: the repo was created 2026-09-08, has a single contributor, 42 stars, and v0.2.6 released 2026-10-15 at the time of writing, with the ROCm backend explicitly untested. Beware a name collision: an unrelated Java/TornadoVM toolkit also called jitllm (beehive-lab/jitllm) will muddy your searches.

jitllm relocates a running model's layers between GPUs mid-reply: 8 ms pause, KV cache moves with the layers

What shipped

Why it matters

Caveats

Sources

Ready to build something like this?

Tell Us About Your Project