Adaptive KV streaming lands on ROCm: Qwen3.8-27B at full 262k context on a 16 GB RX 9070 XT — 9.5 t/s at the full window, 86.5 t/s short-context with MTP

A llama.cpp fork has ported adaptive KV streaming to ROCm, running Qwen3.8-27B at its full 262,144-token context on a single 16 GB RX 9070 XT — 9.5 t/s decode at the full window, up to 86.5 t/s at short context with the model's native MTP head. Every benchmark number is the fork author's own measurement; there is no independent reproduction yet.

What shipped

The fork is achyudh/llama.cpp, branch feature/adaptive-kv-stream-rocm, with commits dated Oct 14–16, 2026. It re-implements for RDNA 4 in ROCm/HIP a technique that originally shipped CUDA-only: keep the full KV history in host RAM and stream only a bounded working set into VRAM as attention needs it, reusing the same GPU space across layers. The original CUDA implementation (RaymondHuang210129/llama.cpp-adaptive-kv-streaming) explicitly lists HIP/ROCm as having "no end-to-end AMD streaming qualification," so the ROCm port is new work by the fork author.

The setup: Qwen3.8-27B with ASCII-condensed IQ4_XS weights (13.56 GB / 12.63 GiB) and a Q8_0 K / Q4_0 V cache. At 262,144 tokens the KV cache needs ~7.4 GB of host RAM; the fork streams ~450 MiB working sets during prefill and ~2,100 MiB during decode over PCIe alongside compute. The README recommends PCIe 5.0 x16 for >80k context and ≥32 GB host RAM.

The numbers, from the fork README (self-reported):

• Decode: 36.53 t/s at 1k context → 32.13 at 65k → 26.96 at 131k → 9.50 at 262k. Stock llama.cpp OOMs past ~65k.
• Decode with MTP: 71.83 at 1k, 86.51 at 4k, 81.11 at 16k, 72.89 at 65k, 78.60 at 131k, 31.08 at 262k.
• Prefill: 1,368.4 t/s at 1k → 585.8 at 262k.

The fork also folds in RDNA kernel work: 4-row multi-column mmvq, 16-byte vectorized loads for Q4_K/Q5_K, SwiGLU fusions, single-queue async HIP graphs, sibling activation reuse across parallel projections, fused residual+RMSNorm, persistent graph arenas, and a reworked MTP speculative path (probabilistic rejection sampling, stacked n-grams, single-head verify catch-up in the first draft step). The README credits stew675/llama-cpp-rdna-boosts as a concept source; the specific kernel commits are the fork author's own.

Power: at a 231 W limit (−24% versus the 304 W default TDP of the RX 9070 XT), the README reports a 3.6% prefill latency penalty and virtually identical decode throughput.

Why it matters

The reason this works is the model's hybrid architecture. Qwen3.8-27B is a 27B dense model with 64 layers, but only 16 are full attention layers; the other 48 are Gated DeltaNet linear-attention layers. That collapses the 262k KV cache from what a full-attention 27B would need to ~7 GB at q8_0/q4_0 (26,624 bytes/token), small enough to plausibly stream over PCIe 5.0. The original CUDA fork's numbers on an RTX 5070 Ti — 2.5 t/s stock with UVM, 26.1 t/s target-only, 59.7 t/s with MTP at 128 Ki — show the technique's ceiling, and the ROCm port extends it to AMD hardware.

For engineers, the takeaway: full native context on a $599 16 GB consumer card is now possible for hybrid-architecture models, but "80+ t/s" is a short-context number. At the full 262k window, decode falls to 9.5 t/s (31.1 with MTP) — usable for offline batch work, not interactive chat. PCIe generation matters: a public replication attempt of the CUDA original on PCIe 4.0 failed to reproduce the throughput curve, with t/s dropping from ~35 to ~24 around 80k context.

Caveats

• Every benchmark number in this post is the fork author's own measurement from the README. No third party has reproduced the ROCm port's results.
• The "80+ t/s" headline only holds at 4k–16k context with MTP enabled; at the full window it is 9.5 t/s (31.1 with MTP).
• A user (azrael-229) reported MTP reliability issues on the CUDA original in discussion #28216 (draft-KV wedge after ~45k cumulative tokens, ctx-size multiple of 256, batch/ubatch ≤ 256); the original author says he root-caused the wedge and pushed a fix. Whether these carry over to the ROCm port is untested publicly.
• The technique is not merged upstream — it lives in an "Ideas" discussion in llama.cpp and a fork branch.
• The ~7.4 GB host-KV figure is slightly above the 6.98 GB arithmetic for q8_0/q4_0 at 262k; the difference is likely MTP draft KV and padding, but it is not itemized.

Sources

• Fork README — achyudh/llama.cpp feature/adaptive-kv-stream-rocm
• Fork commit history
• Original CUDA implementation — llama.cpp-adaptive-kv-streaming
• RDNA kernel boosts — stew675/llama-cpp-rdna-boosts
• Qwen3.8-27B model card
• ASCII-condensed Qwen3.8-27B GGUF (bsaleh03)
• Radeon RX 9000 series specs
• llama.cpp discussion #28216 — adaptive KV streaming for upstream