8 agents × 250K context on two 3090s: sparse KV decode with a rank-64 key sketch and a host-RAM tier
Sparse decode with a rank-64 key sketch and a host-RAM KV tier — 8 × 250K agents on two 3090s, with the quality numbers and caveats included.
An open-source patch set for the HyperQwen vLLM fork claims to run eight concurrent coding agents, each holding a 250K-token context, on two RTX 3090s. It keeps the full KV cache in host RAM, and each attention layer reads only ~1% of the tokens per decode step, picked by a rank-64 key sketch.
What shipped
sparsekv-hyperqwen (Apache-2.0) is not a fork: it ships patches and Triton kernels applied on top of HyperQwen @ ce9615c (a vLLM 0.31 fork) plus the pthfa int8 attention kernel. The model is Qwen3.8-27B W4A16 (AutoRound) with MTP-7 speculative decoding, served from a desktop: i9-12900K, 96 GB DDR5, a 3090 Ti on PCIe x16 and a 3090 on a degraded x4 link, both power-capped at 250 W.
The mechanism, per the repo README:
- Sparse decode. Each decode step, the 16 full-attention layers (3, 7, …, 63; the other 48 are GDN linear attention) score every token through a rank-64 key sketch, then run exact pthfa int8 attention on the top 2048 tokens (~0.8% of 250K), 64 sink tokens, and the last 1024. Prefill and the MTP drafter stay dense.
- Cheap selection. The sketch is int4 (~2.1 KB per token over all 16 layers); top-k is a deterministic Triton radix select; one pick per layer is shared by all four KV heads.
- A frozen basis.
basis27.ptholds rank-64 eigenvectors of real post-RoPE keys, fitted once offline. Because it is the same for every document, prefix caching keeps working. - A host-RAM KV tier. The full KV lives in pinned host RAM at int4 (64-dim groups with fp16 scales, 272 B per token-head, dequantized back to int8 rows on fetch): 4.4 GB per 250K agent versus 8.4 GB at int8. The GPU keeps the sketch, sinks, recent tail, and a 2.5% hot buffer (windowed CLOCK).
- Zero-copy misses. Hot-buffer misses read only the picked rows straight from pinned memory over PCIe (UVA): ~10 GB/s on the x16 card and 6.2 GB/s on the x4 card, even with 512 B rows. Miss rate runs 14–16%, about 30–39 MB per step per server.
- Follow-ups fetch only the picked rows. Before that fix, every turn copied 2.4–15 GB and the agents serialized.
- Dense/sparse switch per step. Two CUDA-graph families exist; short contexts replay the dense graphs, long ones the sparse. All changes are inert unless the
SPARSE_*knobs are set.
Measured results (temperature 0, ~244K-token vLLM source dumps as documents, 1024-token answers):
| setup | combined decode, explain / copy (tok/s) | follow-up TTFT | recall |
|---|---|---|---|
| 8 × 250K, both cards | ~545 / ~790 | 5.2–5.9 s | 31/32 |
| 4 × 250K, 3090 Ti (x16) | 268–271 / 394–415 | 5.2 / 5.5 s | 15/16 |
| 4 × 250K, 3090 (x4) | 268–285 / 375–395 | 5.3 / 5.9 s | 16/16 |
| 4 × 128K, 3090 Ti | 312 / 501 | 4.4 / 5.6 s | 11/13 |
The Reddit announcement adds a six-minute nonstop run at 176–205K per agent, averaging 657 tok/s box-wide (69–90 per agent). Single long chats are faster than dense too: a go-live A/B at ~113K measured +15% (deep3 127→146) to +36% (copy 140→191) on one card, +29%/+41% on the second. GPU cost per agent is ~1.6 GB (0.77 GDN+MTP-7 state, 0.53 int4 sketch, 0.22 hot share, ~0.1 tail), which yields four 250K slots per 24 GB card.
The author also published a teacher-forced KL-divergence test against dense (eval/kld/: 60 samples, 15,300 decode tokens per config, MTP off, temperature 0). The dense-on-a-different-card noise floor is mean 0.0200 nats/token with 96.2% top-1 agreement; GPU-only sparse decode measures 0.0188 / 96.4%. Sparse decode diverges from dense less than dense does from itself on a second card of the same architecture. The production config (sparse + int4 host tier + speed knobs) lands at 0.0269 / 95.6%. The int4 and int8 host copies score the same on shared samples (0.0248 vs 0.0249), so the extra drift comes from the speed knobs, not from int4 host storage. Divergence grows slowly with context: 0.024 at 32K to 0.033 at 192K, and recall-style answers stay at 99.6–99.9% top-1.
Why it matters
The design is a direct answer to TokenGate, the Stanford/Berkeley approach that makes long context cheap by pruning tokens up front. An MLP gate plus LoRA (r=16) at one layer picks the top-K tokens; later layers only process those and never create KV entries for the pruned ones (the question is never pruned). The paper reports 1.4×–11.3× faster TTFT and 1.5×–3.4× lower prefill memory, up to 13.0× at 128K — measured on Qwen3-1.7B and Qwen3.5-2B at batch size 1 on one A100, with the speedup dropping to 1.4× on some tasks.
The sparsekv author tried TokenGate first and reports the failure mode that matters for agents in the Reddit post: "It's fine for one question, but agents keep asking new ones, and by then the tokens they need are gone, so code recall drops hard." So he went the other way — throw nothing away, just don't look at all of it at once. Pruning is destructive and irreversible; per-step selection is not.
The other reason this works is the model itself. Qwen3.8-27B is a hybrid: 64 layers laid out as 16 repeats of 3 × (Gated DeltaNet → FFN) followed by 1 × (Gated Attention → FFN) — 48 linear-attention layers to 16 full-attention ones, with a native 262,144-token context (SGLang cookbook, MindStudio). Only the 16 full-attention layers need the sparse treatment; the GDN layers carry their state in recurrent form, which is the largest single share of the ~1.6 GB per agent. Sparse attention is only tractable here because the model already made 75% of its attention linear.
For people building long-context agent serving on consumer hardware, the transferable ideas are: keep the whole context (host RAM is cheap — 4.4 GB per 250K agent), select per step rather than pruning, and make the selection basis document-independent so prefix caching survives. The quality result is the honest part: sparse decode sits inside the dense cross-card noise floor, and the author says so rather than claiming parity.
Caveats
- Everything is self-reported. All throughput, recall, and KLD numbers come from one author's repo and Reddit post; no independent replication was found.
- The rig is not stock. Per the Reddit post, all numbers were measured with the gpu-unlock kernel module loaded. That project reports up to ~2× tensor throughput on GeForce cards — 3090-class cards measure ~+65–80% BF16 — with the gains mostly in prefill; it claims decode is memory-bound and unchanged. Either way, the tok/s figures are not stock-hardware numbers.
- Attention is approximate. Near-duplicate code lines (CuTe/asm blocks) occasionally get mixed up, and greedy output can diverge from dense after some tokens. The repo offers no formal error bound for the approximation.
- The author missed his own target: 300+ explain tok/s per card at 4 × 250K, got 268–285.
- Fragile internals. The tier hooks monkeypatch vLLM scheduler internals and are acknowledged as fragile across vLLM upgrades.
- Unmeasured edges. No fifth agent per card at MTP-7; the longest KLD test was 192K because the dense comparison server could not hold 250K; cold prefill of one 244K document takes 390 s and prefills run one at a time; follow-up TTFT is 5+ seconds on context switch.
- Two layers of approximation. The model is W4A16-quantized on top of sparse attention.