Richland Technology
Richland Technology
← Back to Blog
llmresearchinferenceservingkv-cache

When the KV cache is slower than the GPU: a 124× p99 fix for context-parallel prefill

A production GLM-5.3 teardown finds remote KV hits slower than recompute — break-even is 2.53 GB/s, and the fix is sharding by rank.

Richland AI Desk·16 October 2026·8 min read
When the KV cache is slower than the GPU: a 124× p99 fix for context-parallel prefill

In a production GLM-5.3 prefill deployment on 8×B300 nodes, enabling a remote KV cache tier (LMCache backed by Valkey) pushed p99 prefill latency at 16k tokens from 3.96 s to 17.3 s — loading a cached prompt took 0.88 s versus 0.31 s to recompute it. A preprint by Muhammad Ibrahim (Multiverse Computing, arXiv:2610.21500) explains the failure with a two-term cost model, then documents the fix — a KV store sharded across prefill ranks, one Valkey instance per rank, and asynchronous loading — which cut a hit-heavy stream's p99 from 83.32 s to 0.67 s, about a 124× reduction.

What shipped

Prefill and decode run on separate 8×B300 nodes, each prefill engine running GLM-5.3 (an FP8 mixture-of-experts model with DeepSeek-style multi-head latent attention, MLA) with prefill context parallelism across its eight GPUs (PCP8), plus expert parallelism (EP8). Every forward step contains NCCL collectives, so the eight ranks advance in lockstep: anything that stalls one rank stalls all of them. LMCache keeps KV in 256-token chunks — 12.4 MB each for this model, about 48.4 KB per token, since MLA's per-token state is a single low-rank latent independent of head count — in host memory (L1) and in a remote Valkey store over the RESP protocol (L2). The SLA workload is enterprise traffic with 9k–200k-token prompts.

Enabling L2 left median TTFT unchanged but inflated the tail: p99 went from 3.96 s to 17.3 s at 16k tokens and from 6.06 s to 25.0 s at 32k, and the deployment was rolled back to L1-only. To find out why, the author rebuilt one production engine's request stream from router logs and replayed it on an isolated, identically configured node. Three effects explained the regression:

A hit was slower than a miss. A 16k-token prompt took 0.31 s to recompute but 0.88 s to load from L2. The B300 node prefills at about 52.2k tokens/s; the L2 path delivered nowhere near the bandwidth needed to keep up.

Synchronous loading is unstable under hit-heavy load. A load executed inside the engine step stalls all eight PCP ranks. When arrival rate times per-hit service time exceeds one, the engine becomes an overloaded queue: median latency grew from 3.52 s to 52.30 s within six minutes while the GPUs sat 7% busy.

Three client bugs caused unbounded stalls, each fixed with a regression test (#5547, #5551, #5550): a lost-completion race that makes fut.result() block forever and freeze the engine; a missing-key read that waits for a fixed-size chunk header when Valkey replies $-1 (a cache eviction between EXISTS and GET wedged the connection permanently); and a key-rewrite bug that stalled every non-zero rank's L2 prefetch after an engine restart.

The cost model

The paper's central contribution is a two-term model (§4). For a prompt of n tokens served from cache, with s bytes of KV per token, prefill rate R, and tier bandwidth B:

  • T_recompute = n / R
  • T_load = T0 + n·s / B

where T0 (~0.33 s here) is the per-hit cost that doesn't scale with bandwidth: scheduling, lookup round-trips, the forward step over uncached tail tokens, and metadata exchange.

Ignoring T0, a hit pays off only if B > B* = s·R — the prompt length cancels, so the threshold depends only on the model's KV footprint and the hardware's prefill speed. For GLM-5.3 FP8 on one B300 node that means the tier must deliver more than 2.53 GB/s per prompt; the single Valkey instance delivered roughly 0.9 GB/s. And because of T0, hits on prompts shorter than ~17k tokens cannot beat recompute at any bandwidth (0.33 s × 52.2k tok/s).

The paper's blunt lesson: "on current GPUs recompute is a demanding baseline. A cache tier has to be judged against the prefill rate of the hardware it serves, not against the latency of the network."

The fix, guided by the model

Four changes, all submitted upstream to LMCache:

  1. Sharded MLA KV store across PCP ranks (#5545). Chunk c is owned by rank c mod 8; only that rank stores, looks up, and loads it, splitting L1 capacity and store traffic evenly. The paper proves the existing min-over-ranks lookup still returns the longest cached prefix: each rank reports the start token of its first owned miss, so if m is the first chunk missing anywhere, no rank's answer is below m and m's owner answers exactly m. The load protocol keeps every rank issuing the same collectives in the same order — a metadata exchange followed by one tensor broadcast per chunk from its owner — and failures are propagated only through that exchange, never raised before a collective, so ranks cannot desynchronize.

  2. Rank-partitioned remote tier (#5548). Valkey executes every command on a single main thread, so one instance's throughput is bounded by one core regardless of client count — the measured bottleneck. Since sharded chunks are partitioned statically by rank, rank r of every engine can use its own server instance (a {rank} placeholder in the connection URL) with no cluster protocol and no client-side hashing, and cross-engine reuse still works because rank r of every engine owns the same chunks. This raised aggregate GET bandwidth 6.3×, from 1.71 to 10.76 GB/s.

  3. Asynchronous loading (#5553). Each rank prefetches only its owned chunks off the lockstep critical path; the equivalence between min-of-async-answers and the sharded prefix is property-tested over 3,000 random cases.

  4. Bug fixes — the three hangs above. The important design point: a KV cache must miss fast, not hang, or it silently converts cache hits into engine freezes with the GPUs idle.

On the hit-heavy replay (450 requests, 98.5% of prompt tokens served from L2), p99 fell from 83.32 s to 0.67 s. The sharded store raises prefill throughput by 14% (43.2k → 49.4k tok/s on one node, per the author's write-up), and by 34% when combined with an upstream FlashInfer kernel fix that routes DeepSeek-style FP8 activation kernels through a compact workset instead of an expanded grid (3.0× faster, bitwise-exact on GLM 5.2 EP8 prefill). The same author reports the production deployment now meets its SLA on 5 B300 nodes instead of 11, at 3.6× tokens/min per node (GitHub profile).

Why it matters

The arithmetic generalizes to anyone running a remote KV tier. Before this paper, the working assumption was "loading KV is cheaper than recomputing it"; now the break-even is computable from two numbers you can measure — the model's KV bytes per token and your engine's prefill rate. On Blackwell Ultra-class hardware, R is large enough ($B^*$ = 2.53 GB/s here) that many "fast" cache backends lose to recompute outright, and the T0 floor means short prompts should never be routed to a cache tier at all. Cache-vs-recompute decisions belong in the router, not in the cache client.

The lockstep result is the sharpest takeaway for context-parallel serving. In PCP, the latency budget for a KV fetch is the per-step time — any synchronous load inside a collective step stalls every rank, which is how an overloaded cache produced 83 s p99 latency with GPUs 7% busy. Cache I/O has to be asynchronous and off the critical path, full stop.

The bottleneck turned out to be structurally mundane: Redis/Valkey's single-threaded command execution capped a whole tier at ~0.9 GB/s, and the fix was static partitioning plus one server per rank, not a cleverer protocol. The bug-fix item itself was three separate hangs that all shared one failure shape: the cache failed by hanging instead of missing — a tier that won't fail open on a missing key is worse than no tier at all. All six PRs are open on GitHub — this is a rare production teardown you can audit yourself, including the proof of the min-over-ranks invariant.

Caveats

This is a single-author preprint with no peer review, all numbers self-reported, and the paper itself notes in §3 that "each condition was run once" — measurements come from isolated trace replay, not a live production A/B. The p99 numbers are prefill-leg latencies, not end-to-end TTFT. There is no independent reproduction yet, and all upstream PRs remain open and unmerged at time of writing.

The numbers are specific to one stack — GLM-5.3 FP8 on B300 with PCP8/EP8. The break-even formula transfers, but s and R (and therefore B*) are model- and hardware-specific, and the ~17k-token T0 floor will shift with engine and deployment. The five-node production claim appears in the paper itself (its introduction reports a five-node deployment passing the workload's SLA) as well as in the author's GitHub profile and blog; none of it is independently verified. GLM-5.3's public specs (1M-token context) are reported via third-party sources citing Zhipu's docs, and the B300 details come from third-party hardware write-ups rather than NVIDIA's own spec sheet.

Sources

Start a Conversation

Ready to build something like this?

Tell us about your project and we'll respond with a clear plan within one business day.

Tell Us About Your Project

Takes 3 minutes. We read every submission personally.