KV²: a self-refining KV cache that replays only proxy-selected tokens — 40+ points on RULER 16K at 2% budget

Karlsruhe Institute of Technology researchers have released KV², a query-agnostic KV-cache compression method that scores cache entries by replaying only a proxy-selected subset of context tokens instead of the full context. The authors report it beats the next-best baseline by more than 40 percentage points on RULER 16K at a 2% cache budget, and it ships as a drop-in KVSquaredPress in an anonymous fork of NVIDIA's kvpress framework.

What shipped

KV² (arXiv:2610.03198v1, submitted October 2, 2026) is a two-stage compression pipeline from Johannes Wesch, Danni Liu, and Jan Niehues. It targets the "reusable" setting: one prefilled context that must later serve many different queries — long-document QA, persistent agent contexts — where compression must be query-agnostic and budgets, the paper argues, sit well below 10% of the original cache.

The mechanism, in two stages:

1. Query selection. A lightweight proxy scorer (default: KeyDiffPress) scores each context chunk (default chunk_size=4096) and selects the top-scoring positions as reconstruction queries — by default 1 - compression_ratio of the chunk.
2. Cache scoring. Only those tokens are replayed through the model against the full cache (num_logits_to_keep=1) with KVzip's forward hooks registered, producing final eviction scores. Low-scoring entries are evicted before decoding reuses the cache.

The design also supports iterative self-refinement: nesting a KVSquaredPress inside itself lets one reconstruction pass provide the proposal signal for the next.

The code is a subclass of KVzipPress in an anonymous fork of NVIDIA's kvpress (Apache-2.0), usable through the existing pipeline:

from transformers import pipeline
from kvpress import KVSquaredPress

model = "Qwen/Qwen3-8B"
pipe = pipeline("kv-press-text-generation", model=model, device_map="auto", dtype="auto")

context = "A very long text you want to compress once and reuse"
question = "\nWhat are the key facts in the context?"

press = KVSquaredPress(compression_ratio=0.9, chunk_size=4096)
answer = pipe(context, question=question, press=press)["answer"]

The authors report evaluations on Llama-3.1-8B-Instruct and Qwen3-8B across RULER 16K, Needle-in-a-Haystack, LongBench, and 200K-scale InfiniteBench. Headline results, from the abstract and README:

• RULER 16K at a 2% cache budget: average score more than 40 percentage points above the next-best baseline.
• LongBench: highest average across 2–10% budgets on both models.
• InfiniteBench 200K: "competitive."
• Single-pass KV² also cuts compression-stage runtime and peak memory relative to full-context reconstruction.

Why it matters

The claim worth taking seriously is not the headline number but the cost structure. Reconstruction-based scoring (KVzip) is the most accurate query-agnostic approach, the authors argue, but it pays for accuracy by reprocessing the entire context: the kvpress library itself warns that KVzip "requires multiple forward passes for chunked context reconstruction, resulting in a computational overhead of 2–3 times the initial prefilling cost." KV² keeps the reconstruction pass but shrinks the query set — the paper's core design choice is "not whether to reconstruct, but which queries to use for reconstruction."

The underlying observation is that long-context models do not use all positions equally: at tight budgets most positions are either trivial to retain (attention sinks) or trivial to evict (filler words, punctuation), so eviction hinges on a sparse set of informative anchors. Scoring a small, well-chosen subset captures most of the signal at a fraction of the replay cost. KV caches already dominate memory at scale — the paper cites roughly 26 GB of KV memory for a 100K-token prompt on Qwen-2.5-32B.

For engineers building long-context or agentic systems, a cache compressed once and reused across many queries no longer forces a choice between cheap-but-dumb estimators and replay-everything scoring. And because KV² ships inside the kvpress fork — the same framework that already implements KeyDiff, Expected Attention, SnapKV, and the other baselines — it is directly comparable and runnable today, not a paper-only artifact.

Caveats

Everything quantitative above is vendor-reported. The paper's results section could not be retrieved from any public route at the time of writing, so the exact RULER scores, LongBench averages, per-baseline tables, and runtime/memory figures could not be independently checked. Treat the >40-point claim as the authors' number, not a verified result.

• The >40-point margin is measured at an extreme 2% budget; the abstract notes the margin over baselines narrows as the budget loosens.
• The paper is an unreviewed preprint (v1, October 2026); the README describes the work as a thesis. An OpenReview forum exists, but its venue and status are not publicly accessible.
• The code lives only in the anonymous review fork — there is no KVSquaredPress in NVIDIA/kvpress main.
• H2O, LAQ, and Judge Q are discussed in related work but do not appear in the repo's evaluation registry; whether the paper actually compared against them is unconfirmed.
• No independent reproduction or third-party coverage was found.

Sources

• arXiv abstract
• Full text (HTML)
• Code README
• kvsquared_press.py
• kvzip_press.py
• Evaluation registry
• NVIDIA kvpress
• KVzip (arXiv:2505.23416)
• OpenReview forum