Latent Compaction: the KV cache as a trainable network — 96.7% on RULER at 32× (logical) compression, code released
Tsinghua's THUNLP optimizes cache entries per document with 2,000 gradient steps of test-time training and frozen weights; the per-document GPU cost is nowhere published.
Latent Compaction treats a document's KV cache as a trainable network: Tsinghua's THUNLP group reports 96.7% on RULER at 32× cache compression with Qwen3-1.7B, where the KVZip+ baseline falls to 40.0%, and has released the code under an MIT licence. The method freezes the model's weights and instead optimizes cache entries per document with test-time training, using only generic prompts.
What shipped
Latent Compaction is a paper plus code release from Tsinghua's NLP group (Yingfa Chen, Xingyu Shen, Chaojun Xiao, Xu Han): one arXiv preprint (2610.20582v1, submitted 14 October 2026, 19 pages) and an MIT-licensed repository with per-benchmark reproduction recipes, pinned dependencies, and a validation record.
The core move is conceptual: the compact cache is not a selection of important KV pairs but a small neural network whose weights are the cache entries themselves, optimized per document against the full-cache model.
The cache is the network. For an average budget of B KV pairs per layer per KV head, the method allocates L·H·B slots. Each slot holds a learned (K, V) pair plus a scalar attention bias initialized to zero; it compensates for the attention mass each head loses to compression. Attention becomes ℓ(q) = s·K_cmp·q + b_cmp, with output A_cmp(q) = V_cmpᵀ softmax(ℓ(q)). Question and generated tokens are appended after the document as ordinary KV pairs outside the budget — no renumbering — and decoding mixes the compressed cache's log-normalizer with the new tokens' normalizer so the two attention pools combine correctly.
Initialization does half the work. Three-quarters of the budget is copied straight from the full cache, selected by reconstruction importance and seeded from KVZip+ scores — the paper calls these "preservers", and the repo describes the pass as "global KVZip+ preservers". The remaining quarter is seeded by teacher attention and key-space diversity ("attention anchors and farthest-key diversifiers"). For a 4K document at 32×, that is 96 preservers plus 32 learned slots per layer/head out of a 128-slot budget.
Two-stage training, weights frozen. Stage 1 holds the preservers fixed and fits the other entries to the full cache's attention outputs and log-normalizers. Stage 2 jointly optimizes the entire cache via end-to-end distillation from the frozen full-cache model.
The released config (configs/paper.json) spells out the schedule: 1,000 updates per stage, a fixed bank of 480 generic prompts used by both stages, greedy teacher continuations of up to 24 tokens, 4,096 evenly spaced captured attention positions per document, and AdamW at 3e-3/1e-4 with zero weight decay. Model computation runs in BF16; cache parameters are FP32. No downstream question or answer is shown to the training loop — only generic prompts about the document.
Each document gets one cache, trained once after prefill, saved as cache.pt, and reused across all held-out questions with no further optimization.
Reported results. The headline is author-reported on RULER with Qwen3-1.7B: 96.7% accuracy at 32× compression versus 40.0% for KVZip+. The repo's recipes cover RULER at ratios 2 through 256 across 4096–32768-token contexts, LongBench (16 English/code tasks) at 4×–32×, and Phonebook Lookup; released baselines are KVZip+, an attention-matching method, SnapKV, PyramidKV, a text-summary cache, and the uncompressed cache. The abstract also claims transfer across model scales and families, with "retrieval gains on both Qwen3 and MiniCPM5"; the repo prescribes replication across Qwen3-0.6B/1.7B/4B/8B and MiniCPM5-1B/2B. I could not independently read the paper's results tables past Section 3.1, so I am quoting only abstract-level claims here — the per-task LongBench, Phonebook, and transfer numbers are unverified from my side.
Why it matters
The standout idea is a KV cache you can fine-tune. The established compression families — eviction (H2O, SnapKV, PyramidKV), quantization, merging — all select, shrink, or round existing entries; the cache is inert data, none of it trainable. Here the entries themselves are parameters, and 2,000 gradient steps per document reshape them to fit that document's attention structure instead of trusting a one-shot importance score. That is the plausible mechanism behind retrieval surviving at 32× while importance-based pruning collapses to 40%.
The method also changes the engineering cost model. Compression normally trades a fixed amount of cache memory against accuracy, once. Latent Compaction adds a per-document compute cost, paid once and amortized: one training run per document, then unlimited questions against the same compact cache. When a fixed set of documents is queried many times — a retrieval corpus, an agent's background reading, a codebase — the trade-off's shape differs from streaming eviction, and per-query savings grow with query volume.
The transfer claim is the part most likely to change how you build. The abstract reports retrieval gains on both Qwen3 and MiniCPM5, and the repo's replication recipe for the family sweep is to retrain the same documents with a different reader model and output location — one pipeline, no new machinery, if the gains hold up.
Caveats
- Unverified preprint, unverified headline. This is arXiv v1 from 14 October 2026. The 96.7/40.0 figure is author-reported on RULER, a synthetic retrieval benchmark; I found no third-party reproduction. Treat every number as a claim pending replication — which the released repo makes cheap to attempt.
- "32×" is a logical budget, not a memory claim. The paper itself says: "These results concern logical KV-pair budgets rather than measured memory savings or inference speedups." The repo concurs: "Padding and bias storage mean physical bytes differ from logical slot counts." A 32×-budget cache is not automatically a 32×-smaller or 32×-faster one.
- The training cost is conspicuously unpublished. No wall-clock, no VRAM figure, no per-document token budget appears in any source I could verify; the environments doc explicitly declines to assert a VRAM requirement. The schedule implies real GPU minutes per document (2,000 updates over 480 prompts, teacher rollouts of 24 tokens). Anyone building this should measure the amortization point — how many questions per document are needed to beat plain eviction on end-to-end latency.
- Operational constraints. The pipeline is per-document: the code fits against 4,096 captured attention positions over a fully prefilled document, and the release describes no streaming variant. The intended path is NVIDIA CUDA with BF16 support. Dependencies are slightly awkward: the repo pins KVPress 0.5.4 with transformers 5.6.0 via a deliberate metadata patch, not upstream-supported versions.
- Scope. Headline results use Qwen3-1.7B; LongBench is capped at 16 English/code tasks; and the released baseline set stops at KVZip+, attention matching, SnapKV, PyramidKV, and a text-summary cache — H2O appears only in the paper's discussion.