Tsinghua's AKV lets agents drop and reposition their own KV cache — 27.7% lower per-task cost, +1.7 points, Apache-2.0
A patch-based SGLang fork treats the KV cache — not the transcript — as the thing agents should edit: drop and RoPE-reposition entries instead of summarizing, with a modeled 27.7% cost cut and an honest losing case.
Tsinghua University (THU-NICS) and CUHK have open-sourced AKV, a serving system that lets LLM agents drop and reposition their own KV cache entries instead of summarizing conversation history. Across 3 open-weight models and 4 agent benchmarks, AKV reports 27.7% lower average per-task cost (API-equivalent, in its own cost model) and +1.7 points average task score versus threshold summarization — per the paper's cost analysis, KV cache reads make up 82.4% of total API spend.
What shipped
AKV is a serving system plus a policy library, not a new model. The code lives in thu-nics/AKV on GitHub under Apache-2.0, built as a fork of SGLang: the repo ships a 311 KB patch pinned to SGLang 0.5.20 (packaged as sglang 0.5.20+akv), applied at build time. Two components: akv.agent, the policy side, and akv.system, the serving side. The paper is arXiv:2610.28577, "Cache Engineering for Agents."
Existing harnesses do what AKV calls context engineering: they summarize or trim the text history when it gets long. Trimming a prefix invalidates every KV state after it, so the retained suffix must be re-prefilled — expensive, and it forces harnesses to summarize late and aggressively, losing details from middle rounds. AKV instead does cache engineering: it never touches the text trajectory, only which KV entries stay visible to attention; retained entries are reused without recomputation.
Three cache operations
AKV defines three operations (paper, repo README):
- Append — the standard update that adds KV states for new tokens.
- Drop — removes selected entries from future attention without changing token positions. The text is still there; it is just no longer attended to.
- Repos — rewrites retained keys to positions 0 through r−1 by rotating each cached key from its old position to its new one (a RoPE rotation). Values are untouched, so no hidden states are recomputed.
Repos is not a nicety: drop alone never resets position indices, so an agent with a shrunken active cache can still run into the model's position limit. In the paper's ablation, adding Repos to a tool-response drop policy raised GPT-OSS-120B's BrowseComp-Plus pass rate from 55% to 62%. (MarkTechPost, Tensor Journal, LLMs Blog)
The hard part is serving
These operations seem trivial, but they break the core assumption every serving engine's prefix cache is built on: identical token sequences can yield different cache states under different operation histories, so plain prefix matching can silently reuse the wrong KV for an identical token prefix. The paper's abstract states this directly and pairs it with a design principle: a "stateless interface compatible with existing serving APIs, while internally managing stateful cache reuse, eviction, and restoration."
AKV's akv.system extends SGLang's radix tree with drop nodes and reposition nodes — cache reuse matches on both tokens and operation history. Dropped KV is freed only when no active request still needs it, and eviction prefers dropped cache over cache from finished requests (more likely to be reused next round). Segment-batch attention reconstructs evicted states in a single prefill pass: attention splits only at drop points while the FFN runs over the full sequence as one batch. (repo system README, LLMs Blog, The Frontier)
A policy API, and a model-callable drop tool
The client interface is stateless and OpenAI-compatible: send the full message history plus two optional fields — drop_message (trigger index → message indices to drop) and reposition (indices after which positions get compacted). On the policy side, a CachePolicy is a (trigger, selector, reposition) triple; built-ins are turn-based, token-based, and model-based. A CacheTool exposes drop_context so the model itself can call for a drop, and successful calls are converted into drop_message operations. Usage is broken out as cached_tokens, drop_skipped_tokens, and repos_tokens in usage.prompt_tokens_details. (repo README)
The flagship policy is a tool-response rolling drop that keeps the KV for all reasoning tokens plus the latest 12 tool responses, dropping older tool-response cache — grounded in an attention analysis showing older tokens and tool responses get less attention.
Benchmark numbers
Evaluation covered 3 models × 4 agent benchmarks (BrowseComp-Plus, a 103-task text-only GAIA validation subset, SWE-bench Verified, and xbench-DeepSearch-2510): GPT-OSS-120B, MiniMax-M2.7 (230B total, 10B active), and MiroThinker-1.7-Mini (30B total, 3B active). Baselines: Standard summarizes at 96K tokens (192K for MiroThinker); Text Engineering at 48K. Average pass rate (%) and cost (US cents per task), from Table 1 of the paper as reproduced by MarkTechPost and LLMs Blog:
| Model | Standard score / cost | Text Eng. score / cost | AKV score / cost |
|---|---|---|---|
| MiniMax-M2.7 | 52.4 / 5.0 | 52.8 / 5.7 | 53.7 / 4.4 |
| GPT-OSS-120B | 51.0 / 11.2 | 48.3 / 10.2 | 53.1 / 7.3 |
| MiroThinker-1.7-Mini | 44.1 / 20.4 | 37.1 / 18.1 | 45.8 / 14.8 |
Best case per the paper: GPT-OSS-120B on SWE-bench Verified drops from 20.0¢ to 9.7¢ per task while score rises 40.0 → 43.2. On serving throughput, the paper reports up to 1.2× output throughput and a 1.5× P95 latency speedup; at concurrency 16, P95 is 229.90 s for Standard vs 30.63 s for AKV, with per-decode-step full-attention time down from 7.082 ms to 2.849 ms. The paper ran on 8× A800-80GB (BF16, TP4, split 4 prefill / 4 decode); the quick start serves GPT-OSS-120B with --tp-size 2 on two GPUs.
Why it matters
The paper's own economics make the case: on its Standard baseline, KV cache reads were 10.8 of 13.1 cents per task on BrowseComp-Plus — 82.4% of total API cost. If that split is anywhere near representative for long-horizon agents, then summarization — which optimizes text while paying re-prefill on every edit — is optimizing the wrong surface. AKV moves the lever to the cache itself and charges for its one trick (Repos re-positioning cached keys) at the cache-hit rate, a deliberately conservative choice in its cost model. (paper, LLMs Blog)
The deeper lesson for people running LLM agents: prefix caching assumes state is a pure function of the token prefix. The moment a policy can drop or reposition cache entries, state becomes history-dependent, and a radix tree that ignores that history silently serves wrong caches. AKV making this work — correctness-preserving reuse, single-pass restore, a stateless exposed API — is the actual engineering contribution, and the part that generalizes beyond these three models.
It also only works as well as the policy. AKV's design reframes "what should the cache forget?" as a callable, model-invocable API (drop_context via CacheTool), which is a genuine improvement: harnesses get a structured, auditable knob instead of a summarizer silently rewriting the transcript.
Caveats
These are the paper's own numbers, not an independent evaluation. Tensor Journal explicitly disclaims reproduction ("the results described here are the authors' own claims... we have not reproduced them"), and nothing has been reproduced independently yet.
- Costs are modeled, not measured. Costs are API-equivalent estimates from published rates (Groq's GPT-OSS-120B rates, official rates for the others), not a deployment bill. Repos being charged at cache-hit rate is conservative by design — but it is still an assumption.
- Not upstreamed. AKV is a patch-based fork pinned to SGLang 0.5.20, not a merged integration. Installing means compiling with Rust and building "Context CPU kernels" at setup, and the constraints are real: page-size 1, TP-only (no PP/CP/DCP), no speculative decoding, LoRA, HiCache, or beam search. You are trading a cheap inference lever for a fork to maintain.
- It loses real cells. MiniMax-M2.7 on BrowseComp-Plus scores 58.9 vs Text Engineering's 62.3. LLMs Blog lists the other losses: MiniMax SWE-Verified 74.4 vs 75.2 (Standard), GPT-OSS on xbench-DeepSearch 37.5 vs 38.5, MiroThinker SWE-Verified 7.4 vs 7.8. In the appendix on the smaller GPT-OSS-20B, AKV wins average score (45.2 vs 44.0) but costs more than Text Engineering (4.8¢ vs 4.6¢) — the authors attribute this to smaller models being more sensitive to cache changes.
- Ghost information. Retained KV entries can still carry information from dropped messages (the paper concedes this). That is fine for most agent workloads and a real problem where the transcript must be byte-exact, e.g. for compliance or debugging. A wrong Repos call can also desynchronize position encodings — the failure mode is reintroducing exactly the state-corruption bugs that made everyone build prefix caching in the first place.
The verdict for a builder: if KV reads dominate your agent bill and you can own a fork, AKV is one of the few open, runnable-today systems that directly attacks the cache instead of the text. If you need an unmodified SGLang line, wait for the patch to be worth merging — the mechanism (history-aware radix tree, segment-batch restore, RoPE Repos) is the part that is likely to survive upstream, whatever happens to the policy library.