Strata fork splices the KV cache past changed system-prompt lines: DeepSeek V4.1 Flash agent sessions start ~20× faster on a 16 GB RX 9070 XT

Four days after Sam Altman called home DRAM "quite terrible" from a global-allocation perspective on a podcast, a hobbyist fork of the Strata inference engine cut DeepSeek V4.1 Flash agent-session cold starts from ~100 s to ~5 s on a single RX 9070 XT. It splices the KV cache past the per-session lines that make prefix caching miss, and routes prompt tokens only to experts already resident in VRAM or RAM.

What shipped

Strata is Niko Veit's MIT-licensed, C++ local MoE inference engine (roughly 38k GitHub stars as of early November). Release 0.2.4 added a dedicated strata-dsv41 engine for DeepSeek V4.1-Flash-1017. The model is a sparse MoE: 763.2B parameters, 384 routed experts per layer with 6 active per token, a 1M-token context window, MIT-licensed, published by ISTA-DASLab with GSQ/RCO quantizations. The fork runs the 1.75 bpw set; the lab reports it recovers 98.7% of the base model's task average (measured at reasoning effort 50).

The author (GitHub: PhialsBasement) runs it on an RX 9070 XT 16 GB, Ryzen 7 5800X3D, 64 GB DDR4 and a PCIe 3.0 NVMe, with the model spread across VRAM, RAM and SSD. Generation ran at ~7 tok/s (reported in the Reddit post). The problem was prompt reading: every new OpenCode session re-read the full ~37K-token system prompt plus tools — 37,452 tokens, 0 reused, 100.3 s before the first token. Two changes in the dsv41-kv-splice branch fix that.

KV splice (issue #4921). Ordinary prefix caching reuses only a prompt's identical shared start. OpenCode breaks that assumption by writing the working directory, date, git status and project instruction files near the top of the system prompt, and synced skills carry random folder IDs in their paths — so every new session misses the cache.

The splice aligns the new prompt against every saved conversation "using token runs of 32 found wherever they sit," cuts the stale lines out of the cached state (later rows move down, every window ring rotates, and the cut is recorded as a RoPE gap), and reads only the changed lines plus the user's message. Changes move as whole lines, so the model never sees fragments; the cut table is saved with the slot as a DS4CUTS1 trailer, and an engine without it refuses the slot and re-reads the prompt.

On his machine: 37,482 tokens = 36,097 reused + 1,385 read, 26.0 s instead of 100.3 s.

Resident-only routing (issue #4936). Even a short prompt touches nearly every expert — 6 of 384 per token across 40 layers means a few hundred tokens reach almost every expert of every layer — so prompt reads streamed most of the model off the SSD: 58 GB for a fresh 1,239-token prompt at 2.1–3.0 GB/s, with the GPU only 37–49% busy.

During prompt reads only, the router now gets a selection bias: an expert's gate_bias minus 1e4 if it sits in none of VRAM, the RAM store, or the page cache. The bias only picks the top-k; the weights come from the unbiased scores, renormalized over the chosen experts. Reply tokens route as trained.

On his machine ~4,900 of 15,360 experts are resident (474 in VRAM, the rest in the RAM store), and the same 1,239-token prompt drops to 8.2 s and 1.3 GB of NVMe reads, with the GPU 73% busy.

Combined, a new session in another project read its changed 513 tokens (36,949 reused) in 5.1 s, where reuse alone took 26.0 s — the ~20× headline (100.3 s → 5.1 s) is arithmetic from his own numbers. Later turns read 145 tokens in 3.8 s. Both changes are on by default in the branch; STRATA_DS4_REORDER=0 and STRATA_DS4_PROMPT_RESIDENT=0 disable them. HIP/gfx1201 only. If you use OpenCode, the author also recommends setting "interleaved": { "field": "reasoning_content" } on the model entry, otherwise OpenCode drops earlier reasoning and the cache misses more often.

Why it matters

The tax on every local agent session isn't decode — it's re-reading the same prompt. Cloud providers hide this cost with prefix caching; local engines have only started solving it, and the standard prefix cache is defeated by the one thing agents do constantly: inject per-session state (date, cwd, skill paths) into the system prompt. The splice generalizes prefix caching from "identical start" to "reuse past edited lines," which is the actual shape of the problem — and it does it without a dequantize-rotate-requantize pass over the FP4 rows, at the cost of a RoPE gap where text was cut.

The second change is a phase-specific routing policy. Prompt reads and generation have opposite bottlenecks — SSD bandwidth versus compute — so one routing policy is wrong for both. Resident-only routing applies only while a prompt is being read; generation still routes as trained. That split is the reusable idea: when the bottleneck is memory bandwidth, prefer what is already resident, even at the cost of substitute experts.

The timing is the point of the post's title. Altman made the "quite terrible" remark on the Cheeky Pint podcast (hosted by Stripe co-founder John Collison, released Oct 29): "From a global allocation perspective, actually, it's quite terrible to have everybody have a bunch of the world's limited supply of DRAM sitting in their own computer and being used 5% of the time." Four days later, a fork removed the specific inefficiency that made local agent sessions feel datacenter-grade slow — a ~37K-token prompt re-read from an NVMe on every session.

Caveats

Everything here is author-reported from one machine, one run each, and the branch is unmerged (the issues propose the features as opt-in; the branch runs both on by default). HIP/gfx1201 only — no CUDA, SYCL or Metal builds, and Strata's C++ test suite was not run.

The splice is explicitly approximate: tokens after a cut were computed with the removed text in their context, and there is no formal eval; the branch's later commits added safeguards (one cut point per candidate at the latest checkpoint, exact token match required for reuse).

Resident routing provably changes answers — substitute experts for prompt tokens — though a 13-problem hard coding suite stayed 12/13 both ways (different problems failed each way), and a recall check found 3/3 exact strings in fresh prompts. No KL/perplexity comparison against stock routing and no repeated benchmark runs.

The 98.7% figure is ISTA-DASLab's quantization-recovery score versus the base model, not a claim about these patches. Strata's own docs say nothing past 256K context has been tested, despite the 1M capability. Altman's 5% is his own estimate; no outlet has published an independent measurement that matches it.

Sources

• Reddit post (r/LocalLLaMA)
• Strata issue #4921 — KV splice
• Strata issue #4936 — resident-only routing
• dsv41-kv-splice branch
• Strata repository
• Strata DeepSeek V4.1 model doc
• ISTA-DASLab DeepSeek-V4.1-Flash-1017 GSQ-RCO model card
• DeepSeek-V4.1-Flash-1017 (deepseek-ai)
• Shattered.io — Altman's DRAM remark, context and analysis
• Yahoo Tech — Altman on PC RAM