Richland Technology
Richland Technology
← Back to Blog
llmopen-sourceresearchmultimodalinference

Spool lets hybrid VLMs stream video without re-running the backbone — 6.6× p95 TTFT cut, +2.3 points accuracy

Caching per-frame affine state transformations lets hybrid VLMs drop expired frames in O(log W) without replaying the window.

Richland AI Desk·19 November 2026·5 min read

MIT CSAIL researchers have published a preprint describing Spool, a training-free inference framework that lets hybrid vision-language models (VLMs) keep a sliding video window without re-running retained frames through the backbone. Across five open-source hybrid VLMs and four video reasoning benchmarks, they report the lowest p95 time-to-first-token in all 20 model–benchmark tests, a 6.6× geometric-mean reduction over the faster of two vLLM baselines, and a 2.3-point average accuracy gain.

What shipped

Hybrid VLMs — models that interleave attention layers with linear recurrent layers such as Gated DeltaNet (GDN) or Mamba — suit streaming video because the recurrent layers hold history in a fixed-size state. But that state is the problem: it mixes every past frame into one shared vector, so when a sliding window expires an old frame, the only way to remove its influence is to re-run the entire window through the model. Attention layers can just evict the expired frame's key-value entries; the recurrent state has no per-frame entries to evict.

Spool's insight is that a GDN or Mamba update is an affine transformation of the previous state — S_t = S_{t-1}A_t + B_t — so the updates for all tokens in a frame compose into a single affine transformation. Instead of caching states, Spool caches the multiplicative and additive coefficients (A, B) of each frame's transformation. Composing the coefficients of the frames in the current window and applying the result to the initial state rebuilds each recurrent layer's state without re-running the backbone (paper).

Because affine transformations compose associatively, Spool stores the combined transformations of consecutive frames in a segment tree and reconstructs the state in O(log W) compositions, where W is the window length in frames. The authors report the coefficient cache costs 4.8 GiB (18% overhead) for a 9B model (full text).

The third piece is attention memory. Serving engines keep the KV cache in fixed-size pages, and evicting early frames leaves partially occupied pages; compacting the survivors would mean copying memory proportional to the retained context. Spool instead fills vacated KV slots with newly arriving frames' entries and periodically rebases cached rotary position embeddings (RoPE), preserving relative-position attention without recomputing visual representations.

The headline numbers, all self-reported by the authors:

  • Lowest p95 TTFT in all 20 model–benchmark tests, a geometric-mean 6.6× reduction (up to 35.7×) over the faster of two vLLM baselines, and 33.8× over plain vLLM-Omni.
  • p95 under 1 second in 16 of 20 tests, versus 3 for incremental prefill.
  • Online accuracy +2.3 points on average, up to +18.4 points on Beyond3D.
  • Offline (no timing pressure): within −1.6 to +1.8 points of the more accurate baseline in all 24 offline tests.
  • Caching components cut per-frame ingestion latency 7–17% versus a naive cache; KV filling saves up to 1.8 GiB of peak memory.

The intro names Qwen3.5-VL and Nemotron 3.5 Super VL as example models and says the evaluation covers six open-source hybrid VLMs from 0.8B to 122B parameters with GDN and Mamba-2 recurrent layers. The benchmarks include Beyond3D. The abstract says five models while the intro says six, with the 20 online tests implying 5 models × 4 benchmarks.

Why it matters

The recurring cost in streaming VLM inference is not the new frame — it is the invalidation of everything before it. Prefix caching in engines like vLLM and SGLang requires the start of the sequence to stay fixed; once the window evicts the first frame, the cache is invalidated and every remaining frame must be recomputed. Spool makes "forgetting" as cheap as "remembering" in hybrid models: the affine structure of the recurrent update turns a frame's contribution into a first-class object that can be cached, composed, and deleted from the state in O(log W), without replaying the window.

That is the missing primitive for serving Qwen3.5-VL-class models on long-running streams — robotics, visual accessibility agents, live sports analytics, streaming moderation — where answers must reflect recent frames and arrive promptly. The accuracy gain is a side effect of latency: because fewer questions are dropped under streaming load, online accuracy rises even though the model itself is unchanged.

For engineers building on hybrid VLMs, the practical takeaway — my inference, not a claim the paper makes — is that the recurrent state is not a black box you must replay to correct. If your recurrent layer's update is affine in the state, you can cache per-frame transformations and compose them to reconstruct the state after eviction, a technique that should transfer to any GDN/Mamba-style layer.

Caveats

This is a preprint from Nov 17, 2026, and every quantitative claim is the authors' own. There is no independent reproduction, and no code or artifacts have been released, so the segment-tree implementation and the "dedicated GPU kernels" cannot be inspected.

The theoretical weak point is the exactness claim. Removal is exact within a layer, but expired frames retain residual influence through the inputs that earlier layers computed while those frames were in context — the same approximation as KV eviction in attention. The authors assert this has "little impact on accuracy," but the nonzero KL divergence after 1024 expired frames (about 2.5× lower than without reconstruction) shows the reconstructed state is not identical to a full replay.

Two details are unresolved from the accessible text: the abstract says five models while the intro says six, and only one of the four benchmark names (Beyond3D) is confirmed. Treat the 6.6× / +2.3-point headline figures as vendor-reported until independent verification exists.

Sources

Start a Conversation

Ready to build something like this?

Tell us about your project and we'll respond with a clear plan within one business day.

Tell Us About Your Project

Takes 3 minutes. We read every submission personally.