S³: a training-free spectral swap that cuts reasoning tokens 27.4% — preprint, no code yet
New arXiv preprint composes paired base and thinking checkpoints per-matrix in spectral space, reporting 27.4% fewer tokens at +1.0pp accuracy across 28 environments.
A new preprint proposes S³ (Spectral Null-Space Swap), a training-free method that composes a "thinking" reasoning checkpoint with its non-thinking base. The authors report a 27.4% average cut in inference tokens and a 1.0 percentage point mean accuracy gain across 28 evaluation environments. The method is a per-matrix spectral operation — no training, no decoding changes — and it rests on an empirical claim about where reasoning lives in weight space.
What shipped
S³: Spectral Null-Space Swap Makes Reasoning Models Efficient (arXiv:2609.37976v1, submitted September 29, 2026) is a 44-page preprint from researchers at UIUC, Tsinghua, Dartmouth, and Peking University. It targets a familiar pain point: reasoning models that emit long chains of thought are accurate but token-hungry, and existing fixes — adaptive stopping, token pruning, checkpoint interpolation — operate at decode time or average whole checkpoints.
The paper starts in weight space. For a paired set of checkpoints — a Non-thinking model W_0 and its Thinking sibling W_t, both descended from the same pretrained base — the authors decompose the post-training difference ΔW = W_t − W_0 into components aligned with and complementary to W_0's singular subspace. On Qwen3-4B over AIME24/25, they report a structural asymmetry: the aligned component carries sizable parameter energy but induces weak functional change, while the complementary component drives most of the functional shift despite lower energy.
S³ turns that into a composition rule. For each target weight matrix, the leading singular directions of W_0 define a protected subspace S_ρ, and the composed weight is:
W_ρ = P_{S_ρ}(W_0) + (I − P_{S_ρ})(W_t)
The base model keeps its own structure inside its dominant subspace; the thinking model supplies everything outside it. A single parameter ρ controls how much of the dominant subspace is protected, and the operation is applied per matrix; the paper evaluates it on dense and mixture-of-experts architectures. The authors frame this as an asymmetric alternative to checkpoint averaging, task arithmetic, sign-resolution/delta-sparsification merging, and direct Non-thinking–Thinking interpolation: instead of blending whole checkpoints, it decides which model supplies each spectral region.
The numbers below are all author-reported from a preprint that has not been independently verified:
- 27.4% average token reduction versus full Thinking models, with +1.0 pp mean accuracy, across 28 evaluation environments spanning math, multimodal, and audio reasoning, on five paired model families from 2B to 30B (dense and MoE).
- Qwen3-4B-S³: +8.3 pp accuracy on HMMT25 alongside a 33.0% token reduction.
- Qwen3-VL-4B-S³: 78.05% on MathVista with 31.9% fewer tokens.
- The ρ=1 ablation isolates the two components: the null-space-only model "nearly matches Thinking accuracy with over a quarter fewer tokens," while the aligned-subspace-only model degrades toward the Non-thinking baseline.
The paper also offers a mechanistic story. Across evaluated datasets it reports a consistent attention-entropy ordering, H_Null < H_Base < H_Sub: the null-space composition concentrates attention more than the base model, while adding only the aligned component diffuses it. A simplified analytical model gives a conditional prediction for small perturbations. The authors say it matches the observed attention patterns and shorter reasoning traces, and they claim it is the first theoretical link between null-space projection in parameter space and attention entropy.
Why it matters
If the result holds, S³ is one of the cheapest levers available for anyone serving reasoning models. It requires no training run and no change to the decoding loop — the paper calls it training-free — just a per-matrix spectral decomposition and a weighted sum of two checkpoints you may already have. For teams running paired base/thinking releases (the Qwen3 family is the named example), that is a weekend-scale experiment with a plausible ~27% token saving, and accuracy is reported as flat-to-positive rather than the usual accuracy cost.
The conceptual claim is the more interesting part for people building on these models. The paper argues that the functional content of thinking-mode post-training is concentrated in the spectral complement of the base model's dominant directions — that the aligned component is functionally nearly inert. That is a testable hypothesis about where reasoning lives in weight space, and it suggests that the right way to compose checkpoints is asymmetric and subspace-specific, not a global interpolation coefficient. If other groups reproduce the asymmetry on non-Qwen families, it changes how merging and "thinking-mode" checkpoint surgery should be designed.
It is also worth noting what S³ is not competing with. It does not shorten the chain of thought the model generates; the authors report that the weight change is what lets the model reach the same answer in fewer tokens. It is a weight-space method, distinct in mechanism from decode-time tricks like adaptive stopping or token pruning.
Caveats
This is a preprint from September 29, 2026, and every headline number above is the authors' own claim. At the time of writing we found no code release, no model release, and no external coverage or replication attempt; the paper is not yet indexed by Semantic Scholar or the Hugging Face papers feed. Treat the 27.4% / +1.0 pp figures as unverified author-reported results.
A few specifics to hold in mind:
- The +1.0 pp mean accuracy gain is an average across 28 environments; no error bars are visible in the readable portion of the paper, so it may be within noise on individual benchmarks.
- The full experimental details — the exact ρ values, the complete list of five model families and 28 benchmarks, and the per-domain breakdown of the token savings — are in sections and 29 tables that are not fully readable from the public HTML, so the per-domain behavior (math vs. multimodal vs. audio) is unverified here.
- The method requires paired checkpoints that descend from the same pretrained base and differ only in thinking-mode post-training. Not every model family ships that pairing, which limits immediate applicability.
- The "null space" in the title is the orthogonal complement of the projector onto the base model's dominant singular subspace — not the kernel of the weight matrix. The distinction matters if you go read the math.
The core asymmetry claim deserves a careful look — and it is new enough that we found no public critique or replication attempt. That is exactly the situation where a weekend replication on your own paired checkpoints is worth more than any summary of the abstract.