Activation steering as memory: facts fail, dislikes invert, and the "leanings transfer" claim failed its own pre-registration
A pre-registered study on Qwen3.5-2B/9B finds a gated activation memory is selective but stores facts as a word to say — and recalls a stored dislike as a like unless the opposite is subtracted at write time.
A pre-registered study on Qwen3.5-2B and Qwen3.5-9B tests whether activation steering can carry user facts and preferences across sessions. The memory is selective — about 98% of unrelated replies are unchanged — but the content mostly does not survive: clean fact use tops out at 15% (against 96–98% in context), and a stored dislike is recalled as a like unless the opposite experience is subtracted when writing.
What shipped
The paper, by Vedant Jumle (TU Delft), builds a training-free "activation memory" beside a frozen LLM. In session 1, an experience ("I can't stand jazz music.") is stored as a shift — the change it causes in late-layer residual-stream states, pooled over the follow-up turns — filed under a whitened key for the conversation's gist. In a later session the shift is added back (h ← h + α·gate·M·q) only when the current gist matches a stored key above a calibrated threshold. The store is the same linear associative memory that ROME and MEMIT write into MLP weights, kept in activations rather than weights (paper, §2).
The design is deliberately minimal: two Qwen3.5 sizes in fp32 with thinking off, shifts applied at three late layers (20/21/23 of 24 on the 2B; 27/28/31 of 32 on the 9B), doses α ∈ {0.5, 1, 2, 3} per layer, and controls for a random direction, a swapped memory, a placebo experience, and the gate removed (PREREG.md). The claims, decision rules, exclusion rule and statistics were written and committed before the run (2026-10-07), and analyze.py implements exactly those rules.
The pre-registered verdicts, per the results handover:
- Selectivity holds. About 98% of unrelated answers are token-for-token identical to the no-memory baseline; remove the gate and only 6% (2B) and 10% (9B) stay unchanged.
- Leanings transfer — the claim failed its pre-registered test. At α=1 the judged gain is +0.20 [0.04, 0.38] on the 2B and +0.11 [0.004, 0.24] on the 9B (on a −1..+1 scale), but at α=2 the confidence interval includes zero because judged incoherence jumps to 61% (2B) and 41% (9B), and the random-direction control's paired CI overlaps. The paper reports this as a failed prediction, not a win.
- Facts almost never come out as clean use. Clean judged use is ≤ 0.15 at every strength, against 0.96–0.98 with the fact in context; at α ≥ 2, 85–94% of answers mention the fact without using it cleanly.
- No premises. Balanced yes/no accuracy never improves on either model; on the 2B, facts at α=2 lowered it by −0.15 (CI excludes 0), which the pre-registered rule counts as a failure.
- The reference effect holds. A shift is
state(with experience) − state(reference), and what you subtract decides what survives. Subtracting the opposite ("I'm a huge jazz fan.") keeps the direction: dislike stays a dislike (+0.23/+0.25). Subtracting a centroid of same-frame alternatives, or nothing, keeps the concept: the stored shift points toward jazz, so a dislike is recalled as a like (−0.38/−0.37, wrong way on every item; −0.25/−0.31 for "without"). - A fact shift is a word, and only late. The stored shift ranks the fact's word in the top 10 of the vocabulary only at relative depth ≥ 0.83 (2B) and ≥ 0.875 (9B); at half depth the median rank is about 70,000. Injected late it loops ("Petra Petra Petra…"), injected earlier it fades or garbles ("My name is Omar"), and the model sometimes claims the memory as its own ("Teal is my favourite colour!").
All 34,200 answers were re-judged by a cross-family panel — gpt-oss-20B, Gemma-4-12B, Llama-3.1-8B and Qwen3.6-35B-A3B — replacing the planned human audit. C3 (facts) and C5 (reference effect) hold under every competent judge; Llama-3.1-8B fails a sanity check and is excluded. C2 is judge-sensitive: it fails under every single judge and holds only under majority votes (HANDOVER.md §4).
Why it matters
In context, attention recomputes an experience's effect for every new question; a stored shift is one fixed average of that effect, so only the part of the experience that is the same for every later question can survive (paper, §1). Facts are question-dependent, so their averaged shift collapses to "say the word". Leanings are question-independent, so they survive — but only in a narrow strength window, and only if written contrastively.
That is a concrete mechanism for the procedural/declarative split that Neural Procedural Memory (Zhao et al., arXiv:2606.29824) assumed when it proposed storing agent skills as retrieved steering vectors; the Seahorse paper reads NPM as explicitly leaving declarative content — facts and preferences about the user — to text retrieval (paper, §1). The paper claims this is the first direct test of that assumption, and its results support it: activation steering suits dispositions, not content.
For anyone building personalization or agent memory, three practical consequences:
- Steering is a push, not a store. If you want a model to remember that the user's dog is named Pepper, a steering vector is the wrong container; that needs weights or retrieval. The author's own framing: the steer channel "is a push, not a store" (HANDOVER.md).
- The write reference decides the sign. A naive write — subtracting nothing, or a centroid of alternatives — inverts a dislike into a like, because negation is weakly encoded. Contrastive writes against the opposite experience are the only construction tested that preserves direction.
- Dose matters and earlier wins were partly incoherent. α≈1 per layer is the usable window; α=2 on three layers is an overdose. The handover notes earlier word-count gains (e.g. a 94% "vegetarian" hit in a prior experiment) included a lot of incoherent text that the coherence-aware judge now catches.
The methodology is also worth copying: pre-registered claims with fixed decision rules, per-item reporting (leanings were strong for early_riser, prefers_quiet, vegan, vegetarian; nothing for celiac, lactose_intolerant, has_young_kids), and a cross-family judge panel instead of a single judge.
Caveats
- Single-author preprint, not peer-reviewed; no independent replication or external commentary is cited.
- All numbers are author-reported. The primary judge is the larger model under test (Qwen3.5-9B judging both sizes); the cross-family panel replaces, but is not, a human audit.
- The leanings claim (C2) failed its own pre-registered test and is judge-sensitive — treat any "leanings transfer" framing with suspicion.
- C4 (no premises) is borderline: it failed on the 2B only as a rule-driven technicality.
- A follow-up documented in the repo (not the paper) finds facts do survive when a stored slice of the model's own attention memory (K,V) is reinstated instead of a steering shift (0.86–0.97 clean use) — but it is handover documentation, not a published result.
Code and full pre-registration: github.com/vedant-jumle/Seahorse.