ParaTTT parallelizes nonlinear TTT memory layers: 32× faster than TTT-MLP at matched error, 4.9× faster 340M training, exact online recurrence as the fixed point

MIT CSAIL, Liquid AI, and MPI-IS researchers have posted ParaTTT (PTTT), an algorithm that parallelizes nonlinear test-time-training (TTT) memory layers by iteratively refining a whole-sequence estimate of the inner gradient trajectory, with the exact online gradient-descent recurrence as its fixed point (arXiv:2610.38255). At 4K context, two refinement rounds match TTT-MLP's block approximation at 32× less time; a full 340M-parameter model trains 4.9× faster, and the resulting 340M/1.3B models match or exceed Gated DeltaNet and Mamba-2 on language modeling and recall.

What shipped

Recurrent language models (Mamba-2, Gated DeltaNet, gated linear attention) process long sequences in linear time with a fixed-size state, and they admit a chunkwise-parallel form: tokens inside a chunk are processed in parallel with dense matmuls while only the chunk-boundary state is propagated sequentially. The catch: their memory is a single linear matrix updated by a delta rule or decay — expressive enough for associative recall, but unable to adapt its own features.

TTT layers (Sun et al.) replace that matrix with the weights of a small neural network trained online by gradient descent on a reconstruction loss. The memory can now learn its features as well as its readout, but each token's update depends on weights changed by every earlier token, so the recurrence is sequential. Existing implementations (TTT-MLP, Titans) recover parallelism by grouping tokens into blocks and evaluating all gradients in a block at the block-entry weights. That is a fidelity–speed dial: smaller blocks are more faithful but need more sequential steps, and the result only approximates token-by-token learning.

ParaTTT replaces the block-size dial with a refinement-count dial. Instead of freezing weights within blocks, it maintains an estimate of the hidden-layer gradients for the whole sequence and improves that estimate in rounds. Each round is three chunkwise-parallel passes:

1. a forward pass that computes the feature trajectory implied by the current gradient estimate;
2. an exact gated-delta-rule pass that computes the readout trajectory and the loss gradients with respect to the features;
3. a backward pass that propagates those gradients through the hidden layers into a new estimate.

Each pass is built from existing chunkwise-parallel recurrences — Gated DeltaNet, Mamba-2, or gated linear attention kernels — so the implementation reuses machinery linear models already have. The exact online gradient-descent recurrence is the iteration's fixed point; finitely many rounds approximate it. The paper proves geometric convergence for a two-layer variant under contraction conditions, and in practice two rounds (N=2, with a damping factor λ=0.8) suffice. Decoding needs no refinement at all: each new token is one exact gradient update from the cached weights, so N only affects prompt prefill.

The headline numbers, all relative to TTT-MLP: at 4K context, two ParaTTT refinements match the output error of TTT-MLP's block approximation (block size ~5) in 32× less time — a trajectory-evaluation comparison on a random inner MLP (d=64, eight heads) measured against exact online gradient descent. The paper reports full 340M-model training on eight AMD MI325X GPUs at 4K sequence length at 37.9k tokens/s versus TTT-MLP's 7.8k tokens/s, a 4.9× speedup.

The quality payoff is on recall. At 340M and 1.3B parameters, models trained on SlimPajama match or exceed Gated DeltaNet and Mamba-2 on language-modeling perplexity and zero-shot accuracy, and beat both on the recall-heavy extraction tasks SWDE and FDA. On SWDE, the paper reports ParaTTT with N=2 at 41.9% versus 32.5% for Gated DeltaNet and 25.8% for Mamba-2; a same-size Transformer still leads at 54.6%.

The repository (github.com/ntumm120/paraTTT) is a placeholder — the README reads "Code to come for ParaTTT." — with no code or license published yet.

Why it matters

Nonlinear memory was the expressivity play for recurrent models, but it was impractical: TTT-MLP's block-wise execution ran several times slower than chunkwise-parallel linear recurrences and only approximated the exact online recurrence. ParaTTT evaluates arbitrarily deep nonlinear memories with chunkwise-parallel kernels while keeping the exact token-by-token recurrence as the target — the fixed point of the refinement, and the exact behavior at decode time. That changes the engineering trade: instead of choosing between a linear memory you can parallelize and a nonlinear memory you must step through sequentially, you pay a constant factor — a few refinement rounds — for the nonlinear memory.

The recall numbers are the concrete reason to care. Linear recurrences underperform attention precisely where recall matters: tasks such as SWDE and FDA require remembering specific values across long contexts, and a fixed matrix memory cannot. A memory that adapts its features closes part of that gap, which is why the authors report better recall than both linear baselines. The honest caveat: attention still wins on SWDE, so this narrows the gap rather than closing it.

Also worth noting for anyone building on this: the speedup is expressiveness-for-parallelism, not a raw win over linear recurrences. At N=2, the paper reports, a ParaTTT layer costs 4.7× a Gated DeltaNet layer, uses 2.5× the peak memory, and runs at roughly one-third the throughput of the linear baselines. The 32×/4.9× figures are speedups over TTT-MLP — the previous way to get nonlinear memory — not over Mamba-2.

Caveats

• Preprint, author-reported, and not independently reproduced. The repo has no code yet, so none of the numbers can be checked by running anything.
• The 32× figure is a matched-error trajectory-evaluation result versus TTT-MLP's block approximation at block size ~5 on a single random inner MLP, not a full-model speedup; per-layer ParaTTT remains ~4.7× costlier than Gated DeltaNet.
• The geometric-convergence proof covers only a simplified two-layer variant (soft clipping, uncentered readout, linear-attention hidden update). The deployed models use a hard unit-ball projection, a hidden-layer delta rule, and centering that the proof does not cover; convergence for those is empirical.
• The finer numbers (SWDE, throughput, layer cost) come from the paper's body as summarized in alphaXiv's page-cited overview; the arXiv HTML full text truncates before the results tables, so I could not read the tables directly.
• Scale is 340M/1.3B; nothing here says the trade holds at frontier scale.

Sources

• ParaTTT abstract — https://arxiv.org/abs/2610.38255v1
• ParaTTT full text (HTML) — https://arxiv.org/html/2610.38255v1
• alphaXiv page-cited overview — https://www.alphaxiv.org/abs/2610.38255
• Repository — https://github.com/ntumm120/paraTTT
• TTT / TTT-MLP (Sun et al.) — https://arxiv.org/abs/2407.04620
• Gated DeltaNet (Yang et al.) — https://arxiv.org/abs/2412.06464
• Mamba-2 (Dao & Gu) — https://arxiv.org/abs/2405.21060
• DeltaNet (Schlag et al.) — https://arxiv.org/abs/2102.11174
• Gated Linear Attention (Yang et al.) — https://arxiv.org/abs/2312.06635
• Titans (Behrouz et al.) — https://arxiv.org/abs/2501.00663