Richland Technology
Richland Technology
← Back to Blog
llmresearcharchitecturetransformerssm

Latent Lattice LMs: per-layer recurrent latents condition attention and MLP — a 406M model matches a 1.5×-larger Transformer, and identical perplexity hides a 10.1-point GSM8K gap

MIT CSAIL and Adobe's LLLM adds per-layer horizontal latent streams alongside the residual stream. The perplexity numbers are promising; the 10.1-point GSM8K decoupling is not yet verified.

Richland AI Desk·18 November 2026·5 min read
Latent Lattice LMs: per-layer recurrent latents condition attention and MLP — a 406M model matches a 1.5×-larger Transformer, and identical perplexity hides a 10.1-point GSM8K gap

MIT CSAIL and Adobe researchers have introduced the Latent Lattice Language Model (LLLM), an architecture that gives each layer its own recurrent "horizontal" latent stream conditioning attention, KV-cache writes, and MLP computation. After 20B tokens of pretraining, a 406M-parameter LLLM reaches 14.40 FineWeb-Edu validation perplexity — matching a 1.5× larger, 40-layer Transformer — yet two models with identical pretraining perplexity differ by 10.1 points on finetuned GSM8K.

What shipped

The paper (arXiv:2611.21608, submitted Nov 16) is by Shivam Duggal (MIT CSAIL, an Adobe summer intern), Zongze Wu, Richard Zhang, and Eli Shechtman (Adobe), and William T. Freeman and Antonio Torralba (MIT CSAIL). It proposes the Lattice of Latents: alongside the vertical residual stream that flows through depth, every Lattice layer maintains a horizontal latent r_t^ℓ, updated recurrently across timesteps (full text).

The horizontal latent is not a second sequence mixer. In hybrid Transformer–SSM models, the recurrent state is internal memory that replaces or augments attention. Here attention (or the backbone's native memory) stays the sequence mixer, and the latent is a companion to the residual stream: it conditions the attention queries, KV-cache writes, MLP update, and — at the top layer — output decoding. The update is an affine Mamba-style recurrence that admits an exact associative scan across token positions, so training stays token-parallel.

Three enhancements build on the base update:

  • Input injection feeds the token embedding directly into every layer's update, so the latent sees the current token without relying on the vertical stream to preserve it.
  • A depth-wise readout stream lets each layer's readout of its latent depend on the latents of layers below, adding nonlinear cross-layer computation.
  • Top-to-bottom feedback feeds the top layer's latent into the bottom layer's update at the next timestep. That breaks token parallelism, so the authors approximate it with token-parallel Jacobi passes — about 2.6× the FLOPs for three passes during training. At inference, the same model rolls out sequentially at base FLOPs.

The headline numbers, all after 20B FineWeb-Edu tokens at 2048-token context (Table 1):

  • A 406M LLLM with input injection + readout stream reaches 14.40 validation perplexity, versus 15.41 for a parameter-matched 406M Transformer — matching a 609M, 40-layer Transformer at 14.40. The Table 1 caption says the 400M LLLM "matches or outperforms the 600M Transformers," though on the CC out-of-distribution set the 600M models still lead (14.12 vs 14.02/13.88).
  • Adding the top-to-bottom refiner (three Jacobi sweeps) reaches 14.03 during training; the same weights rolled out sequentially at inference give 14.73 at 0.72 GFLOP/token, approximately the base Transformer's 0.71.
  • The Lattice composes with prior residual-stream enhancements: with hyper-connections and input residuals it reaches 13.61, versus 14.22 for the same Transformer without the Lattice and 15.41 for the plain base (abstract).
  • On recurrent backbones — Gated DeltaNet, Mamba-2, Triadic GDN, and GDN–Mamba-2 hybrids — it improves perplexity at every native state size tested, and Lattice-GDN and Lattice-Triadic-GDN beat pure baselines holding 4× larger native states. Downstream, the nine-task average improves in nearly every comparison across five state budgets and both zero- and five-shot settings; the Lattice leads in 29 of 30 paired comparisons, per aimodeling.com.

There is also a serving angle: the Lattice models use 22 layers where the matched-quality Transformer uses 40, so at the same attention width they keep a 45% smaller KV cache (project page).

Why it matters

The paper frames scaling as having three existing levers — more per-token computation (deeper or looped layers, latent reasoning steps), richer residual connections, and (in hybrids) bigger recurrent memory — and the Lattice as a fourth: the number of active computational states a model maintains. The horizontal latents are fixed-size per layer and add no KV-cache entries or token positions, so they are a cheap axis to add — and the results say they compose with the other three rather than substitute for them.

The most interesting claim is the perplexity/GSM8K decoupling. Base-40L and the 406M LLLM(908,256) have the same FineWeb-Edu perplexity (14.40) and near-identical zero-shot averages (47.36 vs 47.38), yet after finetuning on GSM8K-Aug they differ by 10.1 points (33.5 vs 43.6) (paper text). If that holds, the horizontal latents buy something perplexity does not measure — a structural difference in how the model reasons after finetuning, not just a better language-modeling loss. An independent write-up reports the same pattern at both scales: the Jacobi-refined LLLM beats the baseline Transformer by 12.08 points at 400M and 7.50 points at 600M (aimodeling.com).

For anyone who uses validation perplexity as a proxy for downstream quality when comparing architectures, that is the sentence to sit with.

Caveats

This is an unreviewed preprint, and the caveats are substantial:

  • No code or weights. The GitHub repo contains only the project page; the paper and code links on the project page read "coming soon." OpenTrain AI found no verified implementation and notes no benchmark numbers could be independently verified.
  • Small scale. Everything is 400–600M parameters, 10–30B tokens, 2048-token context. The paper limits its own conclusions to these configurations.
  • An internal inconsistency. The abstract and Section 4.2 report 13.61 for Lattice + hyper-connections + input residuals, but the Figure 3 caption says 13.76 and an appendix passage gives 13.62 — flagged independently by alphaXiv and aimodeling.com.
  • The Lattice is not the best single enhancement. Alone it reaches 14.47, slightly worse than the input-residual (x0) baseline at 14.34 (alphaXiv). Its value is compositional.
  • The GSM8K result rests on thin evidence. Each finetune uses a single seed, and the coupling used for the recurrent-backbone experiments was selected post hoc — the gains "should be read as those of the best coupling rather than of any single preselected one," per aimodeling.com.
  • Open questions from the community. The Hacker News thread asks whether the horizontal latent is just an EMA-style extension of Short-conv in linear attention, notes the paper never tests beyond 2k context or with needle-in-a-haystack, and questions how much a ~1.0 perplexity point means; one commenter compares it to Infini-attention (arXiv:2404.07143).

The architecture is new and cheap to evaluate — if the authors release the code, a 400M-scale reproduction on FineWeb-Edu is a few days of work. Until then, treat the perplexity numbers as promising and the GSM8K decoupling as unverified.

Sources

Start a Conversation

Ready to build something like this?

Tell us about your project and we'll respond with a clear plan within one business day.

Tell Us About Your Project

Takes 3 minutes. We read every submission personally.