HiSA makes attention sparse in the hidden dimension, not over tokens — and a partial dot product beats the full one
A static, parameter-free RoPE-derived rule computes ~29% of query–key products and beats dense attention's validation loss — with FA3 kernels and a ring-buffer KV cache.

HiSA computes only about a third of the query–key products, and training from scratch it reaches a lower validation loss than dense attention. The result, from EPFL and the Swiss Data Science Center, is a static, parameter-free sparsity rule derived from the geometry of rotary position embeddings (RoPE), with FlashAttention-3 kernels and a ring-buffer KV cache that shrinks memory during decoding.
What shipped
The premise is a question most sparse-attention work skips: does attention actually need the complete query–key dot product? Almost all prior sparsity is over tokens — fixed patterns, sliding windows, KV-cache eviction, or learned key routing. HiSA keeps every key in play but drops parts of the dot product itself.
RoPE gives the decomposition a natural structure. With head dimension d and P = d/2 frequency pairs, the attention score splits exactly into a sum over pairs, each rotated by an angle proportional to the relative distance Δ = i − j times its frequency θ_l = b^(−2l/d). High-frequency pairs complete many full rotations within a few tokens; low-frequency pairs barely rotate over the whole context. HiSA's rule: keep pair l only while its rotation (i−j)·θ_l stays below an angle budget Φ, set to 90° (a quarter period) by default. Φ = ∞ recovers dense attention.
Because the windows Δ_max,l = Φ/θ_l grow with l, the pattern is a clean prefix: nearby keys are compared on all pairs, farther keys only on the slow ones, and keys beyond the largest window — the "reach," 11.8K tokens at RoPE base 10⁴ — leave the softmax entirely. The pattern is set by positions alone: no learned parameter, no runtime router or selection step.
The authors train SmolLM-135M and SmolLM-360M architectures from scratch at 4K context, RoPE base 10⁴, Φ = 90°. The headline numbers (135M, one H100/H200):
| Attention | Kernel | Val. loss | Tokens/s (train) | Q–K products |
|---|---|---|---|---|
| Dense | PyTorch SDPA | 2.906 | 205k | 100% |
| Dense | FlashAttention-3 | 2.904 | 236k | 100% |
| HiSA | Triton (exact rule) | 2.880 | 119k | 29% |
| HiSA, block rule | Triton | 2.888 | 191k | 35% |
| Flash-HiSA | FA3 fork | 2.892 | 238k | 35% |
The best HiSA validation loss (2.880) beats dense (2.904/2.906) by ~0.024–0.026 while computing 29% of the products. Every budget from 45° to 360° beats dense attention at this scale, with 90° chosen as the default.
For inference, HiSA turns the rule into an actual KV-cache reduction rather than just skipped compute. Keys are stored per 16-feature chunk in ring buffers as long as each chunk's slowest-pair window; values live in a single ring as long as the reach. At 4K context and base 10⁴, the four key chunks hold 256 / 256 / 1408 / 4096 tokens instead of 4096 each. At 16K context (base 2·10⁴, batch 128), the KV cache drops from 45.0 GiB to 29.1 GiB and decoding runs at 13.0k vs 9.2k tokens/s against dense FlashAttention-3 (11.6k for dense through the same decode kernel).
The implementation is the part engineers can actually adopt. Flash-HiSA lives in a fork of FlashAttention, one commit on top of FlashAttention 3.0.1, as separate compile-time "Sparse" instantiations (head dim 64, bf16, Hopper). The forward pass slices Q·Kᵀ per 16-feature chunk and skips dropped chunks with straight-line GMMA sequences; the backward pass applies the same rule. A host-side table of dropped pairs per block distance lets the kernel derive skipped chunks without memory loads in the main loop. The block rule groups positions into 128-token blocks and is a slightly denser model than the exact rule — so training and inference must use the same one. The repo also provides a plain-PyTorch reference, Triton kernels, and a patch_llama integration for Hugging Face Llama models.
Why it matters
The counterintuitive result is the point: dropping computation doesn't just save work, it slightly improves the loss. The paper's interpretability analysis claims to explain why — dense attention itself learns to suppress the score parts HiSA drops, shaping its queries and keys to damp the high-frequency pairs at long range. If that holds, HiSA isn't an approximation of dense attention; it's a cheaper way to train toward the same behavior.

For people building long-context systems, the KV-cache reduction is the more immediately useful piece. The concurrent ATFlash and WavePrune methods skip compute but keep every key and value in memory; HiSA is among the few that shrink the cache itself, and it does so with a static rule that needs no calibration or per-query selection at runtime. The repo's HiSADecoder and HiSAKVCache are a reference loop to build on, and the code is Apache 2.0 (the FlashAttention fork keeps its BSD 3-Clause license).
A pretrained dense model can also be converted: switching a trained SmolLM-135M to HiSA without training costs loss (2.906 → 2.937), but continued training recovers it, matching dense continued training after 1.35B tokens. That means HiSA isn't locked to from-scratch training — an existing checkpoint can be migrated.
Caveats
This is a preprint, days old, with no independent replication, and the scale is small. Experiments stop at 360M parameters (a machine review of the paper adds a 20-tokens-per-parameter budget); the central claim — that a partial dot product beats the full one — needs confirmation at the sizes people actually deploy. The paper explicitly defers downstream-task benchmarking to future work, so there are no WikiText-103 or task-accuracy numbers to lean on.
A few of the authors' own choices deserve scrutiny. A machine review of the paper reports the 180° budget gives the best 2K-context validation loss (3.010, at 44.4% compute) yet 90° is the default, and the justification for that choice is thin. The long-context advantage may also shrink: the same review reports PG19 validation loss of 3.170 (HiSA) vs 3.205 (dense) at 16K — an absolute gain of 0.035, smaller than the 0.026 gain at 4K. And part of the reported decoding speedup comes from smaller query tiles in the kernel, not from the sparsity itself.
Training speed is a mixed story. The exact-rule Triton kernel trains slower than dense (119k vs 205k tokens/s); only Flash-HiSA matches dense speed (238k vs 236k), and the backward pass doesn't benefit from the sparsity as much as the forward. The kernels are also limited to head dim 64, causal attention, bf16, and Hopper GPUs — no padding, varlen, or paged KV. The interpretability metric's exact numbers weren't available in the sources I could read, so treat that finding as the authors' claim rather than an independently reproduced result.
Sources
- Paper abstract: https://arxiv.org/abs/2611.26363v1
- Full text (HTML): https://arxiv.org/html/2611.26363v1
- Official code and results table: https://github.com/ambroise-b/hisa
- Kernel documentation: https://raw.githubusercontent.com/ambroise-b/hisa/main/docs/kernels.md
- Usage and limits: https://raw.githubusercontent.com/ambroise-b/hisa/main/docs/usage.md
- Machine review of the paper (used for the PG19, 180°-budget, and training-budget figures): https://pith.science/paper/2611.26363