- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Sliced-Linear Attention addresses the limited associative retrieval capability of finite-feature linear attention by adding content-dependent interactions while keeping full-sequence complexity quasi-linear. The mathematical derivation of the finite-rank separation makes sense, as does the argument that the fixed-state bottleneck does not apply to Sliced-Linear because it retains a growing context. The evaluations are broad enough to show where Sliced-Linear may be useful, but there are limitations in the baselines and model scale that I have mentioned below.
Critical strengths
- Simple and elegant formulation.
- Shows how Sliced-Linear Attention can approximate sharp retrieval with a single feature, while linear attention needs a feature dimension that grows with memory size for a fixed relative error.
- Evaluation shows improved performance over linear attention on MQAR, augmented LRA, causal language modeling, and long-context evaluation.
Critical weaknesses
- Causal language-modeling baselines are limited. Including Mamba-2, RWKV-7, GDN-2, etc. would be useful. GLA is included in the benchmarks and achieves better language-modeling perplexity than Sliced-Linear, but is not included in the end-to-end wall-clock time analysis.
- The model size is limited to 124M parameters. Evaluation at a larger scale, such as 512M–1B, would be informative, although assessing practical use would eventually require even larger models. If compute is a constraint, a lower-cost experiment could train the learned scalar projections for the Laplace kernel on top of a frozen open-source linear-attention model.
- Ablations on positional encoding schemes, data scale, etc. are missing. Since the models already use positional information, I wonder how much additional positional awareness comes from the learned scalar projections and Laplace kernel.
Questions for authors
No specific question apart from the concerns mentioned in the weaknesses.