acceptodds
Withdrawn from ICLR 2027

Sliced-Linear Attention: Beyond the Finite-Feature Bottleneck of Linear Attention

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

To address the quadratic bottleneck of Softmax attention, many efficient attention mechanisms rely on finite-dimensional summaries of the context and, in the causal setting, on fixed-dimensional recurrent states. We introduce Sliced-Linear attention, which multiplies a standard Linear-attention kernel by a Laplace interaction between learned one-dimensional query and key coordinates. This removes the finite-rank constraint of finite-feature Linear attention while retaining exact quasi-linear computation through sorting and scans. We show that, for any prescribed finite key set, a single Sliced-Linear head can approximate arbitrarily sharp associative retrieval, whereas finite-feature Linear attention requires to maintain fixed relative error as memory size grows. In the causal setting, exact retrieval of arbitrary -dimensional values by a continuous recurrent model requires state dimension . Controlled MQAR experiments show the corresponding qualitative capacity separation. On Augmented LRA and in 124M-parameter causal language modeling, Sliced-Linear consistently improves over matched Linear attention, with larger gains after continuation to 16k and 32k context. With , it outperforms Linear attention with across all tested text settings. Measured forward-and-backward runtime scales near-linearly with sequence length.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient Attention, Linear Attention, Long-Context Modeling, Transformers, Associative Retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1SRgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Sliced-Linear Attention addresses the limited associative retrieval capability of finite-feature linear attention by adding content-dependent interactions while keeping full-sequence complexity quasi-linear. The mathematical derivation of the finite-rank separation makes sense, as does the argument that the fixed-state bottleneck does not apply to Sliced-Linear because it retains a growing context. The evaluations are broad enough to show where Sliced-Linear may be useful, but there are limitations in the baselines and model scale that I have mentioned below.

Critical strengths
  • Simple and elegant formulation.
  • Shows how Sliced-Linear Attention can approximate sharp retrieval with a single feature, while linear attention needs a feature dimension that grows with memory size for a fixed relative error.
  • Evaluation shows improved performance over linear attention on MQAR, augmented LRA, causal language modeling, and long-context evaluation.
Critical weaknesses
  • Causal language-modeling baselines are limited. Including Mamba-2, RWKV-7, GDN-2, etc. would be useful. GLA is included in the benchmarks and achieves better language-modeling perplexity than Sliced-Linear, but is not included in the end-to-end wall-clock time analysis.
  • The model size is limited to 124M parameters. Evaluation at a larger scale, such as 512M–1B, would be informative, although assessing practical use would eventually require even larger models. If compute is a constraint, a lower-cost experiment could train the learned scalar projections for the Laplace kernel on top of a frozen open-source linear-attention model.
  • Ablations on positional encoding schemes, data scale, etc. are missing. Since the models already use positional information, I wonder how much additional positional awareness comes from the learned scalar projections and Laplace kernel.
Questions for authors

No specific question apart from the concerns mentioned in the weaknesses.

Reviewer_zAeKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Sliced-Linear attention augments the linear attention kernel with a learned one-dimensional Laplace coupling. This allows for full rank attention maps without the quadratic cost of standard softmax attention. The computational cost is quasi-linear, which is achieved via sorting and scans, compared to the linear cost that linear attention achieves via chunkwise parallel forms. Experiments cover MQAR, long range arena and small scale language models. Sliced-Linear consistently improves over matched Linear baselines on text, although Softmax and GLA achieve better language-modeling performance.

Critical strengths

The paper presents an interesting perspective on full-rank linear attention methods with sub quadratic implementation.

Critical weaknesses
  • State of the art linear attention variants typically use delta-corrected writes (DeltaNet, Gated DeltaNet, KDA). Not only does the paper not compare to those, but it also does not discuss whether the Laplace interaction can be extended to the models. It furthermore does not show sufficient evidence of improved language modeling performance at scale.
  • The practical quality-speed trade-off is unclear. GLA achieves better language-modeling results, but is omitted from the runtime comparison. Also, the logarithmic scale in Figure 1 makes it hard to actually assess the overhead of sliced-linear over linear.
Questions for authors
  • The authors compare their sliced-linear model against softmax and linear attention. How does their method compare to delta-corrected versions like DeltaNet, Gated DeltaNet, and KDA? And can it even be extended to these settings, at least approximately? F.e., could the Laplace coupling be extended (approximately) to DeltaNet?
  • Could the authors clarify why they do not extend their Laplace interaction term to GLA? Given that gates are known to be crucial for linear attention language models (see also table 3 in the paper), this seems like a clear extension which should be possible in the presented framework.
  • Why do the authors continue pre-training on PG-19 and then evaluate exclusively in-distribution (table 3)? Could they additionally report wikitext-2 perplexity in Table 3?
Reviewer_tg3nOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper multiplies the linear attention kernel by a Laplace kernel factor defined on learnable scalar coordinates, in order to address the limited rank of linear attention. The authors analyze the properties of the attention matrix after adding this term, provide a complexity analysis, and design the corresponding algorithms. They theoretically show that, for associative retrieval, a single head with R=1R=1can achieve arbitrarily precise retrieval, whereas the feature dimension of linear attention must grow linearly with the memory size. The authors conduct experiments on MQAR, Augmented LRA, and text modeling, where the proposed method shows consistent improvements over linear attention with the same configuration.

Critical strengths
  • The idea is clear, and the paper gives a clean and precise account of the properties and complexity of the proposed method.
  • The experimental setting is rigorous, and the results show consistent improvements over linear attention.
Critical weaknesses
  • The efficiency evaluation is not yet complete: memory consumption is not empirically measured.
  • In the text modeling experiments, the proposed method is weaker than GLA in all settings; in theory, its complexity is also higher than the linear complexity of GLA.
Questions for authors
  • Could the authors report empirical memory measurements for both training and inference, as well as the time cost of inference?
  • The paper mentions that a learnable λλ is supported, but λλ is fixed to 0.4 in the experiments. Could the authors report results with different values of λλ or with a learnable λλ, and analyze their effects on optimization and performance?
  • Could the authors explain why the method underperforms GLA, which has linear complexity, on text modeling? What are the authors' views on applying the Laplace factor on top of GLA or similar improved linear attention variants?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.