- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper attempts to revisit the previously established relationship between massive activations and attention sinks. The authors argue that the properties associated with attention sinks are not limited to massive activations, but instead involve the distribution of singular values along the principal component directions of hidden states. Their causal intervention experiments largely support this argument.
Critical strengths
The paper provides an alternative explanation for the previously established relationship between attention sinks and massive activations. Indeed, the conventional explanatory paradigm underlying massive activations, which emphasizes individual hidden-state channels, appears rather outdated. In contrast, shifting the focus toward feature directions or principal component directions represents a more advanced perspective.
Critical weaknesses
-
The paper makes rather limited scientific contributions. Several hypotheses have already been proposed to explain the formation of attention sinks. Most of them share a common underlying idea: Transformer computations produce relatively stable constant terms that facilitate the emergence of attention sinks. Such constant terms may originate from massive activations (https://arxiv.org/abs/2402.17762), bias terms associated with attention parameter matrices (https://aclanthology.org/2026.acl-short.9), or other sources.
-
As discussed below, the present paper makes only a limited improvement within this existing framework and does not substantially expand our scientific understanding.
-
The contribution of this paper may even be conceptually at odds with the massive activation explanation, although the authors argue in Section 4.1 that their explanation based on the hidden-state feature spectrum is more accurate than the massive activation account.
-
However, one can argue that the hidden-state feature spectrum and the explicit hidden-state representation are related through a linear transformation. Consequently, the distinction between them may not be fundamental, and it is difficult to establish whether attributing the phenomenon to the feature spectrum constitutes a genuinely new mechanism. Furthermore, the overall argumentation of this paper closely resembles that of earlier studies on massive activations.
-
Admittedly, there are clear benefits to moving beyond the channel-wise perspective of massive activations and instead adopting a spectral perspective based on eigenvalues and a basis of feature vectors. After all, there is no reason to assume that Transformer computations must be understood in a basis consisting of the original coordinate axes. However, this intuition is not original to the present paper. Relevant prior work includes:
- Spectral Filters, Dark Signals, and Attention Sinks. ACL 2024.
- Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin. ICLR 2026.
-
A more scientifically valuable direction would be to explain why the observed spectral structure is preserved through attention-matrix computations. In particular, why should the eigenvectors of attention matrices align with the feature directions of hidden states? Investigating the origin and preservation of this alignment could lead to a broader and more interesting discussion of the underlying mechanism.
-
-
Methodological concerns. Why must the experimental framework be restricted to models in which attention sinks emerge only after more than one Transformer layer, with their onset varying depending on the prompt? One can readily imagine that even when attention sinks emerge in the very first Transformer layer, there is still a preceding hidden-state representation, directly connected to the token embeddings, on which the proposed experiments could be performed. Therefore, I find this restriction in the experimental design unnecessary and somewhat redundant.
-
The manuscript appears to be AI-generated with insufficient manual refinement, and inadequate attention has been paid to human readability. The paper retains numerous characteristics commonly associated with AI-generated scientific manuscripts that make it unnecessarily difficult to read. Substantial revisions throughout the manuscript are needed. For example:
-
Very poor presentation of experimental results. Some numerical results are repeatedly interspersed with the main prose, often in a format that is itself rather unconventional for major AI conferences, such as AP .857 → .913 (∆AP = +.056, [.028, .087]). This presentation substantially distracts readers from the main scientific arguments, making the entire manuscript feel more like an appendix than a coherent research paper. Moreover, this way of presenting numerical results is rather outdated. A more conventional approach would be to organize these results into figures and tables, separating numerical details from the main logical narrative.
-
Even the few figures included in the paper appear to be AI-generated, or at least exhibit many of the shortcomings commonly found in AI-generated scientific visualizations.
-
Excessive use of mathematical notation and equations. Some theoretical details and mathematical derivations could be moved to the appendix, while the main text should focus on the logical arguments and more intuitive explanations. In my view, the mathematical intuition underlying this paper is not particularly complicated, but the authors have presented it in an unnecessarily convoluted manner.
-
Questions for authors
See weaknesses.