- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper attributes the persistence of massive activations to suppressed reading alongside open writing. It studies weight Grams, activation second moments, orthogonal or frozen training interventions, an FFN quadratic approximation, checkpoint timing, and local AdamW updates. MA coordinates have lower relative read-side weight energy while still receiving updates, and constraining one read pathway does not eliminate MAs.
Critical strengths
- Decouple structural weights, input availability, actual update energy, and the sign of updates to avoid conflating non-zero writes with accumulation in the same direction.
- Parameterizing via Cayley orthogonality preserves learning capacity more effectively than simple freezing; with PPL shifting only from 14.21 to 14.24, this represents a structurally meaningful intervention.
Critical weaknesses
This paper highlights an intriguing phenomenon: "massive activation" arises from an asymmetry between the null spaces of the input and output projections—an empirical observation that has largely been overlooked. This, however, also points to the paper's limitations.
In essence, the paper provides an empirical demonstration that these "massive values" are not read out by or . Other claims—such as the assertion that massive activation represents an optimization fixed point—lack rigorous analysis. Consequently, I find the paper to be somewhat lacking in depth.
Questions for authors
See weakness
4 comments
The author’s response fails to explain why "massive activation" serves as a fixed point for the optimizer, and the experiment itself appears potentially susceptible to the fallacy of reverse causality; consequently, my concerns remain unaddressed.
Your comment suggests that you are confused about the primary question our paper investigates. In this paper, we ask whether MAs constitute a fixed point of the optimization (not how that fixed point is reached), i.e., does optimization actively maintain the read–write asymmetry and massive activations once they emerge? Our Hessian and gradient analyses provide strong evidence supporting that the optimizer actively maintains this fixed point, once reached.
Your point that we do not explain "why massive activation serves as a fixed point", concerns their origin. Our paper addresses their persistence. Works such as [1, 2] link MAs to attention sinks, while [3] proposes a mechanism for their emergence. However, we believe these works do not fully explain why training favors such configurations. Why MAs serve as a fixed point remains an open research question, and we welcome concrete experimental suggestions to investigate it.
[1] Sun, M., Chen, X., Kolter, J. Z., & Liu, Z. (2024). Massive activations in large language models. In First Conference on Language Modeling. https://openreview.net/forum?id=F7aAhfitX6
[2] Sun, S., Canziani, A., Lecun, Y. & Zhu, J.. (2026). Anatomy of Massive Activations and Attention Sinks. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:116443-116463 Available from https://proceedings.mlr.press/v306/sun26d.html.
[3] Sun, S., Canziani, A., LeCun, Y., & Zhu, J. (2026). The spike, the sparse and the sink: Anatomy of massive activations and attention sinks. arXiv preprint arXiv:2603.05498.
We hope our response has addressed your concerns and would appreciate your reconsideration of the score in light of these clarifications.
We respectfully disagree that the paper's contribution is limited to observing that massive coordinates are weakly read by input projections. The paper also investigates this asymmetry through training dynamics, structural interventions, and local optimization analysis.
We point to Section 6 in the paper that discusses optimization interplay with read-write asymmetry. Our directional-Hessian analysis finds higher curvature along massive-coordinate directions in read-side matrices, whereas the corresponding write-side directions exhibit lower curvature. This identifies a local loss geometry in which the read-side configuration are being held tightly. Complementing this, our AdamW update analysis finds positive first-order pressure on massive activation magnitude across layers, driven primarily by the optimizer’s accumulated moments. Combined, these results indicate that the AdamW optimizer deliberately maintain the read-blindness and the massive activations during the training.
Further, we also conduct intervention analysis in Section 4.2 where we constrains read-side matrices through frozen orthonormal initialization or Cayley reparameterization. These interventions alter massive activation magnitudes and reveal compensation through other pathways. Thus, we tests the consequences of restricting the proposed mechanism, rather than relying solely on correlations between projection geometry and activations.
Thus, the characterization that our optimization interpretation lacks analysis overlooks these existing experiments. We would appreciate clarification of which specific inference the reviewer considers unsupported.