- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors researched about the late-stage growth of maximum attention logits, which appears when training Transformers with the Muon optimizer, and and find out the main reason is that there exists numerical errors in low-precision FlashAttention (FA), instead of Muon itself. The authors did ablation study and show that BF16 FA has a two-stage logit-growth pattern including a late excess logit growth. Then further showed that the resulting gradient errors are persistently rank-one and has same sign. They provide a theoretical model proving this phenomenon under rank-one gradient errors, and propose a solution to mitigate the precision dependent logit growth.
Critical strengths
- This is a solid paper on an interesting problem: what is the reason for the excess attention-logit growth observed in Muon training? As Muon has became the most widely used optimiser for current industry-size llm, the problem itself is important. The authors took reasonable ablation study to point out the reason, and provide theoretical analysis. I believe this work can deepen our understanding of large scale llm training with muon optimiser.
- The presentation of the idea is clear and easy to follow: the problem to learn, the reason for the phenomenon, and the solution to the problem. I value this more than ever in this autoresearch era.
Critical weaknesses
Theorem 1 depends on 3 strong assumptions, while the authors showed empirically that the numerical errors are rank-one, other assumptions are somehow strong. Can authors explain why they hold conceptually or with a small scale experiment?
Questions for authors
- As we can view adamw and muon as using preconditioners with SGD [1], can authors explain theoretically whether the whole framework can generalise to adamw? If so, I think this can be a good extension of the paper.
- The author mainly report validation loss, while this is convincing, what about the downstream benchmark performance?
[1] Wang, Shaobo, Xuan Ouyang, Tianyi Xu, Yuzheng Hu, Jialin Liu, Guo Chen, Tianyu Zhang et al. "OPUS: towards efficient and principled data selection in large language model pre-training in every iteration." arXiv preprint arXiv:2602.05400 (2026).
3 comments
Q1: As we can view adamw and muon as using preconditioners with SGD [1], can authors explain theoretically whether the whole framework can generalise to adamw? If so, I think this can be a good extension of the paper.
We thank the reviewer for this suggestion and for bringing the relevant work [1] to our attention. We have added a discussion of this work to the Related Work section of the revised manuscript (lines 715–716). Our findings suggest that the absence of persistent signed gradient-error alignment under AdamW may help explain its weaker excess attention-logit growth relative to Muon.
-
Weaker excess logit growth under AdamW than Muon. In our controlled AdamW experiment (FA-16 with RoPE) at its best learning rate, the maximum attention logit does not exhibit the two-stage growth pattern observed with Muon (Appendix B.1 of the revised manuscript). This observation is consistent with the Kimi K2 technical report [2]: "an issue that occurs more frequently with Muon but less with AdamW." (Adam exhibits less logit growth.)
-
Testing the proposed mechanism under AdamW. Our additional experiments show that the structured gradient-error pattern underlying excess logit growth with Muon is not observed under AdamW in the FA-16/RoPE setting (Appendix B.1). Specifically, the gradient discrepancy between FA-16 and FA-32 under AdamW exhibits weak alignment with the rank-one directions defined in Eq. (2), and its signed projection coefficients do not maintain consistent signs throughout training. Thus, AdamW lacks the persistent signed alignment that enables coherent error accumulation in our proposed mechanism for excess logit growth.
-
Implications across optimizers. Although low-rank numerical errors also arise under AdamW, they exhibit weaker alignment with the identified logit-increasing directions and less consistent signed projections than under Muon. These differences may help explain why excess attention-logit growth is less pronounced under AdamW, suggesting that our mechanism also sheds light on the differences in logit dynamics across optimizers.
We therefore view extending our theoretical framework to AdamW as a promising direction for future work.
Q2: The author mainly report validation loss, while this is convincing, what about the downstream benchmark performance?
We agree that downstream benchmark evaluations would provide complementary evidence of the practical impact of our methods. Our experiments already show that the proposed precision interventions substantially suppress excess logit growth while maintaining comparable or improved pre-training validation loss (Tables 1 and 2). Moreover, pre-training validation loss is a widely used proxy for language-model performance, with prior work establishing empirical relationships between validation perplexity and downstream task performance (Gadre et al. [3]). Nevertheless, direct downstream evaluations would provide a more complete assessment and remain an important direction for future work.
References
[1] Wang, Shaobo, Xuan Ouyang, Tianyi Xu, Yuzheng Hu, Jialin Liu, Guo Chen, Tianyu Zhang et al. "OPUS: towards efficient and principled data selection in large language model pre-training in every iteration." arXiv preprint arXiv:2602.05400 (2026).
[2] Team, Kimi, et al. "Kimi k2: Open agentic intelligence." arXiv preprint arXiv:2507.20534 (2025).
[3] Gadre, Samir Yitzhak, et al. "Language models scale reliably with over-training and on downstream tasks." International Conference on Learning Representations. Vol. 2025. 2025.
Dear Reviewer,
As the discussion period is approaching its end, with less than 2 days remaining, we would like to ensure that we have fully addressed all of your concerns. If you have any additional comments or suggestions, please feel free to share them. We are happy to clarify any remaining points and further improve our work.
Thank you so much for your time and effort in reviewing our work.
We thank the reviewer for your positive review and suggestions.
W1: Practical Relevance of Assumptions
All three assumptions in Theorem 1 are rooted in the empirical structures observed in our LLM experiments, rather than being introduced solely for mathematical convenience. We provide conceptual explanations in section 4 and empirical supports for these assumptions in Appendix D.
Assumption 1 is empirically supported by the near-orthogonality of centered token embeddings, as shown in Appendix D, Figure 14. Specifically, we subtract the mean embedding vector from each token embedding and compute the pairwise angles between the resulting centered embeddings. The angles are tightly concentrated around 90∘, with a mean of 89.46∘. This observation supports the assumption that token embeddings share a common component, while their token-specific components exhibit approximately orthogonal geometry.
Assumption 2 is empirically supported by the high projection isotropy scores at initialization, as reported in Appendix D. We evaluate this property using the projection isotropy score of W⊤W, defined as
{\mathrm{rank}(W^\top W)\|W^\top W\|_F^2},$$ which equals 1 for an exact scaled projection. We obtain scores of $0.9206\pm0.0033$ for $W_Q$ and $0.9222\pm0.0044$ for $W_K$, supporting the approximate scaled-projection structure. This assumption characterizes an initialization that is approximately isometric on the query-type and context-type feature subspaces. **Assumption 3 is empirically supported by the persistent sign consistency of $\bar{\gamma}$, as shown in Section 3.2, Figure 3(c).** Specifically, we define the signed projections of the gradient errors onto the identified rank-one directions as $$\gamma_Q=\langle\Delta G_Q,\widehat{k}\overline{x}^{\top}\rangle_F,\qquad \gamma_K=\langle\Delta G_K,\widehat{q}\overline{x}^{\top}\rangle_F,$$ and $\bar{\gamma}$ denotes their exponential moving averages (EMAs). The persistent sign consistency of $\bar{\gamma}$ suggests that numerical errors consistently push the gradients along the identified rank-one directions throughout training. Moreover, the identified directions remain stable after an initial transient (Figure 4) and are approximately orthogonal to the task-gradient directions (Figure 16). Together, these observations support the assumption that the dominant rank-one error components maintain approximately fixed directions and can accumulate coherently throughout training.