acceptodds
Under review as a conference paper at ICLR 2027

Demystifying Maximum Attention-Logit Dynamics in Muon Training

Abstract

Muon has been widely adopted in language-model training due to its superior efficiency, yet the origin of its excess late-stage attention-logit growth remains unclear. Our work systematically investigates the maximum attention-logit dynamics in Muon training. First, by comparing different attention implementations and numerical precisions, we show that excess late-stage growth appears only with FlashAttention (FA) in Brain Floating Point 16 (BF16), indicating that it is not intrinsic to the optimizer design but originates from numerical errors. We further find that, after an early transient, the resulting gradient errors become persistently rank-one, with stable directions and consistent signs, causing them to accumulate coherently rather than cancel and producing structured high-logit heads. Second, motivated by this structure, we analyze a representative retrieval task with additive rank-one gradient errors and prove that the precision-induced logit gap is negligible in the early stage but becomes dominant later, directly linking the observed errors to the two-stage logit growth. Finally, we identify the FA intermediate quantities most responsible for this growth and mitigate it by selectively increasing their precision. Experiments show that these interventions provide a new trade-off between training performance and logit magnitude relative to existing methods.

Primary area:
optimization
Keywords:
Large Language Models, FlashAttention, Muon, Low-Precision Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9GduOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors researched about the late-stage growth of maximum attention logits, which appears when training Transformers with the Muon optimizer, and and find out the main reason is that there exists numerical errors in low-precision FlashAttention (FA), instead of Muon itself. The authors did ablation study and show that BF16 FA has a two-stage logit-growth pattern including a late excess logit growth. Then further showed that the resulting gradient errors are persistently rank-one and has same sign. They provide a theoretical model proving this phenomenon under rank-one gradient errors, and propose a solution to mitigate the precision dependent logit growth.

Critical strengths
  1. This is a solid paper on an interesting problem: what is the reason for the excess attention-logit growth observed in Muon training? As Muon has became the most widely used optimiser for current industry-size llm, the problem itself is important. The authors took reasonable ablation study to point out the reason, and provide theoretical analysis. I believe this work can deepen our understanding of large scale llm training with muon optimiser.
  2. The presentation of the idea is clear and easy to follow: the problem to learn, the reason for the phenomenon, and the solution to the problem. I value this more than ever in this autoresearch era.
Critical weaknesses

Theorem 1 depends on 3 strong assumptions, while the authors showed empirically that the numerical errors are rank-one, other assumptions are somehow strong. Can authors explain why they hold conceptually or with a small scale experiment?

Questions for authors
  1. As we can view adamw and muon as using preconditioners with SGD [1], can authors explain theoretically whether the whole framework can generalise to adamw? If so, I think this can be a good extension of the paper.
  2. The author mainly report validation loss, while this is convincing, what about the downstream benchmark performance?

[1] Wang, Shaobo, Xuan Ouyang, Tianyi Xu, Yuzheng Hu, Jialin Liu, Guo Chen, Tianyu Zhang et al. "OPUS: towards efficient and principled data selection in large language model pre-training in every iteration." arXiv preprint arXiv:2602.05400 (2026).

3 comments

We thank the reviewer for your positive review and suggestions.

W1: Practical Relevance of Assumptions

W1: Theorem 1 depends on 3 strong assumptions, while the authors showed empirically that the numerical errors are rank-one, other assumptions are somehow strong. Can authors explain why they hold conceptually or with a small scale experiment?

All three assumptions in Theorem 1 are rooted in the empirical structures observed in our LLM experiments, rather than being introduced solely for mathematical convenience. We provide conceptual explanations in section 4 and empirical supports for these assumptions in Appendix D.

Assumption 1 is empirically supported by the near-orthogonality of centered token embeddings, as shown in Appendix D, Figure 14. Specifically, we subtract the mean embedding vector from each token embedding and compute the pairwise angles between the resulting centered embeddings. The angles are tightly concentrated around 90∘90^\circ, with a mean of 89.46∘89.46^\circ. This observation supports the assumption that token embeddings share a common component, while their token-specific components exhibit approximately orthogonal geometry.

Assumption 2 is empirically supported by the high projection isotropy scores at initialization, as reported in Appendix D. We evaluate this property using the projection isotropy score of W⊤WW^\top W, defined as

{\mathrm{rank}(W^\top W)\|W^\top W\|_F^2},$$ which equals 1 for an exact scaled projection. We obtain scores of $0.9206\pm0.0033$ for $W_Q$ and $0.9222\pm0.0044$ for $W_K$, supporting the approximate scaled-projection structure. This assumption characterizes an initialization that is approximately isometric on the query-type and context-type feature subspaces. **Assumption 3 is empirically supported by the persistent sign consistency of $\bar{\gamma}$, as shown in Section 3.2, Figure 3(c).** Specifically, we define the signed projections of the gradient errors onto the identified rank-one directions as $$\gamma_Q=\langle\Delta G_Q,\widehat{k}\overline{x}^{\top}\rangle_F,\qquad \gamma_K=\langle\Delta G_K,\widehat{q}\overline{x}^{\top}\rangle_F,$$ and $\bar{\gamma}$ denotes their exponential moving averages (EMAs). The persistent sign consistency of $\bar{\gamma}$ suggests that numerical errors consistently push the gradients along the identified rank-one directions throughout training. Moreover, the identified directions remain stable after an initial transient (Figure 4) and are approximately orthogonal to the task-gradient directions (Figure 16). Together, these observations support the assumption that the dominant rank-one error components maintain approximately fixed directions and can accumulate coherently throughout training.

Q1: As we can view adamw and muon as using preconditioners with SGD [1], can authors explain theoretically whether the whole framework can generalise to adamw? If so, I think this can be a good extension of the paper.

We thank the reviewer for this suggestion and for bringing the relevant work [1] to our attention. We have added a discussion of this work to the Related Work section of the revised manuscript (lines 715–716). Our findings suggest that the absence of persistent signed gradient-error alignment under AdamW may help explain its weaker excess attention-logit growth relative to Muon.

  1. Weaker excess logit growth under AdamW than Muon. In our controlled AdamW experiment (FA-16 with RoPE) at its best learning rate, the maximum attention logit does not exhibit the two-stage growth pattern observed with Muon (Appendix B.1 of the revised manuscript). This observation is consistent with the Kimi K2 technical report [2]: "an issue that occurs more frequently with Muon but less with AdamW." (Adam exhibits less logit growth.)

  2. Testing the proposed mechanism under AdamW. Our additional experiments show that the structured gradient-error pattern underlying excess logit growth with Muon is not observed under AdamW in the FA-16/RoPE setting (Appendix B.1). Specifically, the gradient discrepancy ΔG\Delta G between FA-16 and FA-32 under AdamW exhibits weak alignment with the rank-one directions defined in Eq. (2), and its signed projection coefficients do not maintain consistent signs throughout training. Thus, AdamW lacks the persistent signed alignment that enables coherent error accumulation in our proposed mechanism for excess logit growth.

  3. Implications across optimizers. Although low-rank numerical errors also arise under AdamW, they exhibit weaker alignment with the identified logit-increasing directions and less consistent signed projections than under Muon. These differences may help explain why excess attention-logit growth is less pronounced under AdamW, suggesting that our mechanism also sheds light on the differences in logit dynamics across optimizers.

We therefore view extending our theoretical framework to AdamW as a promising direction for future work.

Q2: The author mainly report validation loss, while this is convincing, what about the downstream benchmark performance?

We agree that downstream benchmark evaluations would provide complementary evidence of the practical impact of our methods. Our experiments already show that the proposed precision interventions substantially suppress excess logit growth while maintaining comparable or improved pre-training validation loss (Tables 1 and 2). Moreover, pre-training validation loss is a widely used proxy for language-model performance, with prior work establishing empirical relationships between validation perplexity and downstream task performance (Gadre et al. [3]). Nevertheless, direct downstream evaluations would provide a more complete assessment and remain an important direction for future work.

References

[1] Wang, Shaobo, Xuan Ouyang, Tianyi Xu, Yuzheng Hu, Jialin Liu, Guo Chen, Tianyu Zhang et al. "OPUS: towards efficient and principled data selection in large language model pre-training in every iteration." arXiv preprint arXiv:2602.05400 (2026).

[2] Team, Kimi, et al. "Kimi k2: Open agentic intelligence." arXiv preprint arXiv:2507.20534 (2025).

[3] Gadre, Samir Yitzhak, et al. "Language models scale reliably with over-training and on downstream tasks." International Conference on Learning Representations. Vol. 2025. 2025.

Dear Reviewer,

As the discussion period is approaching its end, with less than 2 days remaining, we would like to ensure that we have fully addressed all of your concerns. If you have any additional comments or suggestions, please feel free to share them. We are happy to clarify any remaining points and further improve our work.

Thank you so much for your time and effort in reviewing our work.

Reviewer_LT33OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates excess late-stage attention-logit growth during Muon training. Through comparisons across attention implementations and numerical precisions, it attributes this growth to numerical errors in BF16 FlashAttention. The authors characterize these errors as approximately rank one with persistent directions and signs, analyze their consequences in a simplified retrieval model, and propose selective precision increases to mitigate the growth. Experiments on 124M and 1B language models demonstrate substantial reductions in maximum logits with relatively small attention-kernel overheads.

Critical strengths
  1. The empirical investigation connects implementation-level precision choices to the observed dynamics through controlled comparisons, gradient measurements, and targeted interventions extremely rigorously. This provides useful evidence for the proposed mechanism.
  2. The distinction between excess logit growth and training collapse is informative. It helps clarify the trivial fact why a large maximum attention logit alone is insufficient to diagnose training failure.
Critical weaknesses
  1. The conceptual advance over closely related work appears limited. Qiu and Yao, 2026 (which the authors cite) already identify attention-output numerical errors, their propagation through the backward correction term, and the accumulation of structurally aligned low-rank gradient errors. The present paper provides a useful extension to Muon and sustained logit growth, but I am not yet convinced that it establishes a sufficiently substantial new understanding. The theoretical result characterizes the consequences of assumed persistent, aligned rank-one errors in a restricted model; it does not explain when that structure emerges in general language-model training. A sharper account of the new mechanistic insight would strengthen the contribution.
  2. The practical importance of suppressing the observed growth remains insufficiently established. The paper itself demonstrates that large logits can coexist with successful training. Under RoPE, the dramatic reductions in maximum logits yield small validation-loss improvements at both model sizes. The NoPE improvements are more substantial, but QK-Clip recovers most of that benefit. Consequently, the results establish effective control of an internal statistic more clearly than a consequential improvement over existing methods in model quality, training stability, or overall efficiency.
Questions for authors
  1. What specific mechanistic conclusion distinguishes this work from the output-error and low-rank accumulation explanation of Qiu and Yao, beyond the optimizer and the presence or absence of collapse?
3 comments

We sincerely thank the reviewer for reading our paper.

W1&Q1: Similarity between our paper and Qiu & Yao

W1: The conceptual advance over closely related work appears limited. Qiu and Yao, 2026 (which the authors cite) already identify attention-output numerical errors, their propagation through the backward correction term, and the accumulation of structurally aligned low-rank gradient errors. The present paper provides a useful extension to Muon and sustained logit growth, but I am not yet convinced that it establishes a sufficiently substantial new understanding. Q1: What specific mechanistic conclusion distinguishes this work from the output-error and low-rank accumulation explanation of Qiu and Yao, beyond the optimizer and the presence or absence of collapse?

Although our work and Qiu and Yao [2] share some high-level concepts, we investigate substantially different phenomena and reach different mechanisms.

  • Our work and Qiu and Yao [2] share two high-level ideas: (1) Both studies investigate numerical errors in FlashAttention during LLM training; (2) Both examine how these errors propogate through the backward correction term and accumulate over training.

  • However, our work differs from Qiu and Yao [2] in three key respects: (1) We investigate fundamentally different phenomena; (2) We identify distinct mechanisms underlying these phenomena; and (3) We show that the effectiveness of the same mitigation algorithm differs between the two phenomena.

    • Different phenomena: The excess attention-logit growth studied in our work is distinct from the training failure studied by Qiu and Yao [2]. As shown in Figures 1(a)–(b), substantial excess logit growth occurs without a corresponding increase in validation loss. Moreover, Figure 1(c) shows that training failure begins when the maximum attention logit is approximately 1.5×1041.5\times10^4, smaller than the roughly 4×1044\times10^4 reached during successful training in Figure 1(a). Thus, these two phenomena are distinct.

    • Different characterizations of error accumulation: Our work and Qiu and Yao [2] examine error accumulation in connection with different phenomena. Qiu and Yao [2] explain training failure through biased numerical errors accumulating along a shared low-rank direction RR, increasing weight and activation norms and eventually causing loss explosion. In contrast, our work focuses on excess attention-logit growth and identifies the specific signed rank-one components driving this growth: γQk^xˉ⊤\gamma_Q\hat{k}\bar{x}^{\top} and γKq^xˉ⊤\gamma_K\hat{q}\bar{x}^{\top}. Here, q^\hat{q}, k^\hat{k}, and xˉ\bar{x} represent the concentrated directions of queries, keys, and hidden states, respectively, while γQ\gamma_Q and γK\gamma_K are the signed coefficients. We show that coherent accumulation of these components can drive excess logit growth even when training remains stable.

    • Different effectiveness of mitigation methods: The method proposed by Qiu and Yao [2] consistently fails in our setting. Although their method successfully mitigates training failure in their experiments, applying it to our standard NanoGPT training configuration frequently causes numerical underflow, as shown in Line 430 and Appendix B (Figure 10). This result shows that a method effective in mitigating training failure does not necessarily transfer to the excess attention-logit growth setting studied in our work.

W1: Explanations for the persistent, aligned rank-one error.

W1: The theoretical result characterizes the consequences of assumed persistent, aligned rank-one errors in a restricted model; it does not explain when that structure emerges in general language-model training. A sharper account of the new mechanistic insight would strengthen the contribution.

Our empirical results in Section 3.2 show that persistent, approximately rank-one gradient errors always emerge in attention heads exhibiting excess logit growth under Muon with FA-16. Our controlled experiments identify the low-precision attention output OO as the dominant source of these errors. Proposition 1 shows that the resulting query- and key-gradient discrepancies can be expressed as sums of rank-one terms involving hidden states, queries, and keys. Because these features concentrate around their respective mean directions, the constituent rank-one terms become approximately aligned, explaining the observed near-rank-one structure of the aggregate gradient errors.

W2: Necessity to concern the logit growth

W2: The practical importance of suppressing the observed growth remains insufficiently established. The paper itself demonstrates that large logits can coexist with successful training. Under RoPE, the dramatic reductions in maximum logits yield small validation-loss improvements at both model sizes. The NoPE improvements are more substantial, but QK-Clip recovers most of that benefit. Consequently, the results establish effective control of an internal statistic more clearly than a consequential improvement over existing methods in model quality, training stability, or overall efficiency.

Beyond validation loss, training stability is another important practical consideration, and excessively large attention logits can pose numerical risks. As noted in the Kimi K2 technical report [1], "During a mid-scale training run, attention logits rapidly exceed 1000, which could lead to potential numerical instabilities and even training divergence." This observation highlights the value of monitoring maximum attention logits as a potential indicator of training instability. Thus, understanding and controlling excessive attention-logit growth remains practically relevant even when validation loss is unaffected.


We thank the reviewer again for reading our response. We appreciate it if the reviewer could consider raising the score, if our rebuttal has addressed your concerns.

References

[1] Team, Kimi, et al. "Kimi k2: Open agentic intelligence." arXiv preprint arXiv:2507.20534 (2025).

[2] Qiu, Haiquan, and Quanming Yao. "Why low-precision transformer training fails: An analysis on flash attention." arXiv preprint arXiv:2510.04212 (2025).

Dear Reviewer,

As the discussion period is approaching its end, with less than 2 days remaining, we would like to ensure that we have fully addressed all of your concerns. If you have any additional comments or suggestions, please feel free to share them. We are happy to clarify any remaining points and further improve our work.

Thank you so much for your time and effort in reviewing our work.

Reviewer_F2MmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies "excess logit growth" as a problem which occurs during FA16 (Flash-Attention BF16) training with Muon, where the attention logits grow faster than would be expected with FA32 or MATH16 training. The authors attribute this to a rank-1, sign-aligned, numerical error accumulation in the query / key weight gradients, which is persisted by Muon's optimization step. They analyze a toy model of the hypothesized mechanism and suggest practical remedies to this issue, including storing attention outputs in a higher precision. They train 124M and 1B GPT-2 models to validate their claims.

Critical strengths
  • Their empirical results indicate that they have a practical solution to excess logit growth. It seems likely that their proposed solution will help practitioners.
  • While the core results are not novel (see below), this paper extends beyond the analysis of [12] in the following non-trivial ways: it studies excess logit growth in the absence of collapse, it distinguishes between P contributions and O contributions to the error, it extends the error analysis to RoPE and studies multiple positional encodings.
Critical weaknesses
  • There is no null hypothesis for their central explanatory mechanism. They attribute the excess logit growth to properties of the difference ΔZ\Delta Z which compares FA32 quantities to FA16 quantities. However they do not study the properties of the difference between MATH32 and MATH16 quantities to rule out the possibility that their explanatory mechanism is present in cases where excess logit growth does not occur. This omission substantially weakens the strength of their explanatory conclusions. Additionally, there is no null-hypothesis for the optimizer: the authors do not show that excess logit growth is absent in AdamW training. Therefore the role the paper assigns to Muon in accumulating the error is unsubstantiated. Regardless, the authors should run experiments on AdamW to either substantiate their claims or show that this is a Muon specific issue.
  • The strongest results of the paper closely reproduce Qiu & Yao [12] and this is not sufficiently discussed. [12] already localizes low-precision Flash Attention failures to the BF16 attention output O used in the row-sums, and show that computing O in FP32 restores stability. The derive the same error decomposition and Proposition 1 of this paper, and show that the error has a consistent sign which accumulates. All of these findings are billed in the present paper as being novel when in reality they are a reproduction of [12]. [12] also studies Muon in addition to AdamW!
Questions for authors
  1. Can you re-run the analysis of Section 3.2 on differences between MATH32 and MATH16? I am curious to understand the quantitative and qualitative differences between this case and the case in the paper. My weak hypothesis is that you will find that under MATH16 you get the same sign behavior and the same low-rankedness. Since MATH16 does not exhibit excess logit growth, this would imply that two of your central criteria are not explanatory. The only remaining criteria is the bias term, which I do expect to be systematically larger for FA16 than MATH16 training.
  2. Building off of question 1, can you report the ratio of bias to gradient for the runs you did, as well as for the MATH16 and O32/O16 runs?
  3. Does excess logit growth appear in AdamW runs or is it a Muon specific phenomenon? Could you run a control experiment with AdamW?
  4. Under RoPE, excess logit growth does not noticeably affect validation loss. Why should we be concerned? Do you expect excess logit growth to be exacerbated at scale?
  5. How does your paper compare to Qiu and Yao? They find that around step 6k, repeated row maxima became more common shortly before the loss divergence, do you find the same thing and does this coincide with the onset of excess growth?
  6. You use Kimi's attention-logit growth as a motivator in the introduction but then do not study Kimi-type failure cases. Please remove this motivation from the paper or argue for its inclusion despite the lack of established connection.

Minor Questions / Clarifications

  1. Can you clarify in Figure 1 which optimizer was used for the run from reference [12]? [12] reports failures under both AdamW and Muon.
  2. Which attention kernel and version are used for FA16 and FA32? Is MATH16 computed in BF16 internally?
6 comments

We sincerely thank the reviewer for reading our paper.

W1 & Q1: Null Hypothsis for MATH16

W1: There is no null hypothesis for their central explanatory mechanism. They attribute the excess logit growth to properties of the difference ΔZ\Delta Z which compares FA32 quantities to FA16 quantities. However they do not study the properties of the difference between MATH32 and MATH16 quantities to rule out the possibility that their explanatory mechanism is present in cases where excess logit growth does not occur. This omission substantially weakens the strength of their explanatory conclusions.

Q1: Can you re-run the analysis of Section 3.2 on differences between MATH32 and MATH16? I am curious to understand the quantitative and qualitative differences between this case and the case in the paper. My weak hypothesis is that you will find that under MATH16 you get the same sign behavior and the same low-rankedness. Since MATH16 does not exhibit excess logit growth, this would imply that two of your central criteria are not explanatory.

The requested experiments (Appendix B.2) show that MATH16 does not exhibit the persistent alignment and sign consistency observed in FA16, suggesting that the full error structure underlying our proposed mechanism is specific to FA16.

First, low rank is only one component of our proposed mechanism for excess logit growth. The full mechanism involves the following properties: the hidden states, queries, and keys concentrate around shared directions; the approximately rank-one numerical gradient errors align with the corresponding directions; and the aligned error components maintain consistent signs along slowly varying directions throughout training. Together, these properties allow the numerical errors to accumulate coherently and contribute to excess logit growth.

Second, we repeated the analysis of Section 3.2 for MATH16. Although the gradient discrepancy ΔG\Delta G between MATH16 and MATH32 is approximately low rank, it lacks the directional alignment and sign consistency observed in FA16. Specifically, ΔG\Delta G does not align with the rank-one directions in Eq. (2), and its signed projections do not maintain consistent signs during training (Appendix B.2 Figure 9 (c,d)). Weak alignment limits the errors’ contribution to a common upward shift in attention logits, while inconsistent signs undermine their coherent accumulation. Thus, the MATH16 errors lack the structure underlying our proposed mechanism for excess logit growth.

W1 & Q3: Null Hypothsis for Optimizers

Additionally, there is no null-hypothesis for the optimizer: the authors do not show that excess logit growth is absent in AdamW training.

Q3: Does excess logit growth appear in AdamW runs or is it a Muon specific phenomenon? Could you run a control experiment with AdamW?

AdamW does not exhibit consistent excess attention-logit growth in our FA-16 setting with RoPE. In our controlled AdamW experiment at its best learning rate, the maximum attention logit does not show the two-stage growth pattern observed with Muon (Appendix B.1 of the revised manuscript). This observation is consistent with the Kimi K2 technical report [1]: "an issue that occurs more frequently with Muon but less with AdamW" (Adam exhibits less logit growth.).

Our additional experiments show that the structured gradient-error pattern underlying excess logit growth with Muon does not hold under AdamW in the FA-16/RoPE setting (Appendix B.1). Specifically, the gradient discrepancy ΔG\Delta G between FA-16 and FA-32 under AdamW exhibits weak alignment with the rank-one directions defined in Eq. (2), and its signed projection coefficients do not maintain consistent signs throughout training (Appendix B.1 Figure 7 (c,d)). Thus, AdamW does not exhibit the persistent signed alignment that enables coherent error accumulation in our proposed mechanism for excess logit growth.

W2 & Q5: Similarity between our paper and Qiu & Yao[2]

W2: The strongest results of the paper closely reproduce Qiu & Yao [12] and this is not sufficiently discussed. [12] already localizes low-precision Flash Attention failures to the BF16 attention output O used in the row-sums, and show that computing O in FP32 restores stability. The derive the same error decomposition and Proposition 1 of this paper, and show that the error has a consistent sign which accumulates. All of these findings are billed in the present paper as being novel when in reality they are a reproduction of [12]. [12] also studies Muon in addition to AdamW!

Q5: How does your paper compare to Qiu and Yao?

Although our work and Qiu and Yao [2] share some high-level concepts, we investigate substantially different phenomena and reach different mechanisms.

  • Our work and Qiu and Yao [2] share two high-level ideas: (1) Both studies investigate numerical errors in FlashAttention during LLM training; (2) Both examine how these errors accumulate over training.

  • However, our work differs from Qiu and Yao [2] in three key respects: (1) We investigate fundamentally different phenomena; (2) We identify distinct mechanisms underlying these phenomena; and (3) We show that the effectiveness of the same mitigation algorithm differs between the two phenomena.

    • Different phenomena: The excess attention-logit growth studied in our work is distinct from the training failure studied by Qiu and Yao [2]. As shown in Figures 1(a)–(b), substantial excess logit growth occurs without a corresponding increase in validation loss. Moreover, Figure 1(c) shows that training failure begins when the maximum attention logit is approximately 1.5×1041.5\times10^4, smaller than the roughly 4×1044\times10^4 reached during successful training in Figure 1(a). Thus, these two phenomena are distinct.

    • Different characterizations of error accumulation: Our work and Qiu and Yao [2] examine error accumulation in connection with different phenomena. Qiu and Yao [2] explain training failure through biased numerical errors accumulating along a shared low-rank direction RR, increasing weight and activation norms and eventually causing loss explosion. In contrast, our work focuses on excess attention-logit growth and identifies the specific signed rank-one components driving this growth: γQk^xˉ⊤\gamma_Q\hat{k}\bar{x}^{\top} and γKq^xˉ⊤\gamma_K\hat{q}\bar{x}^{\top}. Here, q^\hat{q}, k^\hat{k}, and xˉ\bar{x} represent the concentrated directions of queries, keys, and hidden states, respectively, while γQ\gamma_Q and γK\gamma_K are the signed coefficients. We show that coherent accumulation of these components can drive excess logit growth even when training remains stable.

    • Different effectiveness of mitigation methods: The method proposed by Qiu and Yao [2] consistently fails in our setting. Although their method successfully mitigates training failure in their experiments, applying it to our standard NanoGPT training configuration frequently causes numerical underflow, as shown in Line 430 and Appendix B.3.1 (Figure 10). This result shows that a method effective in mitigating training failure does not necessarily transfer to the excess attention-logit growth setting studied in our work.

Q2: The ratio of bias to gradient

Q1: The only remaining criteria is the bias term, which I do expect to be systematically larger for FA16 than MATH16 training. Q2: Building off of question 1, can you report the ratio of bias to gradient for the runs you did, as well as for the MATH16 and O32/O16 runs?

As requested, we compare the relative gradient discrepancies ∥ΔGQ∥F/∥GQ,hi∥F\Vert\Delta G_Q\Vert_F/\Vert G_{Q,\mathrm{hi}}\Vert_F and ∥ΔGK∥F/∥GK,hi∥F\Vert\Delta G_K\Vert_F/\Vert G_{K,\mathrm{hi}}\Vert_F across FA16--Adam, Math16--Muon, and FA16--Muon. Our experiments show that FA16--Muon exhibits larger relative gradient discrepancies, coinciding with the emergence of excess logit growth. However, these larger discrepancies may also result from accumulated training errors, rather than driving the excess growth. Specifically, each setting is evaluated at checkpoints from its own low-precision training trajectory, rather than at shared model parameters. Thus, this requested comparison cannot establish whether the larger discrepancies cause or result from excess logit growth.

Q4: Necessity to study the logit growth

Q4: Under RoPE, excess logit growth does not noticeably affect validation loss. Why should we be concerned?

Beyond validation loss, training stability is another important practical consideration, and excessively large attention logits can pose numerical risks. As noted in the Kimi K2 technical report [1], "During a mid-scale training run, attention logits rapidly exceed 1000, which could lead to potential numerical instabilities and even training divergence." This observation highlights the value of monitoring maximum attention logits as a potential indicator of training instability. Thus, understanding and controlling excessive attention-logit growth remains practically relevant even when validation loss is unaffected.

Q4: Scale and Logit Growth

Q4: Do you expect excess logit growth to be exacerbated at scale?

Our experiments do not establish a general scaling trend for maximum attention logits with model size. Under RoPE, the maximum attention logits remain nearly unchanged, from 46,720±9,38246,720 \pm 9,382 for the 124M model to 46,592±10,75246,592 \pm 10,752 for the 1B model. We leave a systematic characterization of this scaling behavior across model sizes and architectures to future work.

Q5: Repeated Row Maxima and Excess Logit Growth.

Q5: They find that around step 6k, repeated row maxima became more common shortly before the loss divergence, do you find the same thing and does this coincide with the onset of excess growth?

As requested, we tracked the number of attention rows with near-tied maxima throughout Muon training with FA-16, following the experimental settings recommended by Qiu and Yao [2]. Our observations differ from their findings: the number of such rows gradually decreases during training, rather than increasing (Appendix B.3.2, Figure 11). Moreover, we observe no clear relationship between this metric and the onset of excess logit growth. These results further suggest that the mechanism underlying our phenomenon differs from that identified by Qiu and Yao [2]. One possible explanation for this discrepancy is the difference in attention implementations. Their publicly released code uses a customized reimplementation of FlashAttention-2, whereas our experiments use the standard FlashAttention kernel within NanoGPT, an open-source training codebase that follows a widely adopted and well-established training recipe. Such implementation differences may affect numerical errors and contribute to the contrasting observations.

Q6: Kimi's Failure Cases

Q6: You use Kimi's attention-logit growth as a motivator in the introduction but then do not study Kimi-type failure cases. Please remove this motivation from the paper or argue for its inclusion despite the lack of established connection.

We argue that our work directly investigates the central concern raised by Kimi K2 technical report: excessive attention-logit growth. As noted in the Kimi K2 technical report, "During a mid-scale training run, attention logits rapidly exceed 1000, which could lead to potential numerical instabilities and even training divergence." This statement identifies excessive attention-logit growth itself as an important mode of instability. In our work, we systematically investigate the same phenomenon. Therefore, we believe that Kimi K2 provides a relevant and well-motivated practical context for our study.

Minor questions:

Q1:Can you clarify in Figure 1 which optimizer was used for the run from reference [12]? [12] reports failures under both AdamW and Muon.

The run from reference [12] shown in Figure 1 uses Muon.

Q2:Which attention kernel and version are used for FA16 and FA32? Is MATH16 computed in BF16 internally?

We use FlashAttention-2 (FA2) for our FA-based experiments. For MATH16, we use the PyTorch SDPA math backend with BF16 inputs and outputs, with intermediate computations performed in FP32.

We thank the reviewer again for reading our response. We appreciate it if the reviewer could consider raising the score, if our rebuttal has addressed your concerns.

References

[1] Team, Kimi, et al. "Kimi k2: Open agentic intelligence." arXiv preprint arXiv:2507.20534 (2025).

[2] Qiu, Haiquan, and Quanming Yao. "Why low-precision transformer training fails: An analysis on flash attention." arXiv preprint arXiv:2510.04212 (2025).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Rebuttal and Discussion (Part I)

Dear AC,

We sincerely appreciate your time and effort in reviewing our work. Below, we summarize our discussions with the reviewers.

Reviewer F2Mm's main concerns and our responses are summarized as follows.

  • Null Hypothsis for and Optimizers (W1 & Q3) and MATH16 (W1 & Q1). We add experiments in Appendices B.1 and B.2. Neither MATH16-Muon nor FA16-AdamW exhibits the persistent alignment and sign consistency observed in FA16-Muon, suggesting that our proposed mechanism is specific to the FA16-Muon setting.

  • Comparison with Qiu & Yao [2] (W2 & Q5). We summerize the differences between our papaer and Qiu & Yao [2] in three key respects: phenomenon, mechanism, and algorithm.

  • The ratio of bias to gradient (Q2). We compare relative gradient discrepancies across FA16-Adam, MATH16-Muon, and FA16-Muon. FA16-Muon shows the largest discrepancy, coinciding with excess logit growth, although this may reflect accumulated errors rather than cause the growth.

  • Necessity to study the logit growth (Q4). We emphasized that excessively large attention logits pose numerical risks, as also noted in the Kimi technical report [1].

  • Model Scale and Logit Growth (Q4). Our experiments do not reveal a clear scaling trend in maximum attention logits with increasing model size. Under RoPE, the maximum attention logits remain nearly unchanged between 124M and 1B models.

  • Repeated Row Maxima and Excess Logit Growth (Q5). We tracked attention rows with near-tied maxima throughout FA-16 Muon under the settings of Qiu and Yao [2]. We find no clear relation to excess logit growth (Appendix B.3.2, Figure 11), which suggests a distinct mechanism.

  • Kimi's Failure Cases (Q6). We explained that our work directly investigates the central concern raised by Kimi K2 technical report: excessive attention-logit growth.

  • Minor questions on the optimizer and attention kernel. We clarify that Figure 1 uses Muon, while our experiments use FlashAttention-2 and PyTorch SDPA with BF16 inputs and outputs.

Discussion results:

The reviewer is overall satisfactory with the rebuttal and acknowledged the scientific value of our contributions and raised the score from 2 to 3.

I have read the authors' response and believe that they have fully rebutted all of my concerns regarding the scientific value of their experiments. Therefore I consider Q1, Q2, Q3, Q4 to be "closed satisfactorily".

We will further clarify the relationship between our work and that of Qiu & Yao according to the reviewer's suggestion in the revised manuscript.

AuthorsOpenReview ↗

Summary of Rebuttal and Discussion (Part II)

Reviewer LT33's main concerns and our responses are summarized as follows.

  • Comparison with Qiu & Yao [2] (W1 & Q1). We distinguish our work from Qiu & Yao [2] in three aspects: (1) We investigate fundamentally different phenomena. As shown in Figures 1(a)–(b), substantial excess logit growth occurs without a training failure. Moreover, Figure 1(c) shows that training failure begins when the maximum attention logit is approximately, smaller than the roughly reached during successful training in Figure 1(a). (2) We identify distinct mechanisms underlying these phenomena. Qiu and Yao [2] explain training failure through biased numerical errors accumulating along a shared low-rank direction, while we identifies the specific signed rank-one components driving this growth and show that coherent accumulation of these components can drive excess logit growth even when training remains stable. (3) We show that the effectiveness of the same mitigation algorithm differs between the two phenomena. Although their method successfully mitigates training failure in their experiments, applying it to our standard NanoGPT training configuration frequently causes numerical underflow, as shown in Line 430 and Appendix B.3.1.

  • Explanation for the persistent, aligned rank-one errors (W1). We explained that the persistent, approximately rank-one gradient errors always emerge in attention heads exhibiting excess logit growth under Muon with FA-16. We further provide an explanation for the observed near-rank-one structure based on Proposition 1 and the concentration of features around their respective mean directions.

  • Practical relevance of studying logit growth (W2). We emphasized that training stability is an important practical consideration beyond validation loss, as excessively large attention logits may pose numerical risks, a concern also highlighted in the Kimi technical report [1].

Discussion results:

Despite our active responses to the raised concerns, the reviewer did not engage in the discussion.

Reviewer 9Gdu's main concerns and our responses are summarized as follows.

  • Practical Relevance of Assumptions (W1). We explained that all three assumptions in Theorem 1 are motivated by empirical structures observed in our LLM experiments, with conceptual explanations in Section 4 and empirical support in Appendix D.

  • Generalizing our mechanism to Adam (Q1). We added the suggested work to Related Work (Lines 715–716) and noted that the weaker signed alignment of AdamW gradient errors may explain its weaker excess logit growth than Muon.

  • Downstream benchmark performance (Q2). We explained that our precision interventions substantially suppress excess logit growth while maintaining comparable or improved validation loss (Tables 1–2), which is widely used as a proxy for downstream performance and has been empirically correlated with downstream task performance [3].

Discussion results:

We responded to all of the reviewer’s concerns in our rebuttal, and no further concerns were raised.

Reference

[1] Team, Kimi, et al. "Kimi k2: Open agentic intelligence." arXiv preprint arXiv:2507.20534 (2025).

[2] Qiu, Haiquan, and Quanming Yao. "Why low-precision transformer training fails: An analysis on flash attention." arXiv preprint arXiv:2510.04212 (2025).

[3] Gadre, Samir Yitzhak, et al. "Language models scale reliably with over-training and on downstream tasks." International Conference on Learning Representations. Vol. 2025. 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.