- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether the classical Edge of Stability theory extends to no-momentum Muon, focusing on the relationship between the loss-neutral boundary and temporal orthogonality. The authors derive a coherence-corrected loss-neutral condition for stochastic Muon and theoretically demonstrate that the same effective curvature does not necessarily imply the same alignment of consecutive update directions. Through experiments on MLPs, CNNs, Transformers, and LLMs up to 1B parameters, they show that Muon can approach the loss-neutral boundary without exhibiting complete update-direction reversal, and that learning rate and batch size affect these phenomena differently. Based on these findings, the authors propose a "split Edge of Stability" interpretation for Muon.
Critical strengths
-
I find the theoretical result that the same effective curvature does not necessarily imply the same alignment of consecutive update directions particularly interesting. The matrix quadratic example clearly illustrates why the classical curvature-based relationship between loss neutrality and directional behavior does not directly extend to Muon. This provides a useful perspective on Muon's optimization dynamics that is not captured by the classical EoS framework.
-
The authors conduct extensive experiments across different model architectures and scales, including LLM pretraining. These experiments examine how learning rate and batch size affect loss neutrality and temporal alignment, providing empirical support for the theoretical findings.
-
The learning-rate perturbation experiments are particularly convincing. By branching from the same checkpoint, the authors show that the conditional loss change can switch signs while consecutive update directions remain negatively aligned. This provides more direct evidence for the separation between T1 and T2 than simply observing their trajectories. I also find the frozen-state experiments useful for distinguishing the effects of minibatch randomness from the negative alignment induced by training dynamics.
Critical weaknesses
-
I am not fully convinced that the theoretical results sufficiently support the proposed "split Edge of Stability" interpretation. The loss-neutral boundary in Theorem 2.1 appears to follow directly from the definition of the finite-step curvature remainder, leaving its theoretical novelty unclear. The authors also acknowledge that is not itself a stability criterion. While the matrix quadratic example shows that loss neutrality and temporal alignment can be decoupled, this alone does not establish a new understanding of stability.
-
The paper focuses exclusively on no-momentum Muon, whereas Muon implementations used in LLM pretraining typically incorporate momentum. Since momentum affects the alignment of consecutive updates, the results may not carry over to practical Muon implementations. While isolating the effects of polar normalization is reasonable, the absence of momentum experiments limits the generalizability of the findings.
Questions for authors
-
What additional theoretical insight does Theorem 2.1 provide beyond reformulating the one-step loss change? In particular, is there a theoretical explanation for why Muon training tends to approach the loss-neutral boundary?
-
Given that is not itself a stability criterion, how does the separation between T1 and T2 justify the proposed "split EoS" interpretation? What does this separation reveal about stability beyond the distinct behaviors of the two diagnostics?
-
Do the authors expect the separation between T1 and T2 to persist when momentum is introduced? Preliminary experiments with momentum would help establish whether the findings extend to practical Muon training.
-
The 1B pretraining experiment examines temporal alignment but does not measure the T1 boundary. Is there evidence that the separation between loss neutrality and temporal alignment persists at this scale? Since the 130M and 1B configurations differ in both model size and batch size, how much of the observed difference in alignment can be attributed to batch size?
5 comments
Dear Reviewer V8hT,
As the discussion period draws to a close, we would appreciate it if you could let us know whether our response addresses your concerns. We would be happy to clarify any remaining points. Thank you for taking the time to review our response.
Kind regards, The Authors
Thank you for the detailed response and the additional theoretical and experimental results. I appreciate the authors' efforts in addressing my concerns.
I have one remaining question regarding W1/Q2. The new generalized-sharpness measurements provide more direct evidence that Muon exhibits EoS behavior, which addresses part of my original concern. However, if generalized sharpness already characterizes EoS, what additional insight into stability does the separation between T1 and T2 provide? In other words, what makes the proposed "split EoS" a new characterization of EoS, rather than simply two additional diagnostics of Muon's optimization dynamics?
I would appreciate further clarification on this conceptual distinction. Otherwise, I am satisfied with the authors' responses.
We thank Reviewer V8hT for your constructive comments and positive support.
Theorem 2.1 serves as a loss identity, and we added Proposition 2.1 in the main text, which shows the attraction that effective curvature s^k,bM cannot remain below 2/η.
According to your suggestions, we have included the generalized-sharpness measurements for EoS and conducted experiments for comparions on 130M runs, see Figure 1 in the updated version.
Our 130M runs show that normalized generalized sharpness approaches 2/η, which provides empirical evidence of EoS in LLM Muon pretraining, following the framework of Islamov et al (2026). However, effective curvature (T1, tracks the smaller boundary 2ρ/η) differs such boundary and arrives even eariler.
This is one part of what our finding: effective curvature separates from normalized generalized sharpness, while loss balance does not uniquely determine update alignment (T2). This may suggest to imrpove the WSD learning-rate schedules in practice as there is still some space between T1 and T2.
In summary, before proving dynamics of Muon at EoS, this paper firstly confirms a phenomenon: Muon exhibits generalized-sharpness EoS signatures in LLM pretraining regimes, then identifies two non-interchangeable indicators/diagnostics of this behavior in different charecterizations. Furthermore, our work suggests the possibility of improving learning-rate schedules in practice.
Yes. We added a 130M run with momentum 0.95 (Section3.1 figure6).Loss balance (T1) persists, while update alignment (T2) is positive (median cosine (0.712)), unlike the negative alignment observed without momentum. Thus, the need to examine T1 and T2 separatly extends to momentum Muon.
We added T1 measurements for the 1B model at batch sizes 128, 256, and 512 (Figure 1(d–f), Section 4.3, and Appendix D.6).
Across all three runs, loss-neutrality balance persists. Larger batches show more negative consecutive-update alignment(T2). This comparison isolates batch-size variation at a fixed model scale.
Reference Islamov et al. (2026). Non-Euclidean Gradient Descent Operates at the Edge of Stability. ICML'26.