acceptodds
Under review as a conference paper at ICLR 2027

Beyond Hessian Sharpness: Two Local Stability Edges in Practical Muon

Abstract

Edge of Stability connects Hessian sharpness to the local stability limit of gradient descent. We study this limit for practical Muon with its original five Newton–Schulz steps, normalization, Nesterov momentum, and weight decay. The key quantity combines the loss Hessian with the derivative of the matrix transformation. We derive an exact quadratic equation for two state multipliers per mode. They describe the growth or decay of small weight and momentum perturbations under the local linear map. For real modes, the stable interval has multiplier at the lower edge and at the upper edge. Finite Newton–Schulz steps can introduce negative modes under positive semidefinite curvature. Proposal scaling can preserve the next weight while moving modes across either edge. Two strongly convex quadratic constructions with clamp normalization connect these crossings to stable fixed points and a stable orbit of period two. Language model experiments support each part of the local calculation: negative derivative eigenvalues occur in all 240 sampled matrix and checkpoint pairs from 58M to 1.25B parameters, a convex objective with linearized logits isolates negative modes under positive semidefinite curvature, and interventions cross both edges in 58M and 124M models. In a 16 step replay with non-Muon parameters fixed, successive Jacobians reproduce the measured perturbation growth. Together, these results identify how the implemented matrix transformation and optimizer state determine Muon's local response.

Primary area:
optimization
Keywords:
Muon optimizer, Edge of stability, Local stability, Newton–Schulz iteration, Nesterov momentum, Hessian sharpness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ptbBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the local stability of practical Muon with five Newton–Schulz steps and stored momentum. By analyzing the joint weight–momentum Jacobian, it derives two stability boundaries for each real eigenvalue κ\kappa of K=αDΨ(P)HK=\alpha D\Psi(P)H, corresponding to multipliers +1+1 and −1-1. Its main finding is that the finite Newton–Schulz map can produce lower-edge instability even under positive semidefinite curvature, unlike the differentiable exact polar map. Proposal-scaling interventions change local stability while preserving the next weight under scale-invariant normalization, and quadratic examples exhibit attracting states beyond the boundaries. Language-model experiments support the negative-response mechanism and show that successive Jacobians predict perturbation growth much better than a frozen initial Jacobian. The trajectory validation concerns smooth FP64 updates.

Critical strengths
  • S1. A stability mechanism missed by exact-polar models. The finite Newton–Schulz map can reverse perturbations and generate negative effective modes under PSD curvature. GGN and output-matched controls separate this mechanism from negative loss curvature and changes in the reference update. This is the paper's strongest contribution.

  • S2. Careful theoretical and numerical validation. The analysis handles noncommuting operators, and the quadratic constructions verify attraction in the full weight–momentum state. Independent derivatives, finite-difference checks, and explicit numerical resolution criteria support the results. The 58M replay also shows that frozen-Jacobian extrapolation fails even for identified eigenmodes.

Critical weaknesses
  • W1. The mechanism's finite-horizon contribution remains unclear. GGN and output-matched controls isolate one-step effects but are not extended to Jacobian products. The trajectory experiments use the full Hessian, so negative loss curvature may also contribute. In Table 1's fixed-batch conditional Muon replay, the lower-mode perturbation still grows 442.8×442.8\times, despite being far below the frozen prediction. Successive-Jacobian agreement validates propagation, but does not show whether replacing the finite-map response reduces amplification or explain the large discrepancy from the frozen prediction.

  • W2. Transfer to native mixed precision is not established. The perturbation experiments use smooth FP64 updates, whereas training uses mixed precision. Footnote 2 reports BF16–FP64 matrix-map output differences of 1.9%–11.7% and states that native BF16 finite-difference checks fail, but the corresponding evidence is not provided in Appendix J.11. The key question is whether perturbations that are accurately represented initially still follow the predicted dynamics under native arithmetic.

  • W3. The changing-batch residence measurements do not isolate state dynamics. In Figure 6, the median upper ratio is 0.988 and only 13/64 values fall within ±5% of the edge, failing the paper's residence criterion. As the authors acknowledge, these measurements do not separate batch variability from state changes. Six repeated-batch controls already demonstrate non-residence in small-model continuations. What remains unclear is how much the variation in the changing-batch run reflects state dynamics, and whether the finding extends to larger models with the complete optimizer.

Questions for authors
  • Q1 (W1). Does replacing the finite-map response reduce finite-horizon amplification? Could the output-matched control be re-centered at successive reference states, preserving the reference trajectory while comparing original and controlled Jacobian products in the same fixed norm? A corresponding GGN comparison would help distinguish map effects from negative loss curvature. Does the frozen-versus-ordered discrepancy arise mainly from weaker local amplification or changing perturbation directions?

  • Q2 (W2). Which predictions survive native precision? For identical checkpoints, batches, and perturbation directions, over what amplitudes and horizons do FP32 and native mixed-precision responses agree with the smooth prediction? Please provide the native BF16 protocol and results referenced in footnote 2, distinguishing perturbations lost at initialization from deviations caused by subsequent rounding or finite-amplitude effects.

  • Q3 (W3). What changes when the saved states from the changing-batch continuation are evaluated on a fixed probe batch or panel? Using a consistent positive-mode selection rule, please report temporal averages, dispersion, and residence counts. This would separate probe-batch variability from changes along the original trajectory, complementing the existing repeated-batch training controls. A similar diagnostic at selected 58M checkpoints would test whether the finding extends beyond the small models.

Required Corrections

  • Undefined polynomial (Appendix F.4, p. 29, L1512–L1519). The recurrence uses p,p′,p′′,p′′′p,p',p'',p''' without identifying pp with the previously defined ϕ^\widehat{\phi} (p. 28, L1498–L1501). Define p=ϕ^p=\widehat{\phi} or use consistent notation. Also correct the incomplete sentence preceding the recurrence.

  • Missing native BF16 evidence (p. 6, L321–L323; Appendix J.11, pp. 39–40, L2095–L2142). Footnote 2 refers to failed native BF16 finite-difference checks, but Appendix J.11 does not report their protocol or results. Provide this evidence or correct the cross-reference.

  • Unspecified edge-distance test (Appendix J.6, p. 37, L1970–L1975). Define the edge-distance test and its numerical tolerance, including why only the smallest amplitude passes in the unresolved case behind the reported 11/12 upper crossings.

  • Imprecise scaling description (p. 1, L020–L021; p. 2, L082–L084; Eq. (5), p. 5, L253–L261). “Rescaling stored momentum” may suggest M(c)=cMM(c)=cM, whereas Eq. (5) defines an affine intervention that rescales the proposal. Clarify this distinction and the scale-invariance condition for exact next-weight preservation.

  • Copyediting (p. 5, L218). Replace “the Muon’s matrix map” with “Muon’s matrix map.”

Reviewer_JAR5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

A paper studies additional condition for training with Muon to be stable in theory. Namely, prior optimization stability work has shown that if Hessian eigenvalue is between 0 and 2/lr, the training in the direction of that eigenvector is stable. In case Hessian is positive semi-definite, the eigenvalues will (obviously) never go below the lower bound. This paper shows that for Muon, even if the Hessian is positive semi-definite, certain modes (corresponding to linearization of the update) can hit the lower bound: a real eigenvalue of αRH\alpha RH needs to be above certain threshold -d. Complex values give rise to a different shape in complex plane on which one step is stable. The experiments seem to suggest (as in many cases finding the eigenvalues was 'unresolved') that training with Muon makes these certain modes be both below, within and above these thresholds.

Critical strengths

Showing that, when Hessian is positive semi-definite, Muon can be unstable in a new way (going below the threshold), additional to the way we already knew about (going above the threshold).

Critical weaknesses

The paper is unclear: including the motivation, the purpose and takeaways of theorems, figures and the experiments. Many phrases used are not commonly established, making the paper even harder to understand, for example 'next weight', 'mode', 'roots', 'both optimizers active', 'conditional Muon', '6/12 combinations pass every step of the all-weight response check' etc. Reading the main body of the paper should convey a clear picture, where now it reads as a notation and phrases heavy summary of the results.

Additionally, the paper doesn't convey a clear takeaway. The optimization of deep neural networks is highly non-convex and instabilities happen all the time. In the experiments with small language models (Figure 6) when they found the eigenvalues of αRH\alpha R H do cross the lower threshold, it is not clear that it happened due to RR (i.e. due to using Muon, which is what the paper is about), or just generally because Hessian has both positive and negative eigenvalues. More broadly, I'm not convinced that for practical use of Muon it would even matter whether crossing the threshold comes from non-convexity or from Muon's update.

Questions for authors
  1. What's the practical takeaway of the paper?
  2. Edge of Stability work shows that the eigenvalues of the Hessian oscillate around the upper threshold. Does Muon optimization exhibit some characteristic behavior around the lower threshold?
  3. See second weakness
Reviewer_92ahOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the local stability of Muon, showing that traditional Hessian sharpness alone is insufficient to characterize its training dynamics. By incorporating the Hessian and the derivative of Muon's matrix transformation, the authors derive two local stability boundaries and investigate negative effective modes.

Critical strengths

1. Interesting motivation. The observation that Hessian sharpness alone cannot fully explain Muon's training stability is interesting and provides a valuable perspective on understanding this optimizer.

2. Clear presentation. The paper is generally well-written and clearly organized, with relatively rigorous mathematical derivations and theoretical analysis.

Critical weaknesses

1. Impact of Newton–Schulz iteration count.

The paper focuses on five Newton–Schulz (NS) iterations. How does the negative response mechanism depend on the number of iterations? Does it persist with fewer or more NS steps? An analysis or ablation across different iteration counts would help clarify this relationship.

2. Sensitivity to polynomial coefficients.

The paper uses (a,b,c)=(3.4445,−4.7750,2.0315)(a,b,c)=(3.4445,-4.7750,2.0315). How would the negative response mechanism change with alternative coefficients, such as (2,−1.5,0.5)(2,-1.5,0.5) or (1.5,−0.5,0)(1.5,-0.5,0)? It would be interesting to understand whether the observed phenomenon strongly depends on the specific polynomial coefficients.

3. Prevalence of negative effective modes under GGN curvature.

The negative effective modes under positive semidefinite GGN curvature are interesting. However, how common are these modes across different training stages and model scales? In particular, do negative eigenvalues of K=αRHGGNK=\alpha R H_{\mathrm{GGN}} also appear in larger models, such as 1B-parameter LLMs? Would it be feasible to provide corresponding statistics?

4. Practical implications for LLM training.

While the theoretical analysis is interesting, the practical value remains somewhat unclear. How can these stability boundaries be used in real-world LLM training? Could the proposed analysis provide actionable guidance for understanding or improving Muon's training stability?

5. Computational overhead.

The proposed analysis relies on K=αRHK=\alpha RH. Although Hessian-vector products avoid explicitly constructing HH, they still introduce additional computational costs. Could the authors discuss the computational overhead of estimating the eigenvalues of KK, especially for billion-parameter models, and whether such analysis is practical at scale?

Questions for authors

Overall, I find this paper interesting and technically sound, with a meaningful motivation and rigorous theoretical analysis. My main questions concern the sensitivity of the negative response mechanism to NS iterations and polynomial coefficients, its prevalence in larger models, and its practical utility and computational cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.