acceptodds
Under review as a conference paper at ICLR 2027

Beyond Singular Values: Geometry and Numerical Fidelity in Muon

Abstract

When does spectral flattening improve the loss decrease from a matrix update? We study this question for Muon through rotations that preserve the candidate spectrum, update norm, and entire ideal polar update while changing the orientation of a spectrum-preserving comparator. On one fixed objective, candidates with identical spectra and gradient alignments can favor opposite updates; a local theorem identifies the curvature projection responsible for this variation. At finite step sizes, the gradient at the comparator endpoint determines the first-order response to a small rotation and accounts for curvature along the update path. Tests span 30 frozen state–block combinations from three 124-million-parameter language-model trajectories. In the first layer, scalars fitted on old text recover most of the endpoint predictor's gain. In middle and final layers, the same-objective endpoint expansion remains accurate, while calibration-to-evaluation differences dominate prediction error. We then separate the exact five-round spectral map from its floating-point implementation. In a first-layer same-input test with raw output lengths, single precision (FP32) reduces paired-response squared error by 99.97% compared with brain floating point (BF16). FP32 preserves the ideal response sign in all 36 first-layer and 24 deeper-layer constructed directions; the high-precision finite-round map reproduces the remaining random-direction reversals. These results distinguish orientation and path effects from data-transfer error and numerical distortion at the intervention scale.

Primary area:
optimization
Keywords:
Muon, matrix optimization, polar decomposition, loss curvature

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Bui9OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the geometric and numerical factors underlying the loss reduction achieved by Muon's spectral flattening. It makes three main contributions.

  1. The authors introduce a fixed-polar construction that preserves the candidate's singular values, update norm, gradient alignment, and ideal polar update while varying the orientation of its singular vectors. Through a constructed quadratic example and a local theoretical result, they demonstrate that these quantities alone are insufficient to determine whether spectral flattening improves the loss, highlighting the role of curvature and singular-vector orientation.

  2. The paper proposes using endpoint gradients to better predict the loss response to small orientation perturbations at finite step sizes. Experiments show improved predictive accuracy in early layers, although much of the improvement can be recovered through simple scalar calibration, and generalization across evaluation data is limited in deeper layers.

  3. The authors investigate numerical distortions introduced by finite-round Newton–Schulz iterations and floating-point arithmetic. They show that FP32 substantially improves the fidelity of the measured loss responses compared with BF16, while the remaining discrepancies are largely attributable to the finite-round approximation itself.

The authors validate these findings through analytical constructions, controlled quadratic experiments, and one-step loss measurements on 30 frozen state–block combinations from three training trajectories of a 124M-parameter Transformer.

Critical strengths
  1. Insightful problem formulation. The fixed-polar construction offers an interesting perspective on Muon's many-to-one mapping, raising a meaningful question about whether spectral information alone can explain the benefits of spectral flattening.

  2. Rigorous theoretical analysis. The authors provide a nontrivial mathematical construction demonstrating that identical spectral information and gradient alignment can lead to opposite preferences between Muon and its comparator, highlighting the role of curvature and orientation.

Critical weaknesses
  1. Limited generality and practical relevance. The preference reversal is demonstrated on a carefully constructed quadratic objective. Although the authors provide a local theoretical generalization, its prevalence and significance in realistic LLM training remain unclear. The experiments focus on one-step loss responses rather than long-term training dynamics.

  2. Limited significance of the endpoint-gradient analysis. The improved local prediction follows naturally from a Taylor expansion at the update endpoint. Simple scalar calibration recovers most of the reported gains, and the analysis provides limited actionable insights for understanding or improving Muon.

  3. Unclear optimization implications of numerical fidelity. The authors demonstrate that low-precision Newton–Schulz computations can distort the ideal response, but do not establish whether these deviations improve or degrade actual training performance. In particular, it remains unclear why closer approximation to the exact polar factor should necessarily lead to better optimization.

Questions for authors
  1. How prevalent and significant are the observed effects in realistic LLM training? Specifically, how frequently does the norm-matched comparator outperform Muon, i.e., (A(C)<0)(A(C)<0)? How large are the orientation-induced changes (∣A(CR)−A(C)∣)(|A(C_R)-A(C)|) relative to the baseline advantage (∣A(C)∣)(|A(C)|)?
  2. How sensitive are the conclusions to the choice of Frobenius-norm matching? Given that Muon's polar update is associated with steepest descent under a spectral-norm constraint, would the conclusions change under spectral-norm matching?
  3. To what extent does one-step loss advantage predict long-term training behavior? Since the experiments focus on frozen model states, i wonder whether the observed orientation-dependent effects translate into meaningful differences in convergence. Could the authors provide evidence connecting these local observations to actual training dynamics?
  4. Why should a closer approximation to the exact polar factor necessarily lead to better optimization? The spectral distortions introduced by finite round NS iter may not always be detrimental. Have the authors investigated whether these deviations improve or degrade actual loss reduction or training convergence?
Reviewer_3rDYOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies when Muon’s spectral flattening helps and when it might hurt. The authors answer this question by investigating the orientation of the update’s singular directions relative to the loss curvature. Their main theoretical result is that changing this orientation alone can reverse whether flattening helps, even when the singular values stay the same. The experiments range from toy problems with known curvature to checkpoints of a small GPT model. In both settings, they rotate the singular directions of the momentum with its spectrum fixed, and measure how this changes the benefit of Muon's spectral flattening. Results agree with their theory. They also show that the BF16 Newton-Schulz used in practice is too coarse to measure such small effects.

Critical strengths
  1. The analysis setup is well constructed to facilitate a clean comparison for the study of the alignment with the curvature.
  2. The numerical experiments for verifying the analysis tools are designed with care, especially the consideration of precision, which can cause the gap between the object of the study and the actual practical implementation.
Critical weaknesses
  1. The main theoretical insights and results almost directly follow the second-order view that the paper starts with. Once the loss change is written as a first-order gain plus a curvature cost, it is already clear that the curvature cost depends on not the singular values alone but how the directions line up with the curvature. The theory of this paper only makes this explicit, but not much beyond it. There is also no disucssion of how such theoretical insights could be useful in practice for improving the effectiveness or efficiency of Muon or the training dynamics.
  2. Similarly, the experiments of this paper are mostly for verifying the analysis tools, not to say anything about training practice. For example, the toy problem experiment confirm that the benefit of spectral flattening can be reversed under a well constrcuted condition where the curvature is fully known; the precision study, which takes a significant part of the paper, is merely to show that BF16 Newton-Schulz used in practice is not accurate enought to manifest the tiny loss differences created by their own perturbations, which is expected at this scale.
  3. The paper can greatly benefit from improving the writing style. I understand that the authors have claimed using AI to polish the writing, which adheres to the ICLR policy, but this does not make excuses of a poorly written/organized paper from an academic perspective. Takes the abstract as an example, it simply enumerates results one after another, but rarely states the takeaway, why it matters, or how it connects to the question the paper asks. From the audience view point, it is very difficult to follow the paper when many terms are used without proper definition and introduction, such as “spectrum-preserving comparator”, “endpoint predictor” and “calibration-to-evaluation differences”. The main text follows the same pattern. It defines many new terms and reports a lot of low-level measurement details, without building a coherent logic flow that connects each other, or reasoning the necessity and the broad impact of each result.
Questions for authors
  1. The paper asks when spectral flattening helps versus their unflattening control variant, but never reports how they compare in real training. At the checkpoints studied and with the step size Muon actually uses, does flattening lower or raise the loss compared with the unflattened update? Does this change during training? How often?
  2. The paper shows the benefit of spectral flattening can reverse on a well constructed toy example, but does it also happen in real networks? In a real network the curvature is tied to the layer inputs, so it is not clear whether such cases arise naturally. Some related works look at similar questions from other angles [1][2], especially [2] uses a similar second-order perspective. Can the authors elaborate how this paper differs from these two related works?

[1] Davis, Damek, and Dmitriy Drusvyatskiy. "When do spectral gradient updates help in deep learning?." arXiv preprint arXiv:2512.04299 (2025). [2] Wang, Shuche, et al. "Why Muon Outperforms Adam: A Curvature Perspective." arXiv preprint arXiv:2606.04662 (2026).

Reviewer_Uj95OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work studies when spectral flattening improves a matrix update by constructing rotations that preserve the candidate spectrum, update norms, gradient alignment, and ideal polar endpoint. It identifies the remaining dependence on curvature, develops endpoint-gradient predictions for finite-step responses, and experimentally separates finite-iteration approximation error from floating-point error in Muon.

Critical strengths

This paper studies an interesting question, and contains clear and well defined experimental settings. E.g. the same-objective quadratic preference reversal demonstrates that spectrum and initial gradient alignment are insufficient. I also like the careful numerical analyses.

Critical weaknesses

The practical implications remain unclear. The main neural-network experiments consist of frozen-state, single-block interventions on a 124M model across three training trajectories, using very small constructed orientation perturbations. Their relevance to ordinary optimizer trajectories and larger-scale training is therefore not established. Improved fidelity to these paired loss responses does not by itself imply better training loss or efficiency. Moreover, scalar recalibration recovers most of the first-layer prediction gain, while calibration-to-evaluation differences dominate prediction error in deeper layers. Although the analysis identifies what controls local changes in flattening advantage, it does not establish a broadly validated practical criterion for deciding when to flatten an unmodified training update, or demonstrate an optimizer modification that improves training performance.

Questions for authors

How do the loss-level conclusions change at larger orientation perturbations and on larger models? Can the analysis provide a reliable criterion for deciding whether flattening benefits ordinary, unmodified training updates? Do the precision or approximation changes that improve paired-response fidelity also improve end-to-end training loss or efficiency?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.