acceptodds
Under review as a conference paper at ICLR 2027

Normalization Rigidity and Recovery in Matrix Momentum

Abstract

Matrix optimizers such as Muon normalize a momentum matrix and then change its singular values before updating the model. We study how this transformation responds to gradient noise. Symmetric noise can cancel when the true gradient is zero, while the average update still responds in the wrong direction when a gradient is added. We show that preventing this behavior for every symmetric noise distribution is equivalent to monotonicity of the update map. For methods that apply the same smooth function independently to Frobenius-normalized singular values, this requirement forces the function to be linear once there are at least three singular values. We explain how the shared normalization causes the problem and which gradient directions are affected. We also construct smooth convex tasks on which actual momentum updates keep the gradient away from zero over a rigorously verified time window. Convex spectral potentials provide another design choice: they preserve alignment with the gradient while still bringing singular values closer together. A smoothed nuclear norm additionally controls sensitivity to numerical errors. In our transformer experiments, however, all task-response estimates remain positive, and matched training favors Schatten over the smoothed map. These results identify a mathematical failure mechanism and its limits as an explanation of training behavior.

Primary area:
optimization
Keywords:
Matrix optimizers, muon, transformers

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kEB8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers a form of matrix-valued normalized updates which includes Muon-like iterations. The authors find that such a form may dot descent in a worst-case regime. Overall, I think this paper is based on an interesting observation, and therefore I lean towards an acceptance. However, I cannot provide a higher score since the realistic setting does not exhibit this damage.

Critical strengths
  1. Analysis around the Muon are pretty interesting.
  2. I think finding counterexamples for Muon is of great importance.
Critical weaknesses

I think this paper would benefit much with a polished writing. The message of this paper is clear, however, the organization and the wording of the paper is somehow confusing. Take the abstract as an example, the symmetric noise appears without any introduction. Similar situations happens many times. Besides, for the organization, I think the authors should first introduce the main results and then the details on how the result is constructed.

Questions for authors

I have no major concerns for the authors.

Reviewer_MQAyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is a theoretical paper studying the effect of Neuton-Schulz iteration in practical Muon implementation on the singular structure of the resulting parameter updates. Specifically, normalization with respect to the Frobenius norm creates a inter-dependency in the spectral modes of the noisy parameter updates. The authors theoretically & numerically prove that such an effect can drive the training dynamics in the opposite direction of the gradients. However, in a subsequent series of experiments on Wikitext-103 with a small transformer (~20M), the authors failed to demonstrate this existence of negative propagation and leave this discovery to future work.

Critical strengths
  • The problem of “post-conditioning” momentum causing interference between different spectral mode is a relevant topic worth exploring. I do not oppose to the significance of the problem setting itself.
  • The paper has a strong theoretical results. For example, I value Theorem 2 by its elegance, which has a beautiful results that if dimension is larger than 2, only the scaled Frobeinus-normalized descent guarantees alignment under symmetric noise.
Critical weaknesses
  • Section 9 Discussion is perplexing. I value the authors’ honesty, but the section says that the theoretical claims and assumptions did not appear in the designed small experiments. I think this significantly deteriorates the value of the theoretical works. I highly recommend to have a matching theory and real evidence.
  • The introduction can be made more valuable if it connects the main problem to a real experimental results with some visual demonstration. If there is an actual case that the authors are worrying about happening in real-world practice of Muon, then the whole discourse of this paper will gain a great worth. Currently, such potential value is not yet realized.
  • Although I agree on that having some level of rigor is beneficial, current version of the paper has more numbers than necessary. For example, numbers on lines 226-237 is acceptable, because it shows a concrete example of the failure mode the authors address. However, are these numbers deserve a Corollary (Corollary 4)? If these types of events happen frequently in practice, showing a concrete example with actual numbers is an excellent demonstration. However, as the authors have honestly discussed in page 9, without real-world counterparts, such numeric special cases seem superfluous to the overall discussion of the paper.
Questions for authors
  • A majority of discussion of the paper is based on symmetric noise distribution of the stochastic gradients. The symmetry assumption gives a level of beauty in the theory, but is this practically applicable? If there are some empirical or prior evidences that stochastic gradients are close to or at least approaches unbiased and/or symmetric statistics will greatly increase the significance of the theoretical results.
  • Is this analysis applicable beyond Muon? For example, many follow-up normalization works appear after Muon, such as Scion, Gluon, etc. Some examples like Adamuon mixes preconditioning with Muon-like projection. Can the theory be applicable to those as well?
Reviewer_fwXROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether normalization and spectral transformations in matrix optimizers preserve expected-update alignment under symmetric gradient noise. It connects universal alignment guarantees to monotonicity of the update map and establishes a rigidity theorem for normalized pointwise spectral maps. The authors also construct smooth convex problems whose actual EMA trajectories keep the gradient bounded away from zero over an explicit finite window.

Critical strengths

The rigidity theorem identifies a structural limitation of an entire class of normalized spectral maps, beyond particular polynomial choices or coefficients.

Critical weaknesses

Transformer experiments is not enough

Questions for authors

When responses are evaluated along training-objective gradients while accounting for momentum history and finite-signal effects, is there evidence of alignment failure that materially affects training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.