- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies an idealized Muon update using the exact polar factor, without momentum or weight decay. It derives a nuclear-norm dissipation identity and a finite-time convergence bound under an error-bound assumption. For isotropic Gaussian gradient noise, it calculates the local linear response of the smoothed polar map and its dependence on matrix shape. A local Ornstein–Uhlenbeck approximation then gives a stationary-variance comparison with SGD at matched contraction rates. Synthetic quadratic experiments illustrate the predicted drift, noise floor, and a proposed switch from Muon to SGD near the optimum.
Critical strengths
-
The Gaussian response coefficient is exact and useful. Theorem 6.1 identifies the local coefficient as . Its derivation through Gaussian differentiation and orthogonal invariance is sound, and the matrix-shape dependence provides a clear quantitative insight.
-
The matched-rate comparison addresses a meaningful question. Appendix A.11 accounts for the different natural learning-rate scales and derives an interpretable variance ratio for the local OU models. Figure 4 directly measures the predicted drift. These controlled experiments suit the paper's theoretical scope.
Critical weaknesses
-
The deterministic ODE can fail to have a solution. Definition 3.1 and equation (8) select the compact polar factor at rank-deficient gradients. Consider , with , , and irrelevant off-diagonal entries. This smooth convex objective satisfies . Starting at , the gradient is . Locally, must satisfy and , with . The identity forces , contradicting the prescribed derivative there. The energy calculation is valid along an existing trajectory. The flow formulation needs an existence argument, potentially through a differential inclusion. This is the main obstacle to acceptance.
-
The stationary comparison needs precise local-model scope. Theorem 6.1 is a small-gradient expansion, and Appendix A.11 derives stationary variances for the linearized OU model. The statement should specify positive curvature and the approximation involved. Figure 3 uses equal numerical learning rates and thus checks equation (35). The matched-contraction result is equation (36). At the stated parameters, the paper's OU approximation and square-matrix asymptotic predict RMS , leaving the accuracy of the local approximation uncertain. A measured regime diagnostic would help interpret this experiment. The exact local calculation remains a credible contribution.
Questions for authors
-
Clarification and mathematical repair: What solution concept addresses the rank-deficient example above? Please give the corresponding existence and finite-time statements and explain the relationship to the discrete polar update. A convincing repair is necessary for my support.
-
Clarification of scope: Can you state the stationary comparison explicitly at the approximation level justified by the proof, including positive-curvature assumptions? Please report a diagnostic of Figure 3's local regime and the hybrid switching threshold. Resolving this alongside Question 1 could move my recommendation to weak acceptance. A general invariant-measure theorem is optional.
-
Clarification of novelty: How do the matrix results extend Monzio Compagnoni et al., Adaptive Methods through the Lens of SDEs (ICLR 2025), which develops Gaussian-smoothed signSGD and local covariance analysis? A direct comparison would help assess the significance of the matrix-specific coefficient and shape dependence.