acceptodds
Withdrawn from ICLR 2027

Spectral Flattening Is Not Enough: When Muon Improves Learning Rates and Convergence

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Muon updates matrix-valued parameters by flattening the singular values of each gradient while preserving its singular directions. Despite its practical success, the mechanisms governing its learning-rate capacity and convergence remain insufficiently understood. We analyze deterministic, full-batch Muon through its exact polar update. First, we derive a descent-preserving learning-rate threshold and show that it is controlled by the block-size-weighted average singular scale of the gradient matrices. This characterization makes the role of gradient magnitude explicit and reveals how objective scaling affects the admissible step size. We then analyze Muon's convergence behavior and identify the spectral properties that govern its convergence rate. Our analysis shows that flattening the spectrum within individual parameter blocks is not sufficient to determine global convergence, because scale differences across blocks remain influential. Together, these results provide a theoretical account of how spectral flattening governs the learning-rate capacity and convergence behavior of Muon.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Optimization, Muon

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MNsWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper analyzes the descent-preserving learning-rate of Muon and gradient descent under the deterministic setting. The authors then compare these two descent-preserving learning rates and use experimental results to claim that Muon can achieve a larger descent-preserving learning rate. The authors also track the curvature ratios of Muon and GD in experiments, which is the key quantity in their convergence results. The experiment shows Muon has a substantially more favorable curvature ratio than GD under Kronecker curvature model.

Critical strengths

None

Critical weaknesses
  1. The main idea in this paper is to compare the descent-preserving learning-rate under the deterministic setting, which is the largest stepsize ensuring descent in every step. However, the choice of stepsize is quite different in deterministic and stochastic settings. In the stochastic setting, the noise would dominate the choice of stepsize. Also, in the deterministic setting, we always consider the best descent stepsize, not the largest descent stepsize.

  2. The content cannot support the authors' claims in the abstract and introduction; for example, they claim that spectral flattening is not enough and state their contribution as conditional theory for Muon versus SGD, where the whole work only considers GD.

Questions for authors

Questions

  1. How to transfer your results to the stochastic setting, where you claim you compare Muon versus SGD in the introduction. The choice of stepsize is quite different in deterministic and stochastic settings. In the stochastic setting, the noise would dominate the choice of stepsize.

  2. There is a mismatch between better convergence and the largest stepsize ensuring descent in every step. For example, in [1], the authors achieve a better convergence rate with the silver stepsize, which may increase the function value in some steps.

  3. Eq (9) does not hold everywhere.

  4. I think there is no use of PL condition, since you also assume strong convexity,

  5. There is a large collapse after 500 steps on the left of Figure 1. Could you provide the results after 500 steps?

  6. I am confused about why the results in Theorem 5 and Figure 2 suggest that spectral flattening is not enough. It seems larger curvature ratios can contribute to convergence, following the result in Theorem 4.

  7. The support of Rt>1R_t>1 is purely experimental, could you provide some cases where we can prove Rt>1R_t>1?

[1]Altschuler, J. M., & Parrilo, P. A. (2025). Acceleration by stepsize hedging: Silver Stepsize Schedule for smooth convex optimization. Mathematical Programming, 213, 1105–1118.


Writing problems

  1. Eq (29) is the same as Eq (30)
  2. There is a blank proof in A.6 PROOF OF LEMMA 1
Reviewer_AmqvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies deterministic, full-batch Muon without momentum through its exact polar update. It first compares sufficient one-step descent thresholds for Muon and gradient descent, obtaining a ratio equal to a block-size-weighted average of gradient singular values. It then analyzes Muon as a gradient-dependent preconditioned method and compares curvature ratios under a block-diagonal Kronecker model. A sufficient condition in Appendix A.9 characterizes when the preconditioned curvature ratio improves across multiple parameter blocks. Experiments report spectral diagnostics during GPT training and compare Muon and SGD at selected learning rates.

The question is relevant, and the distinction between within-block spectral flattening and global conditioning is useful. However, I have concerns about the validity and applicability of some theoretical guarantees, the significance of the scalar learning-rate comparison, and whether the experiments establish the predictive value claimed in the introduction.

Critical strengths
  1. The multi-block perspective is useful. The paper highlights that improving the spectrum within individual blocks does not necessarily improve the conditioning of the joint problem. Cross-block scales can change which blocks determine the global spectral endpoints.

  2. Several important distinctions are handled carefully. In particular, the paper distinguishes the representation-gradient second moment CiC_i from the weight-gradient Gram matrix BiB_i. The general transformation is correctly written as C~i=Bi−1/4CiBi−1/4,\widetilde C_i=B_i^{-1/4}C_iB_i^{-1/4}, without assuming that it always equals Ci1/2C_i^{1/2}. The spectral identities in Theorem 5 and the argument in Corollary 1 appear correct under their stated assumptions.

  3. The manuscript explicitly acknowledges several limitations. It distinguishes the Kronecker model from the actual network Hessian, states the additional fixed-metric assumptions needed for nonlinear objectives, and identifies Fig. 2 as a regularized diagnostic rather than a direct verification of the positive-definite theorem. Measuring spectral quantities in a Transformer is also a useful empirical component.

Critical weaknesses

1. The descent guarantee in Eq. (9) is not justified by the stated assumptions

Section 4.2 assumes twice continuous differentiability, while Appendix A.1 derives Eq. (9) by dropping the Taylor remainder at the current iterate and then applying a Rayleigh bound. A current-point Hessian does not, by itself, provide a finite-step upper bound on the loss.

For example, consider the strongly convex scalar function f(w)=w+12w2+w4.f(w)=w+\frac12w^2+w^4. At w=0w=0, both the gradient and Hessian equal one. The proposed current-Hessian threshold therefore permits η=1<2\eta=1<2. Both GD and the scalar exact-polar update move to −1-1, but f(−1)=12>f(0)=0.f(-1)=\frac12>f(0)=0.

The result can be repaired by explicitly assuming a common curvature upper bound along the entire update segment, or appropriate regional smoothness. Under that correction, the subsequent nuclear-norm algebra can remain valid. However, the learning-rate guarantees are not established as currently stated. The distinction between a local quadratic prediction and a guaranteed descent threshold needs to be maintained throughout Section 4.2.

2. The uniform convergence specialization in Eqs. (28)–(30) appears vacuous under the stated infinite full-rank trajectory assumptions

The product bound in Eq. (27) follows from the stated fixed-metric assumptions. The subsequent uniform specialization requires separate bounds

\le \sup_t\widetilde\beta_t<\infty.$$ Let $M=\sum_i m_i$. For full-rank exact-polar updates, $$\|o_t\|_2=\sqrt M.$$ Together with the prescribed learning rate $\eta_t=1/\widetilde\beta_t$, a finite uniform upper bound $\overline\beta$ on $\widetilde\beta_t$ implies $$\|W_t-W_{t-1}\|_2 \ge \frac{\sqrt M}{\overline\beta}>0.$$ However, the claimed geometric decrease, combined with the fixed-metric strong convexity already assumed on the common region $\Omega$ at the first iteration, implies $W_t\to W^\star$. Consecutive step lengths must then vanish, yielding a contradiction. This concern is specific to the infinite-horizon uniform specialization; it does not invalidate the one-step argument or Eq. (27). A uniform lower bound on $\widetilde\alpha_t/\widetilde\beta_t$, allowing $\widetilde\beta_t\to\infty$, would be a natural alternative. The treatment of rank deficiency also needs clarification. Even for $$L(W)=\tfrac12\|W\|_F^2,\qquad W_0=\operatorname{diag}(2,1),$$ the step prescribed by Theorem 4 gives $W_1=\operatorname{diag}(1,0)$. The gradient becomes rank deficient before reaching the optimum, so the inverse-square-root preconditioner is no longer defined. This illustrates that full rank is not automatically maintained even on a simple quadratic. #### 3. The scalar learning-rate comparison has limited scale-independent optimization significance The paper correctly acknowledges objective scaling, but its implications for the main contribution need stronger treatment. Under $L_c=cL$, with $c>0$, $$G_c=cG,\qquad \operatorname{polar}(G_c)=\operatorname{polar}(G),\qquad R_c=cR.$$ Thus, the condition $R>1$ can be reversed by changing the units of the objective. Meanwhile, dividing the GD learning rate by $c$ preserves its trajectory, and the exact-polar direction remains unchanged. Consequently, the observation that $R_t>1$ at 99.8% of steps under a summed objective does not establish a scale-independent optimization advantage. This limitation can also be seen within the paper’s common Euclidean smoothness bound. Write $$S=\sum_i\|G_i\|_*, \qquad M=\sum_i m_i.$$ Optimizing the quadratic descent bound over the scalar step size gives guaranteed decreases $$D_{\mathrm{Muon}}=\frac{S^2}{2\beta M}, \qquad D_{\mathrm{GD}}=\frac{\|g\|_2^2}{2\beta}.$$ Since $S^2\le M\|g\|_2^2$, this particular bound does not yield a stronger progress guarantee for Muon. This observation concerns the explanatory power of the analysis, rather than ruling out actual Muon advantages. Comparisons involving directional curvature, achieved loss decrease, update distance, or independently tuned optimization trajectories would make the learning-rate contribution substantially more informative. #### 4. The empirical section does not directly test the proposed cross-block sufficient condition The introduction promises controlled changes to the spectral quantities and regimes in which the conditions weaken or fail. However, the sufficient condition in Eq. (59) is not evaluated. Figure 2 reports its desired consequence—a larger regularized curvature ratio—without establishing whether Eq. (59) holds or whether changing cross-block scales changes the observed behavior as predicted. A small positive-definite multi-block quadratic experiment would provide a direct and inexpensive test. It could independently vary within-block spectra and cross-block scales, report the condition margin, and compare predicted curvature ratios with actual one-step and trajectory-level progress. Since the condition is sufficient, its failure need not imply that Muon performs worse; nevertheless, examining both sides would clarify its information content and scope. The curvature diagnostic also needs a more complete account of regularization. Each probe uses only 1,024 tokens. For a 3,072-dimensional MLP factor, as in the 12-block architecture specified in Appendix B, its empirical covariance has rank at most 1,024. Such singularity is structural. Please provide the complete matrix shapes used in Section 5.2 and distinguish structural nullspaces from numerical conditioning. I appreciate the qualification on p. 9 that Fig. 2 is not a direct theorem verification. However, the paper does not sufficiently specify which factors are eigenvalue-floored, the reference scale, the order of regularization relative to inverse powers, or sensitivity to the $10^{-6}$ floor. These choices could materially affect the reported 50–2,000-fold improvement. Moreover, the absolute curvature ratios are approximately $10^{-15}$–$10^{-12}$. Substitution into the proposed contraction bound would remain extremely loose over the reported training horizon. Additional evidence is needed to connect this diagnostic to observed optimization speed. #### 5. The optimization comparisons do not establish an advantage after independent learning-rate tuning In the NoNorm experiment, every tested SGD learning rate diverges at step 3. This demonstrates a difference over the selected grid, but no stable SGD baseline is identified. Extending the SGD search to find its stable and effective regime is necessary for a convergence-speed comparison. The normalized experiment compares the methods at the same scalar learning rates and reports only the three smallest values in the sweep. SGD validation loss continues improving as the learning rate increases from 0.03 to 0.3, whereas Muon validation loss worsens. This suggests that the two methods require different tuning ranges. Please report the complete sweep and compare separately selected learning rates, preferably using steps to a fixed objective value. There is also a gap between the analyzed and implemented updates. The appendix uses finite Newton–Schulz iterations, rectangular matrix rescaling, and a reference AdamW optimizer for other parameters. These choices are reasonable, but the theoretical thresholds are derived for the specified exact-polar joint update. The paper should state the precise correspondence, including the effect of block-dependent rescaling. A controlled experiment matching the theoretical update would help resolve this issue. #### 6. The contribution relative to existing theory needs sharper positioning Much of the development consists of standard smoothness/PL descent arguments and spectral identities for Kronecker and block-diagonal matrices. The most distinctive component appears to be the cross-block sufficient condition, whose predictive value is not yet established. The relationship to Davis and Drusvyatskiy, *When do spectral gradient updates help in deep learning?* (arXiv:2512.04299), should be made more explicit, particularly regarding sufficient conditions for spectral versus Euclidean progress. Gonon et al., *Insights on Muon from Simple Quadratics* (arXiv:2602.11948), is also relevant to the limitations of local proxies and the distinction between exact and approximate polar dynamics. These results do not automatically subsume the present work, but they help define the additional explanatory contribution that needs to be demonstrated.
Questions for authors
  1. Is λmax⁡H\lambda_{\max}^H in Section 4.2 intended to be a uniform upper bound over the update segment, rather than the eigenvalue of the current Hessian? How should Appendix A.1 be revised accordingly?

  2. How can the infinite-horizon uniform assumptions in Eqs. (28)–(30) be satisfied while exact-polar step lengths vanish? Would a uniform bound on the curvature ratios provide the intended result? How are rank-deficient gradients handled?

  3. Does Eq. (59) hold in the reported experiments? Can you report its margin and provide controlled examples varying cross-block scales?

  4. How sensitive are the curvature-ratio conclusions to the eigenvalue floor, probe size, and treatment of null directions?

  5. What are the results with independently tuned GD and Muon learning rates? Please also specify the backward loss-reduction convention in the NoNorm experiment and the exact rectangular rescaling used.

Reviewer_fDsGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how the spectral flattening is related to the larger learning rate and faster convergence. Toward this, the authors consider the momentum-free exact polar update for analysis. Using a single-step descent, this paper explains that the advantages of Muon depends on the absolute scale of the gradient singular values, and the LR could vary according to objective scaling. In convergence analysis, the authors provide the conditional bound under the uniform smoothness and strong-convexity in the fixed metric at each step. Next, this paper shows that in the Kronecker block curvature model, spectral flattening within the block alone does not resolve the scale difference between blocks. Then, the authors derive a sufficient condition in which the global curvature ratio improves. Lastly, this work present a comparison of spectral diagnostics and training in small-scale GPT experiments.

Critical strengths
  1. The paper clarifies the dependence of Muon's admissible learning rate on the absolute scale of the gradients and objective function. This provides an interesting perspective on why a larger learning rate does not necessarily imply a scale-independent optimization advantage.

  2. The authors distinguish spectral flattening in individual parameter blocks from the global curvature across blocks. Specifically, this accounts for the difference between the gradient Gram matrix and the curvature factor. Also, it provides a sufficient condition for improving the curvature ratio under a multi-block Kronecker model.

Critical weaknesses

I have several concerns about the paper.

  1. To my understnading, all theoretical claims (theorem 1,2,3) are very classical and easy to derive, thus I could not capture what the main theoretical contribution of this paper is. Specifically, Theorem 1 is highly well-known results and Theorem 2 is nothing but the extension of the Muon update (even with momentum-free exact polar). In this context, the condition for a larger Muon learning rate is very obvious. Could the authors explain what the theoretical contributions of this paper are?

  2. Although the smoothness assumption (A1) is very standard in optimization literature, the global PL condition (A2) is not generally guaranteed for neural network training (in fact, (A2) is also prevalent but is not practical I think). More importantly, Theorem 4 requires additional fixed-metric strong convexity assumptions at each iteration, which appear stringent. Given that the paper is motivated primarily by LLM training, could the authors explain under what circumstances these assumptions are expected to hold?

  3. I am not convinced that the proposed spectral ratios provide practically useful insights into neural network optimization. Though the authors measure these quantities during GPT training, it remains unclear whether they reliably predict the largest stable LR or actual convergence speed. Moreover, some quantities require expensive spectral computations, while the LR ratio is sensitive to objective scaling. In this context, I think that the authors should clarify the practical implications of these conditions and demonstrate if they provide predictive information.

Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.