acceptodds
Under review as a conference paper at ICLR 2027

Muon is Not That Special: Random or Inverted Spectra Work Just as Well

11 views

Abstract

The recent empirical success of the Muon optimizer has renewed interest in non-Euclidean optimization, typically justified by similarities with second-order methods, and linear minimization oracle (LMO) theory. In this paper, we challenge this geometric narrative through three contributions, demonstrating that precise geometric structure is not the key factor affecting optimization performance. First, we introduce Freon, a family of optimizers based on Schatten (quasi-)norms, powered by a novel, provably optimal QDWH-based iterative approximation. Freon naturally interpolates between SGD and Muon, while smoothly extrapolating into the quasi-norm regime. Empirically, the best-performing Schatten parameters for GPT-2 lie strictly within the quasi-norm regime, and thus cannot be represented by any unitarily invariant LMO. Second, noting that Freon performs well across a wide range of exponents, we introduce Kaon, an absurd optimizer that replaces singular values with random noise. Despite lacking any coherent geometric structure, Kaon matches Muon's performance and retains classical convergence guarantees, proving that strict adherence to a precise geometry is practically irrelevant. Third, having shown that geometry is not the primary driver of performance, we demonstrate it is instead controlled by two local quantities: alignment and descent potential. Ultimately, each optimizer must tune its step size around these two quantities. While their dynamics are difficult to predict a-priori, evaluating them within a stochastic random feature model yields a precise insight: Muon succeeds not by tracking an ideal global geometry, but by guaranteeing step-size optimality.

Primary area:
optimization
Keywords:
Optimization, matrix preconditioning, non-Euclidean optimization, Muon, Schatten, QDWH, rational approximation, LMO

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_SCapOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that the success of Muon is not explained by the spectral-norm LMO geometry it is usually attributed to. It proposes two optimizers. The first one, Freon, is an optimizer with updates (GGT)−cG(GG^T)^{-c}G that interpolates between SGD (c=0c=0) and Muon (c=1/2c=1/2), and can extrapolate beyond c=1/2c=1/2. On GPT-2 the best c is reported to lie in (1/2,1](1/2, 1], which lies outside the range of unitarily invariant norms. The paper also shows that c=1c=1 is equivariant under layerwise symmetries. Kaon is an optimizer that replaces the gradient's singular values with (pseudo)random values, produced by a chaotic polynomial iteration. It is reported to match Muon on GPT-2 (124M). The paper gives a convergence guarantee for a class of preconditioned spectral descent updates that includes Kaon and Freon under regularization. It also characterizes which such updates are steepest descent for some unitarily invariant norm.

The authors perform a local theoretical analysis based on Taylor expansion that splits the one-step loss change into batch gradient alignment and directional descent potential. The paper then argues that Muon wins mainly because its optimal step size is stable. Empirical support comes from one-step loss-change heatmaps for GPT-2 and RF experiments.

Validation consists of GPT-2 124M sweeps on WikiText and NanoGPT+FineWeb runs, one-step loss-change heatmaps, and RF experiments. The paper concludes that a precise geometry is practically irrelevant.

Critical strengths
  1. The empirical results are potentially surprising. Many Freon and Kaon runs match Muon on GPT-2 (Fig. 4, Table G.1). This forces the community to reconsider how much of Muon's performance is owed to the exact spectral-norm LMO. It would be a solid contribution if framed as "spectral flattening matters and the exact profile mostly does not". The TruncatedSGD comparison supports this, since suppressing large singular values helps but is not enough.
  2. The paper gives a concrete framework to build on. Theorem 2.1 shows that c>1/2c>1/2 updates cannot be steepest descent for any unitarily invariant norm, which is a limitation of the LMO view. The RF analysis shows that stable step size is an important mechanism. Together these give others something specific to test and extend.
Critical weaknesses
  1. The main conclusions are overstated and do not follow from the evidence, for example:

    • Kaon is not so much "an absurd optimizer that replaces singular values with random noise" as it is Muon (with noise) in expectation. In Theorem 2.5 the entries of EkE_k are i.i.d. and positive, so by exchangeability E[pk]E[p_k] is a constant vector. The expected update is therefore proportional to the polar factor UVTUV^T, and so the algorithm seems to be just stochastic Muon with spectral noise, and claims such as "zero coherent geometry" are overstated. Theorem 2.5 can be reduced to the standard noisy-LMO descent argument. Practical Muon (NS-5) is itself far from the exact LMO (33.5° in Table G.2). Inexact-Muon theory (Shulgin et al.) is cited but never compared against.
    • The data support a weaker claim than "geometry is practically irrelevant". For example, Figure 4 shows that c>0c>0 all behave similarly once the spectrum is reasonably flat, and that TruncatedSGD is insufficient. The matching spread in Table G.2 (0.31 versus 0.33) also points to flattening. The decisive ablations are missing, e.g., heavy-tailed random spectra and spectra with very different spread.
    • "optimal cc lies strictly outside the range of any proper Schatten norm (c>1/2c>1/2)" is not established. It is a one-step greedy optimum over a one-parameter family with a shared LR. The authors themselves report that greedy cc hurts long-term dynamics (Fig. 5). Fig. J.7 shows different near-optimal ranges 0.00-0.21, 0.00-0.26 and 0.00-0.42 for many W_QKV blocks. In Fig. J.2 the c=1/2c=1/2 curve also appears to end at or below the c=3/4c=3/4 and c=1c=1 curves. The text is muddled about which model shows what. Section 3.3 says RF optimal exponents "frequently fall into the strict non-norm regime (c>1/2c>1/2)". Fig. J.3 is the synthetic RF experiment, where the optimum is below 1/21/2. Contribution 1 says the opposite for RF.
    • The J.5 caption cites an "inter-batch SNR analysis of Section 3.2" that does not seem to exist.

    Overall, I believe that the strong wording in the title and abstract ("not that special", "proving", "conclusively", "practically irrelevant") is not supported. What the evidence supports is more like spectral compression/flattening matters and the exact profile mostly does not.

  2. The experimental protocol does not fully support the claims, for example:

    • The LR grid is truncated. All four optimizers select ηmatrix=2e−2\eta_{matrix} = 2e-2, the largest value in the grid (Table A.1). Fig. 3 plots Kaon at ρ=3\rho=3 while the others use ρ=1\rho=1. So the LRs are not properly tuned and the choice of plotted runs seems post hoc.
    • Freon's scale depends on gradient magnitude. For c≠1/2c\neq1/2, Algorithm 4 returns ν1−2c(GGT)−cG\nu^{1-2c}(GG^T)^{-c}G, which has degree 1−2c1-2c in G. Its step size therefore scales with gradient magnitude and interacts with clipping at 1.0. This conflicts with Eq. (2), which is scale-free, and with the hyperparameter-transfer motivation of Section 2.3.1. Since the paper's thesis is that step size is decisive, this may confound the comparisons.
    • The SVD variants use Kaon’s LR rather than their own.
    • Training-length bookkeeping is inconsistent. For example, J.1 labels step 3594 as 50%, J.8 calls the same step 25%, J.4 implies 14000 total steps, and J.1 implies about 7200.

    Overall, comparisons between Muon, Kaon and Freon concluded from this protocol may not be entirely fair. Differences reported (or the absence of differences) are within what LR-grid edge effects and seed noise could produce.

Questions for authors
  1. Could you explain the normalization of Freon for c≠1/2c\neq1/2? How does this interact with clipping, and do the conclusions change with a scale-free version?
  2. The selected ηmatrix\eta_{matrix} is at the grid boundary for all optimizers. Do the conclusions still hold after proper tuning?
  3. Theorem 2.7 is stated for all a/b∈Qa/b \in Q, and its proof is completed by Theorem F.8. The proof of Theorem F.8 uses le≤σmine≤σiel^e \leq \sigma_{min}^e \leq \sigma_i^e with e=(2a−b)/be=(2a-b)/b, which needs a/b≥1/2a/b \geq 1/2. This condition is not assumed. For a/b<1/2a/b < 1/2, which includes the c=1/4c = 1/4 used in the experiments, the explicit constant 1/le1/l^e is not justified by this argument. Could you give a corrected argument?
  4. Table G.1 lists "Kaon, λ=4.1\lambda=4.1" and "Kaon (main experiment)" with different numbers. What differs between them?
  5. The abstract and Fig. 4 caption call c∈[1/2,1]c \in [1/2, 1] the quasi-norm regime, but with c=1−q/2c=1−q/2 and 1/p+1/q=11/p+1/q=1 this range corresponds to p<0p<0. Can you clarify what is meant?
  6. Could you compare against inexact-Muon theory (e.g. Shulgin et al.)?

I would be happy to reconsider my score if the authors address the weaknesses and questions raised above.

7 comments

We thank the reviewer for a careful review. In response we extended the learning-rate sweep beyond the original grid, ran the requested spectral ablations with a separate sweep for each (new Table G.2), and toned down some of the wording. The clarified claims are: matching an LMO geometry is not necessary for Muon-level performance; within mild spreads the exact spectrum matters little; and keeping the gradient's ordering is not needed and can hurt.

  • W1.1. We agree that Kaon in many ways is not absurd, our own Theorems 2.3 and 2.5 show precisely this. It only appears absurd from the perspective of a prescribed LMO or norm geometry, as no single step can come from a unitarily invariant gradient independent LMO. That Kaon is “Muon in expectation” does not count against our claim; it is our point: matching an LMO is not necessary. What is more, that every i.i.d. row of Table G.2 is Muon in expectation, yet their losses range from 3.3315 to 3.6708. So being Muon in expectation does not decide performance. For independent, exchangeable replacement singular values, the expected update is indeed proportional to the polar factor, as the reviewer notes. But an expectation over one step does not make the resulting optimization trajectories identical: after an update, the weights and subsequent gradients differ, so the original exchangeability argument does not identify the later trajectories. On inexact-LMO theory, see Q6.
  • W1.2. Firstly, we want to clarify the reading of Figure 4 – Freon with c=1c=1 creates a largely non-flat spectrum – see Figure 1. More generally, we do not believe that flattening of the spectrum explains our results, and the new ablations in Table G.2 test this directly. In each variant, we take the SVD of the momentum, keep its singular vectors, and replace its singular values. Each variant has its own learning-rate sweep.
    • Flatness is neither sufficient nor necessary: Sorting uniform draws to follow the order of the momentum's singular values increases the final loss by 0.027. Similarly, normalized SGD is 0.16 worse than the same singular values in a random order. Within each pair, the update has the same singular values (for the uniform pair, they follow the same distribution), and therefore the same flatness, yet the results differ substantially. Conversely, Freon with c=1 produces a strongly non-flat, inverted spectrum (Figure 1), yet trains close to Muon (Figure 4c).
    • Suppressing dominant directions helps: We agree with the reviewer on this point, as TruncatedSGD shows. This is precisely the point of Note 2.2. By Theorem 2.1, an LMO of a unitarily invariant norm must preserve the order of the singular values, and therefore cannot suppress these directions. The ordering ablations confirm that preserving the order performs worse.
    • The shape of the spectrum matters once its spread is large: Mild random spectra (U[0,1], Beta(2,5), Pareto with α=3\alpha=3, and lognormal with s<0.5s<0.5) are within 0.0034 of exact-SVD Muon, and at or below tuned Newton–Schulz Muon. Larger spreads increase the final loss, and the increase grows with the spread: by 0.012, 0.085 and 0.34 for lognormal with $ s=1,2,4 $, and by 0.030 for Pareto with α=1.5\alpha=1.5. Finally, the similar spread of Newton–Schulz Muon and Kaon in Table G.4 is consistent with both explanations, and it is the ablations above that distinguish between them.
  • W1.3. We want to clarify a number of points regarding reviewers comments.
    • Regarding greedy-cc: Figure 5 does indeed show that greedy-cc underperforms compared to even Muon. This is not general: Figure I.1 (same setting, but SwiGLU features) shows a setting where greedy-cc significantly outperforms Muon.
    • More importantly, Figure 5 and I.1 are both for a convex random-feature model, in which c>1/2c>1/2 can never be optimal and is never selected.
    • Only in GPT-2 does c>1/2c > 1/2 become favourable. This disagreement between simple models and real networks is what Contribution 1 states (“unlike simple quadratics”). It is not only a one-step effect: in full training (Figure 4c), every c>1/2c > 1/2 still trains at η≈3⋅10−2\eta \approx 3\cdot10^{-2}, where every c≤1/2c \le 1/2 diverges. Among the one-step optima (Fig. J.7), c>1/2c > 1/2 is optimal in 42 of 48 matrices; the WQKVW_{QKV} blocks cited are the exception. We agree that the text was unclear, as it uses "RF" for two different settings, and will rewrite the paragraph to make this distinction explicit.
  • W1.4. Thank you for pointing this out – the caption referred to a section that no longer exists. It now refers to the step-size SNR analysis in Figures J.1 and J.11. We have also softened the wording.
  • W2.1 and Q2. The choice of ρ\rho in Fig. 3 was not post hoc. Fig. 3 shows each optimizer at its best ρ\rho from the sweep in Fig. 2, where all four optimizers were swept over the same eight values; Kaon’s best was ρ=3\rho = 3 and the others was ρ=1\rho = 1.
  • W2.1 and Q2 (continued) The reviewer is right that the grid of Table A.1 ended at ηmatrix=2⋅10−2\eta_{\text{matrix}} = 2\cdot10^{-2} and that all four optimizers selected it. Fig. 2 goes beyond it only by scaling both rates together. We have therefore extended the sweep, raising the matrix rate on its own and retuning the base rate, with rates selected on one seed and losses reported over three (new Table G.2,G.3): | Optimizer | Table A.1 rates | Retuned | |---|---|---| | Muon | 3.3373$\pm$0.0007 | 3.3350$\pm$0.0015 | | Kaon | 3.3456$\pm$0.0015 | 3.3346$\pm$0.0009 | | Freon c=2/3c=2/3 | 3.3452$\pm$0.0020 | 3.3397$\pm$0.0002 | | Freon c=3/4c=3/4 | 3.3580 (seed 42) | no improvement |

With both tuned, Kaon and Muon are indistinguishable (3.3346±0.00093.3346 \pm 0.0009 against 3.3350±0.00153.3350 \pm 0.0015) and Freon remains slightly behind after tuning, with c=3/4c=3/4 being 0.023 worse (seed 42 only). Kaon therefore matches Muon after tuning, while Freon is close to, but slightly behind, Muon on NanoGPT. The SVD variants now also have their own sweeps, with the matrix rate extended to 0.20.2. These runs used one GPU with gradient accumulation and the same batch per step, so we report them alongside, not mixed with, the numbers elsewhere in the paper.

  • W2.2 and Q1. The actual Freon update is shown in equation (2) and is scale-free. Algorithm 4 specifically shows the algorithm used for approximating (GG⊤)(−c)G(GG^\top)^{(-c)}G, not the update used in practice. We have clarified this in the paper. The optimality of step-sizes is explicitly considered in the scale-free update in (2): with a scale-free update, clipping only changes the weights inside the momentum buffer, and all experiments therefore already use the scale-free update, meaning conclusions are unchanged.
  • W2.3. We thank the reviewer for pointing this out. Every SVD variant now has its own sweep over both learning rates (new Table G.2 and G.3). Even at their own best rates, the strongly spread-out and the order-preserving variants remain 0.03 to 0.34 worse than exact-SVD Muon at its best rates, so their gaps are not a tuning artifact. All new numbers use three seeds. Repeating a run with the same seed changes the final loss by 0.0003 to 0.003, so we do not treat gaps below about 0.003 as resolved.
  • W2.4. Thank you for catching this. Our training code keeps two counters: optimizer steps, and gradient-accumulation micro-batches, of which there are two per step. Figs. J.1 and J.8–J.12 use optimizer steps; the run has 7,189 of them ($\approx$118M tokens at 16,384 tokens per step), so step 3,594 is 50% of training. The “25%” in Fig. J.8 was left over from an earlier caption and we apologise for the mistake. Fig. J.4 and all others were labelled in micro-batches, which is where the 14,000 comes from. We will relabel all figures in optimizer steps for the camera-ready version. In the revised PDF, each caption states which counter its step numbers use, and Appendix B gives the run length and the batch composition.
  • Q3. Thank you for catching this, this was a small error that we had missed. This version is now revised in the PDF. The statement simply becomes, (assuming sv(G)∈[l,u]sv(G)\in[l,u]) ∥(GG⊤)−a/bG−OT∥2≤Ca,b,l,u(1−(lu)b)(n+d+1)T\|(G G^{\top})^{-a / b} G-O_T\|_2 \leq C_{a, b, l, u}(1-(\frac{l}{u})^b)^{(n+d+1)^T}, where Ca,b,l,u=max[l1−2a/b,u1−2a/b]C_{a, b, l, u}=\textrm{max} [l^{1-2 a / b}, u^{1-2 a / b}].
  • Q4. The two row are the same setting. They differ by 0.0010.001, within run-to-run noise. We have removed the duplicate for clarity.
  • Q5. Thank you for pointing this out; we have now clarified the term. “Quasi-norm” throughout the paper referred to the dual exponent qq: the Freon direction UΣq−1V⊤U\Sigma^{q-1}V^\top is the gradient of 1q∑iσiq\tfrac{1}{q}\sum_i \sigma_i^q, the Schatten qq-quasi-norm term of the gradient, and q∈(0,1)q \in (0,1) is exactly the quasi-norm range. On the primal side there is no corresponding Schatten norm or quasi-norm (p<0p < 0). We have revised the note in Section 2.3.

Q6. Shulgin et al. analyse an approximate LMO with additive operator-norm error δk<1\delta_k < 1 at each step, with rates degraded by (1+δ)/(1−δ)(1+\delta)/(1-\delta), or (1+δ)1/4/(1−δ)(1+\delta)^{1/4}/(1-\delta) with stochastic gradients and momentum. We agree with the reviewer that Theorem 2.5 is an argument of similar flavour, and our results have a similar structure to their adaptive-step result. Its proof bounds each step’s alignment below by [Ek]min/∥Ek∥[E_k]_{\textrm{min}}/\|E_k\| and averages the square of this bound over steps resulting in δk=(smax−smin)/(smax+smin)\delta_k = (s_{\textrm{max}}-s_{\textrm{min}})/(s_{\textrm{max}}+s_{\textrm{min}}). However, random spectra lie at the edge of this theory: each draw has δk<1\delta_k < 1, but for i.i.d. U[0,1]U[0,1] values at r=768r = 768, δk≈0.998\delta_k \approx 0.998, so the bound is inflated by (1+δk)/(1−δk)=smax/smin≈103(1+\delta_k)/(1-\delta_k) = s_{\textrm{max}}/s_{\textrm{min}} \approx 10^3 and the guarantee is very weak. We have added this relation, with a citation to Shulgin et al., after Theorem 2.6 (2.5 in the original).

Empirically, these bounds get the direction right but not the size or the ordering. Consistent with Shulgin et al., exact-SVD Muon has the lowest mean loss (3.3311±0.00153.3311 \pm 0.0015), although the mildest random spectra are within noise of it. Both bounds worsen as the spread of the spectrum grows, as do the losses in the new Table G.2, but by orders of magnitude more: the constant for uniform values is about 10410^4 times weaker than for lognormal s=0.3s = 0.3, yet their losses differ by only 0.0030.003. Sorted and unsorted uniform draws have the same δk\delta_k and the same constant, and the sorted draws are better aligned with the gradient, yet they are 0.0270.027 worse. Likewise, on the gradient of Table G.4, Kaon is about twice as suboptimal as Newton–Schulz Muon (0.5430.543 against 0.2810.281), yet tuned Kaon matches tuned Muon (3.3346±0.00093.3346 \pm 0.0009 against 3.3350±0.00153.3350 \pm 0.0015). Distance from the LMO therefore does not predict performance.

Reviewer_wEgUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the effectiveness of Muon fundamentally depends on its underlying spectral-norm geometry and linear minimization oracle (LMO) interpretation. The authors introduce Freon, a family of spectral optimizers based on Schatten (quasi-)norm transformations that interpolates between SGD and Muon and extends beyond the regime representable by norm-based LMOs. They also propose Kaon, which replaces gradient singular values through a chaotic transformation, yet achieves performance comparable to Muon in GPT-2 training experiments.

On the theoretical side, the paper characterizes the relationship between preconditioned spectral descent and norm-based steepest descent, establishes convergence results for a broad class of spectral updates, and develops an efficient QDWH-based iterative approximation for Freon. To better understand the empirical observations, the authors further introduce an analytical framework based on two local quantities derived from a Taylor expansion: batch-gradient alignment and directional descent potential. Through random-feature model analysis and empirical observations on GPT-2, they argue that optimizer performance is better understood through the interplay of these quantities and learning-rate dynamics, rather than strict adherence to a particular norm geometry.

Critical strengths

1. A compelling challenge to conventional geometric explanations of spectral optimizers.

  • The paper raises an interesting and fundamental question about whether the precise norm geometry underlying Muon is necessary for its empirical success.
  • The introduction of Kaon and Freon provides complementary perspectives: Kaon challenges the importance of preserving the singular-value spectrum, while Freon explores updates beyond the norm-based LMO regime.
  • The comparable performance of these optimizers across learning-rate sweeps provides meaningful empirical evidence motivating a reconsideration of existing geometric interpretations.

2. Substantial theoretical and algorithmic contributions.

  • The paper provides a characterization of the relationship between preconditioned spectral descent and norm-based steepest descent (Theorem 2.1), as well as general convergence conditions for spectral updates (Theorem 2.3).
  • The proposed QDWH-based iterative method for Freon is also a technically interesting contribution, enabling efficient approximations of spectral transformations in the quasi-norm regime.

3. An insightful attempt to connect optimizer behavior with local loss dynamics.

  • Rather than relying solely on convergence bounds, the paper examines optimization behavior through batch-gradient alignment and directional descent potential, derived from a local Taylor expansion.
  • The combination of random-feature analysis and empirical measurements on GPT-2 provides a useful perspective on the trade-off between gradient alignment, local descent potential, and learning-rate selection.
  • In particular, the discussion of optimal step-size stability offers an interesting potential explanation for why spectral optimizers can perform well despite differences in their spectral transformations.
Critical weaknesses

1. Gap between the convergence guarantees and the practical Kaon algorithm.

  • Theorem 2.5 establishes convergence for random spectral descent under the assumption that the transformed singular values are independently sampled at each iteration. However, the actual Kaon algorithm (Algorithm 2) uses a deterministic chaotic transformation of the gradient's singular values. It is therefore unclear whether the theoretical guarantee directly applies to the proposed implementation.
  • Furthermore, Theorems 2.3 and 2.5 consider full-gradient updates without momentum, whereas the practical experiments employ stochastic minibatch gradients and momentum. This further limits the applicability of the convergence results to the actual training setting.

2. Limited connection between the random-feature analysis and practical Transformer training.

  • The theoretical results in Section 3.2 rely on a well-specified quadratic random-feature model with mean-zero Gaussian activations. However, the authors acknowledge that this approximation does not accurately capture the local loss landscape of GPT-2 (Figure J.5), raising questions about how the derived theoretical insights generalize to practical Transformer optimization.
  • In particular, Proposition 3.1 and Theorem 3.2 identify settings in which SGD achieves more favorable optimal local descent than Muon. The authors subsequently attribute Muon's practical advantage to the stability of its optimal step size, but this mechanism is not sufficiently established in the GPT-2 experiments. A more direct quantitative validation would strengthen the connection between the theoretical analysis and the empirical observations.

3. Insufficient validation of the proposed alignment–descent potential framework as an explanation of optimizer performance.

  • The proposed quantities, batch-gradient alignment and directional descent potential, are derived from a local Taylor expansion. While they provide a useful decomposition of local loss changes, it remains unclear whether they offer predictive insights into long-term optimization dynamics beyond a post-hoc characterization of observed behavior.
  • Although the authors provide empirical measurements of these quantities, the analysis does not sufficiently demonstrate that their combined dynamics consistently explain the comparable performance of Muon, Kaon, and Freon. In particular, a more direct quantitative comparison between predicted descent behavior and actual optimization performance would be valuable.

4. Limited empirical scope and interpretation of the role of spectral geometry.

  • The main empirical comparisons focus on GPT-2 training. Given that the proposed explanation depends on batch-gradient alignment and local curvature, it remains unclear whether the observed similarities among Muon, Kaon, and Freon persist across different architectures, training scales, and batch sizes.
  • Additionally, although Kaon randomizes the singular values, it preserves the gradient's singular vectors. Thus, its empirical success provides evidence that a precise target singular-value spectrum or LMO-based geometry may not be necessary, but does not establish that spectral structure more broadly is irrelevant. A more explicit distinction between these claims would help clarify the scope of the paper's conclusions.
Questions for authors

Please see the Critical Weaknesses above for detailed discussion. In particular, I would appreciate clarification on the following points:

  1. Convergence guarantees: To what extent does Theorem 2.5 apply to the practical Kaon implementation, which uses deterministic chaotic transformations, stochastic gradients, and momentum?

  2. Step-size stability: Can the authors provide direct empirical evidence that the stability of Muon's optimal step size, as observed in the random-feature model, also holds in GPT-2 training and contributes to its practical advantage?

  3. Explanatory power: Can the authors quantitatively demonstrate that the proposed alignment–descent potential framework explains or predicts the comparable optimization performance of Muon, Kaon, and Freon?

2 comments

We want to thank the reviewer for their thorough review, here are our responses:

  • W1.2 and Q1: We have extended Theorems 2.3 and 2.5 to stochastic gradients with momentum under classical assumptions (Appendix E.3). These guarantees are not offered as an explanation of performance, and our results show they cannot be one. They hold for every i.i.d. spectrum in Table G.2, whose final losses range from 3.3315 to 3.6708. Convergence theory of this kind, including the theory used to motivate Muon, does not separate good spectral updates from bad ones.
  • W1.1 and Q1: Theorem 2.5 covers i.i.d. positive spectra under its idealized full-gradient assumptions. It indeed does not cover Kaon defined via chaotic iterations. The i.i.d. counterpart, an exact SVD with singular values drawn i.i.d. from U[0,1]U[0,1], is covered by Theorem 2.5 and reaches 3.3345±0.00083.3345 \pm 0.0008, against 3.3346±0.00093.3346 \pm 0.0009 for Kaon. As such, the chaotic map is simply a cheap replacement for an SVD plus a random generator, not a different algorithm. What is more, technically every implementation of a stochastic method is deterministic: minibatch SGD draws its batches from a pseudo-random generator, and convergence results for SGD apply to it in exactly this sense. And importantly, Kaon destroys ordering of singular values (rank correlation −0.016-0.016 on a GPT-2 gradient, against 0.8930.893 for Newton–Schulz Muon; Table G.4).
  • W2.1: We do not claim the RF results transfer to the GPT-2 setting. As Figure J.5 shows, the curvature of a real architecture breaks the RF approximation. Section 3 uses the RF model for a narrower point: in the settings of Proposition 3.1 and Theorem 3.2, SGD achieves at least as much local decrease as Muon at the optimal step size, which reverses the conclusion of Davis & Drusvyatskiy (2026). Muon’s advantage must therefore come from elsewhere, and in the RF model it comes from the stability of its optimal step size. We partially test this in GPT-2 directly in our answer to Q2.
  • W2.2 and Q2: based on a similar request from 8XpK, we have added direct measurements in GPT-2. For each update direction we compute the optimal step η⋆=γ/λ\eta^{\star}=\gamma / \lambda on each validation batch and report its signal-to-noise ratio across batches (fourth row of Figure J.1; Figure J. 11) and across training (Figure J.12).
  • W3: We want to clarify that the γ\gamma – Φ\Phi decomposition is not a useful predictor of training dynamics, and we do not intend to present it as one. It is a post-factum explanation, which we state in the paper. Even in the RF model, we can only derive properties at initialization; they do not provide a tractable model of the evolving nonlinear training trajectory. Section 3.3 states that local geometry poorly predicts overall performance, and our asymptotic results characterise them at a fixed point, not along the trajectory. Their role is to explain why the conclusions of local analyses reverse once the step size is taken into account. Where a prediction can be checked, it holds: the one-step decrease predicted by the quadratic (GGN) model agrees with the exact one (Figure J.5b,c).
  • W4.2: We completely agree that spectral structure itself is important. All updates we consider keep the gradient’s singular vectors, so our conclusions concern the singular values only: a precise target spectrum, or an LMO geometry, is not necessary, but we make no claim that spectral structure as a whole is irrelevant. We already make this distinction in the introduction's footnote. It would be worth emphasizing that generally speaking spectral structure may not be important either [1], but this is not evaluated here.
  • W4.1: Our results already cover two settings with different data, training lengths and batch sizes: GPT-2 on WikiText-2 (7,189 steps of 16,384 tokens; Appendix B) and NanoGPT on FineWeb-10B (1,907 steps of 524,288 tokens; Table G.2). Kaon matches Muon in both. We have also launched a sweep over a range of sizes of ResNet models over batch sizes, with preliminary results already in the paper in Appendix K (currently still only on one size, but we aim to have a wider range by the camera-ready stage). Anecdotally, in conversation with other optimization researchers, we have found others have observed similar behaviour in other settings, particularly for (GG⊤)−1G(GG^\top)^{-1}G.
  • Q3: The framework explains rather than predicts. At a fixed step size, the realised decrease depends on how close that step is to η⋆\eta^\star, so directions whose η⋆\eta^\star is stable should train better.

[1] Lillicrap, Timothy P., et al. "Random synaptic feedback weights support error backpropagation for deep learning." Nature communications 7.1 (2016): 13276.

Dear Reviewer,

We’re nearing the end of the discussion period. This is a friendly reminder to leave a comment for the authors.

Thank you

~ AC

Reviewer_8XpKOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that Muon’s success isn’t explained by the usual LMO analysis, i.e. steepest descent under the spectral norm. The authors introduce:

  1. Freon. A family of updates (GGT)−cG(GG^T)^{-c} G that interpolates between SGD (with c=0c=0) and Muon (c=1/2c=1/2) and extends to c>1/2c>1/2. Interestingly, updates of c>1/2c>1/2 aren't LMOs of any unitarily invariant norm. The authors also give a QDWH-style rational iteration to compute them stably (which is its own practical contribution).
  2. Kaon. An optimiser that replaces the gradient's singular values with chaotic pseudo-random values. It roughly matches Muon on NanoGPT/GPT-2 and comes with a convergence guarantee.
  3. a local decomposition that splits the update into the batch-gradient alignment γ\gamma and the descent potential Φ\Phi, which has been analysed in a random-features model and measured on GPT-2. The authors conclude that Muon’s real advantage is step-size optimality rather than geometry.
Critical strengths
  1. The paper asks a timely question regarding the necessity of the LMO framework and answers it with simple but interesting counterexamples. The introduced optimiser (or update) Freon at c>1/2c>1/2 is a deterministic, order-reversing update that keeps up with Muon in the tested regime. I found also the Kaon experiments and analysis also very informative.

  2. The authors also provide a specific criterion for when a spectral update is an LMO, that is when we maintain the order of singular values. This makes it easy to explore different types of updates beyond LMOs.

  3. Finally, moving from global bounds to local quantities is a sensitive direction to me. It provides a shift in how we have analysed Muon-type updates so far.

In general, I found the paper quite appealing and informative for better understanding the apparent success that spectral updates have in deep learning.

Critical weaknesses

Below are issues that when addressed will improve the paper in my opinion.

  1. Any spectral update with positive singular-value map is steepest descent under some gradient-dependent norm. For example, Freon is steepest descent under tr(XT(GGT)cX)tr(Xᵀ(GGᵀ)^c X), which is how Shampoo-style whitening is usually derived. So what’s really shown is that Muon-level performance doesn’t need a fixed, unitarily invariant norm. That’s still meaningful, but it should be stated that way. Practical five-step Newton–Schulz Muon isn’t order-preserving either. The standard quintic is non-monotone on (0, 1], consistent with the Spearman correlation of 0.893 in Table G.2. So by Theorem 2.1 it isn’t a unitarily invariant LMO, which supports the thesis but is currently buried in the appendix.

  2. Regarding the γ\gamma-Φ\Phi analysis, Section 3.3 concludes that trading alignment (γ\gamma) for descent potential (Φ\Phi) explains why c>1/2c > 1/2 wins. But at the optimal step the loss drop is proportional to γ2Φ=⟨G,D⟩2/⟨D,λ⟩\gamma^2\Phi = \langle G, D\rangle^2/\langle D, \lambda\rangle. The batch gradient cancels, so γ\gamma and Φ\Phi only matter through their product, and the split between them is bookkeeping. Reading Fig. J.1 roughly, the product is similar for SGD, Muon, Kaon and small cc, lower at c=3/4c = 3/4, and lowest at c=1c = 1, which is the opposite of the claim. The paper also treats high Φ\Phi as "strong local decrease" (e.g. TruncatedSGD at step 1) without accounting for its low γ\gamma. Reporting γ2Φ\gamma^2\Phi for each method would settle this.

  3. Following the previous point, in lines 469-472, I couldn't find a figure that indicates the argument that gradients agree most in the middle of the spectrum, or that c=3/4c = 3/4 targets this band. Also, I couldn't understand this claim as for example $c = 3/4 $weights each direction by σ−1/2\sigma^{−1/2}, which emphasises the smallest singular values rather than the middle of the spectrum. The claim can only come from the ϵ\epsilon regularisation so any mid-spectrum targeting would be set by ϵ\epsilon rather than by cc. And if the update really aligned with high-SNR directions, γ\gamma should go up, not down as the paper reports.

Minor: Some figures (e.g., F.1, F.2) has extremely small fonts. I suggest to break such figures into multiple ones and make sure that they are visible at an appropriate scale.

Questions for authors
  1. Is it fair you restate the main claim as "Muon-level performance doesn’t need steepest descent under a unitarily invariant norm, even one chosen per step"? The c>1/2c > 1/2 updates are still steepest descent under gradient-dependent norms, which is Shampoo-style whitening. How do you position Freon relative to the Shampoo family? This type of discussion is missing from the paper.

  2. Can you add an exact-SVD polar run (the true LMO) next to NS-5 Muon and Kaon in Table G.1? Does the exact LMO do any better?

  3. Can you report γ2Φ\gamma^2\Phi and the spread of a∗=γ/λa^* = \gamma/\lambda for each method in Figs. J.1, J.8 and J.9? How do these fit with Fig. J.7, which favours c≈1c \approx 1 for the same layer?

  4. What is the evidence for the SNR claim in Section 3.3, and is there a direct per-direction SNR measurement? c=3/4c = 3/4 weights directions by σ−1/2\sigma^{−1/2}, which favours the smallest singular values. How does it target the middle of the spectrum, unless the regularisation is doing that? And wouldn’t aligning with high-SNR directions raise γ\gamma rather than lower it?

3 comments

We want to thank the reviewer for their positive review and insightful questions. Here are the responses one-by-one:

  • W1 and Q1: We agree with this framing. Our claim concerns the justification given for Muon: steepest descent under a fixed, unitarily invariant norm. As the reviewer points out, Theorem 2.1 even allows the norm to be chosen per step. Once the norm may depend on GG, however, the notion explains nothing and in fact second order optimizers can start entering the picture, which generally speaking outperform. We have added a footnote to note that practical NS-5 Muon itself need not be an exact LMO, as Theorem 2.1 and Table G.4 already suggest.

  • Q2: We ran an exact-SVD constant-spectrum baseline, together with various separately tuned randomized-SVD variants. Exact LMO contribution is marginal. Exact-SVD Muon reaches 3.3311±0.00153.3311 \pm 0.0015, 0.0040.004 below tuned Newton–Schulz Muon (3.3350±0.00153.3350 \pm 0.0015). Against the Muon used in practice, random spectra lose nothing: uniform draws (3.33453.3345), Beta(2,5) (3.33363.3336), Pareto α=3\alpha=3 (3.33203.3320) and Kaon (3.33463.3346) all finish at or below it. We report these runs in Table G.2 and G.3. as the hardware layout differs from the original Table G.1.

  • W2 / Q3. The reviewer is correct that γ2Φ\gamma^2 \Phi determines the optimal one-step quadratic-model decrease. We have added γ2Φ\gamma^2 \Phi to Figure J.1. In training, however, the optimal batch step-size is not taken: training uses a common scheduled η\eta. In the quadratic model the realised decrease on a batch is 12γ2Φ[1−(1−η/η⋆)2]\frac{1}{2} \gamma ^{2} \Phi [ 1-(1-\eta / \eta^{\star})^2] and γ2Φ\gamma^2 \Phi is the ceiling; the stability of η⋆\eta^{\star} decides how much of it a fixed step captures. This is why we track the two separately and report the SNR of η⋆\eta^{\star} over batches and iterations (Figure J.1 row 4; Figures. J.11, J.12). We have rephrased the end of Section 3.3. In terms of difference to the setting of J.7: it uses a different checkpoint, the actual loss rather than the quadratic model, and are for one layer only.

  • W3 and Q4: The sentence in lines 469–472 was poorly worded, and we have rewritten it. It concerns intermediate exponents cc, not the middle singular values: the “extremes” it contrasts are c=0c = 0 and c=1c = 1. The direct per-direction measurement is the signal-to-noise ratio (mean/std over validation batches) of the optimal step size ηk⋆=γk/λk\eta_k^\star = \gamma_k/\lambda_k for each update direction (new Figure J.11). It is highest for intermediate-to-large exponents, peaks within c∈[1/2,1]c \in [1/2 , 1] and never peaks at c<1/2c < 1/2. This does not conflict with γ\gamma. The SNR measures stability across batches, not alignment, and a direction can have both a lower mean γ\gamma and a more stable optimal step. Increasing cc does exactly this: γ\gamma falls monotonically (Figure J.9), while the SNR of γ\gamma and η⋆\eta^\star rises (Figure J.11). We have rephrased the end of Section 3.3.

And Lastly, figures F.1 and F.2 have been separated for readability.

Dear Reviewer,

We’re nearing the end of the discussion period. This is a friendly reminder to leave a comment for the authors.

Thank you

~ AC

I would like to thank the authors for their reply. I'm covered by their response and I will maintain my acceptance score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.