- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that the success of Muon is not explained by the spectral-norm LMO geometry it is usually attributed to. It proposes two optimizers. The first one, Freon, is an optimizer with updates that interpolates between SGD () and Muon (), and can extrapolate beyond . On GPT-2 the best c is reported to lie in , which lies outside the range of unitarily invariant norms. The paper also shows that is equivariant under layerwise symmetries. Kaon is an optimizer that replaces the gradient's singular values with (pseudo)random values, produced by a chaotic polynomial iteration. It is reported to match Muon on GPT-2 (124M). The paper gives a convergence guarantee for a class of preconditioned spectral descent updates that includes Kaon and Freon under regularization. It also characterizes which such updates are steepest descent for some unitarily invariant norm.
The authors perform a local theoretical analysis based on Taylor expansion that splits the one-step loss change into batch gradient alignment and directional descent potential. The paper then argues that Muon wins mainly because its optimal step size is stable. Empirical support comes from one-step loss-change heatmaps for GPT-2 and RF experiments.
Validation consists of GPT-2 124M sweeps on WikiText and NanoGPT+FineWeb runs, one-step loss-change heatmaps, and RF experiments. The paper concludes that a precise geometry is practically irrelevant.
Critical strengths
- The empirical results are potentially surprising. Many Freon and Kaon runs match Muon on GPT-2 (Fig. 4, Table G.1). This forces the community to reconsider how much of Muon's performance is owed to the exact spectral-norm LMO. It would be a solid contribution if framed as "spectral flattening matters and the exact profile mostly does not". The TruncatedSGD comparison supports this, since suppressing large singular values helps but is not enough.
- The paper gives a concrete framework to build on. Theorem 2.1 shows that updates cannot be steepest descent for any unitarily invariant norm, which is a limitation of the LMO view. The RF analysis shows that stable step size is an important mechanism. Together these give others something specific to test and extend.
Critical weaknesses
-
The main conclusions are overstated and do not follow from the evidence, for example:
- Kaon is not so much "an absurd optimizer that replaces singular values with random noise" as it is Muon (with noise) in expectation. In Theorem 2.5 the entries of are i.i.d. and positive, so by exchangeability is a constant vector. The expected update is therefore proportional to the polar factor , and so the algorithm seems to be just stochastic Muon with spectral noise, and claims such as "zero coherent geometry" are overstated. Theorem 2.5 can be reduced to the standard noisy-LMO descent argument. Practical Muon (NS-5) is itself far from the exact LMO (33.5° in Table G.2). Inexact-Muon theory (Shulgin et al.) is cited but never compared against.
- The data support a weaker claim than "geometry is practically irrelevant". For example, Figure 4 shows that all behave similarly once the spectrum is reasonably flat, and that TruncatedSGD is insufficient. The matching spread in Table G.2 (0.31 versus 0.33) also points to flattening. The decisive ablations are missing, e.g., heavy-tailed random spectra and spectra with very different spread.
- "optimal lies strictly outside the range of any proper Schatten norm ()" is not established. It is a one-step greedy optimum over a one-parameter family with a shared LR. The authors themselves report that greedy hurts long-term dynamics (Fig. 5). Fig. J.7 shows different near-optimal ranges 0.00-0.21, 0.00-0.26 and 0.00-0.42 for many W_QKV blocks. In Fig. J.2 the curve also appears to end at or below the and curves. The text is muddled about which model shows what. Section 3.3 says RF optimal exponents "frequently fall into the strict non-norm regime ()". Fig. J.3 is the synthetic RF experiment, where the optimum is below . Contribution 1 says the opposite for RF.
- The J.5 caption cites an "inter-batch SNR analysis of Section 3.2" that does not seem to exist.
Overall, I believe that the strong wording in the title and abstract ("not that special", "proving", "conclusively", "practically irrelevant") is not supported. What the evidence supports is more like spectral compression/flattening matters and the exact profile mostly does not.
-
The experimental protocol does not fully support the claims, for example:
- The LR grid is truncated. All four optimizers select , the largest value in the grid (Table A.1). Fig. 3 plots Kaon at while the others use . So the LRs are not properly tuned and the choice of plotted runs seems post hoc.
- Freon's scale depends on gradient magnitude. For , Algorithm 4 returns , which has degree in G. Its step size therefore scales with gradient magnitude and interacts with clipping at 1.0. This conflicts with Eq. (2), which is scale-free, and with the hyperparameter-transfer motivation of Section 2.3.1. Since the paper's thesis is that step size is decisive, this may confound the comparisons.
- The SVD variants use Kaon’s LR rather than their own.
- Training-length bookkeeping is inconsistent. For example, J.1 labels step 3594 as 50%, J.8 calls the same step 25%, J.4 implies 14000 total steps, and J.1 implies about 7200.
Overall, comparisons between Muon, Kaon and Freon concluded from this protocol may not be entirely fair. Differences reported (or the absence of differences) are within what LR-grid edge effects and seed noise could produce.
Questions for authors
- Could you explain the normalization of Freon for ? How does this interact with clipping, and do the conclusions change with a scale-free version?
- The selected is at the grid boundary for all optimizers. Do the conclusions still hold after proper tuning?
- Theorem 2.7 is stated for all , and its proof is completed by Theorem F.8. The proof of Theorem F.8 uses with , which needs . This condition is not assumed. For , which includes the used in the experiments, the explicit constant is not justified by this argument. Could you give a corrected argument?
- Table G.1 lists "Kaon, " and "Kaon (main experiment)" with different numbers. What differs between them?
- The abstract and Fig. 4 caption call the quasi-norm regime, but with and this range corresponds to . Can you clarify what is meant?
- Could you compare against inexact-Muon theory (e.g. Shulgin et al.)?
I would be happy to reconsider my score if the authors address the weaknesses and questions raised above.
7 comments
#2
- W2.1 and Q2 (continued) The reviewer is right that the grid of Table A.1 ended at and that all four optimizers selected it. Fig. 2 goes beyond it only by scaling both rates together. We have therefore extended the sweep, raising the matrix rate on its own and retuning the base rate, with rates selected on one seed and losses reported over three (new Table G.2,G.3): | Optimizer | Table A.1 rates | Retuned | |---|---|---| | Muon | 3.3373$\pm$0.0007 | 3.3350$\pm$0.0015 | | Kaon | 3.3456$\pm$0.0015 | 3.3346$\pm$0.0009 | | Freon | 3.3452$\pm$0.0020 | 3.3397$\pm$0.0002 | | Freon | 3.3580 (seed 42) | no improvement |
With both tuned, Kaon and Muon are indistinguishable ( against ) and Freon remains slightly behind after tuning, with being 0.023 worse (seed 42 only). Kaon therefore matches Muon after tuning, while Freon is close to, but slightly behind, Muon on NanoGPT. The SVD variants now also have their own sweeps, with the matrix rate extended to . These runs used one GPU with gradient accumulation and the same batch per step, so we report them alongside, not mixed with, the numbers elsewhere in the paper.
- W2.2 and Q1. The actual Freon update is shown in equation (2) and is scale-free. Algorithm 4 specifically shows the algorithm used for approximating , not the update used in practice. We have clarified this in the paper. The optimality of step-sizes is explicitly considered in the scale-free update in (2): with a scale-free update, clipping only changes the weights inside the momentum buffer, and all experiments therefore already use the scale-free update, meaning conclusions are unchanged.
- W2.3. We thank the reviewer for pointing this out. Every SVD variant now has its own sweep over both learning rates (new Table G.2 and G.3). Even at their own best rates, the strongly spread-out and the order-preserving variants remain 0.03 to 0.34 worse than exact-SVD Muon at its best rates, so their gaps are not a tuning artifact. All new numbers use three seeds. Repeating a run with the same seed changes the final loss by 0.0003 to 0.003, so we do not treat gaps below about 0.003 as resolved.
- W2.4. Thank you for catching this. Our training code keeps two counters: optimizer steps, and gradient-accumulation micro-batches, of which there are two per step. Figs. J.1 and J.8–J.12 use optimizer steps; the run has 7,189 of them ($\approx$118M tokens at 16,384 tokens per step), so step 3,594 is 50% of training. The “25%” in Fig. J.8 was left over from an earlier caption and we apologise for the mistake. Fig. J.4 and all others were labelled in micro-batches, which is where the 14,000 comes from. We will relabel all figures in optimizer steps for the camera-ready version. In the revised PDF, each caption states which counter its step numbers use, and Appendix B gives the run length and the batch composition.
- Q3. Thank you for catching this, this was a small error that we had missed. This version is now revised in the PDF. The statement simply becomes, (assuming ) , where .
- Q4. The two row are the same setting. They differ by , within run-to-run noise. We have removed the duplicate for clarity.
- Q5. Thank you for pointing this out; we have now clarified the term. “Quasi-norm” throughout the paper referred to the dual exponent : the Freon direction is the gradient of , the Schatten -quasi-norm term of the gradient, and is exactly the quasi-norm range. On the primal side there is no corresponding Schatten norm or quasi-norm (). We have revised the note in Section 2.3.
#3
Q6. Shulgin et al. analyse an approximate LMO with additive operator-norm error at each step, with rates degraded by , or with stochastic gradients and momentum. We agree with the reviewer that Theorem 2.5 is an argument of similar flavour, and our results have a similar structure to their adaptive-step result. Its proof bounds each step’s alignment below by and averages the square of this bound over steps resulting in . However, random spectra lie at the edge of this theory: each draw has , but for i.i.d. values at , , so the bound is inflated by and the guarantee is very weak. We have added this relation, with a citation to Shulgin et al., after Theorem 2.6 (2.5 in the original).
Empirically, these bounds get the direction right but not the size or the ordering. Consistent with Shulgin et al., exact-SVD Muon has the lowest mean loss (), although the mildest random spectra are within noise of it. Both bounds worsen as the spread of the spectrum grows, as do the losses in the new Table G.2, but by orders of magnitude more: the constant for uniform values is about times weaker than for lognormal , yet their losses differ by only . Sorted and unsorted uniform draws have the same and the same constant, and the sorted draws are better aligned with the gradient, yet they are worse. Likewise, on the gradient of Table G.4, Kaon is about twice as suboptimal as Newton–Schulz Muon ( against ), yet tuned Kaon matches tuned Muon ( against ). Distance from the LMO therefore does not predict performance.
#1
We thank the reviewer for a careful review. In response we extended the learning-rate sweep beyond the original grid, ran the requested spectral ablations with a separate sweep for each (new Table G.2), and toned down some of the wording. The clarified claims are: matching an LMO geometry is not necessary for Muon-level performance; within mild spreads the exact spectrum matters little; and keeping the gradient's ordering is not needed and can hurt.