- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors propose a modification to the Muon optimizer motivated by a Muon-specific training-time phenomenon that the authors call implicit step decay. Roughly speaking, as training progresses, the ratio of the norm of the optimizer update to the norm of the current parameters (the "angular step") decreases even for a constant learning rate. The authors argue that existing learning-rate schedules and implementations make it difficult to control the angular step, thus motivating the modification to Muon. Several language modeling and image classifications experiments are provided to support the claims.
Critical strengths
I do not believe the current manuscript has substantial strengths. The writing is overall clear to read, but that alone does not make up for the critical weaknesses listed in the next section.
Critical weaknesses
- I am not convinced by the claim that the angular step cannot be controlled by standard implementations of Muon. In particular, it is well-known that Muon with decoupled weight decay imposes a spectral-norm constraint on the parameters, see e.g. [1, 2]. Such works are not cited in the manuscript, and they provide substantially better guarantees on the growth of parameter norms than what is claimed in equation (5). Using these results, we can easily show that for (full-rank) parameters and weight decay coefficient , we have as training progresses. On the other hand, assuming a constant learning rate and exact orthogonalization, we have . It follows that the angular step is lower bounded: . In fact, this is supported by Figure 1(c), where under stronger weight decay and a constant learning rate, the angular step does not decay. If the argument is that here the angular step is being controlled by the weight decay parameter, then I do not see why that is inherently undesirable (also see below).
- Furthermore, even supposing the existence of implicit step decay, I fail to see why the angular step is a relevant quantity to control. If the claim is that better angular step control leads to, e.g., lower validation loss, better generalization, or improved training stability, then the current results do not provide sufficient evidence for such a claim.
- I also have some concerns regarding the experimental setup, including the small scale and unusual choice of compared to the standard [3]. However, the two points listed above are already critical and need to be addressed.
References
[1] Lizhang Chen, Jonathan Li, Qiang Liu. Muon Optimizes Under Spectral Norm Constraints. TMLR 2026.
[2] Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, Volkan Cevher. Training Deep Learning Models with Norm-Constrained LMOs. ICML 2025.
[3] Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, Zhilin Yang. Muon is Scalable for LLM Training. abs/2502.16982, 2025.
Questions for authors
- Can you cite the references above and revise the discussion in Section 4.1?
- Why is the angular step important to control? Why shouldn't it be controlled by weight decay? What does controlling the angular step concretely give in terms of downstream metrics?
- How do the experimental results hold up at larger scales and with standard hyperparameter values?
8 comments
Rebuttal for Reviewer RaZk (Part2)
5. Why weight decay, or a norm constraint, is not a sufficient way to control the angle (Q2).
-
It ties the step to the regularization. As point 4 shows, changing the angle requires changing , and the that keeps the angle from falling is worse.
-
Even a fixed norm only makes the angle follow the learning-rate shape. [1, 2] bound the spectral norm, which does not fix the Frobenius norm that sets the angular step. To test whether holding the norm is enough, we held each matrix's Frobenius norm fixed, as MuonH [4] does (with its level tuned). The angular step then follows the learning-rate shape, and this is no better than Muon, whereas keeping the same fixed norm and replacing the learning-rate shape by ASM's schedule gives a large gain (new runs; final validation NLL; Appendix D.3, Table 13, lines 1020–1039):
| Frobenius norm | Angular schedule | 127M | 300M |
|---|---|---|---|
| Free, weight decay 0.01 (Muon) | Implicit decay | 3.1976 | 3.0086 |
| Fixed (MuonH, tuned) | Learning-rate shape | 3.2034 | 3.0150 |
| Fixed | ASM's schedule | 3.1668 | 2.8817 |
| Free, weight decay 0.01 (ASM) | ASM's schedule | 3.1678 | 2.8725 |
- A single cannot equalize the matrices. At a constant learning rate, Muon's angular step falls by 47% to 71% across the 144 matrices of the 127M model (Fig. 4; lines 402–403). One weight-decay coefficient shared by all of them cannot bring them to a common target, whereas ASM gives every matrix the same angular step at every step.
What matters is therefore how the angular step evolves over training. ASM chooses it directly and leaves weight decay to its role as a regularizer.
W2 and Q2: Why control the angular step, and what does it give?
Response:
We respectfully disagree that the current results provide insufficient evidence: the submitted manuscript reports the downstream metrics and the controls that this weakness asks for, and we summarize them below.
1. Why the angular step is the relevant quantity. How much an update changes the network depends on its length relative to the weights it modifies. For scale-invariant matrices, such as a linear map followed by normalization, the output is unchanged when the weights are rescaled, so only the angle of the update matters [5, 6, 7]; the introduction motivates the angular step in exactly these terms (lines 33–37). In Muon, the raw update length is fixed by the learning rate, so the angular step equals : the step that actually changes the network is the learning-rate schedule multiplied by an extra decay from weight growth. Controlling the angular step is therefore controlling the effective step size itself.
2. What it gives: lower validation loss, better generalization and improved stability. Each of the three outcomes the reviewer lists is measured in the submitted manuscript (Table 1, lines 270–284, discussed in lines 319–325; Table 2, line 332; Table 4, line 426; Table 9, line 819):
| Outcome | Metric | Muon | ASM |
|---|---|---|---|
| Lower validation loss | Validation NLL, 127M ↓ | 3.2017 | 3.1728 |
| Validation NLL, 300M ↓ | 3.0086 | 2.8725 | |
| Validation NLL, 170M ↓ | 3.0622 | 2.9988 | |
| Wall-clock to Muon's last periodic validation loss (3.0258), 170M (min) ↓ | 747.4 | 539.9 | |
| Better generalization | Test top-1 / CE, ResNet-50 | 77.72 / 0.939 | 81.26 / 0.789 |
| Test top-1 / CE, ViT-S | 71.30 / 1.216 | 73.03 / 1.155 | |
| Improved stability (lower learning-rate sensitivity) | Loss range over nominal learning rates 0.005–0.02, 127M ↓ | 0.0200 | 0.0036 |
The language-model losses are measured on held-out tokens, and the vision results are test accuracy and cross-entropy on CIFAR-100 with checkpoints selected on a separate validation split. The last row shows that ASM's final loss changes by only 0.0036 across a fourfold range of learning rates, against 0.0200 for Muon. Held-out validation loss is also the standard metric for evaluating pretraining optimizers: it is used, for example, in Table 5 of [2], which the reviewer cites, in SOAP [8, Appendix A], and in Zhao et al. [9, Section 2.2].
Rebuttal for Reviewer RaZk (Part3)
3. The gains come from controlling the angular step. The submitted manuscript already isolates this. The 2×2 of Table 3 (lines 378–386; protocol in lines 352–363) holds the sum of squared target angular steps fixed and varies the temporal shape and the allocation across matrices (final validation loss at 127M):
| Target shape | Global multiplier | Per-matrix control |
|---|---|---|
| Learning-rate shape only | 3.2207 | 3.1964 |
| ASM shape | 3.1928 | 3.1728 |
| Ordinary Muon (reference) | 3.1976 | |
| Ordinary Muon, updates ×1.5 (reference) | 3.1954 |
Per-matrix control lowers the loss under both temporal shapes, and ASM's temporal shape lowers it under both allocations. Enlarging every Muon update by 1.5× changes the loss by only 0.002 (lines 373–376), so a larger step does not explain the gain. New runs extend these controls (Appendix D.5 and D.3, lines 1055–1072 and 1020–1039):
-
replaying Muon's own angular steps under explicit control gives 3.1977, against 3.1976 for Muon, so explicit control alone does not explain the gain;
-
at 300M, an angle that follows the learning-rate shape at the same total angular budget is 0.140 worse than ASM's schedule.
Controlling the angular step therefore gives lower validation loss, better test accuracy and lower sensitivity to the learning rate, and the controlled comparisons attribute these gains to how the angular step is scheduled. We hope that these results, most of which are in the submitted manuscript, together with the new controls in Appendix D, address this concern, and we would be glad to discuss any further comparison the reviewer has in mind.
W3 and Q3: Scale and the choice of
Response:
1. The choice is not weak weight decay. Decoupled weight decay multiplies the weights by at each step, so its strength is set by the product , not by alone. In [3], is paired with a peak learning rate of for Moonlight and about in its scaling-law runs, i.e., . Our Muon recipe uses and , i.e., , which is 2–5 times stronger per step (Appendix D.1, lines 979–1001). In our setting, applying without rescaling gives at learning rate 0.01, more than ten times the per-step decay of [3]. It makes Muon worse than (3.2192 vs. 3.2068; Sec. 5.3, lines 415–417), so is the stronger baseline. For ASM, no longer sets the step and only shrinks the norm: and 0.01 give the same loss (3.1806 vs. 3.1798), whereas , with this tenfold per-step decay, degrades it to 3.5613 (line 414).
2. Other hyperparameters. On every workload, ASM uses that workload's Muon recipe unchanged, and Muon's learning rate is tuned. ASM only adds the schedule of Eq. (8) (lines 233–234). The 170M recipe starts from the configuration of the dense-model scaling experiments in [3], with learning rate (Appendix A.3, lines 754–757), and there ASM lowers the final validation NLL from 3.0622 to 2.9988 (Table 2, line 332).
3. Scale. Our language-model results cover dense 127M and 300M models trained on 8.4B tokens, a 170M model on a longer recipe of 50,863 steps, and a 127M-active MoE (8 experts, top-2, about 250M parameters in total), in addition to two vision architectures. This is comparable to the language models used to verify the constraint in [1] (100M and 300M). The gain grows from 127M to 300M (0.029 to 0.136 in validation NLL) and carries over to the MoE without retuning (−0.036, against −0.026 for the dense model at the same length; Appendix D.8, Table 18, lines 1100–1109).
References
[1] Chen, Li, Liu. Muon Optimizes Under Spectral Norm Constraints. TMLR 2026.
[2] Pethick et al. Training Deep Learning Models with Norm-Constrained LMOs. ICML 2025.
[3] Liu et al. Muon is Scalable for LLM Training. arXiv 2025.
[4] Wen et al. Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization. arXiv 2026.
[5] van Laarhoven. L2 Regularization versus Batch and Weight Normalization. arXiv 2017.
[6] Li, Arora. An Exponential Learning Rate Schedule for Deep Learning. ICLR 2020.
[7] Kosson, Messmer, Jaggi. Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks. ICML 2024.
[8] Vyas et al. SOAP: Improving and Stabilizing Shampoo using Adam for Language Modeling. ICLR 2025.
[9] Zhao et al. Deconstructing What Makes a Good Optimizer for Language Models. ICLR 2025.
Rebuttal for Reviewer RaZk (Part1)
We thank the reviewer for the review, for finding the writing clear, and for pointing us to [1, 2], which we now cite. We respectfully disagree with the two critical weaknesses, on implicit step decay and on the sufficiency of the evidence, and below we address each weakness (W) and question (Q), pointing to the relevant lines of the paper and to new controlled experiments. All changes in the revised PDF are marked in blue, and the new experiments are collected in Appendix D (lines 972–1109). The new experiments are single-seed reruns on one platform, so their numbers differ slightly from the three-seed means in Table 1 (line 270) and from the controls in Table 3 (line 378). Line numbers below refer to the revised PDF.
W1, Q1 and Q2 (first part): Can the angular step be controlled by weight decay?
Response:
1. What the paper claims. The paper does not claim that the angular step cannot be controlled in standard Muon. It states that controlling this decay directly is "not straightforward" (line 53), that stronger weight decay "keeps the angular step from falling (Figure 1(c))" but "ties the step size to the strength of regularization" (lines 72–76), and that neither option "lets us specify in advance how the angular step should evolve" (line 76); Section 4.1 makes the same point (lines 177–187). Our claim is that in standard Muon the angular step decays at a rate set by weight growth rather than chosen, and that weight decay cannot set this trajectory without also changing the strength of regularization.
2. The bound of [1, 2], and Eq. (5) (Q1). We agree with the bound and now cite [1, 2] in Section 4.1, where we state the bound (lines 187–190); Appendix D.1 (lines 979–1001) gives its derivation. For our update Wt+1=(1−ηλ)Wt−ciηYt with exact orthogonalization, ∥Wt+1∥op≤(1−ηλ)∥Wt∥op+ciη, so the spectral norm is driven into the ball ∥W∥op≤ci/λ. Because ci appears both in the update and in this bound, it cancels, and the angular step is bounded below by ηλ. If the decay is instead written as (1−ciηλ)Wt, the ball is ∥W∥op≤1/λ and the floor is ciηλ, as in the review; the two forms differ only in how λ is scaled. Eq. (5) (line 182) is not a weaker guarantee but an exact one-step identity, ∥Wt+1∥F2/∥Wt∥F2=αt2−2αtrtρt+rt2. The two are complementary: Eq. (5) describes how the norm moves at each step, and [1, 2] bound where it can end up.
3. The bound does not prevent implicit step decay. At 127M, ηλ=0.02×0.01=2×10−4. Muon's measured angular step stays far above this floor while it decays, and the weights keep growing despite weight decay (Appendix D.1, Table 11, lines 979–1001):
A lower bound limits how far the angular step can fall; it does not determine how it falls. Implicit step decay is this fall, by 59.2% between steps 500 and 12,800 at a constant learning rate (Fig. 1; lines 46–47), and it happens entirely above the bound.
4. Figure 1(c) shows the cost of controlling the angle through weight decay. Raising λ from 0.01 to 0.1 raises the floor tenfold, and the angular step stops falling, as the reviewer notes. The final loss, however, worsens from 3.2068 to 3.2192; both values are given in the legend of Figure 1(c) (see its caption, lines 66–67) and in Section 5.3 (lines 415–417), which concludes that "weight decay cannot replace an explicit schedule". Weight decay stops the decay only by shrinking the weights harder, so a single coefficient now sets both the step size and the strength of regularization. It also holds the angular step flat, and a flat angular step is itself worse than a decaying one: removing ASM's decay (b = 0) raises the loss from 3.1798 to 3.2071 (Table 8, line 810).