acceptodds
Under review as a conference paper at ICLR 2027

Angle-Scheduled Muon: Replacing Implicit Step Decay with an Explicit Schedule

Abstract

Muon orthogonalizes momentum so that the norm of each matrix update is fixed by the learning rate and the matrix shape. However, we observe that weight norms keep growing during training while update norms remain nearly constant, so each update moves the weights by a shrinking fraction even at a constant learning rate. We refer to this as implicit step decay: the angular step, i.e., the update norm relative to the weight norm, decays at a rate dictated by weight growth rather than by the learning-rate schedule. Based on this insight, we propose Angle-Scheduled Muon (ASM), which replaces implicit step decay with an explicit schedule. ASM keeps Muon's orthogonalized update direction and sets each update's length to a prescribed fraction of the current weight norm, so that every weight matrix follows the same angular-step schedule. Crucially, ASM is a drop-in modification of Muon that adds only norm computations, requiring no additional orthogonalization or gradient evaluations. Experiments on language modeling and image classification show that ASM consistently improves over Muon and recent Muon variants with limited overhead.

Primary area:
optimization
Keywords:
Muon

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RaZkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose a modification to the Muon optimizer motivated by a Muon-specific training-time phenomenon that the authors call implicit step decay. Roughly speaking, as training progresses, the ratio of the norm of the optimizer update to the norm of the current parameters (the "angular step") decreases even for a constant learning rate. The authors argue that existing learning-rate schedules and implementations make it difficult to control the angular step, thus motivating the modification to Muon. Several language modeling and image classifications experiments are provided to support the claims.

Critical strengths

I do not believe the current manuscript has substantial strengths. The writing is overall clear to read, but that alone does not make up for the critical weaknesses listed in the next section.

Critical weaknesses
  1. I am not convinced by the claim that the angular step cannot be controlled by standard implementations of Muon. In particular, it is well-known that Muon with decoupled weight decay imposes a spectral-norm constraint on the parameters, see e.g. [1, 2]. Such works are not cited in the manuscript, and they provide substantially better guarantees on the growth of parameter norms than what is claimed in equation (5). Using these results, we can easily show that for (full-rank) parameters W∈Rn×d\mathbf{W}\in\mathbb{R}^{n\times d} and weight decay coefficient λ\lambda, we have ∥W∥F≤min⁡(n,d)∥W∥op≤min⁡(n,d)λ\lVert\mathbf{W}\rVert_{\text{F}}\leq\sqrt{\min(n,d)}\lVert\mathbf{W}\rVert_{\text{op}}\leq\frac{\sqrt{\min(n,d)}}{\lambda} as training progresses. On the other hand, assuming a constant learning rate η\eta and exact orthogonalization, we have ∥ΔW∥F=ciηmin⁡(n,d)\lVert\Delta\mathbf{W}\rVert_{\text{F}}=c_i\eta\sqrt{\min(n,d)}. It follows that the angular step is lower bounded: ∥ΔW∥F∥W∥F≥ciηλ\frac{\lVert\Delta\mathbf{W}\rVert_{\text{F}}}{\lVert\mathbf{W}\rVert_{\text{F}}}\geq c_i\eta\lambda. In fact, this is supported by Figure 1(c), where under stronger weight decay and a constant learning rate, the angular step does not decay. If the argument is that here the angular step is being controlled by the weight decay parameter, then I do not see why that is inherently undesirable (also see below).
  2. Furthermore, even supposing the existence of implicit step decay, I fail to see why the angular step is a relevant quantity to control. If the claim is that better angular step control leads to, e.g., lower validation loss, better generalization, or improved training stability, then the current results do not provide sufficient evidence for such a claim.
  3. I also have some concerns regarding the experimental setup, including the small scale and unusual choice of λ=0.01\lambda=0.01 compared to the standard λ=0.1\lambda=0.1 [3]. However, the two points listed above are already critical and need to be addressed.

References

[1] Lizhang Chen, Jonathan Li, Qiang Liu. Muon Optimizes Under Spectral Norm Constraints. TMLR 2026.

[2] Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, Volkan Cevher. Training Deep Learning Models with Norm-Constrained LMOs. ICML 2025.

[3] Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, Zhilin Yang. Muon is Scalable for LLM Training. abs/2502.16982, 2025.

Questions for authors
  1. Can you cite the references above and revise the discussion in Section 4.1?
  2. Why is the angular step important to control? Why shouldn't it be controlled by weight decay? What does controlling the angular step concretely give in terms of downstream metrics?
  3. How do the experimental results hold up at larger scales and with standard hyperparameter values?
8 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer RaZk (Part1)

We thank the reviewer for the review, for finding the writing clear, and for pointing us to [1, 2], which we now cite. We respectfully disagree with the two critical weaknesses, on implicit step decay and on the sufficiency of the evidence, and below we address each weakness (W) and question (Q), pointing to the relevant lines of the paper and to new controlled experiments. All changes in the revised PDF are marked in blue, and the new experiments are collected in Appendix D (lines 972–1109). The new experiments are single-seed reruns on one platform, so their numbers differ slightly from the three-seed means in Table 1 (line 270) and from the controls in Table 3 (line 378). Line numbers below refer to the revised PDF.

W1, Q1 and Q2 (first part): Can the angular step be controlled by weight decay?

Response:

1. What the paper claims. The paper does not claim that the angular step cannot be controlled in standard Muon. It states that controlling this decay directly is "not straightforward" (line 53), that stronger weight decay "keeps the angular step from falling (Figure 1(c))" but "ties the step size to the strength of regularization" (lines 72–76), and that neither option "lets us specify in advance how the angular step should evolve" (line 76); Section 4.1 makes the same point (lines 177–187). Our claim is that in standard Muon the angular step decays at a rate set by weight growth rather than chosen, and that weight decay cannot set this trajectory without also changing the strength of regularization.

2. The bound of [1, 2], and Eq. (5) (Q1). We agree with the bound and now cite [1, 2] in Section 4.1, where we state the bound (lines 187–190); Appendix D.1 (lines 979–1001) gives its derivation. For our update Wt+1=(1−ηλ)Wt−ciηYtW_{t+1} = (1-\eta\lambda)W_t - c_i\eta Y_t with exact orthogonalization, ∥Wt+1∥op≤(1−ηλ)∥Wt∥op+ciη\lVert W_{t+1}\rVert_{\mathrm{op}} \le (1-\eta\lambda)\lVert W_t\rVert_{\mathrm{op}} + c_i\eta, so the spectral norm is driven into the ball ∥W∥op≤ci/λ\lVert W\rVert_{\mathrm{op}} \le c_i/\lambda. Because cic_i appears both in the update and in this bound, it cancels, and the angular step is bounded below by ηλ\eta\lambda. If the decay is instead written as (1−ciηλ)Wt(1-c_i\eta\lambda)W_t, the ball is ∥W∥op≤1/λ\lVert W\rVert_{\mathrm{op}} \le 1/\lambda and the floor is ciηλc_i\eta\lambda, as in the review; the two forms differ only in how λ\lambda is scaled. Eq. (5) (line 182) is not a weaker guarantee but an exact one-step identity, ∥Wt+1∥F2/∥Wt∥F2=αt2−2αtrtρt+rt2\lVert W_{t+1}\rVert_F^2/\lVert W_t\rVert_F^2 = \alpha_t^2 - 2\alpha_t r_t\rho_t + r_t^2. The two are complementary: Eq. (5) describes how the norm moves at each step, and [1, 2] bound where it can end up.

3. The bound does not prevent implicit step decay. At 127M, ηλ=0.02×0.01=2×10−4\eta\lambda = 0.02 \times 0.01 = 2\times 10^{-4}. Muon's measured angular step stays far above this floor while it decays, and the weights keep growing despite weight decay (Appendix D.1, Table 11, lines 979–1001):

StepMuon's angular stepMultiple of the floor ηλ\eta\lambdaWeight-norm growth from initialization (median over matrices)
1,0000.006633×3.0×
4,0000.004121×4.5×
12,8000.003618×5.0×

A lower bound limits how far the angular step can fall; it does not determine how it falls. Implicit step decay is this fall, by 59.2% between steps 500 and 12,800 at a constant learning rate (Fig. 1; lines 46–47), and it happens entirely above the bound.

4. Figure 1(c) shows the cost of controlling the angle through weight decay. Raising λ\lambda from 0.01 to 0.1 raises the floor tenfold, and the angular step stops falling, as the reviewer notes. The final loss, however, worsens from 3.2068 to 3.2192; both values are given in the legend of Figure 1(c) (see its caption, lines 66–67) and in Section 5.3 (lines 415–417), which concludes that "weight decay cannot replace an explicit schedule". Weight decay stops the decay only by shrinking the weights harder, so a single coefficient now sets both the step size and the strength of regularization. It also holds the angular step flat, and a flat angular step is itself worse than a decaying one: removing ASM's decay (b = 0) raises the loss from 3.1798 to 3.2071 (Table 8, line 810).

AuthorsOpenReview ↗

Rebuttal for Reviewer RaZk (Part2)

5. Why weight decay, or a norm constraint, is not a sufficient way to control the angle (Q2).

  • It ties the step to the regularization. As point 4 shows, changing the angle requires changing λ\lambda, and the λ\lambda that keeps the angle from falling is worse.

  • Even a fixed norm only makes the angle follow the learning-rate shape. [1, 2] bound the spectral norm, which does not fix the Frobenius norm that sets the angular step. To test whether holding the norm is enough, we held each matrix's Frobenius norm fixed, as MuonH [4] does (with its level tuned). The angular step then follows the learning-rate shape, and this is no better than Muon, whereas keeping the same fixed norm and replacing the learning-rate shape by ASM's schedule gives a large gain (new runs; final validation NLL; Appendix D.3, Table 13, lines 1020–1039):

Frobenius normAngular schedule127M300M
Free, weight decay 0.01 (Muon)Implicit decay3.19763.0086
Fixed (MuonH, tuned)Learning-rate shape3.20343.0150
FixedASM's schedule3.16682.8817
Free, weight decay 0.01 (ASM)ASM's schedule3.16782.8725
  • A single λ\lambda cannot equalize the matrices. At a constant learning rate, Muon's angular step falls by 47% to 71% across the 144 matrices of the 127M model (Fig. 4; lines 402–403). One weight-decay coefficient shared by all of them cannot bring them to a common target, whereas ASM gives every matrix the same angular step at every step.

What matters is therefore how the angular step evolves over training. ASM chooses it directly and leaves weight decay to its role as a regularizer.

W2 and Q2: Why control the angular step, and what does it give?

Response:

We respectfully disagree that the current results provide insufficient evidence: the submitted manuscript reports the downstream metrics and the controls that this weakness asks for, and we summarize them below.

1. Why the angular step is the relevant quantity. How much an update changes the network depends on its length relative to the weights it modifies. For scale-invariant matrices, such as a linear map followed by normalization, the output is unchanged when the weights are rescaled, so only the angle of the update matters [5, 6, 7]; the introduction motivates the angular step in exactly these terms (lines 33–37). In Muon, the raw update length is fixed by the learning rate, so the angular step equals ciηt∥Yt∥F/∥Wt∥Fc_i\eta_t\lVert Y_t\rVert_F/\lVert W_t\rVert_F: the step that actually changes the network is the learning-rate schedule multiplied by an extra decay from weight growth. Controlling the angular step is therefore controlling the effective step size itself.

2. What it gives: lower validation loss, better generalization and improved stability. Each of the three outcomes the reviewer lists is measured in the submitted manuscript (Table 1, lines 270–284, discussed in lines 319–325; Table 2, line 332; Table 4, line 426; Table 9, line 819):

OutcomeMetricMuonASM
Lower validation lossValidation NLL, 127M ↓3.20173.1728
Validation NLL, 300M ↓3.00862.8725
Validation NLL, 170M ↓3.06222.9988
Wall-clock to Muon's last periodic validation loss (3.0258), 170M (min) ↓747.4539.9
Better generalizationTest top-1 / CE, ResNet-5077.72 / 0.93981.26 / 0.789
Test top-1 / CE, ViT-S71.30 / 1.21673.03 / 1.155
Improved stability (lower learning-rate sensitivity)Loss range over nominal learning rates 0.005–0.02, 127M ↓0.02000.0036

The language-model losses are measured on held-out tokens, and the vision results are test accuracy and cross-entropy on CIFAR-100 with checkpoints selected on a separate validation split. The last row shows that ASM's final loss changes by only 0.0036 across a fourfold range of learning rates, against 0.0200 for Muon. Held-out validation loss is also the standard metric for evaluating pretraining optimizers: it is used, for example, in Table 5 of [2], which the reviewer cites, in SOAP [8, Appendix A], and in Zhao et al. [9, Section 2.2].

AuthorsOpenReview ↗

Rebuttal for Reviewer RaZk (Part3)

3. The gains come from controlling the angular step. The submitted manuscript already isolates this. The 2×2 of Table 3 (lines 378–386; protocol in lines 352–363) holds the sum of squared target angular steps fixed and varies the temporal shape and the allocation across matrices (final validation loss at 127M):

Target shapeGlobal multiplierPer-matrix control
Learning-rate shape only3.22073.1964
ASM shape3.19283.1728
Ordinary Muon (reference)3.1976
Ordinary Muon, updates ×1.5 (reference)3.1954

Per-matrix control lowers the loss under both temporal shapes, and ASM's temporal shape lowers it under both allocations. Enlarging every Muon update by 1.5× changes the loss by only 0.002 (lines 373–376), so a larger step does not explain the gain. New runs extend these controls (Appendix D.5 and D.3, lines 1055–1072 and 1020–1039):

  • replaying Muon's own angular steps under explicit control gives 3.1977, against 3.1976 for Muon, so explicit control alone does not explain the gain;

  • at 300M, an angle that follows the learning-rate shape at the same total angular budget is 0.140 worse than ASM's schedule.

Controlling the angular step therefore gives lower validation loss, better test accuracy and lower sensitivity to the learning rate, and the controlled comparisons attribute these gains to how the angular step is scheduled. We hope that these results, most of which are in the submitted manuscript, together with the new controls in Appendix D, address this concern, and we would be glad to discuss any further comparison the reviewer has in mind.

W3 and Q3: Scale and the choice of λ\lambda

Response:

1. The choice λ=0.01\lambda = 0.01 is not weak weight decay. Decoupled weight decay multiplies the weights by (1−ηλ)(1-\eta\lambda) at each step, so its strength is set by the product ηλ\eta\lambda, not by λ\lambda alone. In [3], λ=0.1\lambda = 0.1 is paired with a peak learning rate of 4.2×10−44.2\times 10^{-4} for Moonlight and about 9×10−49\times 10^{-4} in its scaling-law runs, i.e., ηλ≈(4–9)×10−5\eta\lambda \approx (4\text{–}9)\times 10^{-5}. Our Muon recipe uses η=0.02\eta = 0.02 and λ=0.01\lambda = 0.01, i.e., ηλ=2×10−4\eta\lambda = 2\times 10^{-4}, which is 2–5 times stronger per step (Appendix D.1, lines 979–1001). In our setting, applying λ=0.1\lambda = 0.1 without rescaling gives ηλ=10−3\eta\lambda = 10^{-3} at learning rate 0.01, more than ten times the per-step decay of [3]. It makes Muon worse than λ=0.01\lambda = 0.01 (3.2192 vs. 3.2068; Sec. 5.3, lines 415–417), so λ=0.01\lambda = 0.01 is the stronger baseline. For ASM, λ\lambda no longer sets the step and only shrinks the norm: λ=0\lambda = 0 and 0.01 give the same loss (3.1806 vs. 3.1798), whereas λ=0.1\lambda = 0.1, with this tenfold per-step decay, degrades it to 3.5613 (line 414).

2. Other hyperparameters. On every workload, ASM uses that workload's Muon recipe unchanged, and Muon's learning rate is tuned. ASM only adds the schedule of Eq. (8) (lines 233–234). The 170M recipe starts from the configuration of the dense-model scaling experiments in [3], with learning rate 9.5×10−49.5\times 10^{-4} (Appendix A.3, lines 754–757), and there ASM lowers the final validation NLL from 3.0622 to 2.9988 (Table 2, line 332).

3. Scale. Our language-model results cover dense 127M and 300M models trained on 8.4B tokens, a 170M model on a longer recipe of 50,863 steps, and a 127M-active MoE (8 experts, top-2, about 250M parameters in total), in addition to two vision architectures. This is comparable to the language models used to verify the constraint in [1] (100M and 300M). The gain grows from 127M to 300M (0.029 to 0.136 in validation NLL) and carries over to the MoE without retuning (−0.036, against −0.026 for the dense model at the same length; Appendix D.8, Table 18, lines 1100–1109).

References

[1] Chen, Li, Liu. Muon Optimizes Under Spectral Norm Constraints. TMLR 2026.

[2] Pethick et al. Training Deep Learning Models with Norm-Constrained LMOs. ICML 2025.

[3] Liu et al. Muon is Scalable for LLM Training. arXiv 2025.

[4] Wen et al. Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization. arXiv 2026.

[5] van Laarhoven. L2 Regularization versus Batch and Weight Normalization. arXiv 2017.

[6] Li, Arora. An Exponential Learning Rate Schedule for Deep Learning. ICLR 2020.

[7] Kosson, Messmer, Jaggi. Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks. ICML 2024.

[8] Vyas et al. SOAP: Improving and Stabilizing Shampoo using Adam for Language Modeling. ICLR 2025.

[9] Zhao et al. Deconstructing What Makes a Good Optimizer for Language Models. ICLR 2025.

Reviewer_JDsxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the implicit step size decay phenomenon in the Muon optimizer. Since the update norm of Muon is primarily determined by the matrix shape, and the weight norm continuously increases during training, the relative update magnitude decreases, even if the learning rate remains constant. To address this issue, the authors propose Angle-Scheduled Muon (ASM), which explicitly controls the relative update magnitude of each matrix through time scheduling while preserving the original update direction of Muon. Experiments covering language model pre-training and image classification demonstrate that ASM can improve training performance with relatively small additional computational overhead.

Critical strengths
  1. The research motivation is clear, and the methodology is concise. Starting with the evolution of Muon's weight norm and update norm, the authors analyze the implicit step size decay phenomenon and propose a method for explicitly controlling the relative update magnitude. The paper is clearly written and has good readability.

  2. The experiments are comprehensive and provide some mechanism verification. The experiments involve language models and visual architectures of different sizes. Ablation experiments are used to examine the effects of time scheduling and matrix update allocation, and the additional computational overhead is reported.

Critical weaknesses
  1. The connections with existing research are strong, but the independent contributions are not yet clear enough. Works such as MuonH/Hyperball[1], Rotational Equilibrium[2], and OrScale[3] have studied the relationship between weight norm, relative update magnitude, and angle optimization from different perspectives. In particular, Hyperball May Not Be a Free Lunch[4] has discussed the dynamic changes in the effective learning rate of the angle and the importance of time scheduling. In addition, ASM, under the experimental configuration of weight norm preservation, has the same update form as MuonH, only the scheduling strategy is different. Although this paper discusses some related works, it has not fully explained the essential differences between its core insights and these existing understandings. This is my main concern.

  2. Experimental fairness and mechanism evidence still need to be strengthened. The hyperparameter search budgets of different optimizers are not completely consistent. ASM's gains on the 300M model are significantly greater than those on the 127M model, but there is a lack of sufficient explanation and mechanism ablation of the corresponding scale. In addition, the current experiment cannot completely distinguish the role of explicit angle scheduling itself and the benefits brought by a more suitable update magnitude configuration.

[1]Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization [2]Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks [3]OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling [4]Hyperball May Not Be a Free Lunch

Questions for authors

I have a few key questions. If the authors can clarify the substantial differences from existing work in the rebuttal and provide more convincing experimental support, I am willing to raise the score.

  1. Why does implicit step decay necessarily need to be corrected? Weight growth leading to a decrease in relative update magnitude could also be a beneficial adaptive mechanism. Could the authors further explain whether the gains of ASM come from eliminating this implicit decay or from empirically more suitable time scheduling? Are there training phases or conditions where implicit decay is actually beneficial?

  2. What is the essential difference between ASM and angle optimization methods like MuonH? Especially under the configuration of preserving the weight norm, the update forms of ASM and MuonH are equivalent. Could the authors demonstrate, through comparative experiments with the same training configuration and matched scheduling, how much of ASM's gains come from the new optimization mechanism rather than scheduling adjustments in existing methods?

  3. Why is the performance improvement of ASM on a 300M model much greater than that on a 127M model? Does this mean that explicit angle scheduling has an advantage as the model size increases? Could the authors verify this phenomenon under conditions of fair hpyerparameter tuning and matched update budgets, and further explain the underlying mechanism?

11 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer JDsx (Part1)

We thank the reviewer for the careful and constructive review, and we are glad that the reviewer finds the motivation clear, the method concise, the paper easy to read, and the experiments comprehensive, with ablations that verify the mechanism. Below we address each weakness (W) and question (Q). All changes in the revised PDF are marked in blue, and the new experiments are collected in Appendix D (lines 972–1109). The new experiments are single-seed reruns on one platform, so their numbers differ slightly from the three-seed means in Table 1 (line 270) and from the controls in Table 3 (line 378). Line numbers below refer to the revised PDF.

W1: Independent contributions

Response:

1. Our contribution and how it differs from prior work. We show that in standard Muon, weight growth makes the angular step decay in a way nobody chooses (by 59.2% at a constant learning rate at 127M; Fig. 1, lines 46–47), and ASM replaces this decay with a designed schedule. The cited works control or analyze the angle in different ways:

How the angle is controlledAngle over training
Rotational Equilibrium [2]Balance of learning rate and weight decaySettles to an equilibrium
OrScale [3]Per-layer relative update sizeFollows the learning-rate shape
Hyperball / MuonH [1]Fixed weight normFollows the learning-rate shape
[4] (concurrent)Analysis of Hyperball's phase-dependent effective step; tests decay schedules under a fixed normDoes not propose a schedule for Muon with weight decay
ASMSet to θt\theta_t for every matrix at every step; no norm constraint requiredDesigned schedule: rises to a peak, then decays as a power law

ASM contributes a specific angular schedule, Eq. (8), that rises to a later peak and then decays as a power law, and applies it to standard Muon without requiring a norm constraint: we use it with weight decay for the language models and with norm preservation on the 170M and vision recipes. We now cite [4] as concurrent work in Section 2 (lines 120–122). The experiments below show that this schedule outperforms MuonH and that the gain comes from the schedule, not the update mechanism.

2. ASM's schedule outperforms tuned MuonH [1]. We implemented MuonH, the Muon instance of Hyperball [1], in the same codebase and training recipe (Appendix D.2, Table 12, lines 1002–1019) and tuned its level L over {L*/2, L*, 2L*}, where L* matches ASM's total angular budget ∑tθt2\sum_t\theta_t^2. On ViT-S, where 2L* was selected, we also ran 4L*, which was worse, so every selected level is interior. The table reports final validation NLL and test top-1 (Muon and ASM numbers at 300M and on vision from Table 1, line 270); † marks the selected level.

127M (NLL ↓)300M (NLL ↓)ResNet-50 (top-1 ↑)ViT-S (top-1 ↑)
Muon3.19763.008677.7271.30
MuonH, L*/23.20863.024179.5470.87
MuonH, L*3.2034†3.0150†80.26†71.97
MuonH, 2L*3.25853.075179.4871.65†
ASM3.16782.872581.2673.03
ASM − tuned MuonH−0.036−0.143+1.00+1.38

ASM is better than tuned MuonH on all four workloads.

References

[1] Wen et al. Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization. arXiv 2026.

[2] Kosson, Messmer, Jaggi. Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks. ICML 2024.

[3] Lou, You. OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling. arXiv 2026.

[4] Xiao et al. Hyperball May Not Be a Free Lunch. arXiv 2026.

AuthorsOpenReview ↗

Rebuttal for Reviewer JDsx (Part2)

3. Under norm preservation, ASM and MuonH [1] share the update form; the gain comes from the schedule, not the mechanism. To separate the two, we ran a 2×2 on the same recipe: the update mechanism (free norm with weight decay, as in ASM, or fixed norm, as in MuonH) and the angular schedule (ASM's schedule, or the flat, recipe-shaped schedule that MuonH uses, at the same total angular budget ∑tθt2\sum_t\theta_t^2). The fixed-norm, flat cell is tuned MuonH. On ResNet-50 and ViT-S, ASM itself preserves the norm, so it differs from MuonH only in the schedule (final validation NLL and test top-1; Appendix D.3, Table 13, lines 1020–1039):

WorkloadMechanismASM's scheduleMuonH's scheduleΔ (ASM's − MuonH's)
127M (NLL ↓)Free norm + WD3.16783.2025−0.035
127M (NLL ↓)Fixed norm3.16683.2034−0.037
300M (NLL ↓)Free norm + WD2.87253.0130−0.140
300M (NLL ↓)Fixed norm2.88173.0150−0.133
ResNet-50 (top-1 ↑)Fixed norm81.2680.26+1.00
ViT-S (top-1 ↑)Fixed norm73.0371.65+1.38

Changing the schedule improves NLL by 0.035–0.140 and top-1 by 1.00–1.38 points in every row, whereas changing the mechanism at the same scale moves NLL by at most 0.009. Controlling the angle through weight decay does not substitute for the schedule either: weight decay 0.1 keeps Muon's angle from falling but raises the loss from 3.2068 to 3.2192 (Sec. 5.3, lines 415–417). The essential difference from these works is therefore the angular schedule, and this is where ASM's gain over MuonH comes from.

W2: Experimental fairness and mechanism evidence

Response:

(a) Search budget. For the strongest baselines, AngularMuown and MuonH, we ran an additional round of tuning on every workload, extending each grid until the selected value is interior. The table lists every point, with the selected one in bold (periodic validation NLL for the language models, validation top-1 for vision; Appendix D.4, Table 14, lines 1040–1054, and Appendix D.2, Table 12, lines 1002–1019):

Baseline / workloadGrid
AngularMuown, 127M0.01: 3.2418 · 0.02: 3.1865 · 0.04: 3.1737 · 0.08: 3.2043
AngularMuown, 300M0.01: 3.0372 · 0.02: 2.9771 · 0.04: 2.9755 · 0.08: 3.0242 · 0.16: 3.1674
AngularMuown, ResNet-500.005: 79.04 · 0.01: 80.46 · 0.02: 79.84 · 0.04: 76.70
AngularMuown, ViT-S0.005: 71.08 · 0.01: 71.90 · 0.02: 70.44 · 0.04: 64.42
MuonH, 127ML*/2: 3.2175 · L*: 3.2121 · 2L*: 3.2680
MuonH, 300ML*/2: 3.0319 · L*: 3.0224 · 2L*: 3.0832
MuonH, ResNet-50L*/2: 79.72 · L*: 81.08 · 2L*: 79.94
MuonH, ViT-SL*/2: 70.62 · L*: 71.96 · 2L*: 72.18 · 4L*: 69.76

Every selected value is interior, so these baselines are not limited by their grids. For Muon, the learning rate is tuned over {0.01, 0.02, 0.04}, with an interior optimum at 0.02. ASM keeps Muon's learning rate and fixes a = 0.46; at 127M and 300M it also shares tpt_p and θpeak\theta_{\mathrm{peak}}, so only b is selected (Table 6, line 722).

(b) The larger gain at 300M, and the mechanism ablation at that scale. In Table 1 (line 270), ASM improves over Muon by 0.029 at 127M and by 0.136 at 300M. To explain this, we ran the mechanism ablation at 300M as well as at 127M: a 2×2 of update mechanism (free norm with weight decay, or fixed norm) × angular schedule (ASM's, or a flat, recipe-shaped schedule at the same total angular budget ∑tθt2\sum_t\theta_t^2) on the same recipe. We also measured Muon's implicit decay at both scales (Appendix D.3 and D.6, Tables 13 and 16, lines 1020–1039 and 1073–1087):

127M300M
Muon's angular step at step 2,0000.00500.0051
Muon's angular step at step 12,8000.00360.0037
Muon's weight-norm growth to step 12,800 (median over matrices)5.0×5.1×
Schedule effect, free norm (ASM − flat)−0.035−0.140
Schedule effect, fixed norm (ASM − flat)−0.037−0.133
Mechanism effect, ASM schedule (fixed − free)−0.001+0.009

Muon's implicit decay is nearly identical at the two scales, because Muon's update is normalized by matrix shape, so the larger gain does not come from Muon decaying more at 300M. The mechanism effect stays within 0.009 at both scales, whereas the schedule effect grows from 0.035–0.037 at 127M to 0.133–0.140 at 300M. The larger gain at 300M is therefore attributable to the schedule: the larger model is more sensitive to when the angular budget is spent.

AuthorsOpenReview ↗

Rebuttal for Reviewer JDsx (Part3)

(c) Explicit scheduling versus a more suitable update magnitude. To separate the two, we compared ASM at 127M with variants that change only the size of the updates or only how they are controlled (final validation NLL; Appendix D.5, Table 15, lines 1055–1072):

VariantWhat it testsNLL
MuonReference3.1976
Muon with every update 1.5× larger (Table 3, line 378)A larger update magnitude3.1954
Explicit control, replaying Muon's own angular stepsExplicit control without changing Muon's angular steps3.1977
Explicit control, same total angular budget ∑tθt2\sum_t\theta_t^2 as ASM, following the learning-rate shapeThe same total update magnitude as ASM3.2025
ASM: explicit control with ASM's scheduleThe same total budget, distributed by ASM's schedule3.1678
  • Enlarging Muon's updates changes NLL by only 0.002, so a larger update magnitude does not explain the gain.

  • Explicit control that follows Muon's own trajectory reproduces Muon, so explicit control alone does not explain it either.

  • With the same total angular budget as ASM but distributed along the learning-rate shape, the result is 0.035 worse than ASM, so the gain does not come from a better-calibrated overall magnitude.

The gain therefore comes from how ASM distributes the angular step over training.

Q1: Why must implicit step decay be corrected?

Response:

1. Decay itself is beneficial, and ASM keeps it. We agree that a decreasing relative update can be beneficial, and ASM does not remove it: its angular step decays after the peak at tpt_p, and removing this decay hurts (at 127M, b = 0 is 0.027 worse than b = 0.7; Table 8, line 810). What ASM changes is what sets the decay. In Muon, it is set by weight growth: the angular step peaks around step 150, falls to 0.005 by step 2,000, and then barely decays (≈t−0.3\approx t^{-0.3}) until the final annealing; during the constant-learning-rate phase it also falls by different amounts across matrices, from 47% to 71% (Fig. 4; lines 402–403). In ASM, the decay is chosen in advance and is the same for every matrix.

2. The gain comes from a more suitable schedule, not from eliminating decay. At 127M, we compared schedules that all decay but differ in shape, using the same explicit control (final validation NLL; Appendix D.5, Table 15, lines 1055–1072):

Angular-step scheduleNLL
Muon (implicit decay, no angular control)3.1976
Muon's own decay, replayed exactly under explicit control3.1977
Power law fitted to Muon's decay3.1889
ASM's schedule3.1678

Following Muon's decay exactly gives Muon's loss, a smoother version of it gains only 0.009, and ASM's schedule gains 0.030. Every schedule here decays, so the gain comes from the shape of the decay, not from removing it.

3. When implicit decay helps. Early in training. At the start, Muon's implicit angular step is at its largest while ASM's is still rising, and Muon's loss is lower; ASM overtakes Muon later and finishes lower (Fig. 3, line 298). In our T = 8,000 runs, ASM overtakes Muon near step 3,000 and stays ahead. Implicit decay therefore helps early progress, but ASM's schedule gives the better final model.

Q2: Difference from MuonH: mechanism or schedule?

Response:

1. The essential difference is the angular schedule. Under norm preservation, ASM and MuonH share the update form. They differ in how the angle evolves over training: MuonH's angle follows the learning-rate shape, whereas ASM's follows a designed schedule that rises to a peak and then decays as a power law. ASM also works without norm preservation, with ordinary weight decay, as in our language-model configurations.

2. How much of the gain comes from the mechanism, and how much from the schedule. We ran this comparison on the same training configuration (Appendix D.3, Table 13, lines 1020–1039), crossing the update mechanism (free norm with weight decay, as in ASM, or fixed norm, as in MuonH) with the angular schedule (ASM's schedule, or the learning-rate-shaped schedule that MuonH uses). With fixed norm, the learning-rate-shaped schedule is MuonH with its level tuned over {L*/2, L*, 2L*}; with free norm, it uses the same total angular budget ∑tθt2\sum_t\theta_t^2 as ASM. On ResNet-50 and ViT-S, ASM itself preserves the norm, so it differs from MuonH only in the schedule. The full results are in Appendix D.3, Table 13, lines 1020–1039.

Changing the mechanism at the same scale moves NLL by at most 0.009, whereas changing the schedule improves it by 0.035–0.140 in NLL and by 1.00–1.38 points in top-1. Relative to MuonH, essentially all of ASM's gain therefore comes from the schedule, and tuning MuonH's level does not recover it.

Reviewer_ZZ5EOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Angle-Scheduled Muon (ASM), which explicitly controls the angular step to address the implicit step decay caused by weight norm growth in Muon. The method preserves Muon's orthogonalized update direction while adjusting the update magnitude according to a predefined angular-step schedule.

Critical strengths
  1. Simple and well-motivated idea. The motivation is clear and intuitive. The observation that weight norm growth implicitly reduces the angular step, even with a constant learning rate, is interesting. Explicitly controlling this quantity is a natural and straightforward solution.

  2. Clear presentation. The paper is well-organized and easy to follow. Both the motivation and the proposed algorithm are explained clearly.

Critical weaknesses
  1. Transferability. Equation (9) uses the nominal learning rate ηt\eta_t, while Algorithm 1 introduces a separate angular-step schedule θt\theta_t. However, their relationship is not explicitly characterized. Maintaining two different schedules may increase tuning complexity and make ASM harder to transfer across models and training recipes.

  2. Comparison with Hyperball. Hyperball also explicitly controls angular updates, making it a closely related approach. Although ASM has the advantage of not requiring fixed weight norms, a direct experimental comparison with Hyperball would be valuable to better understand their relative performance.

Questions for authors

1) Robustness and reproducibility. The reported improvements are substantial for a relatively simple modification. Could you clarify how consistently these gains hold across random seeds and the training settings evaluated? Additional information on variability would help assess their reliability. 2) Selection of scheduling parameters. How are aa and bb in Algorithm 2 selected, and how do they depend on the total training budget TT? If TT changes substantially, can the same values be reused, or is retuning required? 3) Connection to effective learning rate. How does ASM relate to the effective learning rate (ELR) studied in [1]? In particular, could an ELR trajectory measured during standard Muon training inform the construction of ASM’s angular-step schedule? An explanation of this relationship would help clarify the rationale for the proposed schedule. 4)Hyperparameter sensitivity and transferability. Based on the sensitivity experiments, which hyperparameters require careful tuning, and which can generally retain their default values across settings? Could you clarify whether the configurations used in the paper were transferred directly or tuned separately? 5)Scope and scalability. The nanoGPT-style experiments provide a useful controlled evaluation. How far do you expect the findings to transfer to larger models and more realistic training setups, and what evidence supports that expectation? As a direction for future validation, experiments on larger dense models, such as Qwen3-0.6B or above, and separately on MoE architectures would help establish practical scalability.

6 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer ZZ5E (Part 1)

We thank the reviewer for the positive assessment and the constructive suggestions, and we are glad that the reviewer finds the idea simple and well motivated, the observation that weight growth shrinks the angular step even at a constant learning rate interesting, and the presentation clear. Below we address each weakness (W) and question (Q). All changes in the revised PDF are marked in blue, and the new experiments are collected in Appendix D (lines 972–1109). The new experiments are single-seed reruns on one platform, so their numbers differ slightly from the three-seed means in Table 1 (line 270) and from the controls in Table 3 (line 378). Line numbers below refer to the revised PDF.

W1: Transferability

Response:

1. Where ηt\eta_t appears in ASM. ηt\eta_t appears only in the weight-decay factor (1−ηtλ)(1-\eta_t\lambda) of Eq. (9) (line 736). The update itself, ΔWi,t=−θt∥Wi,t∥F Yi,t/∥Yi,t∥F\Delta W_{i,t} = -\theta_t\lVert W_{i,t}\rVert_F\, Y_{i,t}/\lVert Y_{i,t}\rVert_F (Eq. 6, line 207), does not contain ηt\eta_t.

2. The speed of learning is set by θt\theta_t. The update length is θt∥Wi,t∥F\theta_t\lVert W_{i,t}\rVert_F, so the angular step of every controlled matrix equals θt\theta_t (Eq. 7, line 215). In Muon, the angular step is ri,t=ηt ci∥Yi,t∥F/∥Wi,t∥Fr_{i,t} = \eta_t\, c_i\lVert Y_{i,t}\rVert_F/\lVert W_{i,t}\rVert_F, where the factor after ηt\eta_t shrinks as the weights grow and differs across matrices. ASM replaces this factor with one that is chosen in advance and shared by all matrices. θt\theta_t is built on the recipe's learning-rate shape q(t)q(t): with ηt=η q(t)\eta_t = \eta\, q(t), Eq. (8) (lines 233–234) gives

θt=θpeak q(t)q(tp) g(t)=κ ηt g(t),κ=θpeakη q(tp),\theta_t = \theta_{\mathrm{peak}}\,\frac{q(t)}{q(t_p)}\,g(t) = \kappa\,\eta_t\,g(t),\qquad \kappa = \frac{\theta_{\mathrm{peak}}}{\eta\,q(t_p)},

where g(t)=(t/tp)ag(t) = (t/t_p)^{a} for t<tpt < t_p and (t/tp)−b(t/t_p)^{-b} for t≥tpt \ge t_p. Eq. (9) and Algorithm 1 (lines 231–238) therefore do not maintain two step schedules: the step follows θt\theta_t alone, which inherits the recipe's warmup and annealing. Because θpeak\theta_{\mathrm{peak}} is set directly, changing η\eta does not change θt\theta_t.

3. Why Eq. (9) keeps Muon's ηt\eta_t in the weight decay.

  • The recipe is unchanged apart from the update length. Keeping Muon's weight decay (1−ηtλ)(1-\eta_t\lambda) makes ASM a drop-in change, and any difference from Muon then comes from the update length alone.

  • The decay no longer has to follow the step. In Muon the decay must scale with the learning rate, because the balance between update and decay sets the weight norm and hence the angular step [2, 3]. In ASM the angular step is prescribed, so the decay only shrinks the weight norm, by ηtλ\eta_t\lambda per step (at most 2×10−42\times 10^{-4} at 127M). There is no reason to tie it to θt\theta_t, and Muon's setting suffices.

4. Tuning and transfer. η\eta adds no tuning and can be kept from the Muon recipe. Varying it over 0.005–0.02, which also changes the decay strength fourfold, changes ASM's final loss by only 0.0036, compared with 0.0200 for Muon (Fig. 6, line 461; Table 9, line 819); removing weight decay altogether changes it by 0.0008 (Sec. 5.3, line 414). When ASM is moved to a new recipe, q(t)q(t), η\eta and λ\lambda are taken from that recipe, and the only ASM-specific choices are the scalars of θt\theta_t.

AuthorsOpenReview ↗

Rebuttal for Reviewer ZZ5E (Part 2)

W2: Comparison with Hyperball

Response:

We implemented MuonH (the Muon instance of Hyperball [3]) in the same codebase and training recipe and compared it directly with ASM (Appendix D.2, Table 12, lines 1002–1019). Its level L was tuned over {L*/2, L*, 2L*}, where L* matches ASM's total angular budget ∑tθt2\sum_t\theta_t^2 (L* = 0.00715, 0.00626, 0.0104 and 0.0082 on the four workloads). As for all methods, the level is selected by periodic validation NLL (language models) or validation top-1 (vision). On ViT-S the best level was 2L*, so we also ran 4L*, which was worse (69.76 validation top-1); every selected level is therefore interior. The table reports final validation NLL on held-out tokens and test top-1 (Muon and ASM numbers at 300M and on vision are from Table 1, line 270); † marks the selected level.

Method127M (NLL ↓)300M (NLL ↓)ResNet-50 (top-1 ↑)ViT-S (top-1 ↑)
Muon3.19763.008677.7271.30
MuonH, L*/23.20863.024179.5470.87
MuonH, L*3.2034†3.0150†80.26†71.97
MuonH, 2L*3.25853.075179.4871.65†
ASM3.16782.872581.2673.03

Tuned MuonH is 0.036 / 0.143 NLL worse than ASM at 127M / 300M, and 1.00 / 1.38 points lower in test top-1 on ResNet-50 / ViT-S.

We attribute ASM's improvement to how it distributes the angular step over training. At L*, MuonH spends the same total angular budget as ASM but follows the recipe's learning-rate shape, whereas ASM raises the angle to a later peak and then decays it as a power law. We thank the reviewer for suggesting this comparison, which helped us clarify the mechanism behind ASM and improved the paper.

Q1: Robustness and reproducibility

Response:

1. Variability across seeds. Table 1 (line 270) reports mean ± standard deviation over three seeds for every method and workload. ASM's own standard deviations are 0.004 / 0.003 NLL at 127M / 300M and 0.27 / 0.57 points of top-1 on ResNet-50 / ViT-S.

2. The gains are large relative to this variability in every setting. We express ASM's gain over Muon as an effect size, d=gain/σpooledd = \mathrm{gain}/\sigma_{\mathrm{pooled}}, with σpooled=(σASM2+σMuon2)/2\sigma_{\mathrm{pooled}} = \sqrt{(\sigma_{\mathrm{ASM}}^2 + \sigma_{\mathrm{Muon}}^2)/2}, computed from the means and standard deviations in Table 1:

127M NLL300M NLLResNet-50 top-1ResNet-50 CEViT-S top-1ViT-S CE
Gain of ASM over Muon0.02890.13613.540.1501.730.061
Effect size d7.238.512.215.83.13.7

In every setting, the gain is at least 3.1 standard deviations, well above 0.8, the conventional threshold for a large effect.

AuthorsOpenReview ↗

Rebuttal for Reviewer ZZ5E (Part 3)

Q2: Selection of scheduling parameters

Response:

1. What a and b do. Eq. (8) (lines 233–234) multiplies the recipe's learning-rate shape q(t)q(t) by (t/tp)a(t/t_p)^{a} before the transition step tpt_p and by (t/tp)−b(t/t_p)^{-b} after it. a shapes how the angular step rises to its peak θpeak\theta_{\mathrm{peak}} at tpt_p, so it plays a role similar to a warmup for the angular step. b shapes how the angular step decays after the peak, so it acts as a decay schedule for the angular step: it replaces the decay that Muon inherits from weight growth with one chosen in advance.

2. What matters is the angular decay after the peak, not the exact exponents. Before tpt_p, the angular step rises with the recipe's warmup and the factor (t/tp)a(t/t_p)^{a}; after tpt_p, it decays through the factor (t/tp)−b(t/t_p)^{-b}. This angular decay is specific to ASM: it acts on the angular step itself, starts at tpt_p, and comes in addition to the recipe's learning-rate annealing in q(t)q(t). The results, however, are not sensitive to how fast the angular step rises or decays. The peak θpeak\theta_{\mathrm{peak}} and q(t)q(t) do not depend on a or b, so the two exponents only change the curve between these fixed parts: a acts only during the first tpt_p steps (12.5–19% of training in our workloads), and for every b > 0 the angular step starts at θpeak\theta_{\mathrm{peak}}, decreases, and is then annealed by q(t)q(t). Removing the angular decay is different: with b = 0, the angular step simply follows the recipe's learning-rate shape.

3. Results. In the 127M sensitivity study (Sec. 5.3, lines 405–408; Table 8, line 810; Fig. 5, line 399), only b is changed, with all other settings fixed; Muon uses the same recipe (final validation loss):

SettingLoss
ASM, b = 0.53.1818
ASM, b = 1.03.1809
ASM, b = 0 (no angular decay)3.2071
Muon3.2068

b = 0.5 and b = 1.0 finish within 0.001 of each other and 0.025–0.026 below Muon, whereas b = 0, without the angular decay, falls back to Muon's level.

4. How we set them. Because the results are not sensitive to the exponents, we keep a = 0.46 on all five workloads and select b by validation performance for each workload: 1.0 at 127M, 1.2 at 300M, 0.7 at 170M, 0.9 on ResNet-50 and 0.7 on ViT-S (Table 6, line 722).

5. Dependence on T, and reuse when T changes. Because a and b act on t/tpt/t_p, they do not need to change with T as long as tpt_p is scaled with T, as the warmup and decay phases of the recipe's q(t)q(t) already are: the schedule then keeps the same shape over the course of training. To test this directly, we reran the paper's 127M configuration at T = 4,000 and 8,000 with tpt_p = T/8 (as at T = 16,000) and a, b and θpeak\theta_{\mathrm{peak}} unchanged. Muon kept its tuned LR, and neither method was retuned (final validation NLL; Appendix D.7, Table 17, lines 1088–1099):

T (steps / tokens)ASMMuonASM − Muon
4,000 / 2.1B3.37263.3823−0.010
8,000 / 4.2B3.24983.2757−0.026
16,000 / 8.4B3.16783.1976−0.030

Without retuning, ASM is better than Muon at every training length. The same a and b can therefore be reused when T changes, with tpt_p scaled with T.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.