acceptodds
Under review as a conference paper at ICLR 2027

Does U-W Ratio Alignment Benefit Muon in LLM Pretraining?

Abstract

Most top-performing entries in the `modded-nanogpt` optimization benchmark use a "u/w floor". The u/w floor lower-bounds the Frobenius-norm ratio of each block's pre-learning-rate update to its weight (*u-w ratio*). Like decoupled weight decay, the u/w floor prevents updates from becoming negligible relative to the current weights during training. Unlike decoupled weight decay, however, the u/w floor also aligns u-w ratios across blocks. Specifically, it aligns the u-w ratios of most weight blocks in a model architecture to a common value on most training steps. We therefore ask whether this u-w ratio alignment is an underlying mechanism of the u/w floor's empirical advantage. To study this question, we remove the u/w floor and directly enforce u-w ratio alignment on Muon updates. This gives *MuonL*, which assigns a common ratio to all Muon-updated blocks, and *typed MuonL (t-MuonL)*, which aligns only blocks with the same Transformer role. To assess these policies independently of particular learning-rate values, we derive a measurable local diagnostic from a theoretical model of one-step loss decrease. When all Muon-updated blocks share the same learning rate, the analysis yields a covariance-based sufficient condition for when MuonL improves upon Muon. When the learning rates can vary across matrix types, it yields a generalized-eigenvalue criterion for when t-MuonL outperforms Muon and vice versa. We then evaluate the diagnostic at checkpoints along Muon training trajectories, using each checkpoint as a starting point to compare Muon's one-step decrease with the counterfactual decreases under MuonL and t-MuonL. It favors t-MuonL over Muon in both cases and MuonL over Muon in the former. Controlled runs using the same training configurations corroborate the diagnostic, with lower final validation losses for MuonL or t-MuonL than for Muon. These results support u-w ratio alignment as a contributing factor to the u/w floor's empirical advantage and a useful intervention for Muon-based LLM pretraining.

Primary area:
optimization
Keywords:
Muon optimizer, update-to-weight ratio, u/w floor, local optimization diagnostics, LLM pretraining dynamics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aKh9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether U-W ratio alignment is a crucial mechanism for improving the pre-training performance of Muon through u/w floor. The authors propose MuonL and t-MuonL, aligning relative update magnitudes between all matrices and within matrices of the same type, respectively. Theoretically, the paper derives sufficient conditions based on covariance and a generalized eigenvalue criterion using local second-order Taylor approximation to compare the local optimization potential of different alignment strategies. Experiments, including Hessian-vector products in the training trajectory, multi-random seed training, and benchmark validation, report performance improvements relative to Muon.

Critical strengths

S1. The research problem is of practical significance, and the method design is relatively simple.

Starting from the empirical benefits of the u/w floor, this paper attempts to isolate the effect of U-W ratio alignment and proposes MuonL and t-MuonL suitable for different learning rate configurations. This problem is closely related to the control of module update scale in LLM pre-training. The method only adjusts the update magnitude without changing the update direction of Muon, which has certain practical value.

S2. The theoretical analysis and experimental evidence are relatively systematic.

The authors not only derived the local optimization criterion but also used Hessian-vector products in the training trajectory to measure directional curvature and cross-matrix interactions, and tested the theoretical criterion at multiple checkpoints. Multi-random seed experiments and benchmark statistical validation also support the performance gains of the method. Compared with simply reporting empirical improvements, this paper attempts to establish a connection between mechanism hypotheses, theoretical diagnosis, and actual training performance.

Critical weaknesses

W1. Section 2 (Equations 1–2) defines the instantaneous U-W ratio alignment, and Sections 3.1 and 3.3 (Equations 5, 12) further derive the local second-order optimization score based on this ideal update rule. However, I found that Appendix A.2 (pages 13–14, Algorithms 1–2) states that the actual algorithm refreshes the scaling factor only every 5 steps and uses EMA (\beta=0.9) for smoothing. The authors also explicitly state in Appendix A.2 (line 667-669, Practical implementation) that the theoretical diagnostic does not consider the refresh interval and EMA smoothing. Therefore, actual updates generally do not satisfy the instantaneous alignment relationship in the theoretical analysis. Furthermore, the theory allows for separate optimization of the learning rate, while actual training uses the learning rate configuration and scheduling specified in Appendix A.1 (page 12). The paper lacks a direct comparison of per-step refresh without EMA, making it impossible to determine whether the performance gains truly come from the alignment mechanism studied in the theory.

W2. The theoretical criteria have not yet revealed a sufficiently explicit optimization structure. Proposition 1 expresses the gains as a relationship between first-order descent, directional curvature, and rescaling costs, while Proposition 2 characterizes the local optimization potential of different strategies through generalized eigenvalue decomposition. These results are mathematically valuable, but primarily provide conditions for determining when alignment is advantageous; they do not explain why real Transformer training systematically satisfies these conditions. For example, the gains in Figure 2(b) mainly come from first-order descent, while the intra-group gains in Figure 3(b) mainly come from curvature improvement; this difference is currently difficult to explain because the alignment strategy and learning rate configuration change simultaneously. The paper also does not fully reveal the actual module structure and stability corresponding to the feature modes.

However, I appreciate the authors' attempt to analyze this method, which has some practical effectiveness, but at present, I believe the depth of the theoretical analysis is insufficient to convince me. If the authors can verify the actual gains of instantaneous alignment through direct ablation and further provide a quantitative link between the theoretical criteria and the real training structure or actual optimization gains, I would be willing to raise the rating to Weak Accept.

Questions for authors
  1. Does the actual performance gain truly stem from the instantaneous U-W ratio alignment analyzed in the theory? Why does the actual implementation in Appendix A.2 (Algorithms 1–2) employ refresh every 5 steps and EMA smoothing? Could the authors compare the four schemes K=1,β=0K=1,\beta=0, K=1,β=0.9K=1,\beta=0.9, K=5,β=0K=5,\beta=0, and the current scheme K=5,β=0.9K=5,\beta=0.9 under the same training configuration, and report the actual updated U-W ratio alignment and validation loss?

  2. In particular, if EMA is removed and alignment is performed at each step, can performance gains comparable to those in the paper still be achieved? This experiment is crucial for establishing the connection between the theoretical diagnostics in Section 3 and the actual training results in Section 4.

  3. What structural patterns in Transformer training do the theoretical criteria reveal?Proposition 1 and Proposition 2 present the covariance condition and the generalized eigenvalue criterion, respectively. But do these conditions correspond to the stable gradient-curvature structure present in real training?

  4. Could the authors further analyze the specific distributions of first-order descent, directional curvature, and cross-module interactions in different matrix types and training stages to explain the different dominant sources of gain in Figure 2(b) and Figure 3(b)? Can these structures predict the true one-step loss reduction under actual learning rates, rather than merely explaining the local second-order scores optimized by the learning rate?

4 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer "aKh9" (Part 1/2)

We thank the reviewer for the careful reading, and in particular for the questions on our theory, which led to the new analysis of the structure of the criteria in Appendix G. Taking into account the comments of all three reviewers, we have revised the paper and uploaded the new version to OpenReview, with newly added content shown in blue. To address the conditions the reviewer stated for raising the rating, we have run the requested ablation of the refresh and EMA scheme, and we have linked the criteria to the structure of real training. The revision also includes results for a 305M-parameter model.

Below, we address the reviewer's comments one by one and describe the corresponding revisions. We would welcome any further suggestions and would be happy to revise the paper accordingly.

  1. "Does the actual performance gain truly stem from the instantaneous U-W ratio alignment analyzed in the theory? Why does the actual implementation in Appendix A.2 (Algorithms 1–2) employ refresh every 5 steps and EMA smoothing? Could the authors compare the four schemes [K = 1, β = 0], [K = 1, β = 0.9], [K = 5, β = 0], and the current scheme [K = 5, β = 0.9] under the same training configuration, and report the actual updated U-W ratio alignment and validation loss? In particular, if EMA is removed and alignment is performed at each step, can performance gains comparable to those in the paper still be achieved?"

    We thank the reviewer for proposing this ablation. We ran all four schemes for both methods and list the final validation losses below. The results show that the gain does stem from u-w ratio alignment. First, every scheme aligns the actual updated u-w ratios after the first few hundred steps (Figure 5). Second, t-MuonL outperforms Muon under all four schemes, including the instantaneous alignment analyzed in our theory (K=1K=1, β=0\beta=0). MuonL also outperforms Muon under every scheme except instantaneous alignment. This exception arises because MuonL equalizes the u-w ratios of the zero-initialized WOW_O and WdownW_{\mathrm{down}} with those of the other blocks from the very beginning, making their updates inappropriately small.

    This explains why our implementation uses refresh and EMA. They let MuonL realize the benefit of alignment when update scales change abruptly or some weight norms are close to zero, just as K-FAC (Martens & Grosse, 2015) and SOAP (Vyas et al., 2024) refresh and smooth their online curvature estimates to apply them reliably. Our scheme with K=5K=5 and β=0.9\beta=0.9 indeed gives the lowest loss for both methods. However, it introduces no separate mechanism and only makes the fullest use of u-w ratio alignment. We now report this ablation in Appendix A.2 (Table 2 and Figure 5).

    SchemeMuonL (shared LR)t-MuonL (type-specific LR)
    K=1K=1, β=0\beta=0 (instantaneous)3.290303.27730
    K=1K=1, β=0.9\beta=0.93.279633.27727
    K=5K=5, β=0\beta=03.278823.27748
    K=5K=5, β=0.9\beta=0.9 (paper)3.277193.27611
    Muon3.280553.27838

  2. "For example, the gains in Figure 2(b) mainly come from first-order descent, while the intra-group gains in Figure 3(b) mainly come from curvature improvement; this difference is currently difficult to explain because the alignment strategy and learning rate configuration change simultaneously."

    We thank the reviewer for pointing out this contrast. The difference is caused by which blocks are aligned, not by the learning-rate configuration. MuonL can be split into two steps. The first step aligns the u-w ratios within each matrix type, which is exactly what t-MuonL does. The second step aligns the resulting ratios across matrix types. Both gains in the MuonL criterion (Eq. 11) are covariances over blocks, and each covariance splits between the two steps as cov⁡π(f,g)=∑GπGcov⁡G(f,g)⏟within-type+∑GπG(fˉG−fˉ)(gˉG−gˉ)⏟across-type,\operatorname{cov}_\pi(f,g)=\underbrace{\sum_{\mathcal G}\pi_{\mathcal G}\operatorname{cov}_{\mathcal G}(f,g)}_{\text{within-type}}+\underbrace{\sum_{\mathcal G}\pi_{\mathcal G}(\bar f_{\mathcal G}-\bar f)(\bar g_{\mathcal G}-\bar g)}_{\text{across-type}}, where fˉG\bar f_{\mathcal G} and cov⁡G\operatorname{cov}_{\mathcal G} are the mean and covariance within type G\mathcal G, and fˉ\bar f is the mean over all blocks. This split reveals opposite trade-offs at all 13 checkpoints: the within-type alignment yields a curvature gain at the cost of first-order descent, and the across-type alignment yields a first-order descent gain at the cost of curvature (see Appendix G.1, Figure 13). The trade-offs hold for both the shared-learning-rate and the type-specific trajectories, so the learning-rate configuration does not cause the difference. t-MuonL, which performs only the within-type step, thus gains through curvature in Fig. 3(b), while MuonL draws its first-order gain in Fig. 2(b) from the across-type step.

AuthorsOpenReview ↗

Authors' Response to Reviewer "aKh9" (Part 2/2)

  1. "These results are mathematically valuable, but primarily provide conditions for determining when alignment is advantageous; they do not explain why real Transformer training systematically satisfies these conditions."

    We thank the reviewer for this comment. For t-MuonL, we explain this through the homogeneity of layers within each matrix type. Motivated by scale invariance, we assume that the layers of each type share the same first-order descent and curvature per unit relative step (Assumption 1). Under this assumption, Proposition 8 (Appendix G.3) shows that within-type alignment yields a curvature gain that outweighs its first-order cost, and that t-MuonL has at least the optimization potential of Muon in the comparison of Proposition 2.

    We test the homogeneity assumption at the 13 checkpoints (Figure 15). Averaged over checkpoints and matrix types, the first-order descent varies by 19% across the layers of a type, against 74% across all 72 Muon-updated blocks, and the curvature by 44%, against 76%. The assumption thus holds approximately within types but fails across them, so it is not surprising that across-type alignment reverses the trade-off, gaining in first-order descent at the cost of curvature.

  2. "The paper also does not fully reveal the actual module structure and stability corresponding to the feature modes."

    We thank the reviewer for this comment. The eigenmodes of Proposition 2 are indeed not stable. They carry no fixed meaning, since they only form the basis in which the criterion splits into independent terms. A structure over matrix types that is stable along training appears instead in the covariance-based criterion of Proposition 1, as we will discuss in the next point.

  3. "Could the authors further analyze the specific distributions of first-order descent, directional curvature, and cross-module interactions in different matrix types and training stages to explain the different dominant sources of gain in Figure 2(b) and Figure 3(b)?"

    We thank the reviewer for this suggestion. We split the two gains of the MuonL criterion by component and by matrix type (Appendices G.1 and G.2). By component, the curvature gain consists of the directional curvatures CiiC_{ii}, the interactions between layers of the same type, and the interactions across types. Attributing these three parts and the first-order descent gain to within-type and across-type alignment yields eight parts, each a consistent gain or cost at all 13 checkpoints on both trajectories (Table 6). By matrix type, the curvature gain of t-MuonL in Fig. 3(b) comes mainly from WVW_V and WOW_O, and the first-order descent gain of MuonL in Fig. 2(b) mainly from WdownW_{\mathrm{down}} (Figure 14). These patterns remain stable throughout training.

  4. "Can these structures predict the true one-step loss reduction under actual learning rates, rather than merely explaining the local second-order scores optimized by the learning rate?"

    We thank the reviewer for this question. Our scores measure a policy's optimization potential: how much it can lower the loss in one step at the best learning rate. Actual training does not use this full potential at every step, since its learning-rate schedule favors stable convergence over the largest immediate decrease. By Taylor expansion, substituting the actual learning rate into the quadratic model would naturally approximate the one-step reduction, but predicting it is not the aim of our criteria.

  5. "Furthermore, the theory allows for separate optimization of the learning rate, while actual training uses the learning rate configuration and scheduling specified in Appendix A.1 (page 12)."

    We thank the reviewer for this comment. In Section 3.4, we analyze our criteria under the actual training schedule, where only the peak learning rate η\eta is optimized. With learning rate ηϕt\eta\phi_t at step tt, the loss decrease over a run is η∑tϕtA(t)(x)−η22∑tϕt2B(t)(x)\eta\sum_t\phi_tA^{(t)}(x)-\frac{\eta^2}{2}\sum_t\phi_t^2B^{(t)}(x) up to a third-order remainder, where A(t)A^{(t)} and B(t)B^{(t)} are the first-order and curvature terms of our one-step analysis (Proposition 3(ii)). Optimizing η\eta in this decrease yields a multi-step score. Evaluated with the actual schedule, this score exceeds Muon's by 1.9% for MuonL and 2.4% for t-MuonL under a shared learning rate, and by 2.5% for t-MuonL under type-specific learning rates (Section 4.2). The advantages found by the one-step criteria thus extend to the entire run.

We thank the reviewer again for the detailed and constructive review. We have run the requested ablation (point 1) and linked the criteria quantitatively to real training structure and the loss decrease over an entire run (points 2–7). As these address the conditions the reviewer stated, might the reviewer consider raising the rating to Weak Accept? We would be happy to clarify any further details.

I thank the author for the supplementary experiments and analyses, which basically respond to the empirical questions I raised earlier. Firstly, Table 2 directly validates the effect of immediate alignment. Although MuonL did not improve, t-MuonL showed a certain performance improvement. Secondly, Appendix G, especially Figures 13–15, further establish the quantitative connection between the theoretical criterion and the actual structure of the Transformer, providing valuable curvature analysis and structural discoveries.

My reservations about the theoretical analysis still exist. Currently, I still do not believe that the analysis framework based on the local second-order Taylor expansion can fully explain the performance gain in actual training.

However, the newly added experimental evidence and structural analysis have basically met the requirements I previously put forward. Therefore, I am willing to raise the score, mainly based on the enhancement of the empirical contribution, rather than the acceptance of the theoretical explanation.

AuthorsOpenReview ↗

Authors' Response to Reviewer "aKh9"

We thank the reviewer for the follow-up and for the willingness to raise the score. We are glad that the added experiments and the structural analysis in Appendix G address the reviewer's earlier questions.

Regarding the theoretical analysis, we use the local quadratic model to provide a measurable diagnostic of when u-w ratio alignment can benefit Muon. The diagnostic sheds light on which terms drive the increase in the optimization potential, consistent with the training performance gains of MuonL and t-MuonL. It also helps explain the performance gain brought by the technique of the u/w floor. However, we do not claim our analysis fully explains the performance gain in actual training, and we expect future work to further study the theoretical benefits of u-w ratio alignment.

We thank the reviewer again for the constructive discussion, which has substantially improved our paper.

Reviewer_wxUJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether u-w ratio alignment contributes to the empirical benefits of the u/w floor in Muon-based LLM pretraining. The authors propose MuonL and t-MuonL, which enforce global and within-type alignment, respectively. Theoretically, they derive local quadratic scores and covariance-based criteria for evaluating the benefits of alignment. Experiments on NanoGPT show that the proposed methods achieve lower validation losses than Muon, supporting the potential benefits of u-w ratio alignment.

Critical strengths

1. Well-motivated investigation.

The paper investigates whether u-w ratio alignment contributes to the empirical benefits of the u/w floor and proposes simple alignment strategies to examine this mechanism.

2. Theoretical analysis supported by empirical evidence.

The paper derives interpretable criteria for when u-w ratio alignment improves local optimization potential. The empirical results are broadly consistent with these theoretical diagnostics, providing evidence for the effectiveness of the proposed methods.

Critical weaknesses

1. Limited theoretical characterization.

The theoretical analysis focuses on a one-step quadratic approximation and derives criteria based on optimized learning rates. However, it does not establish whether improvements in the local score translate into better multi-step convergence or final training performance under practical learning-rate schedules. This limits the theoretical explanation of the observed empirical gains.

2. Limited empirical validation.

The experiments are restricted to a single 162M-parameter NanoGPT model on FineWeb10B. Although the results suggest benefits from u-w ratio alignment, evaluations on larger models and different training configurations are needed to establish the generality and practical significance of the proposed methods.

Questions for authors
  1. Can the authors extend the theoretical analysis beyond the one-step loss decrease to characterize the multi-step convergence behavior of Muon, MuonL, and t-MuonL under practical learning-rate schedules?

  2. Can the authors provide experiments at larger model scales to assess whether the observed improvements persist beyond the current 162M-parameter setting?

3 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer "wxUJ"

We thank the reviewer for the careful reading and for finding the investigation well motivated. Based on the comments of all three reviewers, we have revised the paper and uploaded the new version to OpenReview, with newly added content shown in blue. The main additions are results for a 305M-parameter model with learning-rate sweeps, an extension of the analysis from one step to an entire training run, an ablation of the refresh and EMA scheme, and an analysis of the structure of the criteria along training. We also added quantitative results to the abstract and expanded the background.

Below, we respond to each of the reviewer's comments and describe the corresponding revisions. We welcome any further suggestions and would be glad to revise the paper accordingly.

  1. "Can the authors extend the theoretical analysis beyond the one-step loss decrease to characterize the multi-step convergence behavior of Muon, MuonL, and t-MuonL under practical learning-rate schedules?"

    We thank the reviewer for this important question. Yes, we now characterize the convergence of all three methods under a practical learning-rate schedule ηϕt\eta\phi_t, where η\eta is the peak learning rate and ϕt\phi_t the schedule. Under standard assumptions, MuonL and t-MuonL converge at the same rate O(T−1/4)O(T^{-1/4}) as Muon, provided that alignment changes each u-w ratio by at most a bounded factor CC. This condition holds in training, with C≈5.6C\approx5.6 for MuonL and C≈3.0C\approx3.0 for t-MuonL. We present this result in Section 3.4 (Proposition 3(i)), with the full bound and its verification in Appendix F.

  2. "The theoretical analysis focuses on a one-step quadratic approximation and derives criteria based on optimized learning rates. However, it does not establish whether improvements in the local score translate into better multi-step convergence or final training performance under practical learning-rate schedules. This limits the theoretical explanation of the observed empirical gains."

    We thank the reviewer for this comment. The improvements in the local score do carry over to the entire run under the actual learning-rate schedule ηϕt\eta \phi_t. Up to a third-order remainder, we derive that the loss decrease over TT steps is η∑t=1TϕtA(t)(x)−η22∑t=1Tϕt2B(t)(x),\eta\sum_{t=1}^T\phi_tA^{(t)}(x)-\frac{\eta^2}{2}\sum_{t=1}^T\phi_t^2 B^{(t)}(x), where A(t)A^{(t)} and B(t)B^{(t)} are the first-order and curvature terms of our one-step analysis at step tt (Proposition 3(ii)). Maximizing this decrease over the peak learning rate yields a multi-step score for a certain policy xx. With the actual schedule, this score exceeds Muon's by 1.9% for MuonL and 2.4% for t-MuonL under a shared learning rate, and by 2.5% for t-MuonL under type-specific learning rates (Section 4.2). This matches the lower final validation losses of both methods and connects our criteria to the observed gains.

  3. "Can the authors provide experiments at larger model scales to assess whether the observed improvements persist beyond the current 162M-parameter setting?"

    We thank the reviewer for this question. The improvements persist at a larger scale. We repeated the training runs and the diagnostics on a 305M-parameter model with 16 layers and width 1024, trained for 7680 steps so that the ratio of training tokens to non-embedding parameters is about 20, as for the 162M model. Over three seeds, MuonL lowers the mean final validation loss of Muon by 0.00141 under a shared learning rate and t-MuonL by 0.00116 under type-specific learning rates, with lower loss than Muon for every seed. The diagnostics also agree with those at 162M: the MuonL criterion is positive at all 13 checkpoints and driven by the first-order descent gain, and the criterion of Proposition 2 is positive at 11 of 13 checkpoints with a positive sum. We now report these results in Appendix B.1 (Table 4 and Figures 7–8).

We are grateful to the reviewer for these constructive questions. With the analysis now extended to an entire training run and our findings confirmed on a 305M-parameter model, we believe both concerns have been addressed. Might the reviewer consider raising their score? We would be glad to clarify any remaining questions.

Thank you for the new theoretical and empirical results. My remaining concerns are still with the theoretical results.

  1. Proposition 3(i) characterizes the convergence guarantee with respect to the gradient, but the proposed aligned methods and conventional Muon exhibit the same convergence order; the alignment factor xi/rix_i/r_i (or CC) does not appear. Hence, there is no theoretical evidence of superiority.

  2. Proposition 3(ii) characterizes the loss decrease via a multi-step score, but the theory does not capture whether or how the proposed method outperforms Muon, although the empirical results confirm such an improvement.

AuthorsOpenReview ↗

Authors' Response to Reviewer "wxUJ"

  1. "Proposition 3(i) characterizes the convergence guarantee with respect to the gradient, but the proposed aligned methods and conventional Muon exhibit the same convergence order"

    We thank the reviewer for this question. The equal order is expected. Under the assumptions of Proposition 3(i), O(T−1/4)O(T^{-1/4}) is the best rate that any stochastic first-order method can guarantee (Arjevani et al., 2023), and Muon already attains it. Accordingly, improvements over Muon, such as NorMuon (arXiv:2510.05491) and Newton-Muon (Du & Su, 2026), are supported empirically or in stylized models rather than by faster rates, and even Muon's advantage over gradient descent lies in structure-dependent constants (Shen et al., 2025). An advantage of u-w ratio alignment should therefore appear in the constant, which is consistent with the size of the measured gains, about 2% in the multi-step score.

  2. "the alignment factor xi/rix_i/r_i (or CC) does not appear. Hence, there is no theoretical evidence of superiority."

    We thank the reviewer for raising this point. We agree that Proposition 3(i) does not show an advantage. We have therefore added an analysis in which the alignment factor enters the guarantee explicitly and favors the aligned methods, which we have included in Appendix F.2.

    The key is to measure progress by quantities that do not depend on how the network is parametrized. For example, scaling WVW_V up by cc and WOW_O down by cc leaves the network's output unchanged (Kunin et al., 2021) but changes ∥Gi∥∗\lVert G_i\rVert_{\ast} and Muon's ratios rir_i, whereas wiaiw_ia_i stays the same. Here wi=∥Wi∥Fw_i=\lVert W_i\rVert_F is the weight norm and ai=⟨Gi,Vi⟩Fa_i=\langle G_i,V_i\rangle_F the first-order descent along the update direction ViV_i (Section 3.1).

    For one step, the descent lemma under the assumptions of Proposition 3 gives L(W)−L(W+Δ(x)) ≥ η∑ixiwiai−η22κ∑ixi2wi2.\mathcal L(W)-\mathcal L(W+\Delta(x))\ \ge\ \eta\sum_i x_iw_ia_i-\frac{\eta^2}{2}\kappa\sum_i x_i^2w_i^2 . Bounding the first sum by (min⁡ixi)∑iwiai(\min_i x_i)\sum_i w_ia_i, whose factors are unchanged by the rescalings, and choosing the best η\eta gives L(W)−L(W+Δ(x)) ≥ (∑iwiai)2(min⁡ixi)22κ∑ixi2wi2.\mathcal L(W)-\mathcal L(W+\Delta(x))\ \ge\ \frac{(\sum_i w_ia_i)^2(\min_i x_i)^2}{2\kappa\sum_i x_i^2w_i^2} . This guaranteed decrease is largest if and only if all xix_i are equal. MuonL attains it under a shared learning rate, and t-MuonL under type-specific learning rates. For Muon (xi=rix_i=r_i), it is smaller by the factor ∑imi/(min⁡iri2∑iwi2)≥1\sum_i m_i/(\min_i r_i^2\sum_i w_i^2)\ge1, with equality only when all weight norms are equal.

    The same argument gives a version of our convergence bound in Proposition 7 (Appendix F) for ∑iwi∥Gi∥∗\sum_i w_i\lVert G_i\rVert_{\ast} in place of ∑i∥Gi∥∗\sum_i\lVert G_i\rVert_{\ast}, and its constant is smallest when all relative steps xix_i are equal. MuonL and t-MuonL therefore attain the best constant, and Muon's is larger by the square root of the factor above.

  3. "Proposition 3(ii) characterizes the loss decrease via a multi-step score, but the theory does not capture whether or how the proposed method outperforms Muon, although the empirical results confirm such an improvement."

    We thank the reviewer for this question. Point 2 shows that the aligned methods have the better guarantee over the whole run. How t-MuonL outperforms Muon is characterized in Appendix G.3, which we would like to highlight. Within each matrix type, Muon gives smaller relative steps to layers with larger weight norms. Appendix G.3 assumes that the layers of a type share the same first-order descent and curvature per unit relative step (Assumption 1, motivated by scale invariance). Under this assumption, Proposition 8 shows that equal relative steps within a type maximize its score: t-MuonL gives up some first-order descent but saves more in curvature. The diagnostics confirm this mechanism, with a curvature gain from within-type alignment at all 13 checkpoints (Appendix G.1), mainly in WVW_V and WOW_O (Appendix G.2).

We thank the reviewer again for the careful engagement with our theory. With the alignment factor now entering the convergence constant explicitly, and the mechanism of t-MuonL's advantage explained in Appendix G.3, might the reviewer consider raising their score? We would be glad to clarify any further questions.

Reviewer_hMebOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper discusses the Muon optimizer in the context of the modified NanoGPT benchmark. It discusses the famous phenomenon of the UW floor used by many top submissions. It discusses why this phenomenon is effective. Armed with these insights, the authors introduce two different Muon variants: MuonL t-MuonL

Then the authors use a quadratic approximation to the training loss to come up with some theoretical insights. Then they present experiments over a 12-layer NanoGPT model trained on FineWeb, and at various different checkpoints along training, they show that their two variants have a lower training loss compared to Muon.

Critical strengths
  • I appreciate the fact that the authors bring a fresh perspective on a question that is often theoretically not discussed. Why are the top methods on the modded NanoGPT benchmark working so well with the UW ratio ideas?
  • The theoretical analysis in the paper is quite nicely done. It took me a while to read, but I enjoyed reading it.
  • I appreciate the fact that the authors report mean and standard deviations across many runs.
Critical weaknesses
  • Honestly, the abstract is all over the place. The value add of the new algorithms or variants of Muon is not clear at all. Please consider adding numerical results in the abstract to show that things are substantially better and not just qualitatively being talked about.
  • The background section is a bit weak. The authors should talk a little bit more about the modded nano GPT and the model training size, etc.
  • The experiments are at a single model size, and I feel they are not diverse enough for a strong claim that this method actually works. The learning rates were also not sufficiently tuned. This, to me, is the biggest flaw in the paper and is holding it back. How do we know that the ideas discussed in the paper generalize to settings outside the narrow setting that the experiment section covers?
Questions for authors

Please see weaknesses

1 comment
AuthorsOpenReview ↗

Authors' Response to Reviewer "hMeb"

We thank the reviewer for the careful reading, and for finding our perspective fresh and the theoretical analysis nicely done. Based on the comments of all three reviewers, we have revised the paper and uploaded the new version to OpenReview, with newly added content shown in blue. The main additions are results for a 305M-parameter model with learning-rate sweeps, an extension of the analysis from one step to an entire training run, an ablation of the refresh and EMA scheme, and an analysis of the structure of the criteria along training. We also added quantitative results to the abstract and expanded the background.

Below, we give point-by-point responses to the reviewer's comments and describe the revisions we made to address them. We would be happy to add further clarifications and revisions to address any additional recommendations from the reviewer.

  1. "The experiments are at a single model size, and I feel they are not diverse enough for a strong claim that this method actually works. [...] How do we know that the ideas discussed in the paper generalize to settings outside the narrow setting that the experiment section covers?"

    We thank the reviewer for raising this point. Our findings carry over to a larger model. We repeated the training runs and the diagnostics on a 305M-parameter model with 16 layers and width 1024, trained for 7680 steps so that the ratio of training tokens to non-embedding parameters is about 20, as for the 162M-parameter NanoGPT. Over three seeds, MuonL lowers the mean final validation loss of Muon by 0.00141 under a shared learning rate and t-MuonL by 0.00116 under type-specific learning rates, with lower loss than Muon for every seed. The diagnostics also agree with those at 162M: the MuonL criterion is positive at all 13 checkpoints and driven by the first-order descent gain, and the criterion of Proposition 2 is positive at 11 of 13 checkpoints with a positive sum. We now report these results in Appendix B.1 (Figures 7–8).

  2. "The learning rates were also not sufficiently tuned."

    We thank the reviewer for this comment. The shared-learning-rate setting is the community-tuned Muon baseline of the modded-nanogpt optimization benchmark (entry #12), which our methods reuse directly. For further verification, we conducted a sweep of Muon's peak learning rate over 0.01, 0.02, 0.035, 0.05, and 0.1, where the value 0.035 gives the lowest final validation loss at both 162M and 305M parameters (see below). This guarantees that the Muon baseline is not disadvantaged. The type-specific learning-rate setting adopts the peak learning rates that entry #15 tuned per matrix type for the Newton-Muon optimizer (Du & Su, 2026), each raised by 0.005 so that the largest equals the shared value 0.035.

    Peak learning rate0.010.020.0350.050.1
    162M (full schedule)3.298883.283803.280553.283503.30141
    305M (20%-length schedule)3.335133.322073.321143.326403.34376

  3. "Honestly, the abstract is all over the place. The value add of the new algorithms or variants of Muon is not clear at all. Please consider adding numerical results in the abstract to show that things are substantially better and not just qualitatively being talked about."

    We thank the reviewer for this suggestion. We have revised the abstract and added key numerical results, including the diagnostic results for MuonL and t-MuonL and the step savings of MuonL in the modded-nanogpt optimization benchmark.

  4. "The background section is a bit weak. The authors should talk a little bit more about the modded nano GPT and the model training size, etc."

    We thank the reviewer for this suggestion. We have expanded the background on the modded-nanogpt optimization benchmark in Section 1.2, which now describes its target and validation rule, the model size and training data, and the training budget of the Muon baseline.

We again express our gratitude for the constructive feedback provided by the reviewer. Given the results at a 305M-parameter model and the learning-rate sweeps at both model sizes, might the reviewer consider raising their score? We would also be happy to provide additional clarifications to any details that the reviewer thinks should be further improved.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.