- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether U-W ratio alignment is a crucial mechanism for improving the pre-training performance of Muon through u/w floor. The authors propose MuonL and t-MuonL, aligning relative update magnitudes between all matrices and within matrices of the same type, respectively. Theoretically, the paper derives sufficient conditions based on covariance and a generalized eigenvalue criterion using local second-order Taylor approximation to compare the local optimization potential of different alignment strategies. Experiments, including Hessian-vector products in the training trajectory, multi-random seed training, and benchmark validation, report performance improvements relative to Muon.
Critical strengths
S1. The research problem is of practical significance, and the method design is relatively simple.
Starting from the empirical benefits of the u/w floor, this paper attempts to isolate the effect of U-W ratio alignment and proposes MuonL and t-MuonL suitable for different learning rate configurations. This problem is closely related to the control of module update scale in LLM pre-training. The method only adjusts the update magnitude without changing the update direction of Muon, which has certain practical value.
S2. The theoretical analysis and experimental evidence are relatively systematic.
The authors not only derived the local optimization criterion but also used Hessian-vector products in the training trajectory to measure directional curvature and cross-matrix interactions, and tested the theoretical criterion at multiple checkpoints. Multi-random seed experiments and benchmark statistical validation also support the performance gains of the method. Compared with simply reporting empirical improvements, this paper attempts to establish a connection between mechanism hypotheses, theoretical diagnosis, and actual training performance.
Critical weaknesses
W1. Section 2 (Equations 1–2) defines the instantaneous U-W ratio alignment, and Sections 3.1 and 3.3 (Equations 5, 12) further derive the local second-order optimization score based on this ideal update rule. However, I found that Appendix A.2 (pages 13–14, Algorithms 1–2) states that the actual algorithm refreshes the scaling factor only every 5 steps and uses EMA (\beta=0.9) for smoothing. The authors also explicitly state in Appendix A.2 (line 667-669, Practical implementation) that the theoretical diagnostic does not consider the refresh interval and EMA smoothing. Therefore, actual updates generally do not satisfy the instantaneous alignment relationship in the theoretical analysis. Furthermore, the theory allows for separate optimization of the learning rate, while actual training uses the learning rate configuration and scheduling specified in Appendix A.1 (page 12). The paper lacks a direct comparison of per-step refresh without EMA, making it impossible to determine whether the performance gains truly come from the alignment mechanism studied in the theory.
W2. The theoretical criteria have not yet revealed a sufficiently explicit optimization structure. Proposition 1 expresses the gains as a relationship between first-order descent, directional curvature, and rescaling costs, while Proposition 2 characterizes the local optimization potential of different strategies through generalized eigenvalue decomposition. These results are mathematically valuable, but primarily provide conditions for determining when alignment is advantageous; they do not explain why real Transformer training systematically satisfies these conditions. For example, the gains in Figure 2(b) mainly come from first-order descent, while the intra-group gains in Figure 3(b) mainly come from curvature improvement; this difference is currently difficult to explain because the alignment strategy and learning rate configuration change simultaneously. The paper also does not fully reveal the actual module structure and stability corresponding to the feature modes.
However, I appreciate the authors' attempt to analyze this method, which has some practical effectiveness, but at present, I believe the depth of the theoretical analysis is insufficient to convince me. If the authors can verify the actual gains of instantaneous alignment through direct ablation and further provide a quantitative link between the theoretical criteria and the real training structure or actual optimization gains, I would be willing to raise the rating to Weak Accept.
Questions for authors
-
Does the actual performance gain truly stem from the instantaneous U-W ratio alignment analyzed in the theory? Why does the actual implementation in Appendix A.2 (Algorithms 1–2) employ refresh every 5 steps and EMA smoothing? Could the authors compare the four schemes , , , and the current scheme under the same training configuration, and report the actual updated U-W ratio alignment and validation loss?
-
In particular, if EMA is removed and alignment is performed at each step, can performance gains comparable to those in the paper still be achieved? This experiment is crucial for establishing the connection between the theoretical diagnostics in Section 3 and the actual training results in Section 4.
-
What structural patterns in Transformer training do the theoretical criteria reveal?Proposition 1 and Proposition 2 present the covariance condition and the generalized eigenvalue criterion, respectively. But do these conditions correspond to the stable gradient-curvature structure present in real training?
-
Could the authors further analyze the specific distributions of first-order descent, directional curvature, and cross-module interactions in different matrix types and training stages to explain the different dominant sources of gain in Figure 2(b) and Figure 3(b)? Can these structures predict the true one-step loss reduction under actual learning rates, rather than merely explaining the local second-order scores optimized by the learning rate?
4 comments
Authors' Response to Reviewer "aKh9" (Part 2/2)
-
"These results are mathematically valuable, but primarily provide conditions for determining when alignment is advantageous; they do not explain why real Transformer training systematically satisfies these conditions."
We thank the reviewer for this comment. For t-MuonL, we explain this through the homogeneity of layers within each matrix type. Motivated by scale invariance, we assume that the layers of each type share the same first-order descent and curvature per unit relative step (Assumption 1). Under this assumption, Proposition 8 (Appendix G.3) shows that within-type alignment yields a curvature gain that outweighs its first-order cost, and that t-MuonL has at least the optimization potential of Muon in the comparison of Proposition 2.
We test the homogeneity assumption at the 13 checkpoints (Figure 15). Averaged over checkpoints and matrix types, the first-order descent varies by 19% across the layers of a type, against 74% across all 72 Muon-updated blocks, and the curvature by 44%, against 76%. The assumption thus holds approximately within types but fails across them, so it is not surprising that across-type alignment reverses the trade-off, gaining in first-order descent at the cost of curvature.
-
"The paper also does not fully reveal the actual module structure and stability corresponding to the feature modes."
We thank the reviewer for this comment. The eigenmodes of Proposition 2 are indeed not stable. They carry no fixed meaning, since they only form the basis in which the criterion splits into independent terms. A structure over matrix types that is stable along training appears instead in the covariance-based criterion of Proposition 1, as we will discuss in the next point.
-
"Could the authors further analyze the specific distributions of first-order descent, directional curvature, and cross-module interactions in different matrix types and training stages to explain the different dominant sources of gain in Figure 2(b) and Figure 3(b)?"
We thank the reviewer for this suggestion. We split the two gains of the MuonL criterion by component and by matrix type (Appendices G.1 and G.2). By component, the curvature gain consists of the directional curvatures , the interactions between layers of the same type, and the interactions across types. Attributing these three parts and the first-order descent gain to within-type and across-type alignment yields eight parts, each a consistent gain or cost at all 13 checkpoints on both trajectories (Table 6). By matrix type, the curvature gain of t-MuonL in Fig. 3(b) comes mainly from and , and the first-order descent gain of MuonL in Fig. 2(b) mainly from (Figure 14). These patterns remain stable throughout training.
-
"Can these structures predict the true one-step loss reduction under actual learning rates, rather than merely explaining the local second-order scores optimized by the learning rate?"
We thank the reviewer for this question. Our scores measure a policy's optimization potential: how much it can lower the loss in one step at the best learning rate. Actual training does not use this full potential at every step, since its learning-rate schedule favors stable convergence over the largest immediate decrease. By Taylor expansion, substituting the actual learning rate into the quadratic model would naturally approximate the one-step reduction, but predicting it is not the aim of our criteria.
-
"Furthermore, the theory allows for separate optimization of the learning rate, while actual training uses the learning rate configuration and scheduling specified in Appendix A.1 (page 12)."
We thank the reviewer for this comment. In Section 3.4, we analyze our criteria under the actual training schedule, where only the peak learning rate is optimized. With learning rate at step , the loss decrease over a run is up to a third-order remainder, where and are the first-order and curvature terms of our one-step analysis (Proposition 3(ii)). Optimizing in this decrease yields a multi-step score. Evaluated with the actual schedule, this score exceeds Muon's by 1.9% for MuonL and 2.4% for t-MuonL under a shared learning rate, and by 2.5% for t-MuonL under type-specific learning rates (Section 4.2). The advantages found by the one-step criteria thus extend to the entire run.
We thank the reviewer again for the detailed and constructive review. We have run the requested ablation (point 1) and linked the criteria quantitatively to real training structure and the loss decrease over an entire run (points 2–7). As these address the conditions the reviewer stated, might the reviewer consider raising the rating to Weak Accept? We would be happy to clarify any further details.
I thank the author for the supplementary experiments and analyses, which basically respond to the empirical questions I raised earlier. Firstly, Table 2 directly validates the effect of immediate alignment. Although MuonL did not improve, t-MuonL showed a certain performance improvement. Secondly, Appendix G, especially Figures 13–15, further establish the quantitative connection between the theoretical criterion and the actual structure of the Transformer, providing valuable curvature analysis and structural discoveries.
My reservations about the theoretical analysis still exist. Currently, I still do not believe that the analysis framework based on the local second-order Taylor expansion can fully explain the performance gain in actual training.
However, the newly added experimental evidence and structural analysis have basically met the requirements I previously put forward. Therefore, I am willing to raise the score, mainly based on the enhancement of the empirical contribution, rather than the acceptance of the theoretical explanation.
Authors' Response to Reviewer "aKh9"
We thank the reviewer for the follow-up and for the willingness to raise the score. We are glad that the added experiments and the structural analysis in Appendix G address the reviewer's earlier questions.
Regarding the theoretical analysis, we use the local quadratic model to provide a measurable diagnostic of when u-w ratio alignment can benefit Muon. The diagnostic sheds light on which terms drive the increase in the optimization potential, consistent with the training performance gains of MuonL and t-MuonL. It also helps explain the performance gain brought by the technique of the u/w floor. However, we do not claim our analysis fully explains the performance gain in actual training, and we expect future work to further study the theoretical benefits of u-w ratio alignment.
We thank the reviewer again for the constructive discussion, which has substantially improved our paper.
Authors' Response to Reviewer "aKh9" (Part 1/2)
We thank the reviewer for the careful reading, and in particular for the questions on our theory, which led to the new analysis of the structure of the criteria in Appendix G. Taking into account the comments of all three reviewers, we have revised the paper and uploaded the new version to OpenReview, with newly added content shown in blue. To address the conditions the reviewer stated for raising the rating, we have run the requested ablation of the refresh and EMA scheme, and we have linked the criteria to the structure of real training. The revision also includes results for a 305M-parameter model.
Below, we address the reviewer's comments one by one and describe the corresponding revisions. We would welcome any further suggestions and would be happy to revise the paper accordingly.
"Does the actual performance gain truly stem from the instantaneous U-W ratio alignment analyzed in the theory? Why does the actual implementation in Appendix A.2 (Algorithms 1–2) employ refresh every 5 steps and EMA smoothing? Could the authors compare the four schemes [K = 1, β = 0], [K = 1, β = 0.9], [K = 5, β = 0], and the current scheme [K = 5, β = 0.9] under the same training configuration, and report the actual updated U-W ratio alignment and validation loss? In particular, if EMA is removed and alignment is performed at each step, can performance gains comparable to those in the paper still be achieved?"
We thank the reviewer for proposing this ablation. We ran all four schemes for both methods and list the final validation losses below. The results show that the gain does stem from u-w ratio alignment. First, every scheme aligns the actual updated u-w ratios after the first few hundred steps (Figure 5). Second, t-MuonL outperforms Muon under all four schemes, including the instantaneous alignment analyzed in our theory (K=1, β=0). MuonL also outperforms Muon under every scheme except instantaneous alignment. This exception arises because MuonL equalizes the u-w ratios of the zero-initialized WO and Wdown with those of the other blocks from the very beginning, making their updates inappropriately small.
This explains why our implementation uses refresh and EMA. They let MuonL realize the benefit of alignment when update scales change abruptly or some weight norms are close to zero, just as K-FAC (Martens & Grosse, 2015) and SOAP (Vyas et al., 2024) refresh and smooth their online curvature estimates to apply them reliably. Our scheme with K=5 and β=0.9 indeed gives the lowest loss for both methods. However, it introduces no separate mechanism and only makes the fullest use of u-w ratio alignment. We now report this ablation in Appendix A.2 (Table 2 and Figure 5).
"For example, the gains in Figure 2(b) mainly come from first-order descent, while the intra-group gains in Figure 3(b) mainly come from curvature improvement; this difference is currently difficult to explain because the alignment strategy and learning rate configuration change simultaneously."
We thank the reviewer for pointing out this contrast. The difference is caused by which blocks are aligned, not by the learning-rate configuration. MuonL can be split into two steps. The first step aligns the u-w ratios within each matrix type, which is exactly what t-MuonL does. The second step aligns the resulting ratios across matrix types. Both gains in the MuonL criterion (Eq. 11) are covariances over blocks, and each covariance splits between the two steps as covπ(f,g)=within-typeG∑πGcovG(f,g)+across-typeG∑πG(fˉG−fˉ)(gˉG−gˉ), where fˉG and covG are the mean and covariance within type G, and fˉ is the mean over all blocks. This split reveals opposite trade-offs at all 13 checkpoints: the within-type alignment yields a curvature gain at the cost of first-order descent, and the across-type alignment yields a first-order descent gain at the cost of curvature (see Appendix G.1, Figure 13). The trade-offs hold for both the shared-learning-rate and the type-specific trajectories, so the learning-rate configuration does not cause the difference. t-MuonL, which performs only the within-type step, thus gains through curvature in Fig. 3(b), while MuonL draws its first-order gain in Fig. 2(b) from the across-type step.