acceptodds
Under review as a conference paper at ICLR 2027

The Row Normalization Puzzle in Muon

17 views

Abstract

This paper examines how row-wise renormalization affects Muon, focusing on the gap between worst-case guarantees and practical performance in NorMuon. Despite its growing adoption and promising performance in large language model (LLM) pretraining, NorMuon's worst-case guarantees remain poorly understood. One fundamental question is: does row normalization yield provable convergence gains, potentially through its interaction with approximated polar computation and exponential moving-average momentum? Our results show that row normalization introduces a dimension-dependent factor in the worst-case iteration complexity under the operator-norm geometry, which persists even with exact polar computation and any fixed momentum parameters. Indeed, we establish an algorithm-dependent lower bound and a matching upper bound in deterministic settings, and extend our upper bound analysis to stochastic settings. Our upper bound covers approximated polar computations. Experiments show that NorMuon is slower than Muon on synthetic problems inspired by our worst-case construction, yet outperforms Muon in LLM pretraining. These findings sharpen the puzzle of why row normalization helps in practice and complement the recent findings in Dewulf et al.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language model (LLM) pretraining, Muon, row normalization, worst-case convergence guarantee

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_utW4OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper asks whether the row normalization in NorMuon can make Muon provably faster. To answer it, the authors analyze the worst-case iteration complexity in the operator-norm geometry, under which Muon’s update is the steepest descent direction. They show that NorMuon's row normalization instead increases the complexity by a factor of the number of rows. This cost remains even with exact orthogonalization and for any fixed momentum parameters and step-size rule. The experiments cover synthetic problems built on the worst-case construction and LLM pretraining up to 6.4B parameters. NorMuon is slower than Muon on the synthetic problems but outperforms it in LLM pretraining, which the authors present as sharpening the puzzle of why row normalization helps in practice.

Critical strengths
  1. The theoretical results are clean and complete in the deterministic setting.
  2. The worst-case construction gives a simple and intuitive picture of where the cost comes from, which the synthetic experiments also confirms.
Critical weaknesses
  1. The main theoretical result is largely expected from the setup. In the operator-norm geometry, Muon’s update is the steepest descent direction by definition, so any fixed rescaling of its rows can only move the update away from it in the worst case.

  2. The puzzle that the paper raises is also largely a consequence of the chosen lens, i.e., the worst case over all smooth functions. Aurora [1] already gives a concrete explanation of why row normalization helps in practice. In tall matrices, neurons that receive small updates under Muon keep receiving small updates and eventually become inactive. Such a feedback between past updates and future gradients comes from the structure of neural networks, which the worst case construction does not need to have. The function that the paper constructs for its lower bound has a gradient that stays constant along the iterates, so how much a row is updated never changes the gradient it receives. Moreover, a network with many inactive neurons can still have a small gradient, so reaching a small gradient faster does not mean reaching a better solution. Therefore, being slower in the worst case and better in practice is not a contradiction, and the claim that the findings sharpen the puzzle seems overstated.

  3. The experiments are largely disconnected from the analysis. The synthetic problems are built directly from the worst-case construction, so NorMuon being slower there is expected by design. The LLM part mostly reruns several existing normalization variants against Muon, and reaches the same conclusions as prior work that normalization helps and that the gain comes mainly from the MLP matrices. None of these experiments relates back to the analysis. For example, the paper never checks whether the updates in real training are concentrated on a few rows as in the worst-case construction. As a result, the experiments neither support nor challenge the theory, and only restate the gap between the worst case and practice.

[1] Dewulf, Alec, et al. "Aurora: A Leverage-Aware Spectral Optimizer." arXiv preprint arXiv:2606.27715 (2026).

Questions for authors
  1. The worst-case construction seems to apply to any method that equalizes the rows of the update, whether before or after the orthogonalization. Does the lower bound then also cover Muon+, MuonEq-R and Aurora? If so, stating this would make clear that the result is about row balancing in general, not NorMuon specifically, which would broaden the impact of this work.

  2. The conclusion is tied to the operator-norm geometry, under which Muon’s update is the steepest descent direction. Under a geometry defined by the largest row norm, row normalization is instead the steepest descent direction. How would the results change in such a geometry, and which geometry do the authors think better matches the matrices where NorMuon helps?

  3. The theory analyzes normalization over rows, while the experiments normalize the MLP matrices along their larger dimension. Could the authors clarify how the theory applies to the setting used in the experiments?

5 comments

Thank you for your time and comments. We clarify each point below.

W1: The main theoretical result is largely expected from the setup. In the operator-norm geometry, Muon's update is the steepest descent direction by definition, so any fixed rescaling of its rows can only move the update away from it in the worst case.

Muon is steepest, and we hope to clarify that rescaling steepest descent may not worsen the complexity. E.g., AdaGrad's regret is never worse than that of online gradient descent in ℓ2\ell_2 (Corollary 1 of [1]). NorMuon's rescaling is similar to AdaGrad. Moreover, steepest descent concerns the one step decrease, not the complexity, which we characterize as a Θ(m)\Theta(m) extra cost.

W2(a): ... receives.

We agree that Aurora empirically explains why NorMuon works in LLM, but it gives limited theory on optimization gains. We view Aurora and our paper as complement to each other. Since NorMuon is worse in the worst case, its gain should come from the structure of LLM, which Aurora may help reveal. We agree that the hard instance does not characterize the feedback between past updates and future gradients, and will list this as future work.

W2(b): Moreover ...

We agree that small gradient does not imply a better solution. This is a limitation of worst-case nonconvex analysis.

We do not claim a contradiction between the worst case and LLM. By "sharpen the puzzle," we mean that our analysis and ablations rule out polar and β2\beta_2 as explanations. We have revised the wording.

W3

We hope to make a point that our synthetic experiments, new experiments (Tables R1–R2, Reviewer bVe2), and ablations are closely connected to theory.

For synthetic experiments, we clarify that it is designed with imbalanced rows but different from Theorem 3.1. It is to show that NorMuon remains slower even after tuning. We have added a new experiment to include cosine scheduling.

For LLM, as suggested, we measured the norm imbalance along the normalization dimension, i.e., the rows for square matrices and the larger dimension otherwise (Tables R1–R2). The CV (std/mean) of these norms in the momentum is below 1, compared with >13 in the lower bound. The norms are imbalanced, but far less than in the worst case. Appendix C.3 shows that NorMuon attains a larger alignment than Muon on MLP matrices via a larger momentum. On the same momentum, NorMuon's alignment is smaller than Muon's (Table R2), yet the ratio (0.8–1) far exceeds ~0.08 of the hard instance.

Q1

The lower bound covers Muon+, MuonEq-R, and Aurora. We have added the proof to Appendix B.2, where we establish a unified condition.

Q2(a): How would the results change in such a geometry?

Let ∥X∥R=max⁡i∥Xi,:∥2\Vert X\Vert_R = \max_i \Vert X_{i,:}\Vert_2, with dual ∥G∥R∗=∑i∥Gi,:∥2\Vert G\Vert_{R*} = \sum_i \Vert G_{i,:}\Vert_2. Steepest descent is Xt+1=Xt−ηtR(∇F(Xt))X_{t+1} = X_t - \eta_t R(\nabla F(X_t)), [R(G)]i,:=Gi,:/∥Gi,:∥2[R(G)]_{i,:} = G_{i,:} / \Vert G_{i,:}\Vert_2. If ∥∇F(Y)−∇F(X)∥R∗≤LR∥Y−X∥R\Vert\nabla F(Y) - \nabla F(X)\Vert_{R*} \le L_R \Vert Y - X\Vert_R, then, similar to Eq. (3.1), ηt=∥∇F(Xt)∥R∗/LR\eta_t = \Vert\nabla F(X_t)\Vert_{R*} / L_R gives F(Xt+1)≤F(Xt)−∥∇F(Xt)∥R∗2/(2LR)F(X_{t+1}) \leq F(X_t) - \Vert\nabla F(X_t)\Vert_{R*}^2 / (2 L_R), and thus min⁡t<T∥∇F(Xt)∥R∗≤ϵ\min_{t < T} \Vert\nabla F(X_t)\Vert_{R*} \le \epsilon within T=O(ΔLRϵ−2)T = O(\Delta L_R \epsilon^{-2}) iterations, free of dimension. The steepest descent is not NorMuon, which normalizes polar.

Q2(b): Which geometry do the authors think better matches the matrices where NorMuon helps?

We currently view the operator norm as the best choice and leave this to future work. Given the layerwise structure and normalization layers of LLMs, the operator (ℓ2→ℓ2\ell_2\to\ell_2) norm is regarded as a good model [2,3], underlying Muon's success. No ℓp1→ℓp2\ell_{p_1}\to\ell_{p_2} induced norm makes NorMuon steepest descent. We have not found a natural norm modeling NorMuon better. Still, the operator norm may not fully characterize LLMs; e.g., [2] treats the first and the last layers by different norms, and they are often trained with AdamW in practice. A better norm model for LLMs may eventually explain NorMuon's gain.

Q3

Theorems 3.1-3.3 hold by transposition. Normalizing along the larger dimension can be viewed as normalizing the rows of the possibly transposed max⁡(m,n)×min⁡(m,n)\max(m,n) \times \min(m,n) matrix. Since polar, the operator and nuclear norms, and Assumptions 2.1–2.3 are transposition invariant, Theorems 3.1–3.3 hold with mm replaced by max⁡(m,n)\max(m,n). We have added a remark.

Thank you again, and we look forward to further discussion.

References

[1] J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12(7), 2011.

[2] T. Pethick, W. Xie, K. Antonakopoulos, Z. Zhu, A. Silveti-Falls, and V. Cevher. Training deep learning models with norm-constrained LMOs. In ICML, pp. 49069–49104. PMLR, 2025.

[3] K. Jordan, Y. Jin, V. Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024.

AuthorsOpenReview ↗

Kind Follow-Up on Our Response

Dear Reviewer utW4,

Thank you for your detailed review. We appreciate your time and effort.

As the discussion deadline approaches, we wanted to follow up to check whether our response has addressed your concerns and questions. If you have any further questions, we would be happy to discuss them.

Thank you again, and we look forward to your response.

Sincerely,

Authors of Paper 22475

Dear Reviewer utW4,

Thank you again for your detailed review. As the discussion period ends in 48 hours, we would be grateful if you could let us know whether our responses have addressed your questions and concerns. We would be happy to discuss any further questions you may have.

Sincerely,

Authors of Paper 22475

Reviewer_tW3mOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the effect of row normalization on Muon in two respects. First, it establishes a dimension-dependent lower bound for NorMuon's worst-case convergence complexity, together with a matching deterministic upper bound, and extends the upper-bound analysis to stochastic settings and approximate polar computation.

Second, it examines row normalization in LLM pretraining. The experiments find that most of NorMuon's reported gains come from MLP matrices rather than attention QKVO matrices, and report differences in momentum-update alignment and matrix norms across parameter groups.

Critical strengths

Theoretical analysis. The authors construct a deterministic lower bound from the effect of row normalization on gradient-update alignment and establish a matching upper bound, characterising NorMuon's worst-case iteration complexity as Θ(mΔL/ε2)\Theta(m\Delta L/\varepsilon^2) in the stated setting. The lower bound allows exact polar computation, any fixed momentum parameters, and deterministic adaptive step-size rules. The paper also extends its upper-bound analysis to stochastic gradients and approximate polar computation.

Analysis across parameter groups. The authors examine the effects of normalization on attention QKVO, MLP input, and MLP output matrices. Applying NorMuon to either MLP input or MLP output matrices recovers most of the reported gains, while applying it only to QKVO matrices yields little improvement. The additional measurements of momentum-update alignment, momentum nuclear norms, and update operator norms characterise further differences between these groups.

Critical weaknesses

None

Questions for authors
  1. Have the authors observed similar differences between MLP and attention matrices at other model scales or under different training settings?

  2. Do the authors have any additional brief observations, based on their existing analyses, about the different effects of row normalization on MLP and attention matrices?

5 comments

Thank you for your time and comments. We clarify each point below.

Q1: Have the authors observed similar differences between MLP and attention matrices at other model scales or under different training settings?

We conduct a new experiment and observe similar differences between MLP and attention matrices under a new training setting. We have added it to our revision.

We train a Llama-style decoder-only transformer with 93M parameters, 16 layers, hidden size 512, SwiGLU, RMSNorm, and rotary positional embeddings on FineWeb-Edu, using 1.86B training tokens (20 tokens per parameter). We use a sequence length of 1,024, a batch size of 524,288 tokens, and a peak base step size of 0.02 with a warmup-stable-decay step size schedule. We report the final test loss on 16.78M held-out tokens.

Table R3: NorMuon applied to different groups of matrices on the 93M model.

OptimizerTest loss
Muon3.2150
Full NorMuon3.2126
NorMuon (QKVO only)3.2160
NorMuon (MLP input and gate only)3.2127
NorMuon (MLP output only)3.2124

We observe a similar phenomenon. Applying NorMuon to either the MLP input or the MLP output matrices recovers most of the reported gains, while applying it only to the QKVO matrices yields little improvement.

We include more findings from prior works.

  1. Dewulf et al. [1] train a 340M model with SwiGLU and a gated MLP and compare NorMuon with U-NorMuon, which normalizes only the tall matrices (the MLP input and gate matrices). U-NorMuon attains even a lower validation loss than NorMuon (Table 6 of [1]), so the gain comes from the MLP matrices. In our models in the paper, the MLP input projections are the only tall matrices, and thus our "MLP input only" variant coincides with U-NorMuon. It nearly matches NorMuon but does not improve on it.
  2. Figure 9 of [2] shows that applying NorMuon only to the MLP input matrices recovers part of NorMuon's gain.

Hence, across scales and settings, the gain of normalization comes mainly from the MLP matrices, and its size depends on the training setting.

Q2: Do the authors have any additional brief observations, based on their existing analyses, about the different effects of row normalization on MLP and attention matrices?

  1. In Table R1 of our response to Reviewer bVe2, the MLP gradient momentum has more imbalanced norms along its larger dimension than the attention gradient momentum has along its rows.
  2. In Table R2 of our response to Reviewer bVe2, when Muon and NorMuon are applied to the same momentum (NorMuon's), NorMuon has a smaller alignment than Muon, and this alignment loss is larger on the MLP matrices.

Thank you again, and we look forward to further discussion.

References

[1] A. Dewulf, D. Pai, L. Yang, A. Zhang, and B. Keigwin. Aurora: A leverage-aware spectral optimizer. ArXiv Preprint: 2606.27715, 2026

[2] Z. Li, L. Liu, C. Liang, W. Chen, and T. Zhao. NorMuon: Making Muon more efficient and scalable. In ICML, 2026.

AuthorsOpenReview ↗

Kind Follow-Up on Our Response

Dear Reviewer tW3m,

Thank you for your detailed review. We appreciate your time and effort.

As the discussion deadline approaches, we wanted to follow up to check whether our response has addressed your concerns and questions. If you have any further questions, we would be happy to discuss them.

Thank you again, and we look forward to your response.

Sincerely, Authors of Paper 22475

Dear Reviewer tW3m,

Thank you again for your detailed review. As the discussion period ends in 48 hours, we would be grateful if you could let us know whether our responses have addressed your questions and concerns. We would be happy to discuss any further questions you may have.

Sincerely,

Authors of Paper 22475

Reviewer_bVe2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how row-wise renormalization affects Muon through a theoretical analysis of NorMuon. The authors establish matching deterministic lower and upper bounds, showing a dimension-dependent worst-case penalty under operator-norm geometry. The lower bound holds with exact polar computation, and the upper-bound analysis covers stochastic gradients and approximate polar computation. Experiments also show that NorMuon is slower than Muon on synthetic problems inspired by the lower-bound construction but performs better in LLM pretraining.

Critical strengths
  1. The paper studies the worst-case convergence of NorMuon and establishes matching lower and upper bounds in the deterministic setting. It shows how row normalization can introduce a dimension-dependent slowdown compared with Muon.
  2. The paper evaluates Muon and NorMuon on synthetic problems and LLM pretraining at several model sizes and broad evaluation settings. The experiments demonstrate the empirical slowdown from row normalization in the synthetic setting.
Critical weaknesses
  1. The lower-bound construction in the paper reduces the matrix problem to a single-column vector problem. For the momentum inputs in this construction, Muon performs vector normalization, while NorMuon produces a sign-like update with equal coordinate magnitudes. This is closely related to the gradient-density effect analyzed by [1], which connects uneven gradient magnitudes to the local progress of sign-based updates. Although the matching complexity bounds are nontrivial, the paper should more clearly explain the additional technical insights needed to turn this known geometric effect into a lower bound covering fixed momentum and deterministic adaptive step sizes.
  2. More importantly, the worst-case construction relies on a highly imbalanced gradient structure whose unfavorable alignment persists over many iterations. It remains unclear whether similar situations occur in practical LLM training, how often they occur, and whether they meaningfully affect overall training progress. Although Figure 5 examines alignment during training, it compares different optimizer performance and does not directly test this mechanism. As a result, the practical significance of the worst-case slowdown remains unclear.

[1] Balles, Lukas, Fabian Pedregosa, and Nicolas Le Roux. "The geometry of sign gradient descent." arXiv preprint arXiv:2002.08056 (2020).

Questions for authors
  1. What are the main new technical insights in the lower-bound proof beyond the gradient-density effect of sign-based updates analyzed in [1]?
  2. Do the unfavorable gradient structures in the worst-case construction occur and persist in LLM training? Could the authors compare Muon and NorMuon updates on the same momentum inputs to evaluate whether the alignment loss has a meaningful impact on training progress?
5 comments

Thank you for your time and comments. We clarify each point below.

W1 & Q1: New technical insights in the lower-bound proof beyond the gradient-density effect of sign-based updates analyzed in [1]

New techniques lie in the construction of the hard function (Section 3.3).

  1. The hard instance depends on and covers any initialization and step size. The gradients stay constant along the trajectory, and hence the unfavorable gradient density persists for any fixed β1,β2∈[0,1)\beta_1, \beta_2 \in [0, 1).
  2. We construct ff from bump functions (Section 3.3). This bounds the per-step improvement independently of the step size, whereas a per-step analysis such as [1] is often tied to a specific step size (Eq. (11) of [1]).

These ingredients turn the gradient density effect into a complexity lower bound. We clarify that our analysis does not follow from [1]. [1] uses the gradient density in vector optimization to compare the per-step improvement of GD and signGD at a fixed point (Eqs. (28)–(29) of [1]). However, per-step and complexity analyses are different notions in general, e.g., Nesterov's method can increase the function value in one step yet converges faster than GD.

In conclusion, we acknowledge that [1] identifies a one-step geometric effect for sign-based methods. We construct a rank-one hard instance that lets the effect persist. We will make this relationship explicit in the revision.

W2 & Q2(a): Do the unfavorable gradient structures in the worst-case construction occur and persist in LLM training?

The norm imbalance in our worst-case construction occurs and persists in LLM training, but at a much smaller scale. We run a new experiment in the same setting as the 286M model in the paper (2.20B tokens, 4,200 steps). For each matrix, we take the norms that NorMuon normalizes: the row norms for square matrices and the norms along the larger dimension otherwise. We report their coefficient of variation (CV = standard deviation / mean) for NorMuon's momentum MM and for the Muon and NorMuon updates computed from MM (Table R1). In our hard instance, the CV of MM and the Muon update is (m−1)1.52m−1≥13\frac{(m-1)^{1.5}}{2m-1}\geq 13, and the CV of the NorMuon update is 00. In LLM training, the CV is much smaller. Here, MM is 768×768768\times768 for the QKVO matrices, and 768×3072768\times3072 or 3072×7683072\times768 for the MLP matrices.

Table R1: CV of the row norms (QKVO) or the norms along the larger dimension (MLP) for gradient momentum / Muon update / NorMuon update.

StepQKVOMLP inputMLP output
500.653/0.127/0.0730.730/0.291/0.0800.751/0.398/0.077
2500.597/0.033/0.0110.692/0.199/0.0501.094/0.242/0.054
1,0000.623/0.026/0.0100.797/0.164/0.0491.506/0.193/0.053
2,5000.567/0.019/0.0090.609/0.157/0.0451.240/0.177/0.046
4,0000.400/0.014/0.0080.436/0.149/0.0360.765/0.161/0.030
4,2000.401/0.014/0.0080.443/0.148/0.0350.797/0.160/0.027

Hence, the imbalance behind NorMuon's worst-case slowdown is present in LLM training, but at a much smaller scale. This imbalance is also one motivation for NorMuon and appears in Figures 10–12 of [2].

Q2(b): Could the authors compare Muon and NorMuon updates on the same momentum inputs to evaluate whether the alignment loss has a meaningful impact on training progress?

We compare the alignment of Muon and NorMuon on the same momentum, namely NorMuon's momentum Mt+1M_{t+1}. Let Ot+1O_{t+1} and Dt+1D_{t+1} be the Muon and NorMuon directions (scaled as in Eq. (4.1)), both computed from Mt+1M_{t+1}. We report the ratio ⟨Mt+1,Dt+1⟩/⟨Mt+1,Ot+1⟩\langle M_{t+1}, D_{t+1} \rangle / \langle M_{t+1}, O_{t+1} \rangle in Table R2. For comparison, this ratio is 2m−1m(m2+m−1)≤8%\frac{2m-1}{\sqrt{m(m^2+m-1)}}\leq 8\% in our hard instance.

Table R2: NorMuon/Muon alignment ratio on the same momentum.

StepQKVOMLP inputMLP output
5097.87%89.38%81.39%
25099.49%94.67%90.17%
1,00099.60%96.00%92.56%
2,50099.65%96.65%93.87%
4,00099.74%97.28%95.39%
4,20099.73%97.26%95.38%

On the same momentum, NorMuon does lower the alignment, as in the hard instance, but only slightly. Since the alignment is the first-order decrease term in Eq. (3.1), the alignment loss has a small negative impact on training progress. By design, Muon maximizes the alignment over the operator norm ball, while NorMuon gives up some alignment. Yet, with its own momentum, NorMuon attains a larger momentum and a larger alignment on the MLP matrices (Appendix C.3). We have added both tables to the revision.

Thank you again, and we look forward to further discussion.

References

[1] L. Balles, F. Pedregosa, and N. Le Roux. The geometry of sign gradient descent. ArXiv Preprint: 2002.08056, 2020.

[2] Z. Li, L. Liu, C. Liang, W. Chen, and T. Zhao. NorMuon: Making Muon more efficient and scalable. In ICML, 2026.

AuthorsOpenReview ↗

Kind Follow-Up on Our Response

Dear Reviewer bVe2,

Thank you for your detailed review. We appreciate your time and effort.

As the discussion deadline approaches, we wanted to follow up to check whether our response has addressed your concerns and questions. If you have any further questions, we would be happy to discuss them.

Thank you again, and we look forward to your response.

Sincerely, Authors of Paper 22475

Dear Reviewer bVe2,

Thank you again for your detailed review. As the discussion period ends in 48 hours, we would be grateful if you could let us know whether our responses have addressed your questions and concerns. We would be happy to discuss any further questions you may have.

Sincerely,

Authors of Paper 22475

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Rebuttal and Manuscript Improvement

Dear AC, Reviewer bVe2, Reviewer tW3m, and Reviewer utW4,

Thank you for your time and effort in coordinating and reviewing this paper. We learned a lot from the reviews and the discussion.

To make your assessment easier, we provide a concise summary of the reviews, our rebuttal, and the main changes to the paper.

Overview

  • Reviewers describe the theory as clean and complete with nontrivial matching bounds, and the experiments as covering several model sizes and broad evaluation settings.
  • The main concerns are:
    1. The scope and novelty of our lower bound relative to [1].
    2. Whether the worst case structure appears in LLM training.
    3. The paper "does not explain why row balancing helps in practice."
  • We addressed these with new discussions and clarifications (mainly after Theorem 3.1), an extension of our lower bound to other Muon variants with normalization (Appendix B.2), new LLM experiments on row norm imbalance and same momentum alignment (Appendix C.3, Figures 7–8), and a discussion of how our results relate to the question of why row normalization helps (Section 5).

Reviewer bVe2 (Score: 2)

Reviewer bVe2 notes that our matching bounds are nontrivial and that the experiments cover several model sizes and broad evaluation settings. Their concerns are (i) the technical insight beyond [1] and (ii) whether the unfavorable gradient structure occurs in LLM training.

  • For (i), we clarified that [1] compares a single step of GD and signGD at a fixed point and step size, while our complexity lower bound requires multiple steps. Our hard instance keeps the gradient constant for any initialization, step size rule, and fixed momentum, and bounds the progress of each step independently of the step size. We added this discussion after Theorem 3.1.
  • For (ii), we ran new experiments on the 286M model, tracking the QKVO, MLP input, and MLP output matrices throughout training (Appendix C.3, Figures 7–8). First, we found that the row norm imbalance of the momentum is present but milder than in the hard instance. Second, as the reviewer suggested, we compared Muon and NorMuon on the same momentum and found that NorMuon retains >81% of Muon's alignment, compared with about 8% in the hard instance. Since the alignment is the first order decrease (Eq. (3.1)), the worst case mechanism exists in practice, but at a smaller scale. Moreover, with its own momentum, NorMuon even attains a larger alignment than Muon on the MLP matrices.

We hope our responses and new experiments have addressed the reviewer's concerns.


Reviewer tW3m (Score: 3->3)

Reviewer tW3m finds the paper clear, informative, and sound. Their questions are (i) whether the difference between MLP and attention matrices holds in other settings and (ii) whether we have further observations on this difference.

  • For (i), we trained a new 93M model (Table 14) and observed the same pattern: NorMuon on the MLP matrices recovers most of the gain, while NorMuon on the QKVO matrices yields little.
  • For (ii), our new measurements show that the MLP momentum has more imbalanced norms, and that NorMuon's alignment differs from Muon's mostly on the MLP matrices (Appendix C.3).

After the rebuttal, the reviewer confirmed that their questions were addressed and called the work interesting and meaningful.


Reviewer utW4 (Score: 2->3)

Reviewer utW4 finds the theory clean and complete and the worst case construction simple and intuitive. Their concerns are (i) the result is expected since Muon is steepest descent, (ii) the puzzle is overstated given Aurora, (iii) the experiments are disconnected from the analysis, (iv) the scope of the lower bound, and (v) the choice of geometry.

  • For (i), we clarified that rescaling steepest descent needs not worsen complexity (e.g., AdaGrad).
  • For (ii), we clarified that Aurora provides strong empirical evidence while our work provides theory, and we listed a theory of Aurora's mechanism as future work.
  • For (iii), we ran the new experiments above (Appendix C.3) and edited to clarify where they helps.
  • For (iv), we extended the lower bound to the requested settings (Appendix B.2 and Section 4.3).
  • For (v), we showed that NorMuon is not steepest descent under the max row norm geometry either, and discussed why the operator norm may be more suitable.

After the rebuttal, the reviewer found these substantive improvements and raised their score. They noted that their main concern still holds, namely that the paper does not explain why row balancing helps in practice. We agree that this is an important and challenging open question, and we position our paper as a first step toward it, providing matching optimization complexity bounds and empirical insights into row imbalance and alignment.


Thank you again.

Sincerely,

Authors of Paper 22475

[1] L. Balles, F. Pedregosa, and N. Le Roux. The geometry of sign gradient descent. ArXiv Preprint: 2002.08056, 2020.

AuthorsOpenReview ↗

Summary of requested changes

Again, we thank the reviewers for their time and constructive suggestions. As the discussion period is about to end, we have updated our paper based on our recent discussion with Reviewer utW4. Below, we summarize all changes requested by the reviewers. All changes are highlighted in blue.

Paper

  • Abstract and Section 1: we revise the wording on the puzzle and our ablation studies.
  • Section 3.1: we add a discussion of the relation to [1] and of other row normalization methods.
  • Section 4.3: we add a remark that our theorems hold for normalization along the larger dimension.
  • Section 5: we add a sentence to better position our results and expand the future directions.
  • Appendix B.2: we extend the lower bound to Muon+, MuonEq-R, and Aurora.

Experiments

  • Pages 23–24, Tables 6–7 and Figure 4: additional synthetic experiment with a cosine step size schedule.
  • Pages 26–27, Table 14: parameter group ablation on a new 93M model trained on FineWeb-Edu.
  • Page 28, Figure 7: alignment of Muon and NorMuon on the same momentum.
  • Pages 28–29, Figure 8: row norm imbalance of the momentum and update directions during LLM training.

Sincerely,

Authors of Paper 22475

[1] L. Balles, F. Pedregosa, and N. Le Roux. The geometry of sign gradient descent. ArXiv Preprint: 2002.08056, 2020.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.