acceptodds
Under review as a conference paper at ICLR 2027

Can Normalization Match Muon at Scale?

Abstract

Muon is increasingly replacing AdamW to train large-scale neural networks, but it requires expensive matrix orthogonalization. Recent studies report gains over Muon with simple row or column normalization alone. However, we find that normalization does not match Muon at scale. Our analysis shows that normalization leaves large correlations across rows, while Muon removes them. We also study how normalization should be combined with Muon. Applying normalization after Muon is more effective than applying it before, while using EMA statistics for normalization provides little benefit. The improvement over vanilla Muon decreases at larger scales. We establish these findings in the overtrained regime of frontier production language models. Our models are trained at 37–110 the Chinchilla optimum, whereas NanoGPT-style benchmarks commonly used in recent work are close to 1 Chinchilla. Experiments across vision tasks (e.g., supervised learning, self-supervised learning, diffusion models) lead to similar conclusions.

Primary area:
optimization
Keywords:
Optimizer, LLM Pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sH5HOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether row or column normalization can replace Muon’s orthogonalization, and how the two should be combined. It tests this on overtrained MobileLLM-R1 language models and on ViT, MAE and DiT. It finds that normalization alone falls behind Muon and attributes the gap to cross-row correlations that normalization leaves in the update. It also finds that normalizing after Muon works better than normalizing before, and that the benefit of combining them shrinks at larger scale.

Critical strengths

None

Critical weaknesses
  1. The novelty is limited. The paper proposes no new method, as RowNorm, ColNorm and SinkGD (Scetbon et al., 2025) all come from prior work. The view of RowNorm as a diagonal approximation of Muon’s preconditioner comes from RMNP (Deng et al., 2026), and both Theorem 1 and the cross-row correlation argument follow directly from it. On combining the two, NorMuon (Li et al., 2026c) already found that normalizing after orthogonalization works better than before, and Muon+ (Zhang et al., 2026b) found that a stateless normalization after Muon outperforms EMA-based variants such as NorMuon. What is new is the overtrained setting.
  2. The experiments do not support the central claims. Muon’s tuned learning rate is reused for all normalization methods, and matching update RMS does not make their effective step sizes equal. Most reported gaps are within Muon’s own seed variation, the main tables have no error bars, and the largest language model was trained with a single seed. Normalization alone is never run at the larger scale, the short training regime of prior work is never reproduced under the same protocol, and the small and large runs differ in learning rate, context length and number of seeds.
Questions for authors

Which findings are new relative to RMNP, NorMuon and Muon+? It would help to separate results that are new from those that confirm prior findings in the overtrained setting.

3 comments

We sincerely thank you for your comment. We appreciate that you find our evaluation in the overtrained setting new. We address your concerns and questions below:

W1: The novelty is limited. The paper proposes no new method, as RowNorm, ColNorm and SinkGD (Scetbon et al., 2025) all come from prior work. The view of RowNorm as a diagonal approximation of Muon's preconditioner comes from RMNP (Deng et al., 2026), and both Theorem 1 and the cross-row correlation argument follow directly from it. On combining the two, NorMuon (Li et al., 2026c) already found that normalizing after orthogonalization works better than before, and Muon+ (Zhang et al., 2026b) found that a stateless normalization after Muon outperforms EMA-based variants such as NorMuon. What is new is the overtrained setting.

We agree with you that the methods we study come from prior work, and that the view of RowNorm as a diagonal approximation of Muon's preconditioner comes from RMNP. However, RMNP uses this view to motivate RowNorm as an approximation of Muon. Our analysis of Equation 4 instead shows what this approximation leaves out: normalization does not remove any correlation across rows or columns, whereas Muon removes all of it. Theorem 1 derives the exact condition under which the two are equal, and Figure 2 measures how far real training is from this condition. Neither of these results is in RMNP.

On combining normalization with Muon, our results with per-method learning-rate tuning and five seeds partly differ from the findings you mention:

  • NorMuon finds that normalizing after orthogonalization works better than before. In our results, the ordering makes no clear difference: the combinations differ from Muon by −0.0034 to +0.0006, within one standard deviation, and normalizing before Muon is, if anything, slightly better. Paired by seed, none of these differences is significant at the 0.05 level.
  • Muon+ applies one additional stateless normalization after Muon and reports gains over Muon. This is exactly our Muon → RowNorm and Muon → ColNorm. With the learning rate tuned and five seeds, they differ from Muon by −0.0004 and +0.0006 (paired p = 0.93 and 0.81). So in the overtrained regime, this additional normalization brings no measurable gain over Muon.

We believe these results, obtained far beyond the compute-optimal regime, are useful for understanding which prior findings hold in the overtrained setting.

W2: The experiments do not support the central claims. Muon's tuned learning rate is reused for all normalization methods, and matching update RMS does not make their effective step sizes equal. Most reported gaps are within Muon's own seed variation, the main tables have no error bars, and the largest language model was trained with a single seed. Normalization alone is never run at the larger scale, the short training regime of prior work is never reproduced under the same protocol, and the small and large runs differ in learning rate, context length and number of seeds.

We agree with you that matching the update RMS does not make the effective step sizes of different methods equal, and that the learning rate should be tuned for each method. In response, we rerun all language-model comparisons with the learning rate tuned separately:

  • Standalone methods are swept over {1e-1, 3e-2, 1e-2, 3e-3, 1e-3} at one seed.
  • Combinations with Muon are swept over {3e-2, 1e-2, 3e-3}.
  • We then run five seeds at the best learning rate of each method.

The full sweep at one seed is below (best value in bold):

1e-13e-21e-23e-31e-3
Muon3.04673.01353.00943.02323.0664
AdamW3.11033.03943.02053.03973.0871
RowNorm3.10153.03363.01003.04133.0848
ColNorm3.08553.02443.01933.04943.0871
SinkGD3.07313.02143.01593.05423.0867

At one seed, the best learning rate is 1e-2 for every method, and it always lies in the interior of the grid. For Muon and AdamW, we further run all five seeds at every learning rate. AdamW's best value stays at 1e-2, while Muon's best value becomes 3e-2 (3.0093 ± 0.0044, compared with 3.0128 ± 0.0074 at 1e-2). We therefore compare all methods with Muon at 3e-2. The five-seed results at the best learning rate of each method are below:

losspaired difference vs Muon (p-value)
Muon3.0093 ± 0.0044
AdamW3.0231 ± 0.0069+0.0137 (0.04)
RowNorm3.0117 ± 0.0063+0.0024 (0.61)
ColNorm3.0122 ± 0.0051+0.0029 (0.13)
SinkGD3.0192 ± 0.0077+0.0099 (0.10)
RowNorm → Muon3.0061 ± 0.0054−0.0032 (0.25)
ColNorm → Muon3.0073 ± 0.0060−0.0021 (0.39)
SinkGD → Muon3.0060 ± 0.0056−0.0034 (0.46)
Muon → RowNorm3.0089 ± 0.0082−0.0004 (0.93)
Muon → ColNorm3.0099 ± 0.0029+0.0006 (0.81)
Muon → SinkGD3.0081 ± 0.0034−0.0012 (0.59)

We can see that standalone normalization does not improve on Muon. The combinations differ from Muon by −0.0034 to +0.0006, which is within one standard deviation. Paired by seed (two-sided paired t-test over five seeds), none of these differences is significant at the 0.05 level. We agree with you that the gains of combining normalization with Muon are within seed variation, and we have revised the paper to state that they bring no clear benefit over vanilla Muon.

We have also run standalone normalization on the 950M language model, trained on 2.1T tokens (single seed; all runs use the same learning rate of 1e-3):

950M language modelvalidation loss
Muon2.3166
AdamW2.3344
RowNorm2.3257
SinkGD2.3216

We can see that at 950M, standalone RowNorm and SinkGD finish 0.0091 and 0.0050 behind Muon in validation loss. This is consistent with our findings on the 140M model.

We agree that the 950M model is trained with a single seed. Each 950M run trains on 2.1T tokens, which makes multiple seeds very expensive for us. We therefore run our seeded and learning-rate-controlled comparisons at 140M, and use the 950M model to check whether the same trend holds at a larger scale.

We agree with you that it would be valuable to understand why prior works report gains of normalization over Muon. Their settings differ from ours in several aspects, including the model, data, training horizon, and tuning protocol. Our study focuses on the overtrained regime of production models, where we find that normalization does not improve on Muon.

Finally, the small and large runs now use the same context length of 2048. They still differ in learning rate, since the best learning rate depends on the model size. Within each scale, however, all methods follow the same protocol: at 140M every method is tuned separately, and at 950M all methods share one learning rate.

Q1: Which findings are new relative to RMNP, NorMuon and Muon+? It would help to separate results that are new from those that confirm prior findings in the overtrained setting.

Thank you for the suggestion.

Relative to RMNP, our new results are the analysis of Equation 4 and Theorem 1. RMNP views RowNorm as a diagonal approximation of Muon's preconditioner. We show that this approximation does not remove any correlation across rows or columns, derive the exact condition under which it equals Muon, and measure in Figure 2 that real training is far from this condition. Empirically, RMNP and other works [2, 3] report gains of normalization over Muon in NanoGPT-style settings. In contrast, we find that normalization does not improve on Muon about 110× beyond the Chinchilla optimum, once each method's learning rate is tuned separately.

Relative to NorMuon, our results differ in the ordering of normalization and Muon. NorMuon finds that normalizing after orthogonalization works better than before. In our results, both orderings differ from Muon by amounts within one standard deviation, and neither is clearly better. Paired by seed, none of these differences from Muon is significant at the 0.05 level.

Relative to Muon+, our results differ in the size of the gain. Muon+ applies one additional stateless normalization after Muon and reports improvements over Muon. This corresponds to our Muon → RowNorm and Muon → ColNorm. With the learning rate tuned and five seeds, both are within 0.0006 of Muon, and neither difference is significant when paired by seed (p = 0.93 and 0.81).

None of these previous works studies the overtrained regime at our scale, which reaches about 110× the Chinchilla optimum for both the 140M and the 950M model (310B and 2.1T tokens). We also observe that combining normalization with Muon brings no clear benefit over vanilla Muon, and the same trend holds at the larger scale.

We thank you for your valuable feedback and see resolving your questions and concerns as a great improvement to our paper. If you have any further questions or concerns, we are very happy to answer.

Reviewer_3gTZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether simple normalization methods can match or improve Muon in large-scale training. The authors find that normalization generally underperforms Muon and attribute this gap to remaining correlations across rows that Muon removes. They also compare different ways of combining normalization with Muon, finding that applying normalization after Muon is generally more effective, while EMA-based normalization provides little additional benefit. These gains diminish at larger scales.

Critical strengths
  1. Extensive evaluation. The paper evaluates normalization methods under long training budgets and across several vision tasks. This provides useful empirical evidence.
  2. Practical insights into combining normalization with Muon. The paper systematically compares normalization before and after Muon, as well as EMA-based normalization. These experiments provide useful empirical guidance on the effectiveness and limitations of existing normalization strategies.
Critical weaknesses
  1. Limited novelty and new findings. The main finding that simple normalization falls short of Muon is not surprising, and the paper provides limited new insight beyond this observation. For example, Theorem 1 shows exact equivalence only when the rows are already orthogonal, a strong restrictive special case. It does not provide understanding of the general non-orthogonal setting or why the differences lead to worse optimization performance. So I feel the theoretical analysis is a bit disconnected from the empirical findings. Also, the paper observes that the benefits of combining normalization with Muon diminish at larger scales but without investigating why.
  2. Insufficient emeprical evidence. The paper claims that normalization cannot match Muon at scale, but the largest-scale experiments (table 5) lack isolated normalization baselines. Furthermore, the normalization methods use Muon’s tuned learning rate without their own tuning, so the performance gaps may simply be due to suboptimal hyperparameters.
Questions for authors
  1. Can the authors provide a deeper explanation of why normalization underperforms Muon, beyond Theorem 1? For example, why is there diminishing benefit to combining normalization with Muon at larger scales?
  2. Can the authors provide additional comparisons with independently tuned standalone normalization methods?
2 comments

We sincerely thank you for your comment. We appreciate that you find our paper provides useful empirical findings on normalization and Muon. We address your concerns and questions below:

W1a: Limited novelty and new findings. The main finding that simple normalization falls short of Muon is not surprising, and the paper provides limited new insight beyond this observation. For example, Theorem 1 shows exact equivalence only when the rows are already orthogonal, a strong restrictive special case. It does not provide understanding of the general non-orthogonal setting or why the differences lead to worse optimization performance. So I feel the theoretical analysis is a bit disconnected from the empirical findings. Also, the paper observes that the benefits of combining normalization with Muon diminish at larger scales but without investigating why.

We wish to clarify our contribution here. Several recent works [1, 2, 3] report that normalization alone matches or outperforms Muon in NanoGPT-style settings. Our results show that, far beyond the compute-optimal regime and with the learning rate of each method tuned separately, normalization does not improve on Muon. We believe this is a useful finding for the community, since these methods are proposed as cheaper replacements for Muon.

Our geometric view of normalization comes from prior work [1]. It helps us reveal why normalization does not approximate Muon well in practice: normalization does not remove any correlation across rows or columns, whereas Muon removes all of it. We measure these correlations during training (Figure 2) and find them large across tasks, matrix types, and the entire training process.

We agree with you that Theorem 1 covers a special case: normalization equals Muon if and only if there is no correlation across rows or columns at all. We view it as the boundary of the general setting, and our measurements in Figure 2 show that real training is far from this boundary.

In our updated results, combining normalization with Muon brings no clear benefit over vanilla Muon in language modeling, and the same trend holds at the larger scale. Understanding why normalization adds little on top of Muon's orthogonalization is an interesting direction for future work.

[1] Deng et al. RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization. ICML 2026. [2] Glentis et al. Memory-Efficient LLM Pretraining via Minimalist Optimizer Design. ICML 2026. [3] Xu et al. On the Width Scaling of Neural Optimizers under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer. arXiv 2026.

W2: Insufficient empirical evidence. The paper claims that normalization cannot match Muon at scale, but the largest-scale experiments (table 5) lack isolated normalization baselines. Furthermore, the normalization methods use Muon's tuned learning rate without their own tuning, so the performance gaps may simply be due to suboptimal hyperparameters.

Thank you for pointing this out. We have run standalone normalization on the 950M language model, trained on 2.1T tokens. The results are below (single seed; all runs use the same learning rate of 1e-3):

950M language modelvalidation loss
Muon2.3166
AdamW2.3344
RowNorm2.3257
SinkGD2.3216

We can see that at 950M, standalone RowNorm and SinkGD finish 0.0091 and 0.0050 behind Muon in validation loss. This is consistent with our findings on the 140M model.

We agree with you that the learning rate should be tuned for each method. In response, we rerun all language-model comparisons with the learning rate tuned separately:

  • Standalone methods are swept over {1e-1, 3e-2, 1e-2, 3e-3, 1e-3} at one seed.
  • Combinations with Muon are swept over {3e-2, 1e-2, 3e-3}.
  • We then run five seeds at the best learning rate of each method.

The full sweep at one seed is below (best value in bold):

1e-13e-21e-23e-31e-3
Muon3.04673.01353.00943.02323.0664
AdamW3.11033.03943.02053.03973.0871
RowNorm3.10153.03363.01003.04133.0848
ColNorm3.08553.02443.01933.04943.0871
SinkGD3.07313.02143.01593.05423.0867

At one seed, the best learning rate is 1e-2 for every method, and it always lies in the interior of the grid. For Muon and AdamW, we further run all five seeds at every learning rate. AdamW's best value stays at 1e-2, while Muon's best value becomes 3e-2 (3.0093 ± 0.0044, compared with 3.0128 ± 0.0074 at 1e-2). We therefore compare all methods with Muon at 3e-2. The five-seed results at the best learning rate of each method are below:

MuonAdamWRowNormColNormSinkGD
loss3.0093 ± 0.00443.0231 ± 0.00693.0117 ± 0.00633.0122 ± 0.00513.0192 ± 0.0077
paired difference vs Muon (p-value)+0.0137 (0.04)+0.0024 (0.61)+0.0029 (0.13)+0.0099 (0.10)

We can see that with per-method tuning, RowNorm and ColNorm finish 0.0024 and 0.0029 behind Muon, and SinkGD 0.0099 behind. Since all methods share the same initialization and data order for the same seed, we compare each method with Muon paired by seed (two-sided paired t-test over five seeds). None of the three normalization methods is better than Muon, and AdamW is significantly worse (p = 0.04). So the gap between normalization and Muon is not due to an unsuitable learning rate.

For self-supervised learning (MAE) and diffusion models (DiT), we also sweep the learning rate for every method. We report the final training loss (single seed). The full sweeps are below (best value of each method in bold):

MAE pre-training loss4.5e-61.5e-54.5e-51.5e-44.5e-4
Muon0.41350.40620.40410.40460.4236
AdamW0.40990.40660.40640.4103
RowNorm0.40910.40660.40680.4105
ColNorm0.40960.40690.40690.4110
SinkGD0.41710.40850.40640.40650.4109
DiT training loss3e-51e-43e-41e-3
Muon0.15650.15480.15470.1565
AdamW0.15810.15580.15570.1573
RowNorm0.15730.15560.1556diverged
ColNorm0.15740.15570.15560.1564
SinkGD0.15710.15540.1556diverged

For every method, the best learning rate lies in the interior of the grid. We can see that Muon reaches the lowest loss in both tasks at its best learning rate: 0.4041 in MAE (vs. 0.4064 to 0.4069 for the other methods) and 0.1547 in DiT (vs. 0.1554 to 0.1557).

Q1: Can the authors provide a deeper explanation of why normalization underperforms Muon, beyond Theorem 1? For example, why is there diminishing benefit to combining normalization with Muon at larger scales?

Thank you for this question. Theorem 1 can be extended to the general case where rows are not orthogonal. Consider a wide momentum matrix M∈Rm×nM \in \mathbb{R}^{m\times n} with full row rank, and let C=D−1/2MM⊤D−1/2C = D^{-1/2}MM^\top D^{-1/2} be the normalized Gram matrix in Equation 4, i.e., CijC_{ij} is the cosine similarity between rows ii and jj of MM. Let λ1,…,λm\lambda_1, \dots, \lambda_m be the eigenvalues of CC. The update of RowNorm, nD−1/2M\sqrt{n}D^{-1/2}M, has singular values nλi\sqrt{n\lambda_i}, whereas all singular values of Muon's update are n\sqrt{n}. Since every diagonal entry of CC is one, we have ∑iλi=m\sum_i \lambda_i = m and ∑iλi2=∥C∥F2=m+∑i≠jCij2\sum_i \lambda_i^2 = \|C\|_F^2 = m + \sum_{i \neq j} C_{ij}^2. Therefore,

∑i=1m(λi−1)2=∑i≠jCij2.\sum_{i=1}^{m} (\lambda_i - 1)^2 = \sum_{i \neq j} C_{ij}^2 .

In other words, how unevenly RowNorm distributes the update across singular directions, compared with Muon's flat spectrum, is exactly the total squared correlation across rows. Theorem 1 is the special case where both sides are zero. This identity also connects our two empirical observations: the large off-diagonal entries of the Gram matrices in Figure 2 directly imply the uneven singular values of the normalized update in Figure 4.

This identity quantifies how far normalization is from Muon, but it does not by itself explain how this difference affects the final loss. It would be interesting for future work to study how this gap evolves with the training horizon, and to analyze it within the linear minimization oracle (LMO) framework of steepest descent under different matrix norms. Regarding combining normalization with Muon, our updated results show no clear benefit over vanilla Muon in language modeling, and the same trend holds at the larger scale. Understanding why normalization adds little on top of orthogonalization is also an interesting direction for future work.

We thank you for your valuable feedback and see resolving your questions and concerns as a great improvement to our paper. If you have any further questions or concerns, we are very happy to answer.

Reviewer_a6eZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents an empirical study of whether row/column normalization can match performance of Muon and how can the be combined. The paper develops a common framework in which all methods share the momentum and differ only in the map applied to the momentum. A geometric analysis compares the Gram matrices of various methods and proves that they coincide only when the momentum rows are mutually orthogonal. The paper further studies the order of normalization and orthogonalization. Experiments on language models and on vision tasks show that normalization alone falls behind Muon. They also show that applying normalization after Muon helps more than applying it before, and that this gain shrinks at the larger model size.

Critical strengths
  • The paper provides more results for the literature to understand whether cheap row or column normalization can replace the orthogonalization in Muon, which matters because orthogonalization is costly.

  • The paper evaluates optimizers beyond the usual NanoGPT-style setting. It trains language models far past the compute-optimal token budget and covers supervised, self-supervised and generative vision tasks.

  • The paper is clearly written, and the pseudocode makes every method easy to reproduce.

Critical weaknesses
  • The technical contribution is limited. The paper proposes no new method, and its geometric view of normalization as a diagonal approximation of Muon restates a derivation from previous work [1]. Therefore, the paper mainly adds a benchmark of known methods at a larger training budget.

  • The claim that normalization fails at scale is not fully supported. The paper states that previous work reports gains over Muon from normalization alone, and attributes the different outcome to the shorter training regimes of that work. However, the normalization methods alone are trained only with the smaller language model and the base-size vision models, and the larger-scale validation covers only the combinations with Muon. Besides, the loss curves show that normalization falls behind Muon from the early steps of training rather than losing an early lead. Also, the used normalization baseline differs from previous methods in whether they normalize the gradient or the momentum. This could also be the reason of the gap.

  • The comparison is not fair to the normalization methods. The learning rate is tuned for Muon alone and copied to the other methods, while the paper’s own sweep shows that one grid step in the learning rate shifts the loss by about the size of the reported gap.

  • The gains of combining normalization with Muon is within noise. Most reported gains are smaller than the seed variation the paper reports for a single method.

[1] RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization

Questions for authors
  • What causes the shape-aligned variant's much larger loss in language modeling?
3 comments

We sincerely thank you for your comment. We appreciate you find our work's evaluation setting beyond the usual NanoGPT-style setting. We address your concerns and questions below:

W1: The technical contribution is limited. The paper proposes no new method, and its geometric view of normalization as a diagonal approximation of Muon restates a derivation from previous work. Therefore, the paper mainly adds a benchmark of known methods at a larger training budget.

We wish to clarify that our contributions are the empirical findings and the analysis, not a new method. Prior works [1, 2, 3] all report improvements of normalization over Muon in NanoGPT-style settings. In contrast, we evaluate these methods far beyond the compute-optimal regime (about 110× the Chinchilla optimum) and across tasks. With each method's learning rate tuned separately, we find normalization does not improve on Muon.

Our geometric view of normalization indeed comes from prior work [1], and we credit it in the paper. However, our analysis of Equation 4 is new. It shows that normalization only equalizes the norms of rows or columns and does not remove any correlation across them, whereas Muon removes all of it. We also measure these correlations in practice (Figure 2) and find them large across tasks, matrix types, and the entire training process. Theorem 1 further derives the exact condition under which normalization equals Muon: this happens if and only if there is no correlation across rows or columns at all. Neither of these two results is in RMNP [1].

[1] Deng et al. RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization. ICML 2026. [2] Glentis et al. Memory-Efficient LLM Pretraining via Minimalist Optimizer Design. ICML 2026. [3] Xu et al. On the Width Scaling of Neural Optimizers under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer. arXiv 2026.

W2a: The claim that normalization fails at scale is not fully supported. The paper states that previous work reports gains over Muon from normalization alone, and attributes the different outcome to the shorter training regimes of that work. However, the normalization methods alone are trained only with the smaller language model and the base-size vision models, and the larger-scale validation covers only the combinations with Muon. Besides, the loss curves show that normalization falls behind Muon from the early steps of training rather than losing an early lead.

We would like to clarify that a longer training horizon does not require a larger model. A small model trained on many more tokens is also in the long-horizon regime. In the current draft, we train the 140M model on 310B tokens with a context length of 2048. This is about 110× the Chinchilla optimum, the same ratio as our 950M model trained on 2.1T tokens.

We have also run standalone normalization on the 950M language model, trained on 2.1T tokens. The results are below (single seed; all runs use the same learning rate of 1e-3):

950M language modelvalidation loss
Muon2.3166
AdamW2.3344
RowNorm2.3257
SinkGD2.3216

We can see that at 950M, standalone RowNorm and SinkGD finish 0.0091 and 0.0050 behind Muon in validation loss. This is consistent with our findings on the 140M model.

W2b: Also, the used normalization baseline differs from previous methods in whether they normalize the gradient or the momentum. This could also be the reason of the gap.

Our implementation indeed always applies normalization to the momentum rather than the mini-batch gradient. To address your concern, we add variants that normalize the mini-batch gradient first and then accumulate momentum, as in [2, 4]. We tune the learning rate of each variant separately and run five seeds at its best value. The results on the 140M language model are below:

normalize momentum (ours)normalize gradient
RowNorm3.0117 ± 0.00633.0247 ± 0.0031
ColNorm3.0122 ± 0.00513.0232 ± 0.0069
SinkGD3.0192 ± 0.00773.0274 ± 0.0067
AdamW (reference)3.0231 ± 0.0069

We can see that normalizing the gradient is worse than normalizing the momentum for all three methods, by 0.0130, 0.0110, and 0.0082 for RowNorm, ColNorm, and SinkGD. It is also no better than AdamW (3.0231). Compared with Muon paired by seed, all three gradient variants are significantly worse (p ≤ 0.02). Therefore, normalizing the momentum does not handicap the normalization methods.

[4] Scetbon et al. Gradient Multi-Normalization for Stateless and Scalable LLM Training. 2025.

W3: The comparison is not fair to the normalization methods. The learning rate is tuned for Muon alone and copied to the other methods, while the paper's own sweep shows that one grid step in the learning rate shifts the loss by about the size of the reported gap.

We agree with you that the learning rate is a very important hyperparameter when comparing optimizers, and its optimal value can differ across methods. In response, we rerun all language-model comparisons with the learning rate tuned for each method separately:

  • Standalone methods are swept over {1e-1, 3e-2, 1e-2, 3e-3, 1e-3} at one seed.
  • Combinations with Muon and gradient-normalization variants are swept over {3e-2, 1e-2, 3e-3}.
  • We then run five seeds at the best learning rate of each method.

The full sweep at one seed is below (best value of each method in bold):

1e-13e-21e-23e-31e-3
Muon3.04673.01353.00943.02323.0664
AdamW3.11033.03943.02053.03973.0871
RowNorm3.10153.03363.01003.04133.0848
ColNorm3.08553.02443.01933.04943.0871
SinkGD3.07313.02143.01593.05423.0867
RowNorm → Muon3.02193.00363.0191
ColNorm → Muon3.01113.00583.0196
SinkGD → Muon3.01012.99763.0162
Muon → RowNorm3.00972.99743.0158
Muon → ColNorm3.01113.00903.0245
Muon → SinkGD3.01483.00973.0157

At one seed, the best learning rate is 1e-2 for every method, and it always lies in the interior of the grid. For Muon and AdamW, we further run all five seeds at every learning rate. AdamW's best value stays at 1e-2, while Muon's best value becomes 3e-2 (3.0093 ± 0.0044, compared with 3.0128 ± 0.0074 at 1e-2). We therefore compare all methods with Muon at 3e-2. The five-seed results at the best learning rate of each method are below:

MuonAdamWRowNormColNormSinkGD
loss3.0093 ± 0.00443.0231 ± 0.00693.0117 ± 0.00633.0122 ± 0.00513.0192 ± 0.0077
paired difference vs Muon (p-value)+0.0137 (0.04)+0.0024 (0.61)+0.0029 (0.13)+0.0099 (0.10)

We can see that with per-method tuning, RowNorm and ColNorm finish 0.0024 and 0.0029 behind Muon, and SinkGD 0.0099 behind. Since all methods share the same initialization and data order for the same seed, we compare each method with Muon paired by seed (two-sided paired t-test over five seeds). None of the three normalization methods is better than Muon, and AdamW is significantly worse (p = 0.04). So normalization still does not improve on Muon once each method uses its own best learning rate.

For self-supervised learning (MAE) and diffusion models (DiT), we also sweep the learning rate for every method. We report the final training loss (single seed). The full sweeps are below (best value of each method in bold):

MAE pre-training loss4.5e-61.5e-54.5e-51.5e-44.5e-4
Muon0.41350.40620.40410.40460.4236
AdamW0.40990.40660.40640.4103
RowNorm0.40910.40660.40680.4105
ColNorm0.40960.40690.40690.4110
SinkGD0.41710.40850.40640.40650.4109
DiT training loss3e-51e-43e-41e-3
Muon0.15650.15480.15470.1565
AdamW0.15810.15580.15570.1573
RowNorm0.15730.15560.1556diverged
ColNorm0.15740.15570.15560.1564
SinkGD0.15710.15540.1556diverged

For every method, the best learning rate lies in the interior of the grid. We can see that Muon reaches the lowest loss in both tasks at its best learning rate: 0.4041 in MAE (vs. 0.4064 to 0.4069 for the other methods) and 0.1547 in DiT (vs. 0.1554 to 0.1557).

W4: The gains of combining normalization with Muon is within noise. Most reported gains are smaller than the seed variation the paper reports for a single method.

We agree with you. To check this more carefully, we rerun all combinations of normalization and Muon with the learning rate tuned for each combination and five seeds at its best value. The results on the 140M language model are below:

losspaired difference vs Muon (p-value)
Muon3.0093 ± 0.0044
RowNorm → Muon3.0061 ± 0.0054−0.0032 (0.25)
ColNorm → Muon3.0073 ± 0.0060−0.0021 (0.39)
SinkGD → Muon3.0060 ± 0.0056−0.0034 (0.46)
Muon → RowNorm3.0089 ± 0.0082−0.0004 (0.93)
Muon → ColNorm3.0099 ± 0.0029+0.0006 (0.81)
Muon → SinkGD3.0081 ± 0.0034−0.0012 (0.59)

Here Muon uses its best learning rate over five seeds (3e-2). We can see that the combinations differ from Muon by −0.0034 to +0.0006, which is within one standard deviation of a single method. Since all methods share the same initialization and data order for the same seed, we also compare each combination with Muon paired by seed (two-sided paired t-test over five seeds). None of the differences is significant at the 0.05 level. We have removed the claim that applying normalization after Muon is more effective than before. In the new results, the ordering makes no clear difference.

We thank you for your valuable feedback and see resolving your questions and concerns as a great improvement to our paper. If you have any further questions or concerns, we are very happy to answer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.