- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks whether row or column normalization can replace Muon’s orthogonalization, and how the two should be combined. It tests this on overtrained MobileLLM-R1 language models and on ViT, MAE and DiT. It finds that normalization alone falls behind Muon and attributes the gap to cross-row correlations that normalization leaves in the update. It also finds that normalizing after Muon works better than normalizing before, and that the benefit of combining them shrinks at larger scale.
Critical strengths
None
Critical weaknesses
- The novelty is limited. The paper proposes no new method, as RowNorm, ColNorm and SinkGD (Scetbon et al., 2025) all come from prior work. The view of RowNorm as a diagonal approximation of Muon’s preconditioner comes from RMNP (Deng et al., 2026), and both Theorem 1 and the cross-row correlation argument follow directly from it. On combining the two, NorMuon (Li et al., 2026c) already found that normalizing after orthogonalization works better than before, and Muon+ (Zhang et al., 2026b) found that a stateless normalization after Muon outperforms EMA-based variants such as NorMuon. What is new is the overtrained setting.
- The experiments do not support the central claims. Muon’s tuned learning rate is reused for all normalization methods, and matching update RMS does not make their effective step sizes equal. Most reported gaps are within Muon’s own seed variation, the main tables have no error bars, and the largest language model was trained with a single seed. Normalization alone is never run at the larger scale, the short training regime of prior work is never reproduced under the same protocol, and the small and large runs differ in learning rate, context length and number of seeds.
Questions for authors
Which findings are new relative to RMNP, NorMuon and Muon+? It would help to separate results that are new from those that confirm prior findings in the overtrained setting.
3 comments
At one seed, the best learning rate is 1e-2 for every method, and it always lies in the interior of the grid. For Muon and AdamW, we further run all five seeds at every learning rate. AdamW's best value stays at 1e-2, while Muon's best value becomes 3e-2 (3.0093 ± 0.0044, compared with 3.0128 ± 0.0074 at 1e-2). We therefore compare all methods with Muon at 3e-2. The five-seed results at the best learning rate of each method are below:
| loss | paired difference vs Muon (p-value) | |
|---|---|---|
| Muon | 3.0093 ± 0.0044 | |
| AdamW | 3.0231 ± 0.0069 | +0.0137 (0.04) |
| RowNorm | 3.0117 ± 0.0063 | +0.0024 (0.61) |
| ColNorm | 3.0122 ± 0.0051 | +0.0029 (0.13) |
| SinkGD | 3.0192 ± 0.0077 | +0.0099 (0.10) |
| RowNorm → Muon | 3.0061 ± 0.0054 | −0.0032 (0.25) |
| ColNorm → Muon | 3.0073 ± 0.0060 | −0.0021 (0.39) |
| SinkGD → Muon | 3.0060 ± 0.0056 | −0.0034 (0.46) |
| Muon → RowNorm | 3.0089 ± 0.0082 | −0.0004 (0.93) |
| Muon → ColNorm | 3.0099 ± 0.0029 | +0.0006 (0.81) |
| Muon → SinkGD | 3.0081 ± 0.0034 | −0.0012 (0.59) |
We can see that standalone normalization does not improve on Muon. The combinations differ from Muon by −0.0034 to +0.0006, which is within one standard deviation. Paired by seed (two-sided paired t-test over five seeds), none of these differences is significant at the 0.05 level. We agree with you that the gains of combining normalization with Muon are within seed variation, and we have revised the paper to state that they bring no clear benefit over vanilla Muon.
We have also run standalone normalization on the 950M language model, trained on 2.1T tokens (single seed; all runs use the same learning rate of 1e-3):
| 950M language model | validation loss |
|---|---|
| Muon | 2.3166 |
| AdamW | 2.3344 |
| RowNorm | 2.3257 |
| SinkGD | 2.3216 |
We can see that at 950M, standalone RowNorm and SinkGD finish 0.0091 and 0.0050 behind Muon in validation loss. This is consistent with our findings on the 140M model.
We agree that the 950M model is trained with a single seed. Each 950M run trains on 2.1T tokens, which makes multiple seeds very expensive for us. We therefore run our seeded and learning-rate-controlled comparisons at 140M, and use the 950M model to check whether the same trend holds at a larger scale.
We agree with you that it would be valuable to understand why prior works report gains of normalization over Muon. Their settings differ from ours in several aspects, including the model, data, training horizon, and tuning protocol. Our study focuses on the overtrained regime of production models, where we find that normalization does not improve on Muon.
Finally, the small and large runs now use the same context length of 2048. They still differ in learning rate, since the best learning rate depends on the model size. Within each scale, however, all methods follow the same protocol: at 140M every method is tuned separately, and at 950M all methods share one learning rate.
Q1: Which findings are new relative to RMNP, NorMuon and Muon+? It would help to separate results that are new from those that confirm prior findings in the overtrained setting.
Thank you for the suggestion.
Relative to RMNP, our new results are the analysis of Equation 4 and Theorem 1. RMNP views RowNorm as a diagonal approximation of Muon's preconditioner. We show that this approximation does not remove any correlation across rows or columns, derive the exact condition under which it equals Muon, and measure in Figure 2 that real training is far from this condition. Empirically, RMNP and other works [2, 3] report gains of normalization over Muon in NanoGPT-style settings. In contrast, we find that normalization does not improve on Muon about 110× beyond the Chinchilla optimum, once each method's learning rate is tuned separately.
Relative to NorMuon, our results differ in the ordering of normalization and Muon. NorMuon finds that normalizing after orthogonalization works better than before. In our results, both orderings differ from Muon by amounts within one standard deviation, and neither is clearly better. Paired by seed, none of these differences from Muon is significant at the 0.05 level.
Relative to Muon+, our results differ in the size of the gain. Muon+ applies one additional stateless normalization after Muon and reports improvements over Muon. This corresponds to our Muon → RowNorm and Muon → ColNorm. With the learning rate tuned and five seeds, both are within 0.0006 of Muon, and neither difference is significant when paired by seed (p = 0.93 and 0.81).
None of these previous works studies the overtrained regime at our scale, which reaches about 110× the Chinchilla optimum for both the 140M and the 950M model (310B and 2.1T tokens). We also observe that combining normalization with Muon brings no clear benefit over vanilla Muon, and the same trend holds at the larger scale.
We thank you for your valuable feedback and see resolving your questions and concerns as a great improvement to our paper. If you have any further questions or concerns, we are very happy to answer.
We sincerely thank you for your comment. We appreciate that you find our evaluation in the overtrained setting new. We address your concerns and questions below:
We agree with you that the methods we study come from prior work, and that the view of RowNorm as a diagonal approximation of Muon's preconditioner comes from RMNP. However, RMNP uses this view to motivate RowNorm as an approximation of Muon. Our analysis of Equation 4 instead shows what this approximation leaves out: normalization does not remove any correlation across rows or columns, whereas Muon removes all of it. Theorem 1 derives the exact condition under which the two are equal, and Figure 2 measures how far real training is from this condition. Neither of these results is in RMNP.
On combining normalization with Muon, our results with per-method learning-rate tuning and five seeds partly differ from the findings you mention:
We believe these results, obtained far beyond the compute-optimal regime, are useful for understanding which prior findings hold in the overtrained setting.
We agree with you that matching the update RMS does not make the effective step sizes of different methods equal, and that the learning rate should be tuned for each method. In response, we rerun all language-model comparisons with the learning rate tuned separately:
The full sweep at one seed is below (best value in bold):