- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies when Muon's normalization balances the directions under stochastic gradient noise. The central argument is that balancing the noisy gradients does no necessarily balance the patterns that should be learnt. Noise structure (isotropic or not), task coverage (batch size), and momentum all matter in this process.
The main contributions are
- This paper neatly shows the limitation of Muon under isotropic noise assuming low SNR.
- With the associative-memory model, this paper proves an exact two-task batch size threshold for balanced margin growth, and derives the many-task results based on .
- This paper then separates task converage from gradient reliability and predicts the square-root dependence of learning strength on task frequency, with some assumptions.
- This paper further discusses how momentum can help and characterizes momentum's tradeoff between variance reduction and tracking latest gradients.
Validation primarily includes the synthetic and associative-memory experiments testing the mechanisms. An additional 0.12B LM experiment shows lower validation loss with an existing fast-slow momentum method.
Overall, this paper provides a controlled theoretical explanation of stochastic Muon behavior, with plausible broader insights.
Critical strengths
-
Descriptive strength. This paper connects much realism, including noise geometry, task frequency, label noisem and temporal averaging, to explain when Muon's spectral normalization leads to balanced learning and when that benefit disappears. The paper's combination of exact results, associative-memory model analyses, and controlled experiments makes this paper a substantial contribution to understanding Muon beyond the full-batch setting.
-
Prescriptive strength. This paper identifies how batch size and momentum should accommodate task occurrence, gradient reliability, and the speed of gradient evolution. These provide a statsitical rationale for heterogeneous momentum, supported by a preliminary language-model experiment. These results make the theory potentially useful for optimizer design.
Critical weaknesses
- The practical validation does not test those mechanisms. Besides the synthetic experiments, section 6 and Appendix H.8 report only a 0.12B-model comparison of standard Muon against Zhu et al.’s existing fast–slow method: final validation losses are 3.0097 and 2.9888, with one run per method. There are no larger scale experiments on real, practical model training, varying batch size, task frequency, or momentum to test the theoretical predictions, nor component ablations isolating the proposed mechanism. Thus, while the synthetic experiments support the theoretical mechanisms and the pretraining experiment demonstrates an improvement in validation loss, it remains unclear whether the predicted phenomena, such as frequency-dependent learning and the averaging–tracking trade-off, also occur in practical deep models.
These predictions may change in real deep models because shared parameters allow updates to affect unobserved tasks, learned representations evolve during training, gradient strength depends on task difficulty and interactions as well as frequency, and practical Muon uses approximate polar normalization across multiple interacting layers. These differences motivate testing whether the predicted quantitative relationships persist beyond the analyzed settings.
Questions for authors
Clarification requests
- Do you have real training results showing that rare-task learning changes with , as predicted in Sections 4.2–4.3? Similar transitions across different batch sizes and task frequencies would support the analyzed sampling mechanism.
- Section 6 reports lower validation loss with the complete fast–slow method. Can you clarify what evidence connects this improvement to Section 5’s averaging–tracking mechanism? For example, does varying the slow buffer’s memory while keeping the same update structure reveal the predicted benefit from averaging and deterioration from tracking lag?
3 comments
Thank you for the thoughtful response and the new experiments. They help address my questions, and I will maintain my positive score.
The small-Transformer experiment (Fig. 15) is a nice addition: it shows the frequency-limited to balanced transition in a model with shared parameters and learned representations. The slow-memory sweep (Fig. 14) also illustrates the averaging–tracking trade-off clearly.
Two small optional suggestions for the final version. First, it may help to briefly discuss why the Fig. 15 curves do not fully collapse onto Bp_j (for example, whether larger batches seeing more data under the fixed step schedule plays a role). Second, referencing Appendix I from the main text would make these results easier to find.
Thank you for your thoughtful feedback and continued positive assessment. We appreciate both suggestions. In the final version, we will discuss possible reasons why the curves in Fig. 15 do not fully collapse onto , including the potential role of larger batches seeing more data under the fixed-step schedule. We will also add a reference to Appendix I in the main text to make these results easier to find.
We thank the reviewer for the helpful suggestions on further testing the proposed mechanisms. We have added a small-Transformer experiment on frequency-dependent learning and a slow-memory sweep in the toy model to examine the averaging–tracking trade-off.
To study the predicted behavior in a setting closer to natural-language pretraining while retaining control over task frequency, we used a controlled factual-recall task. Specifically, we trained a two-layer, 108k-parameter Transformer end-to-end on 128 facts divided into four frequency groups, and compared Muon with AdamW at B∈{8,32,128,512,2048} using three paired replicates.
Fig. 15 plots each group's margin-growth rate against Bpj, using the same fitting window (steps 64∼256) for all runs. Rare-fact growth is strongly limited at small batch sizes, whereas larger batches lead to approximately balanced growth across groups under Muon. Although the curves do not collapse exactly with Bpj, they show a qualitatively similar transition in a model with learned representations.
To examine the proposed trade-off, we varied the slow momentum coefficient while keeping the fast–slow update structure fixed in the associative-memory toy model (Fig. 14). We kept the fast momentum coefficient, batch size, learning rate, mixing weight, and normalization and projection rules unchanged. We varied only βs over six values from 0.9 to 0.999, using 64 paired sampling trajectories.
Longer memory reduces relative sampling error from 0.2358 to 0.0228. However, the slow branch's directional error, 1−cos(Ds,Dspop), decreases from 0.0905 to 0.0040 at βs=0.995, then rises to 0.1981 at βs=0.999. Here, Dspop is formed by applying the same normalization and projection to the current population gradient. These measurements illustrate how longer memory improves averaging but can impair tracking of the current gradient. They provide evidence for Section 5's mechanism at the level of update directions, rather than a complete explanation of the LM improvement in Section 6.