- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how stochasticity affects Muon’s learning dynamics. In a simplified matrix-factorization setting, full-batch Muon learns singular modes in parallel, while minibatch updates can produce more sequential learning. The authors explain this through noise smoothing the spectral-sign operator. They derive effective continuous dynamics under a structured noise model, characterize implicit regularization for diagonal networks, and present experiments on matrix reconstruction, vision transformers, and language models.
Critical strengths
-
Understanding how stochasticity interacts with gradient normalization is a relevant question for explaining Muon’s behavior.
-
The controlled matrix-factorization experiments demonstrate a qualitative difference between full-batch and actual minibatch updates. Experiments with injected Gaussian and Cauchy noise provide additional comparisons.
Critical weaknesses
-
Rhee, Simon and Karkada (2026), Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics, already describe parallel mode growth and balanced solutions. Dragutinovic et al. (2026) also establish the contrast between sequential learning under GD and parallel learning under spectral descent. Both are cited in this paper. The main contribution therefore rests on the stochastic extension, whose connection to actual training remains limited.
-
The analysis assumes diagonal noise in an aligned basis, with variance proportional to the loss, and uses a large-noise approximation. Actual minibatch gradients have correlations and low-rank structure. The paper does not quantify when this approximation preserves their dynamics or relate its noise parameter precisely to batch size. The diagonal-network implicit-bias result does not resolve this gap for general matrix updates.
-
The spectral theory assumes aligned and balanced factors from initialization. It does not explain how these properties emerge under stochastic training or establish how departures from them affect the proposed mechanism.
-
Lower final effective rank does not really demonstrate sequential feature learning or explain generalization.
Questions for authors
-
In the batch-size-one matrix-factorization experiment, each layer’s gradient is rank one. Does averaging the actual Muon updates give the drift predicted by Theorem 4.2?
-
Appendix E acknowledges that the two-layer smoothed flow lacks the sequential delays observed in Figure 2. Can the theory predict when each mode reaches half its target singular value, using the stated time change?
-
In the transformer experiments, can you hold batch size, learning rate, and number of updates fixed and vary only injected gradient noise? Does this reproduce the lower effective rank observed with smaller batches, and does validation performance improve alongside it?
4 comments
Q1 To test the directional prediction of Theorem 4.2, we conduct the following batch-size sweep. We find that small batches align well with the gradient direction while less with full batch spectral update direction.
We added a deep-linear batch-size sweep to test our noise model under minibatch sampling. At nine frozen checkpoints, we compute the exact full-data gradient matrix for each layer. For each batch size , we apply Muon to the minibatch gradient matrices and average the resulting directions to estimate . We then compute the Frobenius cosine similarity between and , and between and the exact matrix sign of . The expected gradient stays fixed across batch sizes at each checkpoint.
From batch size one to full batch, the median cosine with decreases from to , and the median cosine with its matrix sign increases from to . In the corresponding training runs, the ratio of the last to the first singular-value threshold crossing time falls from a median of to , indicating weaker sequential learning at larger batch sizes.
Thus, small-batch Muon updates align more closely with the gradient direction, supporting the directional prediction of Theorem 4.2 without relying on the idealized noise model.
We additionally extend this probe to frozen Transformer checkpoints, using CIFAR-10 DeiT-Tiny at epochs 10, 100, and 210. At each checkpoint, we vary the probe batch size and average the resulting Muon updates. We measure Frobenius cosine similarity with the full-training-set gradient and its exact matrix-sign direction, using full-batch Muon as a baseline. We also report gradient signal-to-noise ratios and comparisons with averaged Frobenius-normalized minibatch gradients. Small to intermediate batches yield stronger gradient alignment than the full-batch Muon baseline, although this improvement diminishes at the smallest batch sizes. As batch size increases further, the signal-to-noise ratio increases, gradient alignment decreases toward the full-batch baseline, and the average update approaches the full-batch Muon direction. These results complement our controlled experiments by demonstrating batch-size-dependent changes in update geometry at fixed Transformer checkpoints, with the strongest gradient alignment occurring at intermediate noise levels. The experimental details and results across the probed layers are included in the revised appendix.
Q2. Predicting threshold crossing times.
Theorem 4.2 characterizes the effective dynamics within the aligned noise model, but does not quantitatively reproduce the pronounced sequential delays of Figure 2, as already discussed in Appendix E. We therefore do not claim that the current approximation accurately predicts the observed crossing times. Instead, it explains how stochastic smoothing restores magnitude-dependent updates, providing a mechanism for sequential learning.
Q3. Controlled noise injection.
We agree that this would be an interesting additional experiment. However, independently injected isotropic Gaussian noise is not equivalent to minibatch noise, which is data-dependent, anisotropic, and correlated. Consequently, varying only the injected noise scale need not reproduce the effect of changing batch size. Our current experiments establish the batch-size dependence of spectral concentration, while a causal connection to validation performance remains open. On top of that, spectral concentration can be a useful property on its own, namely for low rank compression.
Thanks to the authors for the detailed rebuttal, however some concerns still remain unadressed.
For my second question, the authors clarify that predicting mode-learning times is beyond their theory’s scope, so this remains a limitation.
The new averaged-update experiments support the predicted update direction, but the theoretical noise structure remains too ideal, and its applicability to correlated minibatch noise that mixes singular modes is not established.
So, I would stick with my original rating.
We thank the reviewer for their response and for clarifying their remaining concerns. Our work builds on analyses by Rhee et al. (2026) and Dragutinović et al. (2026), which use tractable models to reveal properties of Muon’s full batch deterministic dynamics. Our contribution explains how stochastic smoothing qualitatively changes these dynamics and restores sequential feature learning. The simplifying assumptions allow us to isolate this mechanism and derive consequences that we subsequently examine beyond the theoretical setting.
Regarding mode-learning times, our theory establishes how noise restores mode-dependent growth. Predicting precise learning times in general networks would require substantially stronger control of the evolving training dynamics. This requirement applies equally to gradient-flow explanations of sequential learning, whose explicit learning-time predictions also depend on simplifying structural assumptions. A general quantitative predictor is therefore a broader challenge in understanding feature-learning dynamics, extending beyond the mechanism studied here.
Our noise model enables a tractable analysis of stochastic smoothing, as in SGD analyses such as [1], which studies isotropic, loss-dependent noise while leaving low-rank structure for future work. Our extension to symmetric heavy-tailed noise and experiments with Gaussian noise, Cauchy noise, and actual minibatch gradients support the mechanism’s robustness beyond the specific noise model used in the main analysis.
We respectfully believe these remaining concerns reflect limitations shared by many theoretical studies, which use simplified settings to isolate and explain a particular phenomenon. More precise predictions and broader guarantees would be valuable extensions, but their absence should be considered in that wider context. We hope the assessment also recognizes what the present work establishes: a new explanation for how noise changes Muon’s learning behavior, supported by experiments that go beyond the assumptions of the analysis. We would be grateful if you could reconsider your rating in light of these clarifications regarding the scope and contribution of our work.
[1] Wojtowytsch’s Stochastic Gradient Descent with Noise of Machine Learning Type, Part II
We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. Below, we elaborate on your concerns in a detailed point-by-point response. In case of any open questions, we would be happy to discuss them.
W1. Novelty and relevance to practical training.
We agree that Dragutinović et al. (2026) and Rhee et al. (2026) establish parallel mode growth under full-batch spectral descent and we now indicate this clearly at the start of the theory section. However, our contribution is to show that minibatch stochasticity can reverse this behavior. Spectral normalization removes singular-value magnitudes from the full-gradient update, but averaging spectrally normalized minibatch gradients can restore sensitivity to gradient magnitude. Figure 2 demonstrates this change with actual minibatch gradients, where stronger modes are learned first.
This distinction is particularly relevant for practical Muon training, which relies on stochastic updates. Therefore, conclusions based solely on deterministic spectral flow need not extend to minibatch training. We further strengthen this connection through the new experiment described in Q1.
W2. Noise structure and approximation.
We agree that our noise model is idealized. However, this simple model captures the qualitative behavior observed with actual minibatch gradients in Figure 2: full-batch updates learn modes in parallel, whereas small-batch updates learn stronger modes first.
Moreover, the new experiment in Q1 directly measures the averaged Muon updates under actual minibatch sampling, without imposing Gaussian or diagonal noise assumptions.
Within the aligned model, we can also extend the smoothing perspective to anisotropic noise by assigning a separate noise scale σi to each mode. This gives a leading-order response proportional to (λi−θi)/σi at fixed loss. While anisotropy can affect the relative learning rates, it does not remove the smoothing mechanism. We will add this observation as a remark (Remark 4.3).
W3. Alignment and balanced initialization.
Alignment and balance play distinct roles. Alignment keeps the modes decoupled under our noise model. With aligned but unbalanced factors, their difference remains conserved, and smoothing still acts mode-wise, although the imbalance can change relative growth rates. Balance alone does not ensure alignment, and the dynamics can mix modes.
These assumptions permit an explicit characterization, but are not imposed in Figure 2, where the transition is observed with random initialization and actual minibatch gradients.
W4. Effective rank and sequential learning.
We agree that lower final effective rank does not by itself establish sequential learning. However, sequential learning provides a mechanism for low rank under early stopping. Stronger modes grow first, leaving an approximately low-rank predictor at a finite training budget.
The Transformer experiments provide complementary evidence of spectral concentration. Figure 9 shows that the batch-size trend also appears in Q⊤K, the bilinear interaction determining attention logits. This extends the observation to a functionally relevant quantity. We do not claim that these measurements establish the order of Transformer feature acquisition or a causal link with generalization.