acceptodds
Under review as a conference paper at ICLR 2027

From Parallel to Sequential: Small-Batch Noise Restores Simplicity Bias of Muon

Abstract

Simplicity bias, which can arise from sequential feature learning during training, has been proposed as a general explanation for the success of deep learning. An important driver of this phenomenon is saddle-to-saddle behavior, in which the gradient flow trajectory remains close to a sequence of saddle manifolds while learning one feature at a time. However, Muon, a state-of-the-art optimizer based on spectral descent, does not exhibit such dynamics in the full-batch setting and instead learns features in parallel. How is it then that Muon is so successful in deep learning settings? We show that stochastic gradient noise fundamentally changes this behavior: small-batch noise restores sequential feature learning, providing a mechanism for simplicity bias that is absent from full-batch Muon. We explain this transition through stochastic smoothing of the spectral descent operator underlying Muon. When gradients are small relative to the noise scale, the expected spectral update becomes approximately proportional to the actual gradient, which restores gradient-flow-like dynamics. We formalize this mechanism for deep linear models by deriving the effective continuous dynamics of noisy spectral descent. Experiments on low-rank matrix reconstruction, vision transformers, and large language models (LLMs) support these findings.

Primary area:
optimization
Keywords:
Muon, simplicity bias, implicit regularization, small batches, noise, saddle to saddle, steepest descent, mirror descent, sign descent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bi4hOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how stochasticity affects Muon’s learning dynamics. In a simplified matrix-factorization setting, full-batch Muon learns singular modes in parallel, while minibatch updates can produce more sequential learning. The authors explain this through noise smoothing the spectral-sign operator. They derive effective continuous dynamics under a structured noise model, characterize implicit regularization for diagonal networks, and present experiments on matrix reconstruction, vision transformers, and language models.

Critical strengths
  1. Understanding how stochasticity interacts with gradient normalization is a relevant question for explaining Muon’s behavior.

  2. The controlled matrix-factorization experiments demonstrate a qualitative difference between full-batch and actual minibatch updates. Experiments with injected Gaussian and Cauchy noise provide additional comparisons.

Critical weaknesses
  1. Rhee, Simon and Karkada (2026), Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics, already describe parallel mode growth and balanced solutions. Dragutinovic et al. (2026) also establish the contrast between sequential learning under GD and parallel learning under spectral descent. Both are cited in this paper. The main contribution therefore rests on the stochastic extension, whose connection to actual training remains limited.

  2. The analysis assumes diagonal noise in an aligned basis, with variance proportional to the loss, and uses a large-noise approximation. Actual minibatch gradients have correlations and low-rank structure. The paper does not quantify when this approximation preserves their dynamics or relate its noise parameter precisely to batch size. The diagonal-network implicit-bias result does not resolve this gap for general matrix updates.

  3. The spectral theory assumes aligned and balanced factors from initialization. It does not explain how these properties emerge under stochastic training or establish how departures from them affect the proposed mechanism.

  4. Lower final effective rank does not really demonstrate sequential feature learning or explain generalization.

Questions for authors
  1. In the batch-size-one matrix-factorization experiment, each layer’s gradient is rank one. Does averaging the actual Muon updates give the drift predicted by Theorem 4.2?

  2. Appendix E acknowledges that the two-layer smoothed flow lacks the sequential delays observed in Figure 2. Can the theory predict when each mode reaches half its target singular value, using the stated time change?

  3. In the transformer experiments, can you hold batch size, learning rate, and number of updates fixed and vary only injected gradient noise? Does this reproduce the lower effective rank observed with smaller batches, and does validation performance improve alongside it?

4 comments

We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. Below, we elaborate on your concerns in a detailed point-by-point response. In case of any open questions, we would be happy to discuss them.

W1. Novelty and relevance to practical training.

We agree that Dragutinović et al. (2026) and Rhee et al. (2026) establish parallel mode growth under full-batch spectral descent and we now indicate this clearly at the start of the theory section. However, our contribution is to show that minibatch stochasticity can reverse this behavior. Spectral normalization removes singular-value magnitudes from the full-gradient update, but averaging spectrally normalized minibatch gradients can restore sensitivity to gradient magnitude. Figure 2 demonstrates this change with actual minibatch gradients, where stronger modes are learned first.

This distinction is particularly relevant for practical Muon training, which relies on stochastic updates. Therefore, conclusions based solely on deterministic spectral flow need not extend to minibatch training. We further strengthen this connection through the new experiment described in Q1.

W2. Noise structure and approximation.

We agree that our noise model is idealized. However, this simple model captures the qualitative behavior observed with actual minibatch gradients in Figure 2: full-batch updates learn modes in parallel, whereas small-batch updates learn stronger modes first.

Moreover, the new experiment in Q1 directly measures the averaged Muon updates under actual minibatch sampling, without imposing Gaussian or diagonal noise assumptions.

Within the aligned model, we can also extend the smoothing perspective to anisotropic noise by assigning a separate noise scale σi\sigma_i to each mode. This gives a leading-order response proportional to (λi−θi)/σi(\lambda_i-\theta_i)/\sigma_i at fixed loss. While anisotropy can affect the relative learning rates, it does not remove the smoothing mechanism. We will add this observation as a remark (Remark 4.3).

W3. Alignment and balanced initialization.

Alignment and balance play distinct roles. Alignment keeps the modes decoupled under our noise model. With aligned but unbalanced factors, their difference remains conserved, and smoothing still acts mode-wise, although the imbalance can change relative growth rates. Balance alone does not ensure alignment, and the dynamics can mix modes.

These assumptions permit an explicit characterization, but are not imposed in Figure 2, where the transition is observed with random initialization and actual minibatch gradients.

W4. Effective rank and sequential learning.

We agree that lower final effective rank does not by itself establish sequential learning. However, sequential learning provides a mechanism for low rank under early stopping. Stronger modes grow first, leaving an approximately low-rank predictor at a finite training budget.

The Transformer experiments provide complementary evidence of spectral concentration. Figure 9 shows that the batch-size trend also appears in Q⊤KQ^\top K, the bilinear interaction determining attention logits. This extends the observation to a functionally relevant quantity. We do not claim that these measurements establish the order of Transformer feature acquisition or a causal link with generalization.

Q1 To test the directional prediction of Theorem 4.2, we conduct the following batch-size sweep. We find that small batches align well with the gradient direction while less with full batch spectral update direction.

We added a deep-linear batch-size sweep to test our noise model under minibatch sampling. At nine frozen checkpoints, we compute the exact full-data gradient matrix GG for each layer. For each batch size BB, we apply Muon to the minibatch gradient matrices and average the resulting directions to estimate UˉB=E[S(GB)]\bar U_B=\mathbb{E}[S(G_B)]. We then compute the Frobenius cosine similarity between UˉB\bar U_B and GG, and between UˉB\bar U_B and the exact matrix sign of GG. The expected gradient stays fixed across batch sizes at each checkpoint.

From batch size one to full batch, the median cosine with GG decreases from 0.9850.985 to 0.8860.886, and the median cosine with its matrix sign increases from 0.6690.669 to 0.7370.737. In the corresponding training runs, the ratio of the last to the first singular-value threshold crossing time falls from a median of 11.2611.26 to 1.751.75, indicating weaker sequential learning at larger batch sizes.

Thus, small-batch Muon updates align more closely with the gradient direction, supporting the directional prediction of Theorem 4.2 without relying on the idealized noise model.

We additionally extend this probe to frozen Transformer checkpoints, using CIFAR-10 DeiT-Tiny at epochs 10, 100, and 210. At each checkpoint, we vary the probe batch size and average the resulting Muon updates. We measure Frobenius cosine similarity with the full-training-set gradient and its exact matrix-sign direction, using full-batch Muon as a baseline. We also report gradient signal-to-noise ratios and comparisons with averaged Frobenius-normalized minibatch gradients. Small to intermediate batches yield stronger gradient alignment than the full-batch Muon baseline, although this improvement diminishes at the smallest batch sizes. As batch size increases further, the signal-to-noise ratio increases, gradient alignment decreases toward the full-batch baseline, and the average update approaches the full-batch Muon direction. These results complement our controlled experiments by demonstrating batch-size-dependent changes in update geometry at fixed Transformer checkpoints, with the strongest gradient alignment occurring at intermediate noise levels. The experimental details and results across the probed layers are included in the revised appendix.

Q2. Predicting threshold crossing times.

Theorem 4.2 characterizes the effective dynamics within the aligned noise model, but does not quantitatively reproduce the pronounced sequential delays of Figure 2, as already discussed in Appendix E. We therefore do not claim that the current approximation accurately predicts the observed crossing times. Instead, it explains how stochastic smoothing restores magnitude-dependent updates, providing a mechanism for sequential learning.

Q3. Controlled noise injection.

We agree that this would be an interesting additional experiment. However, independently injected isotropic Gaussian noise is not equivalent to minibatch noise, which is data-dependent, anisotropic, and correlated. Consequently, varying only the injected noise scale need not reproduce the effect of changing batch size. Our current experiments establish the batch-size dependence of spectral concentration, while a causal connection to validation performance remains open. On top of that, spectral concentration can be a useful property on its own, namely for low rank compression.

Thanks to the authors for the detailed rebuttal, however some concerns still remain unadressed.

For my second question, the authors clarify that predicting mode-learning times is beyond their theory’s scope, so this remains a limitation.

The new averaged-update experiments support the predicted update direction, but the theoretical noise structure remains too ideal, and its applicability to correlated minibatch noise that mixes singular modes is not established.

So, I would stick with my original rating.

We thank the reviewer for their response and for clarifying their remaining concerns. Our work builds on analyses by Rhee et al. (2026) and Dragutinović et al. (2026), which use tractable models to reveal properties of Muon’s full batch deterministic dynamics. Our contribution explains how stochastic smoothing qualitatively changes these dynamics and restores sequential feature learning. The simplifying assumptions allow us to isolate this mechanism and derive consequences that we subsequently examine beyond the theoretical setting.

Regarding mode-learning times, our theory establishes how noise restores mode-dependent growth. Predicting precise learning times in general networks would require substantially stronger control of the evolving training dynamics. This requirement applies equally to gradient-flow explanations of sequential learning, whose explicit learning-time predictions also depend on simplifying structural assumptions. A general quantitative predictor is therefore a broader challenge in understanding feature-learning dynamics, extending beyond the mechanism studied here.

Our noise model enables a tractable analysis of stochastic smoothing, as in SGD analyses such as [1], which studies isotropic, loss-dependent noise while leaving low-rank structure for future work. Our extension to symmetric heavy-tailed noise and experiments with Gaussian noise, Cauchy noise, and actual minibatch gradients support the mechanism’s robustness beyond the specific noise model used in the main analysis.

We respectfully believe these remaining concerns reflect limitations shared by many theoretical studies, which use simplified settings to isolate and explain a particular phenomenon. More precise predictions and broader guarantees would be valuable extensions, but their absence should be considered in that wider context. We hope the assessment also recognizes what the present work establishes: a new explanation for how noise changes Muon’s learning behavior, supported by experiments that go beyond the assumptions of the analysis. We would be grateful if you could reconsider your rating in light of these clarifications regarding the scope and contribution of our work.

[1] Wojtowytsch’s Stochastic Gradient Descent with Noise of Machine Learning Type, Part II

Reviewer_SmMUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper reports that sequential dynamics, a common one in deep matrix factorization in gradient flow, can be observed in spectral flow (the continuous version of spectral gradient descent) with small-batch noises. This contrasts with full-batch spectral flow, where every feature is simultaneously learned. This paper further claims that the stochastic smoothings affect the implicit bias: as the depth increases from 2 to ∞\infty, the smoothed sign descent, which is the spectral descent on diagonal linear networks, exhibits an implicit regularization of type L3/2L_{3/2} to L1L_1. Empirical results support that trajectories under smaller batches favor sparse structure.

Critical strengths
  1. This paper reported another interesting dynamic of gradient orthogonalization in the factorized setting, especially under a small-batch setting. This highlights that the implicit bias of a stochastic optimizer cannot be inferred from its deterministic part.
Critical weaknesses
  1. The practical relevance of the assumptions underlying the explicit dynamics remains unclear. The leading-order dynamics in Theorem 4.2 depend on the gradient signal being small relative to the noise scale. The paper neither establishes how prevalent this regime is in small-batch training nor empirically reports whether the assumption holds throughout training. The results in Section 4 also assume initialization is balanced and aligned with the target singular vectors, yet the experiments use generic initialization; the paper lacks comparisons assessing the effects of these assumptions.

  2. Lower effective rank does not establish sequential learning. The transformer experiments measure effective rank at the end of training, which reflects the spectra of the weight matrices rather than sequential feature learning. Hence, these measurements do not imply that the proposed sequential learning mechanism occurs in these models.

Questions for authors
  1. How prevalent is the assumption that the signal-to-noise ratio is small in the minibatch setting? Can the authors measure these ratios and report whether sequential feature learning still occurs when the gradient signal is not smaller than the noise?

  2. Does the application of Theorem B.5 require nonzero imbalance in every coordinate, that is, ∣λ0,j∣>0|\lambda_{0,j}|>0 for all jj? When λ0,j=0\lambda_{0,j}=0, the potential contains 23∣θj∣3/2\frac23|\theta_j|^{3/2}, which is not twice differentiable at θj=0\theta_j=0. Can the authors distinguish training at zero imbalance from taking the limit as imbalance goes to zero?

  3. How crucial are balancedness and target alignment for the dynamics in Section 4? Which conclusions hold under aligned but unbalanced initialization, and vice versa? Please explain how critical each assumption is for the dynamics and the proof.

7 comments

We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. Below, we elaborate on your concerns in a detailed point-by-point response. In case of any open questions, we would be happy to discuss them.

W1.

Practical relevance of the flow assumptions. Full-batch flow gives a deterministic baseline for Muon. Figure 2 shows that its prediction of parallel mode growth changes when Muon uses actual minibatch gradients on the same factorization problem. Stronger modes grow first even from small random initialization. Theorem 4.2 explains how this can happen. Averaging after spectral normalization can restore sensitivity to mode strength, so predictions about learning order in minibatch Muon need to account for gradient noise. This contrasts with the gradient flow setting where small initialization is sufficient to induce sequential learning.

Q1. Signal-to-noise ratio.

The leading-order approximation in Theorem 4.2 requires a small mode-wise ratio ∣ri∣|r_i|, where ri=(λi−θi)/(σL)r_i=(\lambda_i-\theta_i)/(\sigma\sqrt{\mathcal L}). The exact response in the Gaussian model is erf⁡(ri/2)\operatorname{erf}(r_i/\sqrt{2}). It remains sensitive to the residual at ratios of order one and gradually saturates toward the deterministic sign as ∣ri∣|r_i| grows. We added a deep-linear batch-size sweep to test our noise model under minibatch sampling in Appendix F. At nine frozen checkpoints, we compute the exact full-data gradient matrix GG for each layer. For each batch size BB, we apply Muon to the minibatch gradient matrices and average the resulting directions to estimate UˉB=E[S(GB)]\bar U_B=\mathbb{E}[S(G_B)]. We then compute the Frobenius cosine similarity between UˉB\bar U_B and GG, and between UˉB\bar U_B and the exact matrix sign of GG. The expected gradient stays fixed across batch sizes at each checkpoint.

First, we report the alignment, from batch size one to full batch, the median cosine with GG decreases from 0.9850.985 to 0.8860.886, and the median cosine with its matrix sign increases from 0.6690.669 to 0.7370.737. We also report the gradient signal-to-noise ratio ∥G∥F/E∥GB−G∥F2\|G\|_F/\sqrt{\mathbb{E}\|G_B-G\|_F^2}, whose median across frozen checkpoints and both layers rises from 0.300.30 at batch size 11 to 13.4513.45 at batch size 10001000. Moreover, when the signal to noise ratio increases, the ratio of the last to the first singular-value threshold crossing time falls from a median of 11.2611.26 to 1.751.75, indicating weaker sequential learning at larger batch sizes. Second, we extend this probe to frozen Transformer checkpoints, using CIFAR-10 DeiT-Tiny at epochs 10, 100, and 210. At each checkpoint, we vary the probe batch size and average the resulting Muon updates. We measure Frobenius cosine similarity with the full-training-set gradient and its exact matrix-sign direction, using full-batch Muon as a baseline. We also report gradient signal-to-noise ratios and comparisons with averaged Frobenius-normalized minibatch gradients.

Small to intermediate batches yield stronger gradient alignment than the full-batch Muon baseline, although this improvement diminishes at the smallest batch sizes. As batch size increases further, the signal-to-noise ratio increases, gradient alignment decreases toward the full-batch baseline, and the average update approaches the full-batch Muon direction. These results complement our controlled experiments by demonstrating batch-size-dependent changes in update geometry at fixed Transformer checkpoints, with the strongest gradient alignment occurring at intermediate noise levels. The experimental details and results across the probed layers are included in the revised appendix.

Moreover, at the batch size used during training (B=128B=128), we observe a gradient SNR below one, for example in the first attention block’s QKV matrix at epoch 100. This supports the practical relevance of studying training dynamics in the low-SNR regime.

Q2. Zero imbalance.

Yes. Our application of Theorem B.5 requires δi=∣ui,0∣−∣vi,0∣≠0\delta_i=|u_{i,0}|-|v_{i,0}|\ne 0 in every coordinate. The corresponding mirror Hessian entry is (4∣θi∣+δi2)−1/2(4|\theta_i|+\delta_i^2)^{-1/2}, which is singular at θi=0\theta_i=0 when δi=0\delta_i=0. Remark 5.2 takes the limit of the potentials as imbalance vanishes, giving an L3/2L_{3/2}-type objective when θ0=0\theta_0=0. At exact balance, a coordinate initialized with both factors zero stays fixed because both factor gradients vanish. The distinction between vanishing initialization and training from exact zero also appears in diagonal linear networks trained with gradient flow [1]. In contrast to gradient flow, at exact balance, smoothed sign descent can drive an initially nonzero coordinate to zero in finite time, where the mirror potential fails the smoothness assumptions of Theorem B.5.

Q3. Alignment and balance.

Alignment keeps the updates diagonal in the target basis under our noise model. In the aligned two-layer setting with positive factors, both factors receive the same signed update in each mode, so their difference δi\delta_i is conserved. The resulting mobility is 4∣θi∣+δi2\sqrt{4|\theta_i|+\delta_i^2}, up to a common time scaling. Smoothing therefore still acts mode by mode with unequal factor magnitudes, and the imbalance can change the relative growth rates.

With balance alone, the factors’ singular directions can differ from the target’s, and the updates can mix modes. The proof in Appendix C uses the shared target basis to reduce the matrix dynamics to scalar equations. This is also a standard starting point for studying sequential learning under gradient flow [2].

W2. Sequential learning and effective rank. Sequential learning can lead to an approximately low-rank predictor through early stopping. Once the leading modes have grown, there is a period in which the weaker modes are still small. A fixed epoch or token budget can end during this period, leaving a predictor dominated by the modes learned first.

We agree that the Transformer results characterize the learned spectra at the end of training. Figure 9 shows that smaller batches also yield lower effective rank in Q⊤KQ^\top K, the bilinear interaction that determines the attention logits. This shows spectral concentration in a functionally relevant part of the model. The order in which Transformer features are learned remains open.

[1] Woodworth et al. Kernel and Rich Regimes in Overparametrized Models. ICML, 2020.

[2] Saxe, Andrew M. et al. “Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.” arXiv: Neural and Evolutionary Computing (2013): n. pag.

Thank you for the clarifications.

My remaining concern is how often individual modes have a low signal-to-noise ratio (SNR) in practice. I could not find the deep-linear batch-size sweep in Appendix F of the revised manuscript. Based on the description in the rebuttal, the gradient SNR result does not verify the mode-wise SNR condition underlying the leading-order approximation in Theorem 4.2. Reporting mode-wise ratios in the deep linear setting would address this concern.

I appreciate the authors’ acknowledgment that the final effective ranks of Transformer weights do not establish the order in which features are learned; that question remains open.

The responses on Q2 and Q3 address my questions. Nonetheless, the fact that the initial imbalance must be nonzero in every coordinate should be in the revision, as it is required to apply Theorem B.5.

Reviewer_cXKEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper revisits sequential feature learning and its connection to simplicity bias in deep learning, where simpler or dominant features are learned first and intermediate predictors may remain low-rank or sparse before more complex features emerge. Full-batch Muon behaves differently in the studied setting: its spectral normalization removes singular-value magnitude information, causing singular modes to grow more nearly in parallel. The paper shows that small-batch gradient noise smooths the spectral sign operator, approximately restoring dependence on gradient magnitude and producing more gradient-flow-like sequential dynamics.

Critical strengths
  1. The paper addresses an interesting tension between Muon’s empirical success and the proposed role of sequential feature learning in producing simplicity bias. The observation is well motivated, and the theoretical mechanism is presented clearly.
  2. The paper presents diverse empirical evaluations, ranging from controlled matrix factorization to vision and language Transformers, together with ablations comparing mini-batch noise and explicitly injected noise. The empirical analyses support the paper’s general hypothesis, particularly in the controlled linear settings.
Critical weaknesses
  1. Muon’s strongest empirical adoption has been in LLM pretraining, while its benefits are less established across other architectures and deep-learning tasks. Connecting Muon to simplicity bias therefore requires more than showing that noise can recover sequential feature learning. It is also necessary to demonstrate that this simplicity bias is important in realistic LLM pretraining and contributes to generalization. The existing experiments show a correlation between batch size and effective rank but do not establish this causal connection.
  2. Practical Muon training rarely approaches the full-batch regime considered by the theory. In LLM pretraining, models are trained on enormous token streams, and sequential feature acquisition may also be influenced by feature prevalence, with frequent patterns receiving earlier and more consistent gradient signals than rare patterns. This mechanism can coexist with stochastic smoothing, but the paper does not disentangle or rule it out as an alternative explanation.
  3. The main theoretical results assume diagonal, isotropic noise aligned with the target singular-vector basis. This noise model is substantially more idealized than real mini-batch gradient noise, which is generally low-rank, anisotropic, and capable of changing the singular-vector directions.
Questions for authors
  1. Could the authors consider the practical-relevance and feature-prevalence perspectives above and provide stronger evidence connecting the proposed theoretical mechanism to practical gains?
  2. Would the same batch-size trends appear with SGD or AdamW? Such comparisons would help determine whether the observed effective-rank behavior is specific to Muon’s spectral normalization or is a general consequence of small-batch training.
  3. Why should summed parameter effective rank be interpreted as simplicity bias, and how strongly does it correlate with function-level complexity or generalization?
7 comments

We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. Below, we elaborate on your concerns in a detailed point-by-point response. In case of any open questions, we would be happy to discuss them.

W1.

Sequential learning provides a mechanism for low-rank bias under early stopping. Once the leading modes have grown, there can be a period during which weaker modes remain small. A fixed epoch or token budget ending within this interval yields an approximately low-rank predictor.

Besides a potential mechanism for improved generalization as demonstrated in the sparse recovery example in the regression setting, low rank can be a desired property in its own right for downstream applications. For instance, it could enable better low rank projections to reduce the model size. We will explicitly mention this in the experimental section to not put all weight on generalization performance with the following sentence:

Beyond its observed correlation with improved generalization, low rank is also a desirable property in its own right, as it can facilitate model compression and computational efficiency.

The probes in W2 further show that minibatch averaging can increase alignment between Muon’s update and the full gradient.

W2.

We agree that frequent patterns can produce stronger and more consistent gradient signals during LLM pretraining. This gives a concrete reason why the mechanism we study matters. Feature prevalence can shape both the mean gradient and its minibatch variability. Our question is how Muon translates those gradients into updates. We highlight this connection in the revised manuscript as follows:

Even when pretraining LLMs uses a long token stream, each update sees only a single batch. Common patterns can have a larger effect on the mean gradient ∇L\nabla L, and the mix of patterns varies from batch to batch, which then contributes to the noise ξ\xi. The full-batch limit gives us a baseline: the spectral sign acts directly on the mean gradient and gives all its nonzero singular modes the same weight. With minibatches, the spectral sign acts on each batch gradient separately. As we show below, this can give stronger modes more weight in the average update.

Moreover, we added a deep-linear batch-size sweep to test our noise model under minibatch sampling in Appendix F. At nine frozen checkpoints, we compute the exact full-data gradient matrix GG for each layer. For each batch size BB, we apply Muon to the minibatch gradient matrices and average the resulting directions to estimate UˉB=E[S(GB)]\bar U_B=\mathbb{E}[S(G_B)]. We then compute the Frobenius cosine similarity between UˉB\bar U_B and GG, and between UˉB\bar U_B and the exact matrix sign of GG. The expected gradient stays fixed across batch sizes at each checkpoint.

First we report the alignment, from batch size one to full batch, the median cosine with GG decreases from 0.9850.985 to 0.8860.886, and the median cosine with its matrix sign increases from 0.6690.669 to 0.7370.737. We also report the gradient signal-to-noise ratio ∥G∥F/E∥GB−G∥F2\|G\|_F/\sqrt{\mathbb{E}\|G_B-G\|_F^2}, whose median across frozen checkpoints and both layers rises from 0.300.30 at batch size 11 to 13.4513.45 at batch size 10001000. Moreover, when the signal to noise ratio increases, the ratio of the last to the first singular-value threshold crossing time falls from a median of 11.2611.26 to 1.751.75, indicating weaker sequential learning at larger batch sizes. We additionally extend this probe to frozen Transformer checkpoints, using CIFAR-10 DeiT-Tiny at epochs 10, 100, and 210. At each checkpoint, we vary the probe batch size and average the resulting Muon updates. We measure Frobenius cosine similarity with the full-training-set gradient and its exact matrix-sign direction, using full-batch Muon as a baseline. We also report gradient signal-to-noise ratios and comparisons with averaged Frobenius-normalized minibatch gradients.

Small to intermediate batches yield stronger gradient alignment than the full-batch Muon baseline, although this improvement diminishes at the smallest batch sizes. As batch size increases further, the signal-to-noise ratio increases, gradient alignment decreases toward the full-batch baseline, and the average update approaches the full-batch Muon direction. These results complement our controlled experiments by demonstrating batch-size-dependent changes in update geometry at fixed Transformer checkpoints, with the strongest gradient alignment occurring at intermediate noise levels. The experimental details and results across the probed layers are included in the revised appendix.

W3.

First we would like to highlight the contrast with previous theoretical analysis on full-batch by Dragutinović et al. (2026) and Rhee et al. (2026), which explain why spectral descent can learn modes in parallel. However we reveal that this can be reversed by minibatch training. Spectral normalization removes singular-value magnitudes from the full-gradient update, but averaging spectrally normalized minibatch gradients can restore sensitivity to the gradient magnitude. Figure 2 shows this change with actual minibatch gradients, where stronger modes are learned first.

We do agree that our noise model is idealized. However, this simple model does capture the qualitative behavior observed with actual minibatch gradients in Figure 2. Full-batch updates learn modes in parallel, whereas small-batch updates learn stronger modes first. We use additive Gaussian noise as a standard starting point, with a loss-dependent scale that reflects how gradient noise decreases near interpolation in least squares as has been used in [1,2] to study SGD.

Within the aligned noise model, the same smoothing perspective extends to mode-dependent variances σi2>0\sigma_i^2>0 in the shared singular basis with each mode having leading-order response proportional to ((\lambda_i-\theta_i)/\sigma_i) at fixed loss. This would not change the smoothening perspective, however, these scales can potentially affect the order in which modes are learned. We will add this as a remark near the noise modelling:

In our noise model we have assumed that the variance is the same along each singular value direction characterized by σ2\sigma^2. The same smoothening perspective can be directly extended to anisotropic noise modeled by replacing σ2\sigma^2 with a vector diag(σ12,...,σd2)\text{diag}(\sigma^2_1,..., \sigma_d^2).

[1] Wojtowytsch, Stephan. “Stochastic Gradient Descent with Noise of Machine Learning Type Part II: Continuous Time Analysis.” Journal of Nonlinear Science 34 (2021): 1-45.

[2] Pesme, Scott et al. “Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity.” ArXiv abs/2106.09524 (2021): n. pag.

Questions

Q1. See the answers to W1 and W2.

Q2. Yes. For SGD, stochasticity can strengthen implicit regularization, as established for diagonal linear networks in [2]. However there is a key theoretical distinction here with Muon, as noted by Reviewers SmMU and bi4h. The noise here is needed to restore the simplicity bias. In contrast, gradient flow with small initialization would also exhibit sequential learning while spectral flow does not, which is quantified in Theorem 4.1.

Q3. Summed parameter effective rank measures how concentrated the spectra of the weight matrices are. Figure 9 shows that the trend also appears in Q⊤KQ^\top K, the bilinear interaction that determines the attention logits. Its effective rank increases by roughly $ 42%$ across the tested batch-size range in DeiT-Small. This provides evidence of spectral concentration in a functionally relevant part of the model. Figure 5 reports validation accuracy alongside effective rank.

C1: The early-stopping explanation clarifies how sequential learning can temporarily produce low-rank predictors. However, low parameter effective rank does not directly establish practical compressibility, since the paper does not evaluate whether low-rank projections preserve accuracy or produce actual memory and computational savings.

C2: The authors’ response clarifies that feature prevalence and minibatch noise can coexist, and the DeiT experiment improves the practical relevance of the analysis. Reproducing the batch-size-dependent dynamics in autoregressive language-model pretraining would better support the claimed applicability to LLMs.

C3: The anisotropic extension still assumes that mode-dependent noise lies in the shared target singular-vector basis. The paper does not demonstrate that real minibatch gradient noise is aligned with this basis or that off-diagonal correlations are negligible.

C4: Measuring the effective rank of the attention-logit bilinear interaction is more functionally relevant than summed parameter rank, and reporting validation accuracy strengthens the observed correlation. The causal relationship between rank dynamics and generalization is not established, while this is a common weakness for many deep learning studies.

C5: The response theoretically distinguishes Muon from gradient flow and cites stochastic implicit regularization for SGD, but it does not answer whether the same batch-size trends empirically occur with SGD or AdamW. Can batch-size sweep experiments with SGD, AdamW, and Frobenius-normalized SGD be provided to isolate which rank dynamics are specific to Muon?

Comments on the paper

1 comment

Dear Area Chair,

Earlier deterministic spectral-flow analyses by Rhee et al. (2026) and Dragutinović et al. (2026) establish parallel mode growth. We show how stochastic smoothing restores magnitude-dependent updates and sequential feature learning under minibatch noise, with the mechanism persisting under heavy-tailed noise, including infinite-variance Cauchy noise. We also characterize the diagonal dynamics’ implicit bias through a mirror potential that remains convex at arbitrary depth in contrast to the gradient flow equivalent, and show how sequential learning coexists with faster initial mode growth and finite-time convergence under the analyzed assumptions.

During the rebuttal, we clarified the initialization assumptions, the nonzero-imbalance requirement for the implicit-bias result, and the early-stopping connection to low rank. We also added averaged-update probes with actual minibatch gradients in deep linear networks and Transformers, together with AdamW comparisons. These additions were positively acknowledged: Reviewer SmMU raised their rating to weak acceptance, finding their concern largely addressed. Reviewer cXKE acknowledged that many concerns were addressed. Reviewer bi4h recognized that the new probes support the predicted update direction.

The remaining concerns largely request broader applicability to correlated, mode-mixing noise, precise predictions of learning times, or causal explanations of downstream performance. These are valuable extensions, but reflect limitations common to theoretical analyses that use tractable models to isolate learning mechanisms. Existing gradient-flow and deterministic Muon analyses likewise rely on simplifying assumptions. We believe the contribution should be assessed on the new mechanism and consequences it establishes, supported by experiments beyond the theoretical setting.

We hope this summary clarifies our contributions, the points addressed during the rebuttal, and the scope of the remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.