acceptodds
Under review as a conference paper at ICLR 2027

When and How Does Muon Precondition? The Role of Structured Gradient Noise

Abstract

Muon has become a popular optimizer for large language model training, yet its theoretical foundations remain poorly understood. In this work, we study how gradient noise geometry shapes Muon’s preconditioning benefits. We first show, theoretically and empirically, that under the classical isotropic noise assumption, Muon offers no acceleration beyond a rescaling of the learning rate. This motivates us to investigate structured noise arising from mini-batch sampling in associative memory with heterogeneous task frequencies. We establish an exact balancing threshold for two tasks and extend the analysis to multiple tasks, revealing how task coverage and gradient reliability jointly shape learning. Our analysis shows that Muon can soften the dependence on task frequency from linear to square-root, helping balance the learning of frequent and rare tasks. We further identify rotation-induced negative transfer in SignSGD that Muon avoids, and explain how momentum reduces sampling noise while tracking changing gradients. Controlled numerical experiments validate these mechanisms, and a preliminary LLM pretraining study illustrates the practical promise of fast–slow momentum.

Primary area:
learning theory
Keywords:
Muon, stochastic optimization, spectral normalization, gradient noise, associative memory, momentum

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_v3hEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies when Muon's normalization balances the directions under stochastic gradient noise. The central argument is that balancing the noisy gradients does no necessarily balance the patterns that should be learnt. Noise structure (isotropic or not), task coverage (batch size), and momentum all matter in this process.

The main contributions are

  • This paper neatly shows the limitation of Muon under isotropic noise assuming low SNR.
  • With the associative-memory model, this paper proves an exact two-task batch size threshold for balanced margin growth, and derives the many-task results based on BpjBp_j.
  • This paper then separates task converage from gradient reliability and predicts the square-root dependence of learning strength on task frequency, with some assumptions.
  • This paper further discusses how momentum can help and characterizes momentum's tradeoff between variance reduction and tracking latest gradients.

Validation primarily includes the synthetic and associative-memory experiments testing the mechanisms. An additional 0.12B LM experiment shows lower validation loss with an existing fast-slow momentum method.

Overall, this paper provides a controlled theoretical explanation of stochastic Muon behavior, with plausible broader insights.

Critical strengths
  • Descriptive strength. This paper connects much realism, including noise geometry, task frequency, label noisem and temporal averaging, to explain when Muon's spectral normalization leads to balanced learning and when that benefit disappears. The paper's combination of exact results, associative-memory model analyses, and controlled experiments makes this paper a substantial contribution to understanding Muon beyond the full-batch setting.

  • Prescriptive strength. This paper identifies how batch size and momentum should accommodate task occurrence, gradient reliability, and the speed of gradient evolution. These provide a statsitical rationale for heterogeneous momentum, supported by a preliminary language-model experiment. These results make the theory potentially useful for optimizer design.

Critical weaknesses
  • The practical validation does not test those mechanisms. Besides the synthetic experiments, section 6 and Appendix H.8 report only a 0.12B-model comparison of standard Muon against Zhu et al.’s existing fast–slow method: final validation losses are 3.0097 and 2.9888, with one run per method. There are no larger scale experiments on real, practical model training, varying batch size, task frequency, or momentum to test the theoretical predictions, nor component ablations isolating the proposed mechanism. Thus, while the synthetic experiments support the theoretical mechanisms and the pretraining experiment demonstrates an improvement in validation loss, it remains unclear whether the predicted phenomena, such as frequency-dependent learning and the averaging–tracking trade-off, also occur in practical deep models.

These predictions may change in real deep models because shared parameters allow updates to affect unobserved tasks, learned representations evolve during training, gradient strength depends on task difficulty and interactions as well as frequency, and practical Muon uses approximate polar normalization across multiple interacting layers. These differences motivate testing whether the predicted quantitative relationships persist beyond the analyzed settings.

Questions for authors

Clarification requests

  • Do you have real training results showing that rare-task learning changes with BpjBp_j, as predicted in Sections 4.2–4.3? Similar transitions across different batch sizes and task frequencies would support the analyzed sampling mechanism.
  • Section 6 reports lower validation loss with the complete fast–slow method. Can you clarify what evidence connects this improvement to Section 5’s averaging–tracking mechanism? For example, does varying the slow buffer’s memory while keeping the same update structure reveal the predicted benefit from averaging and deterioration from tracking lag?
3 comments

We thank the reviewer for the helpful suggestions on further testing the proposed mechanisms. We have added a small-Transformer experiment on frequency-dependent learning and a slow-memory sweep in the toy model to examine the averaging–tracking trade-off.

[W1-1&Q1] Do you have real training results showing that rare-task learning changes with BpjBp_j?

To study the predicted behavior in a setting closer to natural-language pretraining while retaining control over task frequency, we used a controlled factual-recall task. Specifically, we trained a two-layer, 108k108\text k-parameter Transformer end-to-end on 128128 facts divided into four frequency groups, and compared Muon with AdamW at B∈{8,32,128,512,2048}B\in\{8,32,128,512,2048\} using three paired replicates.

Fig. 15 plots each group's margin-growth rate against BpjBp_j, using the same fitting window (steps 64∼25664\sim256) for all runs. Rare-fact growth is strongly limited at small batch sizes, whereas larger batches lead to approximately balanced growth across groups under Muon. Although the curves do not collapse exactly with BpjBp_j, they show a qualitatively similar transition in a model with learned representations.

[W1-2&Q2] Can you clarify what evidence connects Section 6's improvement to Section 5’s averaging–tracking mechanism?

To examine the proposed trade-off, we varied the slow momentum coefficient while keeping the fast–slow update structure fixed in the associative-memory toy model (Fig. 14). We kept the fast momentum coefficient, batch size, learning rate, mixing weight, and normalization and projection rules unchanged. We varied only βs\beta_s over six values from 0.90.9 to 0.9990.999, using 6464 paired sampling trajectories.

Longer memory reduces relative sampling error from 0.23580.2358 to 0.02280.0228. However, the slow branch's directional error, 1−cos⁡(Ds,Dspop)1-\cos(D_s,D_s^{\mathrm{pop}}), decreases from 0.09050.0905 to 0.00400.0040 at βs=0.995\beta_s=0.995, then rises to 0.19810.1981 at βs=0.999\beta_s=0.999. Here, DspopD_s^{\mathrm{pop}} is formed by applying the same normalization and projection to the current population gradient. These measurements illustrate how longer memory improves averaging but can impair tracking of the current gradient. They provide evidence for Section 5's mechanism at the level of update directions, rather than a complete explanation of the LM improvement in Section 6.

Thank you for the thoughtful response and the new experiments. They help address my questions, and I will maintain my positive score.

The small-Transformer experiment (Fig. 15) is a nice addition: it shows the frequency-limited to balanced transition in a model with shared parameters and learned representations. The slow-memory sweep (Fig. 14) also illustrates the averaging–tracking trade-off clearly.

Two small optional suggestions for the final version. First, it may help to briefly discuss why the Fig. 15 curves do not fully collapse onto Bp_j (for example, whether larger batches seeing more data under the fixed step schedule plays a role). Second, referencing Appendix I from the main text would make these results easier to find.

Thank you for your thoughtful feedback and continued positive assessment. We appreciate both suggestions. In the final version, we will discuss possible reasons why the curves in Fig. 15 do not fully collapse onto BpjBp_j, including the potential role of larger batches seeing more data under the fixed-step schedule. We will also add a reference to Appendix I in the main text to make these results easier to find.

Reviewer_vgPaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when Muon provides preconditioning benefits under stochastic gradient noise. The authors first show that, under isotropic Gaussian noise and low signal-to-noise ratio, Muon's mean update behaves like SGD with a rescaled learning rate. They then use an associative-memory model to study the effects of batch size, task frequency, and label noise. The paper also analyzes momentum and compares Muon with SignSGD. The theoretical findings are supported by synthetic experiments and a preliminary LLM pretraining experiment.

Critical strengths

The paper addresses an interesting question about Muon's behavior under stochastic gradients. The exact two-task balancing threshold is a useful theoretical result. I also appreciate the analysis of how task frequency, label noise, and momentum affect learning. The paper is generally well organized, and the synthetic experiments provide good support for the theoretical findings.

Critical weaknesses

1. Limited generality of the theoretical analysis. The theoretical results rely on simplified associative-memory models with orthogonal embeddings. The square-root frequency scaling also depends on additional covariance and task-frequency approximations. It is unclear whether these conclusions hold in more realistic neural networks with complex gradient correlations.

2. Limited empirical validation. Most experiments are conducted on synthetic tasks that closely match the theoretical setting. The LLM experiment uses only a 0.12B model with one run per method. There are no multi-seed experiments or component ablations. This makes it difficult to assess how robust the observed improvements are.

3. Unclear novelty compared with prior work. Several related studies have already explored Muon's behavior in associative-memory learning. The fast–slow momentum method is also adopted from previous work. While the stochastic sampling analysis provides useful insights, the paper could better explain how its main theoretical contributions differ from existing results.

Questions for authors
  1. Does the square-root frequency scaling still hold under more general anisotropic gradient noise or non-orthogonal embeddings? Can the authors provide additional evidence beyond the current settings?

  2. Can the authors provide multi-seed results or ablation studies for the LLM experiment? It would help clarify whether the improvement is consistent and which components contribute most to the gain.

  3. Could the authors clarify the key differences between their theoretical results and prior work on Muon in associative-memory learning? In particular, which findings go beyond the existing full-batch analyses?

3 comments

We thank the reviewer for the constructive feedback. We provide additional empirical evidence and clarify our theoretical contributions below.

[W1&Q1] Does the square-root frequency scaling still hold under more general anisotropic gradient noise or non-orthogonal embeddings?

We added an experiment with K=6K=6 tasks in which both input and output embeddings are independently sampled Gaussian vectors in d=1536d=1536 dimensions. We retain the full categorical label-noise model and keep the task frequencies and optimization settings unchanged. Results are averaged over 88 independent pairs of input and output embedding matrices.

Fig. 13 shows that Muon's local contraction coefficient retains a near-square-root dependence on task frequency, while SGD exhibits approximately linear scaling. This suggests that the square-root frequency scaling is not restricted to exactly orthogonal representations.

[W2&Q2] Can the authors provide multi-seed results or ablation studies for the LLM experiment?

We acknowledge that the LLM experiment uses only one run per method and therefore does not establish the robustness of the observed validation-loss improvement across random seeds. However, we clarify that the fast–slow momentum method evaluated in Section 6 is adopted from Zhu et al. and is not a new optimizer proposed in this work. The LLM experiment is intended as a preliminary illustration of the practical relevance of our analysis, rather than a comprehensive evaluation of the optimizer.

Separately, to provide more direct evidence for the theoretical mechanism underlying the fast–slow design, we conducted a controlled slow-memory sweep in the toy model (Fig. 14). Using 6464 paired sampling trajectories for each setting, we find that increasing the slow-memory length consistently reduces sampling error, while excessively long memory increases directional tracking error. This directly supports the averaging–tracking trade-off analyzed in Section 5.

[W3&Q3] Could the authors clarify the key differences between their theoretical results and prior work on Muon in associative-memory learning?

The associative-memory analyses of Wang et al. and Li et al. apply spectral normalization to population gradients. We instead ask when the resulting balancing behavior persists under finite-batch sampling.

In the two-task setting, Theorem 4.1 gives an exact batch-size threshold: below it, the margins grow at different linear rates; above it, their gap has uniformly bounded exponential moments. For multiple tasks, we identify local sampling regimes governed by BpjBp_j.

We also distinguish task coverage from gradient reliability under label noise: even a frequently observed task can have an unreliable gradient near the population optimum. These results characterize when population-gradient balancing survives stochastic sampling and when it breaks down.

Reviewer_vgPaOpenReview ↗

Follow-up after author response

Thank you for the clarification and additional experiments. The non-orthogonal embedding tests, controlled Transformer experiment, and slow-memory sweep strengthen the evidence, while the revised discussion clarifies the finite-batch contribution relative to prior work. Generalization to large-scale training remains open, but this does not negate the scoped theoretical results. I retain my original weak-acceptance rating.

Thank you for your thoughtful assessment and continued support. We are glad that the clarifications and additional experiments have addressed your concerns and strengthened the evidence for our results. We agree that generalization to large-scale training remains an open question, and we appreciate your recognition of our theoretical contributions within the scope of this work.

Reviewer_veveOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether Muon's balanced learning across frequent and rare tasks, previously established in the full-batch setting, survives mini-batch stochasticity. In a linear associative-memory model with heterogeneous task frequencies, the authors show that this benefit is governed by the expected per-batch task count BpjBp_j, and derive an exact balancing threshold Bbal≈log⁡2/pmin⁡B_{\text{bal}} \approx \log 2/p_{\min} in the two-task case. The analysis is further extended to the multi-task setting, label noise, a comparison with SignSGD, and momentum, and is complemented by synthetic simulations and a preliminary LM pretraining run.

Critical strengths

The theoretical analysis is rigorous and well scoped. The exact finite-batch transition (Theorem 4.1) identifies a simple and intuitive mechanism: Muon's update depends on whether a rare task is present in the batch rather than on its count. This mechanism consistently explains the subsequent multi-task and label-noise results. The authors also clearly distinguish exact results from approximations.

Critical weaknesses

The paper lacks an experiment that connects its central theoretical prediction to practical training. The synthetic experiments simulate the same idealized model being analyzed, and therefore verify the derivations rather than the practical relevance of the mechanism. The pretraining experiment, which the authors acknowledge is preliminary, evaluates an existing method with a single seed and is not designed to test the paper's own predictions. A targeted experiment would substantially strengthen the claimed practical implications, for example by examining whether rare tokens or facts benefit from Muon as a function of BpjBp_j when batch size varies.

Questions for authors
  1. Could the authors add a small-scale experiment in a realistic LLM setting that directly tests the central prediction? For example, training a small Transformer on a long-tail factual-recall task, or bucketing tokens by frequency during LM pretraining, and checking whether the gap between Muon and AdamW on rare buckets changes with batch size as predicted by BpjBp_j and BbalB_{\text{bal}}.
  2. I would appreciate a clearer statement of the core contribution. Several parts (the associative-memory setup, full-batch balancing, label noise, and the batch-size dependence of Muon's preconditioning) overlap with [1], [2], and [3]. Could the paper be restructured to foreground the new finite-batch, occurrence-based results, and to state more explicitly what is new relative to these works?

[1] Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Cunxiao Du, Chao Du, Tianyu Pang, Zhuoran Yang, Mingyi Hong, and Vincent Y. F. Tan. Muon outperforms Adam in tail-end associative memory learning. In International Conference on Learning Representations (ICLR), 2026. arXiv:2509.26030.

[2] Binghui Li, Kaifei Wang, Han Zhong, Pinyan Lu, and Liwei Wang. Muon in associative memory learning: Training dynamics and scaling laws. In International Conference on Machine Learning (ICML), 2026. arXiv:2602.05725.

[3] Elliot Paquette, Noah Marshall, Lucas Benigni, Guangyuan Wang, Atish Agarwala, and Courtney Paquette. Phases of Muon: When Muon eclipses SignSGD. arXiv:2605.09552, 2026.

3 comments

We thank the reviewer for the helpful feedback and the suggestion of a targeted factual-recall experiment. Following this suggestion, we have added a small-Transformer experiment to examine how task frequency and batch size affect learning, and clarified what is new in our finite-batch analysis.

[W1&Q1] Could the authors add a small-scale experiment in a realistic LLM setting that directly tests the central prediction?

We trained a two-layer, 108k108\text k-parameter Transformer on 128128 facts split equally into four frequency groups, with per-fact probabilities 1/60,1/120,1/240,1/4801/60,1/120,1/240,1/480. All parameters are trained. We compare Muon and AdamW at B∈{8,32,128,512,2048}B\in\{8,32,128,512,2048\} using three paired replicates and 512512 updates per run. Within each optimizer, the learning-rate schedule is the same across batch sizes. Larger batches therefore process more training examples.

Fig. 15 plots each group's margin-growth rate, fitted over steps 64∼25664\sim256, against BpjBp_j. At small batch sizes, rare-fact growth is strongly limited; with larger batches, Muon produces approximately balanced growth across frequency groups. For example, at B=32B=32, the most and least frequent groups grow at rates of 0.012640.01264 and 0.006180.00618, respectively. At B=128B=128, all four rates lie between 0.012920.01292 and 0.013730.01373. The curves do not fully collapse with BpjBp_j, so we view the results as support for the qualitative sampling-related transition rather than an exact scaling law.

[Q2] I would appreciate a clearer statement of the core contribution.

The associative-memory analyses of Wang et al. and Li et al. apply spectral normalization to population gradients. We instead ask when the resulting balancing behavior persists under finite-batch sampling. In the two-task setting, Theorem 4.1 gives an exact batch-size threshold: below it, the margins grow at different linear rates; above it, their gap has uniformly bounded exponential moments. For multiple tasks, we identify local sampling regimes governed by BpjBp_j. These results characterize when population-gradient balancing survives stochastic sampling and when it breaks down.

Paquette et al. already study stochastic mini-batch preconditioning in Gaussian matrix least squares. Our focus is different: discrete task occurrence in softmax associative memory and label fluctuations that persist near the population optimum. The expected task count BpjBp_j and expected corruption count BpjραBp_j\rho_\alpha define distinct sampling scales, and frequent observation alone does not guarantee a reliable gradient. The new results therefore concern the occurrence-driven balancing transition and the distinction between coverage and reliability, not batch-size dependence in itself.

Thank you for the response and the additional experiments. The experiment and the clarification address my questions well. I would suggest restructuring the paper to present the new finite-batch results and stating the contributions relative to prior work in the main body. Thank you again for the excellent work. I maintain my positive assessment.

Thank you for your thoughtful feedback and positive assessment. We have revised the main text to incorporate the new finite-batch results and clarify our contributions relative to prior work. We appreciate your constructive suggestions and continued support.

Reviewer_ke73OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how structured gradient noise affects Muon's preconditioning behavior. The authors first show that under low-SNR isotropic noise, Muon's expected update is approximately equivalent to SGD with a rescaled learning rate. They then use an associative memory model to study how mini-batch sampling and task frequency influence Muon's ability to balance learning across frequent and rare tasks. The paper also analyzes the effects of label noise, compares Muon with SignSGD in terms of negative transfer, and discusses the trade-off between noise reduction and tracking delay introduced by momentum. The theoretical analysis is supported by synthetic experiments. The authors also evaluate an existing fast–slow momentum variant on a 0.12B language model, showing a modest improvement in validation loss over standard Muon.

Critical strengths
  • Theoretical insights. The paper clarifies when and why Muon's preconditioning benefits emerge or disappear under stochastic gradient noise.
  • Practical implications. The analysis of task balancing and momentum provides useful insights for understanding Muon and designing better optimizers.
Critical weaknesses
  • Limited generalizability of the theoretical results. The main findings rely on simplified associative memory models with orthogonal task representations. The exact balancing threshold only applies to two tasks, while the square-root frequency scaling relies on additional approximations. It remains unclear whether these conclusions hold for realistic Transformer training with correlated gradients and evolving representations.

  • Weak connection between theory and practical validation. Most experiments are conducted in synthetic settings closely aligned with the theoretical assumptions. The LLM experiment only compares standard Muon with an existing fast–slow momentum method on a 0.12B model, without directly validating the predicted task-balancing behavior or gradient noise mechanisms. The lack of multiple seeds and ablation studies further weakens the empirical evidence.

  • Limited clarity on the novelty over prior work. Previous studies, particularly Wang et al. (2026) and Li et al. (2026), have already investigated Muon's task balancing, heterogeneous task frequencies, and label noise in associative memory. Although this paper provides additional theoretical results, the significance of these advances over existing work could be more clearly established.

Questions for authors
  1. Generalizability: Can the authors provide theoretical arguments or empirical evidence showing whether the task-balancing results hold with non-orthogonal task representations?

  2. Theory–practice connection: Can the authors provide evidence from actual LLM training that supports the predicted relationship between gradient noise structure, task frequency, and Muon's optimization behavior?

  3. Novelty: Could the authors clarify the key theoretical advances over Wang et al. (2026) and Li et al. (2026), particularly regarding task balancing and heterogeneous task frequencies?

  4. Experimental reliability: Can the authors provide multi-seed results or ablations to establish the robustness of the reported fast–slow momentum improvements?

3 comments

We thank the reviewer for the constructive feedback. We provide additional empirical evidence and clarify our theoretical contributions below.

[W1&Q1] Generalizability

We added an experiment with K=24K=24 tasks in which both input and output embeddings are independently sampled Gaussian vectors in d=1536d=1536 dimensions. We keep the task frequencies and optimization settings unchanged. Results are averaged over 88 independent draws of the input and output embedding matrices, with 6464 sampling trajectories per draw and batch size.

Fig. 12 shows the same qualitative behavior as in the orthogonal setting: growth is frequency-limited at small BpjBp_j and becomes approximately balanced across frequency groups as BpjBp_j increases. This shows that the phenomenon is not confined to exactly orthogonal representations.

[W2-1&Q2] Theory–practice connection

To study the predicted behavior in a setting closer to natural-language pretraining while retaining control over task frequency, we used a controlled factual-recall task. Specifically, we trained a two-layer, 108k108\text k-parameter Transformer end-to-end on 128128 facts divided into four frequency groups, and compared Muon with AdamW at B∈{8,32,128,512,2048}B\in\{8,32,128,512,2048\} using three paired replicates.

Fig. 15 plots each group's margin-growth rate against BpjBp_j, using the same fitting window (steps 64∼25664\sim256) for all runs. Rare-fact growth is strongly limited at small batch sizes, whereas larger batches lead to approximately balanced growth across groups under Muon. Although the curves do not collapse exactly with BpjBp_j, they show a qualitatively similar transition in a model with learned representations.

[W3&Q3] Novelty

The associative-memory analyses of Wang et al. and Li et al. apply spectral normalization to population gradients. We instead ask when the resulting balancing behavior persists under finite-batch sampling.

In the two-task setting, Theorem 4.1 gives an exact batch-size threshold: below it, the margins grow at different linear rates; above it, their gap has uniformly bounded exponential moments. For multiple tasks, we identify local sampling regimes governed by BpjBp_j.

We also distinguish task coverage from gradient reliability under label noise: even a frequently observed task can have an unreliable gradient near the population optimum. These results characterize when population-gradient balancing survives stochastic sampling and when it breaks down.

[W2-2&Q4] Experimental reliability

We acknowledge that the LLM experiment uses only one run per method and therefore does not establish the robustness of the observed validation-loss improvement across random seeds. However, we clarify that the fast–slow momentum method evaluated in Section 6 is adopted from Zhu et al. and is not a new optimizer proposed in this work. The LLM experiment is intended as a preliminary illustration of the practical relevance of our analysis, rather than a comprehensive evaluation of the optimizer.

Separately, to provide more direct evidence for the theoretical mechanism underlying the fast–slow design, we conducted a controlled slow-memory sweep in the toy model (Fig. 14). Using 6464 paired sampling trajectories for each setting, we find that increasing the slow-memory length consistently reduces sampling error, while excessively long memory increases directional tracking error. This directly supports the averaging–tracking trade-off analyzed in Section 5.

Thanks for your response. I also read the other reviews. I think this paper is novel and recommend acceptance.

Thank you for your positive assessment and recommendation for acceptance. We appreciate your recognition of the novelty of our work and your time and effort in reviewing our paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank all reviewers for their valuable feedback. We have added several experiments and further clarified the contributions and scope of our theoretical analysis.

1. Main contributions. This work investigates how stochastic gradient noise affects the preconditioning behavior of Muon, with particular attention to task frequency, finite-batch sampling, label noise, and momentum. Our analysis identifies when Muon's balancing behavior persists under stochastic sampling and when it breaks down. In particular, we establish an exact balancing threshold in the two-task setting, extend the analysis to multiple tasks, and distinguish task coverage from gradient reliability under label noise. We also characterize the averaging–tracking trade-off introduced by momentum.

2. Additional empirical evidence. We have added four experiments to examine the generality and practical implications of our theoretical findings: (i) task balancing with non-orthogonal embeddings (Fig. 12); (ii) square-root frequency scaling with non-orthogonal embeddings (Fig. 13); (iii) a controlled slow-memory sweep examining the averaging–tracking trade-off (Fig. 14); and (iv) frequency-dependent learning in a small Transformer trained on a controlled factual-recall task (Fig. 15).

3. Relationship between theory and experiments. These experiments provide complementary evidence for different aspects of our analysis. The non-orthogonal embedding experiments examine whether the predicted frequency-dependent phenomena persist beyond orthogonal representations. The slow-memory sweep directly examines the averaging–tracking mechanism, while the Transformer experiment investigates whether the sampling-related transition remains observable with shared parameters and learned representations. The existing LLM experiment in Section 6 provides an additional illustration of the practical relevance of our analysis using the fast–slow momentum method of Zhu et al.

In summary, our theoretical analysis characterizes the effects of stochastic sampling, task frequency, label noise, and momentum on Muon's preconditioning behavior, while the additional experiments provide complementary empirical evidence for the generality of the predicted phenomena and their relevance beyond the idealized theoretical setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.