acceptodds
Under review as a conference paper at ICLR 2027

DynMuon: A Dynamic Spectral Shaping View of Muon

10 views

Abstract

In recent years, Muon has emerged as the dominant method for training large language models, and transformers more broadly. The essential difference, when compared to standard gradient descent methods, is to replace the usual update matrix with its polar factor . In this work, we consider a class of Muon-like updates, where we replace the update with for some parameter . We call this a "spectral-shaping" operation, and develop an idealized, noise-aware local model of how to pick which depends on (a) local curvature of the loss function, (b) noise stemming from stochastic gradients and label noise, and (c) training stage. Our analysis and experimentation reveal a previously overlooked behavior: positive helps early by emphasizing high-curvature directions and accelerating signal contraction, while mildly negative helps later by reallocating update strength toward low-curvature directions that still contain useful training signals. Building on the insight, we propose DynMuon, an efficient dynamic spectral shaping method that schedules from positive to mildly negative over training. Extensive experiments across model sizes, architectures, and training settings show that DynMuon consistently achieves lower validation loss than Muon, while requiring 10.6–26.5% fewer steps to reach the same target loss.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Matrix-based Optimizers, Dynamic Spectral Shaping, Muon, Language Model Pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2sPQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the spectral transform of Muon should be dynamically adjusted during the training phase. The authors extend Muon to UΣpV⊤U\Sigma^pV^\top and, through a local model considering curvature and gradient noise, demonstrate that in the early stages of training, a positive exponent pp favors convergence in high-curvature directions; in the later stages, a moderately negative exponent is beneficial for optimizing low-curvature directions where residual signals still exist. Based on this observation, the authors propose DynMuon, which adjusts the spectral exponent through a pre-defined logistic schedule and uses a low-order polynomial approximation to avoid expensive SVD. Experiments show that this method outperforms Muon on multiple model scales, reducing the number of training steps by approximately 10.6%–26.5% to achieve the same validation loss.

Critical strengths
  • Valuable insights. The paper doesn't limit itself to finding better fixed spectral transforms, but focuses on the potential changes in optimal spectral preferences during training. The observation of moderately enhancing updates in low-curvature directions later on, in particular, warrants further investigation.

  • The theoretical motivation and experimental observations are consistent. The authors explain the role of the spectral exponent using a local signal-noise model, supported by curvature estimation, noise measurement, and ablation experiments at different training stages.

  • The method is simple and yields practical benefits. Experiments cover different model sizes, training budgets, and architectures, showing consistent performance improvements with low additional computational overhead.

Critical weaknesses

W1: The theory primarily provides a qualitative explanation and has not yet yielded sufficient predictive conclusions. The core analysis relies on assumptions such as the approximate alignment of the gradient spectrum and curvature spectrum, and the residual signal is indirectly estimated through the ratio of gradient to curvature. Existing evidence supports the authors' explanation, but is insufficient to prove that the mechanism can quantitatively determine the optimal spectral exponent and its switching timing. I would like to ask the authors whether, using the curvature, residual signal, and noise statistics measured during training, the optimal p and its switching timing can be directly predicted? In particular, can the phenomenon of the optimal p changing with batch size in Figure 10 be quantitatively explained?

W2: The contribution of spectral scheduling itself has not been sufficiently isolated. Algorithm 1 actually uses a segmented switching between the original momentum update and the Muon update in the positive exponential interval, rather than a continuous spectral transformation. This may also change the update norm. Existing ablation methods have not fully ruled out the benefits from staged update amplitude changes or simple segmented scheduling. Therefore, I have some concerns about whether the benefits come from the redistribution of spectral directions or from a staged update strategy.

W3: The validation of the core mechanism is still insufficient. The primary evidence for the shift of residual signals towards lower curvature comes from single-matrix training with other parameters frozen, while the final method is used for full model pre-training. Furthermore, the performance improvement on larger models lacks sufficient multi-random seed validation.

Questions for authors

Q1: If comparing Muon with update norm matching, and DynMuon with a three-stage p=1→0→−0.25p=1\rightarrow0\rightarrow-0.25 scheduling at the same time point, does DynMuon still have a significant advantage?

Q2: Is the shift of residual signals towards lower curvature a general optimization phenomenon in the later stages? Does this phenomenon also occur in full model training and optimizers such as SGD and AdamW? Additionally, can the authors rule out the bias caused by the dependence of 1/hi21/h_i^2 in the residual signal surrogate quantity on this observation?

If the authors can clarify the aforementioned mechanistic issues in their rebuttal, especially by providing more compelling predictive evidence and ablation experiments, I am willing to reconsider and raise the rating.

Reviewer_BozCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether the spectral transformation used by Muon should change over training. It considers updates of the form UΣpV⊤U\Sigma^{p}V^{\top} and develops an idealised local model relating spectral shaping to curvature-dependent signal contraction and noise amplification. The resulting hypothesis is that positive exponents are useful early, whereas mildly negative exponents can help after useful residual signal becomes relatively more concentrated in flatter directions. DynMuon implements this idea through an initial raw-momentum stage, an intermediate Muon stage, and a bounded polynomial correction in the negative-exponent regime. Language-model pretraining experiments report lower validation loss than Muon across several scales and settings, with 10.6%–26.5% fewer steps to reach a target defined by Muon's loss at 80% of its training schedule and approximately 0.3%–2.5% additional per-step time.

Critical strengths
  1. The stage-dependent hypothesis has useful intervention-based evidence. Beyond plotting spectral statistics, the paper varies the exponent and its ordering and batch size. The observation that larger batches favor more negative exponents is consistent with the proposed noise-amplification trade-off.

  2. The practical gains are consistent across several relevant settings at low reported overhead. Table 1 reports validation-loss improvements across three GPT-style model scales and two training budgets, together with target-loss step savings and per-step timings. The additional architecture, corpus, learning-rate, seed, exact-SVD, and NorMuon comparisons provide useful robustness checks.

Critical weaknesses

My major concern with this paper is:

  • The experiments do not isolate the dynamic spectral shape from changes in overall update magnitude. For a fixed input MM, the exact spectral update has squared Frobenius norm ∑iσi2p\sum_i \sigma_i^{2p}, so changing pp generally changes both its shape and its magnitude. The efficient implementation also retains the scalar factor ∥M∥Fp\lVert M \rVert_F^{p} (Section 3 and Algorithm 2), while Algorithm 1 switches between raw momentum, the Muon update, and the corrected update. The paper does not report an output-norm-matched comparison that separates these effects.
Questions for authors

Clarification and decisive control for update magnitude:

  • Is any additional output normalisation applied in the experimental implementation beyond Algorithms 1–2?
  • Could the authors specify the complete update scaling and report how its Frobenius norm compares with that of Muon across matrices and training stages?
  • Does the advantage persist when the transformed matrix updates of both methods are normalised to the same Frobenius norm before a common learning-rate schedule is applied?
Reviewer_BWFUOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DynMuon, which changes how Muon transforms the singular values of the momentum over the course of training. The authors consider a family of updates that raise the singular values of the momentum to a power. SGD with momentum and Muon are two members of this family, with the powers one and zero. Based on a local analysis of curvature and gradient noise, the paper argues that a positive power is preferred early in training. Later in training, the remaining signal moves to directions of low curvature. A mildly negative power puts more weight on these directions and becomes preferable. DynMuon follows this by decreasing the power from one to a small negative value during training. The experiments mainly cover GPT-style models up to 1.1B parameters trained on up to 20B tokens, and also test another architecture and dataset at a smaller scale. In all these settings, DynMuon reaches a lower validation loss than Muon, and needs fewer steps to reach the same loss.

Critical strengths
  1. The intuition that the preferred transformation of the singular values should change across training stages is natural and interesting. The analysis of curvature and gradient noise gives a simple picture of why this can happen.

  2. The negative power is approximated on top of the Newton-Schulz iteration without computing the exact SVD.

Critical weaknesses
  1. The method adds considerable complexity to Muon, while the necessity of this complexity is not fully established. In practice, DynMuon uses the raw momentum at the start of training, then switches to Muon, and then to a negative power of the momentum’s singular values. When to switch is controlled by a schedule with several new hyperparameters. Compared with this added complexity, the gain over Muon is modest, around 0.01 to 0.02 in validation loss. The gain also becomes smaller as the model size or the number of training tokens grows.

  2. The paper attributes the gain to the change in the spectrum of the update, but it could also come from the change in the size of the update. The size of Muon’s update does not depend on how large the momentum is, since all its singular values are set to one. This is not the case for the raw momentum and the update with a negative power. Therefore, with the same learning rate, the first and last stages of DynMuon also take steps of a different size from Muon. The paper does not report these sizes. The only experiment on this question runs Muon with twice the learning rate for the first 500 steps. This variant performs worse than Muon. However, this only rules out that the first stage acts like a larger step. If the raw momentum in fact takes smaller steps than Muon, the first stage lowers the effective learning rate at the beginning of training, which works like a longer warmup.

  3. The comparison with related work is also limited, which makes it hard to judge whether this introduced complexity is worthwhile. The main text only compares with Muon and AdamW. NorMuon only breifly appears in the appendix, and the paper does not report whether it is a fully tuned setting. The paper positions itself against existing methods that apply a fixed transformation to the singular values. Yet it does not compare with any of them, such as a fixed positive power [1] or the suppression of the dominant singular directions [2]. Other variants of Muon such as AdaMuon [3], Aurora [4], Muon+ [5] and Muon^2 [6] also report gains over Muon without any schedule across training stages. If DynMuon cannot compete with these alternatives, the benefit of its added complexity is unclear.

  4. The schedule of the power is also not mature enough for practical use. It is fixed before training and does not respond to the state of training, and the paper leaves an adaptive choice to future work. The analysis only suggests that the power should be positive early and mildly negative later. How negative the power should be and when to switch are chosen empirically. The paper’s own controlled study shows that the best power for the late stage ranges from -0.1 to -0.5 as the batch size grows. These choices likely interact with each other, but the ablations only vary them one at a time on the smallest model. As each choice can only be evaluated by a full training run, applying the method to a new setting, especially at a larger scale, may require tuning all of them together, which is intractable in practice. Therefore, it is crucial to have a principled tuning strategy that allows tuning on small scales and is guaranteed to work on larger scales, similar to what muP [7] achieves.

[1] Qi, Xianbiao, et al. “Delving into Muon and Beyond: Deep Analysis and Extensions.” arXiv preprint arXiv:2602.04669 (2026).

[2] Huang, Zhendong, et al. “Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy.” arXiv preprint arXiv:2602.11185 (2026).

[3] Si, Chongjie, Debing Zhang, and Wei Shen. “Adamuon: Adaptive muon optimizer.” arXiv preprint arXiv:2507.11005 (2025).

[4] Dewulf, Alec, et al. "Aurora: A Leverage-Aware Spectral Optimizer." arXiv preprint arXiv:2606.27715 (2026).

[5] Zhang, Ruijie, et al. “MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training.” arXiv preprint arXiv:2602.21545 (2026).

[6] Liu, Ziyue, et al. “Muon^2: Boosting Muon via Adaptive Second-Moment Preconditioning.” arXiv preprint arXiv:2604.09967 (2026).

[7] Yang, Ge, et al. "Tuning large neural networks via zero-shot hyperparameter transfer." Advances in Neural Information Processing Systems 34 (2021): 17084-17097.

Questions for authors
  1. How large are the updates of the raw momentum and of the negative power compared with Muon’s update? If the raw momentum takes smaller steps, would Muon with a longer warmup recover part of the gain? This would help separate the effect of the spectrum from that of the step size.

  2. Does the gain remain if the switch happens at a fixed step instead of a fixed fraction of training? For example, the 20B-token run could reuse the switching step of the 10B-token run. If so, the method could be used without fixing the training length in advance. It would be even better if the switch could be triggered by a quantity measured during training, such as how much signal remains in the low-curvature directions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.