- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether the spectral transform of Muon should be dynamically adjusted during the training phase. The authors extend Muon to and, through a local model considering curvature and gradient noise, demonstrate that in the early stages of training, a positive exponent favors convergence in high-curvature directions; in the later stages, a moderately negative exponent is beneficial for optimizing low-curvature directions where residual signals still exist. Based on this observation, the authors propose DynMuon, which adjusts the spectral exponent through a pre-defined logistic schedule and uses a low-order polynomial approximation to avoid expensive SVD. Experiments show that this method outperforms Muon on multiple model scales, reducing the number of training steps by approximately 10.6%–26.5% to achieve the same validation loss.
Critical strengths
-
Valuable insights. The paper doesn't limit itself to finding better fixed spectral transforms, but focuses on the potential changes in optimal spectral preferences during training. The observation of moderately enhancing updates in low-curvature directions later on, in particular, warrants further investigation.
-
The theoretical motivation and experimental observations are consistent. The authors explain the role of the spectral exponent using a local signal-noise model, supported by curvature estimation, noise measurement, and ablation experiments at different training stages.
-
The method is simple and yields practical benefits. Experiments cover different model sizes, training budgets, and architectures, showing consistent performance improvements with low additional computational overhead.
Critical weaknesses
W1: The theory primarily provides a qualitative explanation and has not yet yielded sufficient predictive conclusions. The core analysis relies on assumptions such as the approximate alignment of the gradient spectrum and curvature spectrum, and the residual signal is indirectly estimated through the ratio of gradient to curvature. Existing evidence supports the authors' explanation, but is insufficient to prove that the mechanism can quantitatively determine the optimal spectral exponent and its switching timing. I would like to ask the authors whether, using the curvature, residual signal, and noise statistics measured during training, the optimal p and its switching timing can be directly predicted? In particular, can the phenomenon of the optimal p changing with batch size in Figure 10 be quantitatively explained?
W2: The contribution of spectral scheduling itself has not been sufficiently isolated. Algorithm 1 actually uses a segmented switching between the original momentum update and the Muon update in the positive exponential interval, rather than a continuous spectral transformation. This may also change the update norm. Existing ablation methods have not fully ruled out the benefits from staged update amplitude changes or simple segmented scheduling. Therefore, I have some concerns about whether the benefits come from the redistribution of spectral directions or from a staged update strategy.
W3: The validation of the core mechanism is still insufficient. The primary evidence for the shift of residual signals towards lower curvature comes from single-matrix training with other parameters frozen, while the final method is used for full model pre-training. Furthermore, the performance improvement on larger models lacks sufficient multi-random seed validation.
Questions for authors
Q1: If comparing Muon with update norm matching, and DynMuon with a three-stage scheduling at the same time point, does DynMuon still have a significant advantage?
Q2: Is the shift of residual signals towards lower curvature a general optimization phenomenon in the later stages? Does this phenomenon also occur in full model training and optimizers such as SGD and AdamW? Additionally, can the authors rule out the bias caused by the dependence of in the residual signal surrogate quantity on this observation?
If the authors can clarify the aforementioned mechanistic issues in their rebuttal, especially by providing more compelling predictive evidence and ablation experiments, I am willing to reconsider and raise the rating.