acceptodds
Under review as a conference paper at ICLR 2027

Musec: MomentUm SpEctral Clipping for Stable Muon-type Training

Abstract

Muon has emerged as a highly effective optimizer for large language model training, often achieving superior convergence and performance compared with the widely adopted Adam and AdamW optimizers. Nevertheless, Muon is prone to training instability due to its spectral flattening, manifested by loss spikes and unbounded growth of model weights. Existing approaches primarily rely on weight or attention-logit clipping, which require architecture-specific modifications and do not directly address instability across all model components. We propose MomentUm SpEctral Clipping (Musec), which replaces Muon's spectral flattening with spectral clipping: rather than setting all singular values of the momentum matrix to approximately one, Musec clips singular values that exceed a threshold while preserving the underlying spectral structure of the momentum. Our strategy provides an optimizer-level, architecture-agnostic mechanism for stabilizing Muon training. Theoretically, we establish convergence guarantees for Musec in nonconvex nonsmooth stochastic optimization. Practically, we develop Soft Musec, an efficient implementation that uses a smooth spectral saturation function approximated by coupled Newton–Schulz iterations. Empirically, Soft Musec consistently improves training stability over existing Muon variants across a wide range of learning rates and model sizes, remaining stable in settings where existing Muon variants diverge while matching their performance under well-tuned configurations.

Primary area:
optimization
Keywords:
Muon, nonsmooth analysis, nonconvex optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9gnyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new optimizer, Musec, which replaces Muon’s spectral flattening with singular-value clipping. Musec feeds the clipped momentum back into the exponential moving-average buffer, thereby spectrally bounding the momentum state. The exact hard-clipping update is proven to converge under the stated assumptions. Soft Musec is proposed as a practical variant that uses smooth spectral saturation and coupled Newton–Schulz iterations to avoid explicit SVD computation. Experiments on 491M–1.63B NanoGPT models show improved stability over Muon at larger learning rates, although the improvement is less significant compared with the closely related SPECTRA method.

Critical strengths
  1. The paper demonstrates Muon’s instability and proposes a well-motivated approach to address it. The presentation is clear and complete.
  2. It provides a clear optimization interpretation and a convergence analysis for exact hard Musec.
  3. It presents extensive experiments across three datasets, multiple model sizes, learning-rate and weight-decay sweeps, and ablation studies. The reported computational overhead of Soft Musec is small.
Critical weaknesses
  1. The theory covers exact hard clipping, while the experiments use smooth clipping. No theoretical connection or approximation analysis between the two is provided.
  2. The tuning budget is large. Weight decay is tuned for every method and learning rate, while Musec introduces an additional clipping-threshold hyperparameter. This may require expensive hyperparameter tuning and limit transferability to large-scale training or different architectures.
  3. The incremental practical benefit of Soft Musec over the concurrent SPECTRA method is not sufficiently discussed or clearly isolated. The reported improvement over SPECTRA appears limited.
Questions for authors
  1. Can the authors provide an approximation or convergence analysis for Soft Musec with smooth saturation?
  2. Can the authors isolate the benefit of clipped-state feedback by comparing Soft Musec and independently tuned SPECTRA? Since SPECTRA retains an unclipped momentum state while achieving similar performance, in which settings is clipped-state feedback necessary?
  3. Could the authors report the spectral norms of the update matrices throughout training? Figure 9 reports effective rank, but update spectral norms would provide complementary information about the actual update magnitudes.
  4. How should hyperparameters such as parameter-group-specific weight decay and the clipping threshold be tuned and transferred in practical large-scale pretraining scenarios?
Reviewer_VcTCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Musec, which replaces Muon's spectral flattening with momentum spectral clipping to improve training stability. An efficient variant, Soft Musec, implements smooth spectral clipping using coupled Newton–Schulz iterations. The authors establish convergence guarantees for Musec in nonconvex nonsmooth optimization and demonstrate improved stability across different model sizes, datasets, and learning rates with negligible computational overhead.

Critical strengths

1. Theoretical contribution. The paper provides a principled interpretation of spectral clipping and establishes convergence guarantees for Musec in nonconvex nonsmooth optimization.

2. Comprehensive experiments. The paper evaluates multiple optimizers across different model sizes, datasets, and learning rates, with additional analyses of weight spectral norms and training dynamics.

Critical weaknesses

1. Questionable motivation. In practical large-scale LLM training, Muon is generally observed to be more stable than AdamW for standard matrix parameters [1]. The paper does not convincingly establish that Muon's instability is a significant issue under realistic, well-tuned training settings.

2. Potential numerical instability. Computing matrix inverse square roots via Newton–Schulz iterations can be less numerically stable than Muon's NS orthogonalization. The paper lacks a direct numerical stability comparison between these approaches.

[1] On the design of qwen3. 8-next architecture: Evaluation, efficiency, and training stability. https://arxiv.org/pdf/2608.30320

Questions for authors
  1. Is there evidence that Muon consistently exhibits the reported instability in realistic, well-tuned large-scale LLM pretraining?

  2. How does the numerical stability of the coupled Newton–Schulz inverse-square-root computation compare with standard Muon's Newton–Schulz orthogonalization? In particular, how sensitive is Soft Musec to matrix conditioning, clipping thresholds, matrix dimensions?

Reviewer_Y28DOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Musec, which replaces Muon’s spectral flattening with spectral clipping of the momentum matrix. Its principal distinction from SPECTRA applied to momentum SGD is that the clipped momentum is stored and fed back into the moving-average recurrence, rather than used only to compute the parameter update. The paper establishes a constrained quadratic characterization and a Goldstein-stationarity guarantee for the exact hard-clipping algorithm under Lipschitzness, weak convexity, and stochastic-oracle assumptions.

The practical implementation, Soft Musec, uses a smooth spectral saturation function approximated by coupled Newton–Schulz iterations. Experiments on decoder-only language models from 491M to 1.63B parameters show better stability than Muon and MuonClip at large nominal learning rates, while validation losses, stability, and runtime are broadly comparable to SPECTRA.

Critical strengths

The paper addresses a practically relevant training-stability problem with a simple optimizer-level intervention. The experiments cover several datasets and model sizes, include SPECTRA as a relevant baseline, and report loss trajectories, weight norms, update effective rank, and runtime. The measured overhead is small. These are useful aspects of the evaluation, although they do not establish a substantial contribution beyond SPECTRA.

Critical weaknesses

The main distinction from SPECTRA is not shown to matter. Spectral clipping and the smooth saturation operator already appear in SPECTRA. Musec's distinctive clipped-state feedback is motivated by reduced persistence of historical spectral components, but the paper does not directly measure momentum recovery after anomalous gradients, directional distortion, or their effect on subsequent loss. Effective-rank curves do not establish this mechanism. Figures 1 and 8 show broadly comparable losses and stability, without repeated-run uncertainty to resolve small differences. The hard-clipping theorem also does not establish an advantage over SPECTRA or directly cover the implemented soft, finite-iteration recurrence.

The model scale and training budgets are too limited to establish broader training-stability benefits. The experiments use one model family, ranging from 491M to 1.63B parameters, for 1,390–8,040 steps. Actual token totals are not reported; final batch sizes imply approximate upper bounds of 0.55B, 2.48B, and 8.43B tokens, with lower totals after accounting for batch ramp-up. These runs do not establish an advantage during larger or substantially longer pretraining. Moreover, clipping changes update magnitude, so nominal-learning-rate comparisons cannot isolate spectral shape from a smaller effective step size. The current evidence is insufficient to establish a consequential improvement over SPECTRA.

Questions for authors

With the soft spectral operator and other settings fixed, does clipped-state feedback improve recovery after anomalous gradients, directional accuracy, and subsequent loss? Please compare feedback on and off across repeated runs and include controls matching applied update magnitudes.

Does an advantage over SPECTRA persist on larger models or longer training runs under comparable tuning and compute? Please report actual token counts, batch schedules, seed counts, and all divergence events.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.