- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a new optimizer, Musec, which replaces Muon’s spectral flattening with singular-value clipping. Musec feeds the clipped momentum back into the exponential moving-average buffer, thereby spectrally bounding the momentum state. The exact hard-clipping update is proven to converge under the stated assumptions. Soft Musec is proposed as a practical variant that uses smooth spectral saturation and coupled Newton–Schulz iterations to avoid explicit SVD computation. Experiments on 491M–1.63B NanoGPT models show improved stability over Muon at larger learning rates, although the improvement is less significant compared with the closely related SPECTRA method.
Critical strengths
- The paper demonstrates Muon’s instability and proposes a well-motivated approach to address it. The presentation is clear and complete.
- It provides a clear optimization interpretation and a convergence analysis for exact hard Musec.
- It presents extensive experiments across three datasets, multiple model sizes, learning-rate and weight-decay sweeps, and ablation studies. The reported computational overhead of Soft Musec is small.
Critical weaknesses
- The theory covers exact hard clipping, while the experiments use smooth clipping. No theoretical connection or approximation analysis between the two is provided.
- The tuning budget is large. Weight decay is tuned for every method and learning rate, while Musec introduces an additional clipping-threshold hyperparameter. This may require expensive hyperparameter tuning and limit transferability to large-scale training or different architectures.
- The incremental practical benefit of Soft Musec over the concurrent SPECTRA method is not sufficiently discussed or clearly isolated. The reported improvement over SPECTRA appears limited.
Questions for authors
- Can the authors provide an approximation or convergence analysis for Soft Musec with smooth saturation?
- Can the authors isolate the benefit of clipped-state feedback by comparing Soft Musec and independently tuned SPECTRA? Since SPECTRA retains an unclipped momentum state while achieving similar performance, in which settings is clipped-state feedback necessary?
- Could the authors report the spectral norms of the update matrices throughout training? Figure 9 reports effective rank, but update spectral norms would provide complementary information about the actual update magnitudes.
- How should hyperparameters such as parameter-group-specific weight decay and the clipping threshold be tuned and transferred in practical large-scale pretraining scenarios?