- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies optimizer mismatch when Muon is used to fine-tune AdamW-pretrained language models. It proposes Spectral Momentum Steering (SMS), which uses the pretrained left singular basis to restrict updates and reduce plasticity along directions with large singular values. The update is derived from an anisotropic spectral-norm constraint and retains Muon's Newton-Schulz implementation. Experiments on Qwen3-4B-Base and Llama3.1-8B-Base show substantial improvements in capability preservation with comparable or better math adaptation. Additional experiments examine code adaptation, full-weight fine-tuning, and controls for update magnitude and spectral ordering.
Critical strengths
- The preservation gains are substantial and practically relevant. Llama also benefits, particularly in knowledge retrieval and reading comprehension. These results support SMS as a promising way to use Muon with existing AdamW-pretrained checkpoints.
- The method has a clear optimization formulation. Deriving the update from an anisotropic spectral-norm constraint makes the role of directional plasticity explicit. SMS retains Muon's Newton-Schulz implementation and avoids requiring calibration data for its construction. This provides an interpretable optimizer-level approach that can be applied to both LoRA and full-weight fine-tuning.
- The ablations address important alternative explanations. Appendix B.1 separates subspace restriction from spectral steering, B.2 compares approximately matched update magnitudes, and B.3 reverses the plasticity ordering. Together, these controls support a contribution from update geometry. The code experiment in B.5 also provides useful evidence that the benefits extend beyond math adaptation.
Critical weaknesses
- The mechanistic interpretation remains partly a hypothesis. In Section 3.1, output variance already contains a squared singular-value factor, so high variance alone does not identify knowledge-critical directions. Figure 6 measures changes in sorted singular values rather than rotation of the principal output subspace. The inverse-profile control supports the spectral ordering, but more direct representation measurements would strengthen the claim about preserving optimizer-specific foundational geometry.
- The comparisons leave some uncertainty about the best achievable preservation-adaptation trade-off. The Muon learning-rate sweep and matched-magnitude control in Appendices B.1-B.2 are reassuring. However, the forgetting-aware baselines are evaluated only with Muon. Their poor performance therefore does not establish superiority over their standard AdamW implementations. A comparison at similar adaptation accuracy with one representative baseline would strengthen the broader claim.
- The average preservation score can obscure substantial forgetting in individual capabilities. For Llama in Table 1, SMS achieves Avg1 of 101.30, although knowledge retrieval and reading comprehension retain only 61.42% and 49.42% of their base scores. Improvements in instruction following compensate for these losses in the average. The authors acknowledge ratios above 100%, but the overall preservation claims should more clearly distinguish these improvements from actual retention.
Questions for authors
- Can the authors report principal left-singular-subspace overlap or activation drift for a few representative layers, or qualify the mechanistic interpretation more explicitly? This would help distinguish spectral preservation from preservation of knowledge-critical representations.
- Can the authors summarize preservation at comparable adaptation accuracy and clarify whether the baseline ranking holds with standard AdamW implementations?