- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes sMuon, a low-rank optimizer that adapts Muon's full-layer update geometry to LoRA. Rather than orthogonalizing the two factor updates independently, sMuon projects a Muon-inspired update onto the feasible tangent space and exploits low-rank structure to perform the required computations through small matrices. The method also incorporates momentum transport and weight decay. The paper shows experiments results across 4 language models to show competitive performance, with promising efficiency and convergence results.
Critical strengths
-
By reducing the main orthogonalization computations to a small core matrix, the proposed sMuon avoids materializing full-layer gradients and achieves computational complexity that scales favorably with the adapter rank.
-
The ReLoRA experiments demonstrate promising optimization efficiency. sMuon achieves a validation loss comparable to Riemannion and outperforming AdamW, while substantially reducing optimizer overhead.
-
The evaluation covers various models and tasks, including commonsense reasoning, knowledge benchmarks, code generation, and continued pretraining.
Critical weaknesses
-
Some theoretical properties derived under exact arithmetic do not directly carry over to the implemented algorithm. In particular, the regularized pseudo-inverses used for numerical stability do not generally preserve exact reparameterization invariance, and the resulting approximate projections need not retain the idempotence assumed in the analysis. The full-rank assumptions also need more clarification in the presence of zero initialization or degenerate gradients. The paper should distinguish the properties of the idealized formulation from those of the implemented optimizer and discuss the effect of numerical regularization.
-
The empirical advantages are not sufficiently consistent or statistically validated. Although the ReLoRA results are encouraging, the SFT experiments show substantial regressions on several model-task combinations, including code generation on Qwen and DeepSeek and ARC-C on Llama.
-
The experiments do not fully isolate the contribution of the proposed full-layer update geometry. The proposed sMuon and its competitors differ in factor initialization, momentum handling, weight decay, and update scaling, making it difficult to attribute performance differences specifically to the projected Muon direction.
Questions for authors
-
How sensitive is sMuon to numerical regularization and rank deficiency? To what extent do the reparameterization invariance and projection properties established in the analysis hold for the implemented optimizer?
-
How stable are the reported improvements across random seeds and learning rates, particularly for the ReLoRA experiment and the SFT settings showing substantial regressions?