- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper provides a theoretical study of spectral optimization techniques via algorithms such as Muon and related methods. The paper aims to establish descent guarantees for different variants of spectrally regularized gradient updates. In particular, the authors show that a properly calibrated "spectral clipping" procedure attains the optimal descent in their problem formulation. They also provide an efficient approximate implementation of spectral clipping, and study the effect of finite precision and gradient approximation on the descent guarantees.
Critical strengths
The early observations of the paper (Theorem 1 and Corollary 2), comparing calibrated clipping and the polar factor, are interesting.
Also, the authors give an algorithm for implementing spectral clipping in the matrix-vector query model (Lemma 5 and Appendix D), which seems interesting.
Critical weaknesses
The paper becomes progressively more and more difficult to follow as it goes on, with more and more results and notations being introduced.
I found the finite-precision model somewhat confusing and not particularly helpful. I think I would have preferred if the paper spent more time on discussing the implementation of calibrated clipping, instead of relegating it to the appendix (I'm not sure whether this is a significant novelty of the paper, or if similar constructions appeared in prior works).
Not sure what was the point of some of the later results, such as for example why do you introduce the "fractional family" in section 5, if you already have an "optimal" clipping procedure.
It was not immediately clear to me how Section 4 fits with the remainder of the paper. I see the point of obtaining broader optimization guarantees from the rest of the theory, but I just didn't get how it fits together.
Questions for authors
See the weaknesses section.
Dear Reviewers,
As the rebuttal deadline is approaching, please review the authors' responses and participate in the discussion. Kindly indicate whether your concerns have been addressed and update your scores if appropriate.
Thank you for your time and effort!
Best regards,
--AC