- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Spectral-GD replaces a weight-matrix gradient with its polar factor and can be viewed as a full-batch, momentum-free simplification of Muon. The paper analyzes Spectral-GD under the empirical squared loss of a two-layer ReLU network with randomly initialized and fixed second-layer weights, in the neural tangent kernel (NTK) regime. The paper derives a non-asymptotic training guarantee for Spectral-GD to achieve a prescribed training-loss accuracy, with explicit bounds on the required network width, step size, and number of iterations. Experiments on synthetic networks and language models provide additional empirical evidence.
Critical strengths
- Timely and relevant theoretical problem. The paper addresses an important question motivated by the growing use of Muon-family optimizers in LLM training: whether the polar-factor gradient update can provably reduce neural-network training loss despite discarding singular-value magnitude information.
- Careful treatment of ReLU activation changes. The analysis explicitly handles activation-boundary cases and controls the linearization error caused by changing ReLU gates, rather than assuming that the activation patterns remain fixed throughout training. This strengthens the technical treatment of the non-smooth network.
Critical weaknesses
- Gap between the theorem statement and its proof. Theorem 3.1 (Eq. 9) claims that the empirical loss remains below the target threshold for all . However, Step 3 of the proof only establishes that the iterates remain within the neighborhood for , and the descent argument is only applied within this finite horizon. Step 5 therefore appears to establish the guarantee at iteration , rather than for all subsequent iterations. Could the authors clarify how the required neighborhood and kernel conditions are maintained beyond , or otherwise weaken the theorem statement to match the proved result?
- Extremely conservative width requirement. Although the paper improves the sufficient width scaling from to relative to the cited GD analysis, the explicit constant in Appendix D.2 (Eq. 45) appears to make the guarantee quantitatively vacuous. Substituting the parameters reported for the synthetic experiment (, , , , and ) yields a sufficient width of approximately , whereas the experiment uses . The authors should discuss this enormous gap and clarify the practical interpretation of the theoretical width improvement.
Questions for authors
- Unclear role of the spectral transformation. The theoretical analysis establishes that Spectral-GD can achieve low training loss under the stated assumptions. However, it remains unclear which parts of the guarantee specifically depend on the polar-factor transformation, as opposed to more general normalized-gradient updates.
2 comments
Thank you for the detailed response and for tightening the paper’s claims. The revisions make the paper more rigorous and complete, even though they narrow the scope of the theoretical guarantees.
I am keeping my rating at 2, and I lean toward rejection. But since my work focuses primarily on the practical use and stability of optimizers, and my confidence rating of 2 reflects my limited expertise in the theoretical analysis at issue here, I would not argue against it If the AC recommends acceptance.
This year’s four-point scale is too coarse to convey that distinction. Given that I have heard that roughly two-thirds of submissions received a rating of 2, I wanted to state my position explicitly in writing. I would appreciate it if the AC considered this clarification carefully when weighing my review.
We thank the reviewer for their detailed feedback. We present the following responses to the key issues raised. All revisions made in response to the reviewers’ comments are highlighted in blue in the updated manuscript.
Response. Thanks for the spotting the typo in the theorem statement; and it is corrected in the updated uploaded file. Theorem 3.1 now states L(WT)≤ε (Eq. 9), as does Section 1. The iterates stay in B for t≤T (Eqs. 26--27), and Step 5 runs over t⋆≤t<T, where both endpoints lie in B, so Lemma 4.3 gives L(Wt)≤ε on [t⋆,T]. Also a general version of the Theorem 3.1 has been given in Appendix A as Theorem A.3
Response. The width in (7) is a worst-case sufficient condition. The first NTK bounds for GD had similar character (n6/(λ04δ3) in Du et al.), and were later reduced to scale as n4 by Song & Yang (2019) --- which is the same scaling with n as encompassed by our general Theorem A.3 that applies to a class of algorithms.
This philosophy of neural net theory can be traced back to the work since Neal (1996) who had argued qualitatively about large width limits being analytically trainable. Our work can be seen as the latest embodiment of that philosophy in the context of the most modern optimizers.
The width sweep (new Figure 3, m∈{500,…,8000}) evidences the mechanism of the proof, shrinking gate changes and unit movement, already at m in the hundreds.
Response. New Theorem A.3 (Appendix A) gives a generalization of the claimed main theorem 3.1 which is specific to spectral-GD. The general theorem clarifies that three properties of the direction matrix Dt are important, (1) ⟨Gt,Dt⟩≥∥Gt∥F , (2) ∥Dt∥2≤1, and (3) ∥Dt∥F≤ρ. Theorem A.3 shows that every update rule with these properties gets L(WT)≤ε with the same η and T as for Spectral-GD and the second term of the width bound (Eq. 46) ∝ρ2n4; Spectral-GD corresponds to ρ=n.