- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper designs a new algorithm for neural network optimization. The main idea is to replace the Frobenious norm constraint in the factored NGD update with the spectral norm constraint. Based on this desing choice the authors develop FISMO algorithm, analyze its convergence and perform preliminary numerical experiments.
Critical strengths
The main strengths are useful closed-form update rule in Theorem 4.2 and promising preliminary experiments.
Critical weaknesses
The proofs in this paper contain many mathematical mistakes. The most severe mistakes are the following:
-
Theorem 4.1 is incorrect: function is not convex in P for fixed Q and not convex in Q for fixed P. The proof of this statement is missing, and a simple 1-d counterexample is available: $ J(p,q) = (f + \mu)/(pq) + \ln p + \ln q. $ Choosing and implies .
-
The proof of Lemma F.8 is incorrect: the inequality (65) is plugged into (64) with an opposite sign. Similarly, the proof of Lemma F.9 is incorrect: the inequality on lines 1649-1650 has wrong direction.
-
The proof of Lemma F.10 is incorrect. In particular, the expectation on lines 1671-1673 can be non-zero: it is not justified that is an unbiased estimator of , since , , and may be correlated. This issue further propagatesn to Lemma F.13, Theorem F.14 and Theorem 5.8. The relation on lines 1652-1656 may be also incorrect even in the noiseless setting, since is an EMA of and is just the current transformed gradient.
The errors above invalidate the key steps in the convergence analysis of Algorithm 1. Consequently, the main convergence results are not established by the proofs in the paper.
Besides, the main design choice of this paper is replacing frobenius norm in the factored NGD (line 192) with the spectral norm. However, while the original Frobenius norm constraint follows from Fisher geometry, the choice of spectral norm here is not adequately justified. In particular, the justification on lines 194-200 is weak: Frobenius norm already bounds the "largest directional amplification", and there is no explanation of why the "updates that are less dominated by large singular directions" are preferable in this setting. In contrast, for Muon, there are many theoretical results showing, that orthogonalization can be beneficial under different structural assumptions.
Questions for authors
n/a