- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper asks whether equivalent factorizations that achieve nearly optimal values of a spectral criterion also produce similar matrix updates. The analysis focuses on the non-uniqueness of internal coordinates in matrix factorization. Specifically, for , any invertible matrix gives an equivalent factorization . These factorizations represent the same matrix, but gradient descent on the factors can produce different matrix updates. Therefore, choosing factor coordinates by optimizing spectral criteria does not necessarily lead to similar updates. Building on this, the authors develop a theoretical framework that connects the accuracy of spectral optimization to differences in the resulting matrix updates. The key condition is that the internal directions constrained by the spectral criterion must cover those used by the gradient update. Using a coverage matrix and coefficient, they derive an upper bound on the update error in terms of the spectral objective gap. For the largest-eigenvalue objective, they further characterize when different optimal coordinates lead to the same update. The paper also provides a formula for choosing coordinates that maximize the first-order loss decrease within a fixed region. Toy digit classification experiments support the theoretical predictions.
Overall, I think the paper presents an interesting theoretical idea, but the motivation and practical implications are not sufficiently clear to me. The presentation could also be improved.
Critical strengths
-
The idea of relating spectral accuracy to the consistency of low-rank updates is original.
-
The paper provides explicit conditions and error bounds, and the toy experiments support the predictions of the theory.
Critical weaknesses
-
The motivation is not clear. The authors study guarantees of update consistency without showing what training problems arise when such guarantees are absent. In Abstract and Introduction, the authors unnecessarily emphasizes specific numerical results without a clear explanation of why the underlying question matters for training.
-
The paper is hard to follow. Some figures are difficult to read because the text overlaps with the plots. The writing is sometimes defensive, and the text repeats many unnecessary details. These issues make it hard to focus on the main ideas.
Questions for authors
-
Why are the data reduced to 12 dimensions using PCA? Is there a theoretical reason for this choice? Do the conclusions still hold with the original features or a different number of PCA components?
-
How can the theory guide low-rank optimization in practice? For example, in real low-rank problems (low-rank matrix recovery, LoRA fine-tuning, etc.), what would we do differently based on the theory, and how would that improve optimization?
Thank you for the response. I will keep the current rating.