- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies when adaptive spectral updates can be interpreted as optimization under a fixed convex geometry. It gives conditions under which these updates correspond to a convex potential or norm-derived direction, and shows that some monotone Newton-Schulz style updates still fail this property. The paper supports the theory with proofs, counterexamples, approximation results, and experiments on spectra collected from model training.
Critical strengths
-
The main theoretical result is strong and interesting. The paper shows that preserving singular value order is not enough for a spectral update to correspond to a fixed convex geometry. In particular, deeper Newton-Schulz style updates can fail this property even when they are monotone. This seems like a useful distinction for understanding spectral optimizers.
-
The theory is developed carefully. The paper gives both positive and negative results, counterexamples, and also makes clear that failure of exact integrability does not automatically imply worse training.
Critical weaknesses
-
My main concern is that the practical meaning of the result is still unclear. The paper shows a geometric difference, but does not show that this leads to a meaningful difference in optimization or model performance.
-
The empirical validation is fairly limited. Much of it uses stored spectra from Tiny-Shakespeare rather than full training runs using the different spectral updates.
Questions for authors
-
Could the authors clarify more directly what part of the main result is new relative to the earlier integrability results discussed in the paper?
-
Is there evidence that the nonintegrability shown here leads to a measurable difference during actual model training?
-
Could the different spectral updates be compared in full training runs rather than only on stored spectra?