- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper derives the second-order tangential defect of QR retraction on the Stiefel manifold and constructs examples showing that this defect is non-conservative, obstructing a first-order modified-loss interpretation. It also analyzes permutation-equivariance breakdown, validates early trajectory differences in CNNs and RNNs, and benchmarks a Newton–Schulz implementation of polar retraction.
Critical strengths
- The paper explicitly characterizes the QR defect and its non-conservative behavior
- It also honestly acknowledges that the effect on final accuracy remains unresolved.
Critical weaknesses
Although the theoretical analysis is interesting, its practical significance remains unclear.
- It is already known that QR is a first-order retraction and polar is second order, so better preservation of local geometry is expected.
- However, does this actually benefit neural network training, which is itself approximate, especially at large scale, such as [a] and accelerated orthogonal training [b]. Does the proposed approach offer advantages over these existing orthogonal training?
- Unfortunately, evaluation is limited to toy problems and a toy datasets, CIFAR-10, without task performance at scale.
Therefore, the experimental evaluation needs to be significantly strengthened.
References
[a] Riemannian Optimization for LoRA on the Stiefel Manifold
[b] POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformation
Questions for authors
See weaknesses.