acceptodds
Under review as a conference paper at ICLR 2027

Gram–Schmidt Breaks the Gradient Structure of Riemannian Gradient Descent: First-Order Retractions on the Stiefel Manifold

Abstract

Orthogonality constraints are commonly found in deep learning, appearing in applications such as unitary recurrent networks, orthogonal convolutions, and Stiefel-constrained layers. These constraints are typically enforced by retracting each update using a thin QR factorization, which involves Gram-Schmidt orthogonalization. However, QR factorization is known to be only a first-order retraction. We demonstrate that this has structural implications for the optimizer itself. Specifically, for Riemannian gradient descent on the Stiefel manifold, we derive, in closed form, the tangential defect introduced by QR factorization at second order in the step size. Notably, this defect disappears when using the polar retraction. Furthermore, we prove that using explicit objectives validated through exact rational arithmetic, the -form dual to this defect is not closed. As a result, gradient descent employing QR factorization does not support any modified loss at first order in the step size, in contrast to methods based on the exponential map and other second-order retractions. Since the defect vanishes at critical points, it goes unnoticed in standard local convergence and stability analyses. Additionally, because Gram-Schmidt processes columns sequentially, the optimization based on QR is not equivariant under permutations of the weight columns. This means that in a Stiefel-constrained convolutional network and an orthogonal recurrent network, permuted copies of the same model diverge within the first few updates, while updates based on polar retraction remain consistent to machine precision. We cannot determine whether this lack of equivariance affects final accuracy since, over the course of full training, even the polar control eventually loses equivariance due to accumulated rounding errors. Our primary concern is with well-posedness and reproducibility. Lastly, we show that the polar retraction can be computed by a short Newton–Schulz iteration built entirely from matrix multiplications. This method restores equivariance at the tested sizes and performs faster than thin QR factorization at larger dimensions.

Primary area:
optimization
Keywords:
Riemannian Optimization, Stiefel Manifold, QR factorization, Backward Error Analysis, Implicit Regularization, Polar Retraction, Newton–Schulz Iteration, Gram-Schmidt Orthogonalization, Orthogonal Recurrent Networks, Stiefel Convolutional Neural Networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_h9jPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper derives the second-order tangential defect of QR retraction on the Stiefel manifold and constructs examples showing that this defect is non-conservative, obstructing a first-order modified-loss interpretation. It also analyzes permutation-equivariance breakdown, validates early trajectory differences in CNNs and RNNs, and benchmarks a Newton–Schulz implementation of polar retraction.

Critical strengths
  • The paper explicitly characterizes the QR defect and its non-conservative behavior
  • It also honestly acknowledges that the effect on final accuracy remains unresolved.
Critical weaknesses

Although the theoretical analysis is interesting, its practical significance remains unclear.

  • It is already known that QR is a first-order retraction and polar is second order, so better preservation of local geometry is expected.
  • However, does this actually benefit neural network training, which is itself approximate, especially at large scale, such as [a] and accelerated orthogonal training [b]. Does the proposed approach offer advantages over these existing orthogonal training?
  • Unfortunately, evaluation is limited to toy problems and a toy datasets, CIFAR-10, without task performance at scale.

Therefore, the experimental evaluation needs to be significantly strengthened.

References

[a] Riemannian Optimization for LoRA on the Stiefel Manifold

[b] POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformation

Questions for authors

See weaknesses.

Reviewer_YkPHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the second order tangent defect induced by the QR retraction on the stiefel manifold. They show that a lack of permutation invariance can appear and propose a Newton schultz implementation of the polar retraction

Critical strengths

The motivation of the paper is clear and the proofs seem correct

Critical weaknesses

I think that the contribution is pretty limited, especially from an optimization or ML perspective and especially to a submission to ICLR. Furthermore the proofs seems to rely mostly on standard techniques and I could not see any new idea being introduced More specifically concerning the contributions: 1Tangential deflect: it seems to be a straightforward taylor expansion of the QR factorization. 2 This is a property of smooth first order retraction, the second order term in the taylor expansion of the exponential map does not matter for the convergence of the Riemannian gradient descent based on a retraction 3 the non closedness is interesting but I cannot understand what is the relevance in an optimizaiton context. 4 That is a nice observation but again there does not seem to be any meaingfull consequence discussed in the paper. 5 This seems to be already investigated by Grishina et al. (arXiv:2506.10935, 2025). THe novelty and practical advantage need to be discussed

Questions for authors

1 Can the authors clarify the practical importance of the permutation equivariance, particulary regarding convergence ? 2 Can the authors explain if there are anything non-standard that has been used in the proofs ? 3 Can the authors discuss the difference with the Grishina et al. paper for the Newton Schultz implementation?

Reviewer_vk8pOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper analyzes Riemannian gradient descent on the Stiefel manifold under thin-QR versus polar retractions. It defines the tangential defect, proves a closed form for QR, τqf(X)[V,V]=X(up(VTV)T−up(VTV))\tau_{qf}(X)[V,V]=X(up(V^{T}V)^{T}-up(V^{T}V)), and shows polar has zero defect. The QR defect yields a non-closed one-form, so no order-η\eta modified energy exists; it also breaks column-permutation equivariance and produces asymmetric Floquet matrices on periodic orbits. Numerical checks, exact rational witnesses, float64 CNN/RNN controls, and a Newton–Schulz benchmark support the theory.

Critical strengths

The central result is crisp, falsifiable, and directly relevant to widely deployed thin-QR retractions: a closed-form tangential defect plus a non-conservativity proof with exact rational witnesses. This materially sharpens backward-error/implicit-regularization guarantees for manifold optimization, which previously applied to second-order retractions but not first-order QR. The float64 permutation controls are well designed and isolate the retraction from floating-point associativity. The practical comparison with Newton–Schulz adds actionable guidance. These strengths strongly support acceptance as a rigorous theoretical contribution with clear warnings for practice.

Critical weaknesses

The empirical evidence does not establish practical significance. The authors explicitly avoid final-accuracy claims and confine deep-network measurements to an early window because chaos amplifies rounding; thus the CNN/RNN results show symmetry breaking, not degraded optimization. The non-conservativity claim is proved for p≤n−2p\leq n−2 with specific witnesses; p=np=n is excluded and p=n−1p=n−1 untested, limiting scope. The Newton–Schulz benchmark is useful, but finite-iteration NS may not be an exact second-order retraction, so its defect and consequences are unclear. These issues weaken impact and generality.

Questions for authors
  1. Clarification: Does Theorem 4/Corollary 5 hold for all 2≤p≤n−22\leq p\leq n−2, or only for the exhibited witnesses? What behavior do you expect at p=n−1p=n−1, and why is it left untested?

  2. Clarification: Is the finite Newton–Schulz map an exact retraction with zero tangential defect at 4–5 iterations? If not, what is its residual defect, and does the practical recommendation still hold?

  3. Suggestion: Can you add a controlled, non-chaotic experiment isolating QR vs polar—e.g., a quadratic objective with known optimum—to show a measurable convergence, stability, or final-loss difference? This would materially strengthen practical relevance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.