- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes two Riemannian extensions of the classical Adam and AMSGrad algorithms, called RAdam and RAMSGrad, for optimization on Riemannian manifolds. The central idea is to use intrinsic representations of tangent vectors and vector transport by parallelization to maintain first- and second-order gradient moment estimates across successive iterations. This enables coordinate-wise adaptive updates on manifolds that are not necessarily product manifolds.
For RAMSGrad, the authors introduce an additional constraint on the variability of the coordinate-wise second-moment estimates and establish a sublinear regret bound under geodesic convexity and other regularity assumptions. This additional constraint has no counterpart in the original Euclidean AMSGrad algorithm.
Both algorithms are evaluated on three optimization problems: Poincaré embeddings, Karcher mean computation on symmetric positive definite matrices, and joint diagonalization on the Stiefel manifold. The numerical results show competitive performance compared with Riemannian SGD and existing Riemannian adaptive optimization methods.
Critical strengths
The use of intrinsic tangent-vector representations and vector transport by parallelization provides an interesting framework for extending Adam-type algorithms beyond Euclidean spaces and product manifolds. In particular, representing tangent vectors in local coordinates allows the authors to perform component-wise operations and transport gradient information between iterations without explicitly computing more expensive transport operators. This is an appealing approach to adaptive optimization on manifolds with more general geometric structures.
--
The paper provides a theoretical analysis of RAMSGrad through a sublinear regret bound under geodesic convexity assumptions. Although the scope of this result is restricted, it provides theoretical support for the proposed algorithm. The implementation in ROPTLIB and the experiments on three different optimization problems are also valuable. The results suggest that the proposed methods can perform quite competitively with existing Riemannian optimizers.
Critical weaknesses
The proposed algorithms rely on an orthonormal basis of the tangent space at each iterate to define the intrinsic representation of gradients and apply component-wise adaptive scaling. However, unlike scalar gradient scaling, coordinate-wise adaptation is generally not invariant under changes of orthonormal basis. Consequently, different choices of tangent-space bases may produce different optimization trajectories, even when starting from the same point and using the same objective function.
The manuscript does not adequately discuss this dependence or explain how the bases are selected and updated throughout the iterations. Moreover, the construction of vector transport by parallelization assumes a smoothly varying family of tangent-space bases, which may impose additional geometric requirements.
These aspects deserve particular attention. A more precise characterization of the underlying geometric requirements and the influence of basis selection would be necessary.
--
An important feature of RAMSGrad is the additional constraint introduced in Steps 7–11 of Algorithm 2, which bounds the difference between the largest and smallest square roots of the adaptive second-moment estimates by a prescribed decreasing sequence phi(t). While this modification facilitates the convergence analysis, it progressively restricts the differences between the coordinate-wise scaling factors, making the adaptive scaling increasingly close to a scalar scaling.
This raises an important question about the extent to which RAMSGrad preserves the coordinate-wise adaptivity that constitutes one of the main motivations of the proposed approach. The manuscript does not investigate this issue, nor does it provide an experimental ablation study demonstrating the influence of this additional constraint on convergence speed and optimization performance.
Finally, the numerical evaluation, although encouraging, remains relatively limited in scope, with insufficient investigation of the sensitivity to algorithm parameters and the geometric characteristics of the manifolds.
These limitations make it difficult to assess the actual advantages of the proposed methods over existing Riemannian adaptive optimization algorithms.
Questions for authors
How sensitive are RAdam and RAMSGrad to the choice of orthonormal bases used for the intrinsic representation of tangent vectors? In particular, would different admissible choices of these bases lead to different optimization trajectories?
The additional constraint in Steps 7–11 of Algorithm 2 progressively reduces the differences between the coordinate-wise second-moment estimates. To what extent does this modification preserve the benefits of coordinate-wise adaptivity, particularly after a large number of iterations? Could the authors clarify the influence of the choice of phi(t)?