acceptodds
Withdrawn from ICLR 2027

Adaptive Optimization Algorithms on Manifolds

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In machine learning training, minimizing accumulated objective functions is a common task, while most optimization algorithms are designed for variables in Euclidean spaces. This paper generalizes the classical Adam and AMSGrad algorithms to manifold settings, which is often necessary for problems with geometric structures, introducing the RAdam and RAMSGrad algorithms. Under the geodesical convexity assumption, we establish the global convergence of RAMSGrad by deriving a regret bound. RAdam and RAMSGrad are implemented efficiently in the C++ package ROPTLIB. In the numerical experiments, RAdam and RAMSGrad are compared to the state-of-the-art methods including the Riemannian stochastic gradient and existing Riemannian adaptive optimization algorithms using the Karcher mean computation, the Poincaré embedding taxonomies, and the joint diagonalization. The proposed methods perform at least competitively or more effectively than the compared algorithms.

Primary area:
optimization
Keywords:
Adaptive optimization algorithm, Riemannian optimization, Vector transport by parallelization, Machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_n6CuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes two Riemannian extensions of the classical Adam and AMSGrad algorithms, called RAdam and RAMSGrad, for optimization on Riemannian manifolds. The central idea is to use intrinsic representations of tangent vectors and vector transport by parallelization to maintain first- and second-order gradient moment estimates across successive iterations. This enables coordinate-wise adaptive updates on manifolds that are not necessarily product manifolds.

For RAMSGrad, the authors introduce an additional constraint on the variability of the coordinate-wise second-moment estimates and establish a sublinear regret bound under geodesic convexity and other regularity assumptions. This additional constraint has no counterpart in the original Euclidean AMSGrad algorithm.

Both algorithms are evaluated on three optimization problems: Poincaré embeddings, Karcher mean computation on symmetric positive definite matrices, and joint diagonalization on the Stiefel manifold. The numerical results show competitive performance compared with Riemannian SGD and existing Riemannian adaptive optimization methods.

Critical strengths

The use of intrinsic tangent-vector representations and vector transport by parallelization provides an interesting framework for extending Adam-type algorithms beyond Euclidean spaces and product manifolds. In particular, representing tangent vectors in local coordinates allows the authors to perform component-wise operations and transport gradient information between iterations without explicitly computing more expensive transport operators. This is an appealing approach to adaptive optimization on manifolds with more general geometric structures.

--

The paper provides a theoretical analysis of RAMSGrad through a sublinear regret bound under geodesic convexity assumptions. Although the scope of this result is restricted, it provides theoretical support for the proposed algorithm. The implementation in ROPTLIB and the experiments on three different optimization problems are also valuable. The results suggest that the proposed methods can perform quite competitively with existing Riemannian optimizers.

Critical weaknesses

The proposed algorithms rely on an orthonormal basis of the tangent space at each iterate to define the intrinsic representation of gradients and apply component-wise adaptive scaling. However, unlike scalar gradient scaling, coordinate-wise adaptation is generally not invariant under changes of orthonormal basis. Consequently, different choices of tangent-space bases may produce different optimization trajectories, even when starting from the same point and using the same objective function.

The manuscript does not adequately discuss this dependence or explain how the bases are selected and updated throughout the iterations. Moreover, the construction of vector transport by parallelization assumes a smoothly varying family of tangent-space bases, which may impose additional geometric requirements.

These aspects deserve particular attention. A more precise characterization of the underlying geometric requirements and the influence of basis selection would be necessary.

--

An important feature of RAMSGrad is the additional constraint introduced in Steps 7–11 of Algorithm 2, which bounds the difference between the largest and smallest square roots of the adaptive second-moment estimates by a prescribed decreasing sequence phi(t). While this modification facilitates the convergence analysis, it progressively restricts the differences between the coordinate-wise scaling factors, making the adaptive scaling increasingly close to a scalar scaling.

This raises an important question about the extent to which RAMSGrad preserves the coordinate-wise adaptivity that constitutes one of the main motivations of the proposed approach. The manuscript does not investigate this issue, nor does it provide an experimental ablation study demonstrating the influence of this additional constraint on convergence speed and optimization performance.

Finally, the numerical evaluation, although encouraging, remains relatively limited in scope, with insufficient investigation of the sensitivity to algorithm parameters and the geometric characteristics of the manifolds.

These limitations make it difficult to assess the actual advantages of the proposed methods over existing Riemannian adaptive optimization algorithms.

Questions for authors

How sensitive are RAdam and RAMSGrad to the choice of orthonormal bases used for the intrinsic representation of tangent vectors? In particular, would different admissible choices of these bases lead to different optimization trajectories?

The additional constraint in Steps 7–11 of Algorithm 2 progressively reduces the differences between the coordinate-wise second-moment estimates. To what extent does this modification preserve the benefits of coordinate-wise adaptivity, particularly after a large number of iterations? Could the authors clarify the influence of the choice of phi(t)?

Reviewer_duQrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends the Euclidean Adam and AMSGrad algorithms to manifold settings, and proposes RAdam and RAMSGrad algorithms. Convergence of RAMSGrad by limiting the regret value is proved under geodesic convexity. Comparisons with existing Riemannian adaptive algorithms demonstrate the performance of the proposed methods.

Critical strengths

Using the intrinsic representation of tangent vectors in Huang et al. (2017) and the vector transport by parallelization in Huang et al. (2015), the paper extends Adam and AMSGrad to Riemannian setting, in particular being able to appropriately devising a way to handle element-wise adaptivity in general (non-product) manifold.

Critical weaknesses
  1. The paper only proves the convergence of the proposed RAMSGrad, while the convergence of RAdam is left open. With this in mind, along with the following reasons listed below, I strongly believe that the contributions of this manuscript is very limited to be suitable for top ML conferences such as ICLR.

(a) RAMSGrad in the present paper can be viewed as the non-product manifold extension of Becigneul and Ganea (2019). In particular, the main result Theorem 3.5 which provides a regret bound rate is an extension of Theorem 1 of Becigneul and Ganea (2019). The proof development appears to be similar as well. The novelty of the extension is not sufficiently established.

(b) Between the two proposed algorithms, RAdam outperforms RAMSGrad in most instances (in the reported metrics), specifically the running time. The performance of RAdam is comparable to the performance of other existing methods, while RAMSGrad generally performs worse than RAdam. Hence, the main practical contribution is coming from the proposed algorithm with no convergence guarantee (i.e., a heuristic).

  1. The empirical performance of the proposed algorithms is not truly tested on a reasonable problem scale. More importantly, there is insufficient experimentation on non-product manifolds, which is the main motivation of this paper's framework. The Poincare embedding experiment demonstrates good performance of the non-theoretically-guaranteed RAdam in terms of computational time. However, the running time comparisons are not reported for the other two problems, namely Karcher mean and joint diagonalization. Moreover, the joint diagonalization experiment, in particular, should comprise the more meaningful setting of the proposed framework since this is the non-product manifold instance.
Questions for authors
  1. The additional steps in lines 9 to 11 of Algorithm 2 enforce that (\sqrt{\hat{v}{t,\max}} - \sqrt{\hat{v}{t,\min}}\leq C/t^a) with a∈(0,1)a \in (0,1), as already shown in Theorem 3.5. What is the implication of this in the algorithm's parameters when t→∞t\to\infty? Moreover, Corollary 3.6 further assumes that β1t=β1λt−1\beta_{1t}=\beta_1\lambda^{t-1} with λ∈(0,1)\lambda \in (0,1), and this coefficient tends to zero as well. Along with v^t,max⁡−v^t,min⁡≤C/ta\sqrt{\hat{v}_{t,\max}} - \sqrt{\hat{v}_{t,\min}}\leq C/t^a, what does this imply on the algorithm parameters?

  2. Assumption 3.4 assumes boundedness of the Riemannian gradient. Is this condition satisfied by the objective functions in the experiments?

  3. Can the authors provide more intuition and justification for the use of regret bound? Can the authors use other measures of stationarity, for example the expected squared Riemannian gradient norm in Sakai & Iiduka (2025), instead of directly extending the regret bound result of Becigneul and Ganea (2019)?

  4. Lines 468–469 refer to the theoretical contributions of both RAdam and RAMSGrad, yet the paper establishes a regret bound only for RAMSGrad. Given that Zhang et al. (2022), as cited in the paper, established convergence guarantees for Euclidean Adam without modifying its update rules, could the authors clarify why no theoretical guarantee is provided for RAdam? In particular, what additional difficulties, if any, arise from the Riemannian setting?

Reviewer_vZw9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends Adam and AMSGrad to general Riemannian manifolds, giving RAdam and RAMSGrad. Tangent vectors are written in coordinates with respect to an orthonormal basis of each tangent space (the intrinsic representation), and the momentum is moved between tangent spaces by vector transport by parallelization, which is the identity in these coordinates. Adam and AMSGrad are then applied coordinate-wise, and the iterate is updated by the exponential map. RAMSGrad additionally rescales v^t\sqrt{\hat v_t} so that the maximum difference among its components is at most C/taC/t^a. Under geodesic convexity and bounded iterates, the paper proves an o(T)o(T) regret bound for RAMSGrad. The methods are implemented in ROPTLIB and tested on Poincaré embeddings, the Karcher mean of SPD matrices and joint diagonalization on the Stiefel manifold.

Critical strengths
  1. Relevant problem. Adaptive methods on manifolds that are not products of simple factors are of practical interest, and the paper explains clearly why the product-manifold approach of Bécigneul & Ganea (2019) loses adaptivity in this case.
  2. Simple and cheap implementation. With the intrinsic representation, the vector transport is the identity, so the methods are easy to implement and cheap per iteration. The implementation in ROPTLIB (Huang et al., 2018) is a useful practical contribution.

References

  • Gary Bécigneul and Octavian-Eugen Ganea. Riemannian adaptive optimization methods. In International Conference on Learning Representations (ICLR), 2019.

  • Wen Huang, P.-A. Absil, Kyle A. Gallivan, and Paul Hand. ROPTLIB: An object-oriented C++ library for optimization on Riemannian manifolds. ACM Transactions on Mathematical Software, 44(4):1–21, 2018.

Critical weaknesses

W1. Limited novelty. The method applies Euclidean Adam and AMSGrad to the intrinsic coordinates of tangent vectors (Huang et al., 2017), and moves the momentum by vector transport by parallelization (Huang et al., 2015). The regret analysis follows Reddi et al. (2018) and Bécigneul & Ganea (2019). Apart from Steps 7–11 of Algorithm 2 (see W2), the paper appears to be a direct combination of these existing tools.

W2. The element-wise adaptivity is neither well defined nor supported by the theory.

  • The coordinate-wise scaling depends on the choice of the orthonormal basis BxB_x: with a different basis, such as BxQB_xQ for an orthogonal QQ, the algorithm takes a different step. On a general manifold there is no natural choice of this basis (Sec. 2.3), and the paper does not discuss how BxB_x should be chosen or how this choice affects the results.
  • Theorem 3.5 requires that the difference between the largest and smallest components of v^t\sqrt{\hat v_t} be at most C/taC/t^a, and Steps 7–11 of Algorithm 2 enforce this. Since C/ta→0C/t^a\to0, all components of v^t\sqrt{\hat v_t} become almost equal for large tt, so every coordinate gets almost the same stepsize and RAMSGrad becomes Riemannian SGD with momentum. This seems to be needed because the basis BxtB_{x_t} changes from one iterate to the next, so the coordinate-by-coordinate argument of the Euclidean AMSGrad proof cannot be used. In other words, the proof works only because it removes the element-wise adaptivity, which is the main point of the paper.
  • RAdam, which performs best in most experiments, has no convergence guarantee. The Poincaré experiments use a fixed stepsize, and joint diagonalization is not geodesically convex, so Theorem 3.5 does not cover these experiments either.

W3. The experiments are small and do not show a clear advantage.

  • The problems are small: the Karcher mean of 1,000 SPD matrices of size 10×1010\times10, joint diagonalization with n=40n=40 and p=30p=30, and 5-dimensional Poincaré embeddings of the WordNet mammal subtree.
  • The differences are very small. For the Karcher mean, RASA-LR (Kasai et al., 2019) reaches the lowest objective value, and the best final values of RSGD, RAdam, RAMSGrad and RASA-LR agree to about four digits (Table 5). For joint diagonalization, the best final values of RSGD, RAdam and RAMSGrad differ by less than 0.05% (Table 6). No standard deviations over multiple runs are reported.
  • On the Poincaré ball, BxB_x is a rescaled canonical basis, so the update direction of RAdam is coordinate-wise Adam applied to the rescaled Euclidean gradient. This experiment therefore does not test the general case that the paper targets.
  • There are no experiments with neural networks, such as networks with orthogonality constraints or hyperbolic neural networks (Ganea et al., 2018), and no comparison with Sakai & Iiduka (2025).

Minor.

  • The name RAdam is already used for Rectified Adam (Liu et al., 2020).
  • Assumption 3.4 bounds the gradient on all of M\mathcal M, which fails for the Karcher mean; it should be stated on the bounded set of Assumption 3.3.
  • The proof calls FtF_t a "retraction convex function" and Exp−1\mathrm{Exp}^{-1} the inverse of the retraction, while the assumptions and the algorithm use geodesic convexity and the exponential map. The constant ϵ\epsilon in Algorithms 1 and 2 is dropped in the analysis.

Summary. The paper studies a general problem, but the method is a direct combination of known tools, the convergence analysis removes the element-wise adaptivity that is the main point of the paper, and the experiments are small and show no clear advantage over existing methods. I therefore recommend rejection.

References

  • Gary Bécigneul and Octavian-Eugen Ganea. Riemannian adaptive optimization methods. In International Conference on Learning Representations (ICLR), 2019.

  • Wen Huang, P.-A. Absil, and Kyle A. Gallivan. A Riemannian symmetric rank-one trust-region method. Mathematical Programming, 150(2):179–216, 2015.

  • Wen Huang, P.-A. Absil, and Kyle A. Gallivan. Intrinsic representation of tangent vectors and vector transports on matrix manifolds. Numerische Mathematik, 136:523–543, 2017.

  • Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.

  • Hiroyuki Kasai, Pratik Jawanpuria, and Bamdev Mishra. Riemannian adaptive stochastic gradient algorithms on matrix manifolds. In Proceedings of the 36th International Conference on Machine Learning (ICML), PMLR 97:3262–3271, 2019.

  • Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations (ICLR), 2020.

  • Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of Adam and beyond. In International Conference on Learning Representations (ICLR), 2018.

  • Hiroyuki Sakai and Hideaki Iiduka. A general framework of Riemannian adaptive optimization methods with a convergence analysis. arXiv:2409.00859, 2025.

Questions for authors
  1. How sensitive are RAdam and RAMSGrad to the choice of the orthonormal basis BxB_x? For example, on the SPD manifold, how do the results change if the basis is built from an eigendecomposition instead of the Cholesky factor? Is there a principled choice?
  2. Since C/ta→0C/t^a\to0, Steps 7–11 make all components of v^t\sqrt{\hat v_t} nearly equal for large tt. In what sense is RAMSGrad still element-wise adaptive? Can the analysis be carried out without Steps 7–11, or for RAdam?
  3. Could you report means and standard deviations over several runs, and add larger experiments in which element-wise adaptivity matters, for example neural networks with orthogonality constraints or hyperbolic neural networks (Ganea et al., 2018)?

References

  • Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.