Adaptive Optimization Algorithms on Manifolds

In machine learning training, minimizing accumulated objective functions is a common task, while most optimization algorithms are designed for variables in Euclidean spaces. This paper generalizes the classical Adam and AMSGrad algorithms to manifold settings, which is often necessary for problems with geometric structures, introducing the RAdam and RAMSGrad algorithms. Under the geodesical convexity assumption, we establish the global convergence of RAMSGrad by deriving a regret bound. RAdam and RAMSGrad are implemented efficiently in the C++ package ROPTLIB. In the numerical experiments, RAdam and RAMSGrad are compared to the state-of-the-art methods including the Riemannian stochastic gradient and existing Riemannian adaptive optimization algorithms using the Karcher mean computation, the Poincaré embedding taxonomies, and the joint diagonalization. The proposed methods perform at least competitively or more effectively than the compared algorithms.