Ambient integration rules for Riemannian manifold Hamiltonian Monte Carlo

Riemannian manifold Hamiltonian Monte Carlo (RMHMC) adapts the metric of Hamiltonian Monte Carlo to the local geometry of the posterior, but every integration step requires the solution of an implicit system that contains derivatives of the metric. We consider target posterior distributions $\text{pdf}(q\mid x)$ of hierarchical models, where the joint is given by a diffeomorphic map $\xi \to (q,x)$, where $\xi$ is a standard multivariate normal distributed random variable. We use the fact that conditioning on $x$ restricts $\xi$ to a submanifold with chart $q$ to derive integrators of RMHMC that take paths along secants through ambient space. To do this we consider variational integration methods and rephrase their kinetic term in ambient coordinates which makes them invariant under reparamterisation of the manifold. We show that each implicit integration method allows for an analogous secant version. We show that within our framework the solution of the implicit RATTLE integrator can be derived as a special case and is identical to the Generalized Leapfrog integrator in ambient space. We compare eight solvers for four implicit integrators, including the different solvers for RATTLE, on four test problems, including Neal's funnel, a supernova dust emission model and a hierarchical Gaussian process. Our experiments show that ambient secant formulations are superior in both robustness to the chosen step size as well as solver cost, where we observe that ambient formulations can allow for a 10x larger step-size. We then compare the resulting implicit MCMC schemes to the explicit LMC scheme and find that when matched for trajectory length, secant based methods can be competitive and can even improve over LMC especially for higher dimensional problems, even when using an efficient implementation of LMC that makes use of the geometric structure.