Riemannian Manifold Steering Stays on a Data-Informed Track

Activation steering aims to control a model's behavior by moving activations along a linear direction or interpolating between concepts. However, steered activations can lie far from those the model naturally produces, degrading its output. Prior manifold-aware steering (Wurgaft et al., 2026) addresses this with cubic splines, but requires labeled concept centroids and assumptions about the manifold's topology. Instead, we model the activation space as a Riemannian manifold whose metric is pulled back from the output distributions. We consider geodesics (shortest paths) under that metric with a regularizing term that penalizes paths moving away from observed activations. We call this unsupervised method Riemannian Manifold Steering (RMS). We first examine path recovery and surface drift on synthetic surfaces with known geometry, isolating the roles of the metric and data support regularizer. We then show that the resulting paths produce outputs closer to the natural output distributions of the model across a bunch of tasks ranging from toy to real world LLM tasks.