- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Riemannian Manifold Steering (RMS), an unsupervised method for activation steering and interpolation in language models. To keep steering trajectories on the data manifold without requiring labeled intermediate concepts or pre-specified topology, the authors combine a pullback metric induced by the output token distribution (measured via Hellinger distance) with an activation support penalty based on the distance to observed activations. In practice, trajectories between activation endpoints are parameterized by a small neural bridge network and optimized via finite-difference approximations of the pullback length. The method is evaluated on 2D synthetic surfaces, toy concept interpolation tasks in Llama-3.1-8B (weekdays and ages), and behavioral steering datasets in Llama-2-7B (hallucination, corrigibility, and myopia).
Critical strengths
-
The geometric motivation combining output pullback metrics with an input support penalty is intuitive and well-analyzed on synthetic manifolds. As shown in Figure 1 and Table 1, neither the pullback metric alone nor the support penalty alone is sufficient to keep trajectories both on the data manifold and away from high output distortion regions. The synthetic surface ablations clearly demonstrate how the two terms complement each other.
-
The paper is written with commendable transparency regarding its approximations and failure modes. The appendices thoroughly document the practical relaxations, hyperparameter sensitivities, and negative results, which makes it much easier to understand what the method actually does under the hood.
Critical weaknesses
-
In-sample toy LLM evaluation where test metrics mirror the training loss. In Section 5.2 (Table 2), each neural bridge is trained to minimize pullback length and k-NN distance to training activations, and then evaluated using the exact same metrics on that same training reference set (Appendix C.1, C.2.6). Showing that a path optimized to stay close to training points achieves a smaller distance to those exact points than unoptimized Linear or Spline baselines is trivial by construction and does not validate better semantic steering.
-
Real-world steering requires ground-truth test answers during optimization, cannot generalize to unseen prompts, and fails to beat straight lines. In Section 5.3 (Table 8), the algorithm trains a separate neural bridge per prompt and optimizes its endpoint using the ground-truth positive and negative answer log-likelihoods for that specific prompt (Equation 6), then reports that same log-likelihood gap as the test metric. Because finding the path requires knowing the contrastive answers in advance, this setup cannot steer unseen prompts at inference time. Even in this in-sample setting, curved RMS trajectories fail to consistently outperform a simple optimized straight line on Corrigibility and Myopia (Table 8).
-
Failure to achieve ordered concept traversal on the toy LLM benchmarks. Although weekday interpolation is meant to show that RMS can naturally traverse intermediate concepts in order without labels, Table 5 (Appendix C.4) shows that RMS achieves 0 out of 21 pairs with ordered dominant concept visits (matching the 0/21 failure rate of Linear interpolation, versus 7/21 for Spline). Only 12.7% of interior peaks along RMS paths ever become the dominant weekday, with a tiny median prominence of 0.72%.
-
Disconnect between the theoretical guarantee and the LLM implementation. Lemma B.1 proves that the support penalty bounds manifold deviation only under a constant Riemannian speed constraint; otherwise, a path can linger near support points and jump rapidly across empty regions. While the synthetic experiments approximate constant speed by minimizing quadratic energy, the LLM implementation minimizes a reparameterization-invariant piecewise-linear length without any speed constraint, invalidating the theoretical bound.
-
Heavy test-time compute overhead and single-token evaluation scope. Unlike standard activation addition, RMS trains a 404K-parameter network for every prompt by decoding 51 path points and 8 subdivided graph connections through the upper transformer layers and LM head, requiring over 4,500 partial forward and backward passes for just 10 Adam steps. Moreover, all experiments only inspect the probability distribution of the first generated token rather than full multi-token generation.
Questions for authors
-
In Section 5.3, since optimizing the endpoint in Equation 6 requires access to the target positive and negative answer tokens for each individual prompt and trains a separate bridge per prompt, how can RMS be applied to steer unseen test prompts where ground-truth answer pairs are unavailable? Can the learned steering trajectories transfer to held-out prompts, and how does the method affect full multi-token autoregressive generation beyond the first token distribution?
-
Given that Table 5 shows 0 out of 21 weekday pairs achieve ordered dominant intermediate visits under RMS, and Table 8 shows that curved bridges do not outperform optimized straight lines on Corrigibility and Myopia, what concrete downstream behavioral benefit does optimizing a curved trajectory provide in LLMs beyond simply optimizing the endpoint or applying standard activation addition?
-
Why did the LLM implementation minimize the piecewise-linear length functional instead of the quadratic energy functional used in the synthetic experiments, which would naturally encourage constant metric speed and align with the assumptions of Lemma B.1?
6 comments
Response 2/3: W2 + W3
W2 / Q1 Answer Pairs, Unseen Prompts and Multi-Token Generation
Real-world steering requires ground-truth test answers during optimization, cannot generalize to unseen prompts, and fails to beat straight lines. [...]
Q1: [...] how can RMS be applied to steer unseen test prompts where ground-truth answer pairs are unavailable? Can the learned steering trajectories transfer to held-out prompts, and how does the method affect full multi-token autoregressive generation beyond the first token distribution?
The reviewer is right that Section 5.3, as written, has these limitations. They come from its per-prompt setup, not from the method. We ran a new experiment on MAxBench Countries that removes them (full table and setup are in our response to Reviewer opUL, See Responses 3/3 W3):
- No test answers. The endpoints are two continent centroids, not per-prompt answers: the same supervision as activation addition is needed. The training reward uses labelled Countries sentences only and test prompts need no answers (no groundtruths are needed). Since our centroids are unordered, splines do not work.
- Unseen prompts. We fit 15 paths once (one per continent pair; Llama-3.1-8B-Instruct, layer 19) and apply them unchanged to 100 AlpacaEval instructions that are not used to fit the paths and are held out.
- Multi-token generation. Steering acts at the last prompt token and at every generated token. MAxBench's LLM judge scores 30 full responses per instruction (up to 128 tokens) from 0 to 2 for concept, fluency and instruction following.
Result. RMS has the highest combined score of the steered methods within the steerable strength range. At (steering magnitude), it raises the concept score from 0.39 (unsteered) to 1.08, with fluency and instruction following on par with difference-of-means steering (combined 0.99 vs 0.98). At , its fluency and instruction gains become significant (+0.17 [+0.08, +0.26], +0.16 [+0.07, +0.25]; combined 0.68 vs 0.50). At equal , RMS expresses the concept less than difference of means (−0.17 and −0.13).
Section 5.3. We will present it as a per-prompt study of how the support term trades steering strength for output proximity, and move its limitation (Section 6, Appendix E.6) into the section itself. Against tuned mean-difference steering, RMS has lower output distance on all three behaviours and higher strength on two (Table 8; ten prompts, one seed).
W3 / Q2 Ordered Traversal and What the Path Adds
Failure to achieve ordered concept traversal on the toy LLM benchmarks. [...] RMS achieves 0 out of 21 pairs with ordered dominant concept visits [...]
Q2: [...] what concrete downstream behavioral benefit does optimizing a curved trajectory provide in LLMs beyond simply optimizing the endpoint or applying standard activation addition?
Ordered traversal. We agree: RMS does not visit the intermediate weekdays in order as the top answer. The spline is given the class labels and their order, and passes through the centroids by construction; RMS uses neither. Ordered visits are also not what RMS's objective targets. We will remove any wording that suggests ordered traversal, and qualify the sentence in Section 5.2 about peaks at intermediate concepts.
Q2: what the path adds. We do not claim a consistent benefit of curvature. Our claim is that RMS paths stay closer to the outputs the model naturally produces, which keeps steered outputs more natural. Curvature is the freedom that lets the path follow the data; it is not the goal in itself.
- Closer to natural outputs. On held-out prompts, RMS paths stay closer to the model's natural outputs than linear and spline paths on both toy tasks (W1).
- More fluent generations. On MAxBench, RMS is the most fluent and most on-task steered method at both strengths, against both difference-of-means steering and the optimised straight path (W2).
- Strength, in some cases. RMS is stronger than tuned mean-difference steering on hallucination and myopia (Table 8), and curvature improves strength over the straight path (optimised linear method) on hallucination (0.749 vs 0.733). It does not on corrigibility and myopia.
- A trade-off. The support term trades strength for proximity: raising its weight from 0 to 10 lowers output distance on all three behaviours at a small cost in strength (Table 8). On MAxBench, RMS expresses the concept less than difference of means at equal strength. We will state this trade-off explicitly.
Response 3/3: W4 + W5
W4 / Q3 Lemma B.1 and Length Versus Energy
Disconnect between the theoretical guarantee and the LLM implementation. Lemma B.1 proves that the support penalty bounds manifold deviation only under a constant Riemannian speed constraint [...]
Q3: Why did the LLM implementation minimize the piecewise-linear length functional instead of the quadratic energy functional used in the synthetic experiments [...]?
- We agree. Lemma B.1 needs constant speed, and the LLM runs do not enforce it: path length is invariant to reparameterisation, but our support and proximity averages, taken at evenly spaced , are not. We will state in Appendix B and in the main text that the bound does not cover the LLM runs.
- Why length (Q3). The support graph measures distance as shortest-path length, so we minimised the same quantity along the path. We had not tested energy in the LLM tasks; we now have.
- What we tested. On held-out prompts (21 pairs per task): (a) training with energy, as in our synthetic experiments, which halves speed variation; (b) re-scoring the same paths at constant speed, weighting each stretch by its length instead of each -step equally. The full table is in our response to Reviewer vSue (W10/Q8).
- Weekdays: no change. RMS stays closer than the spline under both changes (output difference −0.005 by default, −0.004 with energy, −0.005 at constant speed).
- Ages: the reviewer's concern holds. RMS covers half its length in 20% of its -steps, exactly where the output switches age: the uneven speed the reviewer describes. At constant speed, RMS still beats linear (−0.015, 17/21 pairs) but no longer the spline (+0.011, 7/21), and training with energy does not close this gap (+0.005, 8/21).
- We will report constant-speed scores alongside the default ones and state this limitation.
W5 Compute and Single-Token Evaluation
Heavy test-time compute overhead and single-token evaluation scope. Unlike standard activation addition, RMS trains a 404K-parameter network for every prompt [...] Moreover, all experiments only inspect the probability distribution of the first generated token rather than full multi-token generation.
- We agree that RMS costs more than activation addition, by design. Activation addition estimates one direction. RMS optimises a whole path, and every point on it is a candidate intervention chosen to stay near the data, rather than a rescaling of one fixed direction. Optimising a path through the representation space, under the model's output geometry, is a harder problem than estimating one vector, and it costs more. We will state this cost in the main text as highlighted by the reviewer.
- We note that the cost is paid per concept, not per prompt, when the endpoints are concept centroids. On MAxBench, the 15 paths are fit once and reused for every new instruction (W2); steering a new prompt then costs the same as activation addition. Only the per-prompt setting of Section 5.3 fits a path for each prompt.
- Single-token evaluation. The toy tasks and the output-proximity scores use the next-token distribution. For free multi-token generation, we now show that MAxBench scores full responses of up to 128 tokens with an LLM judge (W2). We introduce this experiment in the rebuttal.
Response 1/3: Summary + W1
We thank the reviewer for the careful reading, and for noting the synthetic ablations and our transparency about failure modes. We agree with several points and ran new experiments to address them. In summary:
W1 In-Sample Evaluation
We agree that the original evaluation was in-sample, and we now evaluate on held-out prompts. Support graphs, endpoints and spline knots use training prompts only; every score below uses held-out prompts. On held-out prompts, RMS is closer to the model's natural outputs than linear and spline on both tasks (weekdays 0.126 vs 0.131 and 0.130; ages 0.258 vs 0.284 and 0.272; lower is closer). The weakest margin is weekday output against the spline (11/21 pairs). The full split and table, including activation proximity, are included in our response to Reviewer vSue (W11/Q9).
Our main score is not a training term. Output proximity is the Hellinger distance between the outputs along the path and the 5 nearest natural outputs (previously measured on the training points but now also measured on held-out prompts). As stated in Appendix C.2.6, it was never part of the training objective; it was, however, computed on the training references, which we now fix. Activation proximity has the same form as the support term, so we report it as secondary and it is more of a sanity check of the optimisation working as intended.
Is proximity trivial by construction? We test this with a baseline that directly optimises closeness to the training activations (B1, suggested by Reviewer opUL; full table in our response to opUL, W2). If staying close to training points were enough, B1 would win on held-out prompts. On weekdays it does not: B1 is the closest method on training prompts (0.079, vs 0.080 for RMS at the same budget) but the farthest on held-out prompts (0.133 vs 0.126), with RMS closer on 15/21 pairs. On ages, following the activation density is enough: B1 beats RMS at the paper's setting, and RMS matches it with the same 500 updates (0.228 vs 0.237, RMS closer on 14/21).
Semantic steering. We agree that proximity is not semantic steering. Our claim is that RMS paths stay close to the outputs the model naturally produces. We will qualify the sentence in Section 5.2 that RMS outputs "peak at intermediate concepts" (see W3), and state the claim as proximity throughout.