acceptodds
Withdrawn from ICLR 2027

Watch the Model Think: On-Policy Extraction of Activation Steering Vectors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When a model solves a problem on one attempt and fails it on the next, what separates the two is rarely the final answer token; it is the trajectory that reached it. Contrastive activation steering leaves that signal unused: CAA, SADI, RepE and ITI build their direction from experimenter-supplied text, recorded while the model reads rather than reasons. That choice also caps what the vector can express, since polarity must be written into the text, and a task judged only by outcome offers nothing to write it with. ROAST makes the trajectory itself the contrast: sample rollouts, let an outcome verifier split them into successes and failures, and contrast the reasoning that worked against the reasoning that did not. The only thing that changes is which text the activations are read from, and a matched teacher-forced control — rollouts, labels, answer text and pair counts held fixed, the trajectory alone stripped — isolates what that buys: on GSM8K at B the pairs alone give points and restoring the trajectories gives , the larger step on all six columns tested and the only one of the two to clear seed variance anywhere. Holding token count and anchor fixed, substituting an equal-length neutral prefix or another question's reasoning falls below no intervention — worse than supplying no reasoning at all. The two corpora are far apart geometrically as well: the two readings of one direction sit a median and apart at the two scales we probe, within of that net of a split-half null. Reading from rollouts forces two corrections — keep the full difference vector, since Top- masking discards half its energy, and give each question one vote rather than one per pair — and grouped aggregation is what keeps GSM8K above baseline under verifier noise. We rest the comparison on the benchmarks scored without an answer parser — GSM8K, MATH500 and IFEval — where ROAST is best in all six cells across two models, by up to over no intervention, for wall-clock and no added context; on the parser-scored multiple-choice benchmarks it also leads every steering baseline on average at all three scales. Across nine benchmarks and nine models (B–B, four families), ROAST improves the per-model average at every scale. https://anonymous.4open.science/r/ORBIT-E014/

Primary area:
interpretability and explainable AI
Keywords:
activation steering, on-policy extraction, representation engineering, large language model reasoning, inference-time intervention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UYrFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ROAST builds a task specific steering vector by using model generated rollouts split by an outcome verifier. It compares the final token activations between correct and incorrect generations for each question. Then takes the mean direction of each question and renormalizes them. The resulting vector is then added to the intermediate MLP layer activation while decoding. There were many models and tasks tested however not all combinations. By using matched answers they show benefit from keeping generation context and outperform no steer by 9.66% on GSM8K/Qwen3-0.6B.

Critical strengths
  1. Ablating with matched corpus is good. Table 5 keeps rollouts, labels, final answers and number of pairs. Removing only trajectories from the prompt. Restoring the trajectory improves all six matched columns, including +6.05 points on GSM8K/0.6B. It's better to have these types of ablations compared to an unmatched metric vs an answer.
  2. Its fairly simple method to deploy and test on multiple models. No parameters changed and can be used without any added context. The disclosure of negative results, strength sensitivity, and evaluation limitations helps delimit the findings.
Critical weaknesses

The contribution is insufficiently distinguished from closely related prior work. ALS [1] already constructs mean-difference steering vectors from automatically verified successful and unsuccessful model generations, with final-token extraction explicitly described in Section 2 of its original version. AdaRAS [2] self-samples multiple trajectories for the same input, pairs opposite correctness outcomes, and uses activation mean differences to steer MLP activations (Sections 3.1–3.4). Neither work is discussed or empirically compared against. ROAST differs in its aggregation, normalization, and intervention details, and its matched source-context ablations are useful. However, without comparisons to these predecessors, it remains unclear whether these differences establish a sufficiently substantial contribution beyond the existing trajectory-based steering recipe.

References

[1] Nathan Egbuna, etal. Amortized Latent Steering: Low-Cost Alternative to Test-Time Optimization. 2025. arXiv:2509.18116v1. https://arxiv.org/abs/2509.18116v1

[2] Fangan Dong, etal. Identifying and Transferring Reasoning-Critical Neurons: Improving LLM Inference Reliability via Activation Steering. 2026. arXiv:2601.19847v1. https://arxiv.org/abs/2601.19847v1

The context controls do not fully establish what the steering direction captures. Table 6 tests one model and task at a single strength, α=3. It shows that intact trajectory context produces a better direction under this setting. However, corrupted contexts may produce harmful directions, and different directions may require different strengths. The results therefore do not establish that the vector captures a reusable correct-reasoning mechanism. A strength sweep and readouts that exclude the final answer would help clarify this.

The theory does not yet explain the practical benefit of grouping. Assumption A2 directly assumes that within-question averaging does not reduce variance as the pair count grows. Sharing rollouts across pairs does not, by itself, establish this assumption. The proof also compares an unnormalized naive estimator with a normalized grouped estimator. Appendix A.4 acknowledges both limitations, but without a comparison between equally normalized estimators, the theoretical advantage attributable to grouping remains unclear.

The noise results are too limited to establish broad robustness. Table 7 uses one seed on two tasks. Grouping performs better than nongrouped aggregation, but retaining an improvement over no intervention is task-dependent: at 20% label flips, grouped ROAST remains above baseline on GSM8K but falls to 46.02 on TruthfulQA, below the 48.78 baseline. The experiment also tests independent symmetric flips, rather than systematic verifier errors. It supports a benefit of grouping in these tests, but not a general robustness claim.

Questions for authors
  1. What are the main advances over ALS [1] and AdaRAS [2]? Can controlled comparisons using the same rollout pool, evaluation protocol, and comparable tuning budgets demonstrate the value of those advances?

  2. Section 4.4 intervenes at the MLP post-activation output, while §1/§2/App. E describe the method as operating on the residual stream and Related Work excludes 'localized-component' methods from ROAST's family. Which characterization holds, and do baseline vectors (extracted at the residual stream) use the same injection component?

  3. Does the ordering of the extraction contexts in Table 6 hold across steering strengths and on another reasoning model or task? Do the gains persist when steering vectors are extracted from activations just before the final answer begins, to test their dependence on terminal-answer features?

  4. What supports A2 under the actual rollout-sampling process? For a fixed question, how does the variance of its mean-difference estimate change across repeated rollout draws as the number of independently sampled rollouts increases? Can the theoretical comparison use Norm(v_naive) and the grouped estimator, with the roles of question count and rollout count stated explicitly?

Reviewer_y8HBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ROAST, a training-free activation steering method that extracts steering vectors from the model’s own successful and failed trajectories. ROAST uses an outcome verifier to construct contrastive rollout pairs and introduces continuous normalization and grouped aggregation to obtain stable steering directions. Experiments across multiple models and benchmarks show that ROAST can achieve consistent improvement.

Critical strengths

The paper is generally clear. The idea is simple yet effective, and the experimental results demonstrate its effectiveness.

Critical weaknesses
  1. The rollout construction introduces extra computational cost. When applying ROAST to a new LLM, a large number of trajectories must first be generated, introducing additional computational and time overhead.

  2. There has been extensive work on activation steering, but this paper lacks comparisons with stronger baselines.

  3. The method depends on reliable outcome verifiers. ROAST requires a verifier to split model rollouts into successful and failed trajectories. In the experiments, this is conveniently provided by benchmark-specific metrics, but open-ended tasks would require learned reward models, whose errors can be biased and correlated.

Overall, I do not think this paper provides sufficient additional insight beyond existing work. The experimental evaluation is not sufficiently solid, and the paper falls below the acceptance threshold.

Questions for authors

see weaknesses

Reviewer_ivqgOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors proposed to extract activation steering vectors from the model's own rollout trajectories, which are split into correct and incorrect ones by an outcome verifier, instead of from ground truth. They also replace Top-k masking with full-vector normalization, and normalize per question before averaging. They've done experiments on a series of benchmarks and models, with controls showing that the gain mainly comes from the model's own reasoning trajectory.

Critical strengths

Reading the steering direction from the model's own verifier-labeled rollouts is a simple and easy-to-implement idea, and the matched teacher-forced control, which keeps the rollouts, labels, answer text, and pair counts fixed and strips only the trajectory, is a reasonable design for attributing the gain to the trajectory.

Critical weaknesses
  1. This paper is quite hard for human readers to follow. Generally I would not assess a paper by its clarity, but this paper gives me a strong feeling that its overall structure is different from most of the papers I read: the main claims are spread across densely packed sentences, many qualifications, and frequent cross-references to the appendix, which makes the core argument difficult to extract on a first read. Also, even though I do not want to make accusations casually, and I note that the AI Use Statement discloses AI assistance for prose editing, the writing style still gives me a very strong impression that the text is largely AI-generated rather than AI-polished.

  2. Then there are also problems, ranging from specific experimental problems, like the narrow evidence for the central attribution claim (the matched teacher-forced control uses only two models at N=100N=100, and the length-matched context controls use only one model, one task, and a fixed α=3\alpha=3), or missing baselines that matter for the claim (training on the same verified rollouts, e.g., rejection-sampling fine-tuning or STaR, and a linear probe on the same activations are not compared, while IFEval is compared against RepE only), to inconsistencies in the core logic. When arguing against Top-k masking, the paper treats activation magnitude as signal: the top 10% of dimensions carry only about half of the L2 energy, so masking is said to "discard half the signal". However, when motivating per-question normalization, it treats magnitude as noise, stating that ∥Δhq∥\lVert \Delta h_q \rVert carries no quality signal and that "loud prompts" should not dominate, and its own analysis shows that magnitude is only weakly and even negatively correlated with directional consistency. If magnitude does not indicate quality across questions, it is unclear why squared magnitude (energy) should indicate signal across dimensions.

Both of them, including the following questions, make me think the paper is below the requirements of an accepted work.

Questions for authors
  1. Can the authors discuss in more detail how generative AI was involved in the whole research process (e.g., ideation, experiment design and execution, analysis, and writing), beyond the prose editing and LaTeX formatting stated in the AI Use Statement?

  2. Can the authors provide a plan for restructuring/revising the paper that can increase clarity?

  3. Can the authors add some complementary experiments that address W2, e.g., the matched teacher-forced control and the context controls on more models and tasks, and comparisons with training on the same rollouts and with a probe-based direction?

  4. Can the authors discuss how their ideas iterated towards finally choosing L2 energy as the evidence against Top-k masking, and towards adopting per-question normalization? Describing the reasoning and the alternatives considered along the way would help readers understand why these two design choices were made, especially given that they treat activation magnitude differently.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.