- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies Prefix Steering: applying an existing steering direction only over a short span beginning at the final input token. It derives sufficient conditions for matching prompted attention outputs under fixed-state assumptions, then evaluates intervention duration and strength across four models and five tasks. Short interventions often retain much of full steering’s behavioral control with less capability degradation.
Critical strengths
-
The paper identifies a useful distinction between initiating a behavior and continually enforcing it. Figures 4–5 show that repeatedly applying the same direction can add capability cost without a corresponding gain in control. Obtaining better trade-offs simply by shortening the intervention is a practically useful finding, particularly because it works with existing directions and both Additive steering and COAST.
-
The final-answer formatting experiments strengthen the empirical contribution. Unlike the refusal example, where the first generated word already signals the target behavior, these tasks evaluate a property expressed after intervening reasoning. They provide evidence that the benefit of a short intervention extends beyond changing the response opening.
Critical weaknesses
-
The attention-matching theory provides motivation, but its explanatory value for practical Prefix Steering remains unclear. The constructed displacement depends on the query and context, whereas the behavioral experiments use a fixed DiM direction across inputs. Appendix D.1 establishes partial directional alignment on one model and concept, but does not show that this alignment yields small attention-matching errors or predicts successful control. Likewise, a smaller sufficient matching subspace with more steered tokens does not establish why longer interventions damage capability. The paper acknowledges these limitations, but the connection between its two main contributions remains incomplete.
-
The comparison with adaptive steering policies is less conclusive than the comparison with full steering. Appendix D.3 fixes DAS’s maximum coefficient and ACT’s parameters to values from their original papers, while Prefix Steering’s strength is selected for the present models and tasks. Since the paper itself shows that strength substantially changes the control–capability trade-off, Figure 6 does not establish that the advantage comes from stopping early rather than from the chosen operating points. A comparable tuning procedure is needed to support the claimed advantage over these policies.
Questions for authors
-
Does the attention-matching error obtained with the actual DiM intervention predict behavioral success? For the delayed formatting results, can you distinguish the contribution of modified cached representations from that of the generated text—for example, by replaying the same initial generated tokens with a freshly computed, unsteered cache?
-
Does the advantage in Figure 6 persist when DAS and ACT are tuned on the same validation data and compared at matched levels of behavioral control?
2 comments
Thank you again for your feedback. We would like to check whether our response has addressed your concerns. If any question remains, we are happy to clarify before the discussion period ends.
All revisions made in response to this review are marked in blue in the revised paper.
W1 and Q1. Explanatory value of the attention-matching theory
Thank you for the thoughtful comments. Our goal is to connect prompting and activation steering at the attention level, rather than to construct a displacement that exactly matches each prompt, which would offer little beyond prompting itself. The theory shows that steering a single position can reproduce a prompt's attention-level effect, and this connection motivates Prefix Steering, which applies reusable directions such as DiM over a short span. It does not claim that a lower matching error predicts better control, since prompting itself does not always succeed. Following your suggestion, we replay the initial generated tokens with a fresh, unsteered cache (Table 1). Relative to prompting, the attention-output error is much smaller with the modified cache than with the replayed text alone. For safety, where refusal appears in the first tokens, replay keeps most of the control. For delayed formatting, replay loses most of the gain and restores MATH-500 accuracy, so both the lasting control and the capability cost come mainly from the modified cached representations. Regarding longer interventions, we present the smaller matching subspace as a geometric picture consistent with the capability cost, not as its cause. We have added the replay experiment as a new Section 4.3.
\hline & & \text{Safety} & & & & \text{IF-Boxed} & \\\\ \text{Condition} & \text{Control} & \text{MMLU-Pro} & \text{Attn. diff} & \quad & \text{Format} & \text{MATH-500} & \text{Attn. diff} \\\\ \hline \text{Unsteered} & 36.0 & 39.4 & \text{–} & & 48.7 & 48.3 & \text{–} \\\\ \text{Prefix-1, steered} & 80.7 & 39.0 & 0.421 & & 91.6 & 47.8 & 0.513 \\\\ \text{Prefix-1, replay} & 77.6 & 40.5 & 2.847 & & 57.2 & 48.7 & 3.426 \\\\ \text{Prefix-5, steered} & 85.4 & 36.6 & 0.368 & & 94.8 & 45.4 & 0.482 \\\\ \text{Prefix-5, replay} & 81.9 & 38.3 & 1.935 & & 64.1 & 48.9 & 4.127 \\\\ \hline \end{array}$$ Table 1: Replay on OLMo 3 7B with additive steering. *Steered*: Prefix Steering as in the paper. *Replay*: the same first $k$ generated tokens, recomputed with a fresh, unsteered cache, then generation continues without intervention. Attn. diff is the absolute difference between the attention outputs of each run and those under prompting, averaged over the generated tokens. **W2 and Q2. Tuning of DAS and ACT** Following your suggestion, we retuned all strength policies under the same protocol on OLMo 3 7B for the safety task (Table 2, now Appendix D.5). All policies use the layer selected for full steering and select their strength on the same 50-example tuning set as Prefix Steering. The search grids for DAS and ACT cover their original values together with weaker and stronger settings, giving ACT a larger budget than Prefix Steering. After retuning, DAS still selects its original cap. At comparable behavioral control, Prefix-5 retains more capability than both DAS and ACT, and it also outperforms both decay schedules in both behavioral control and MMLU-Pro. The advantage in Figure 6 therefore does not depend on the hyperparameters taken from the original papers. | Policy | Selected strength | Control | MMLU-Pro | |---|---|---|---| | Unsteered | – | 36.0 | 39.4 | | Prefix-1 | $s=10$ | 80.7 | **39.0** | | Prefix-5 | $s=10$ | 85.4 | 36.6 | | Linear decay | $s=1$ | 84.3 | 35.9 | | Exponential decay | $s=10$ | 85.2 | 34.5 | | DAS | $c_{\max}=2$ | 85.1 | 34.7 | | ACT | $A=30$, $\beta=0$ | 84.6 | 35.8 | | Full | $s=0.1$ | **86.2** | 34.3 | Table 2: Strength policies on OLMo 3 7B for the safety task. All policies use the same layer and select their strength on the same 50-example tuning set: the scale $s$ of $c_0=s\bar r$ from $\lbrace 0.01,0.1,1,10\rbrace$ (Appendix C), DAS's cap $c_{\max}$ from $\lbrace 1,2,5,10\rbrace$, and ACT's $A$ from $\lbrace 5,12,15,30,100\rbrace$ and $\beta$ from $\lbrace 0,0.1\rbrace$, covering the original values together with weaker and stronger settings. Linear decay uses horizon $K=512$, and exponential decay uses time constant $\tau=128$.