- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Authors investivate a phenomenon what they call "behavior offloading". It is adding a steering vector during SFT leads to behavior related to that steering vector being not acquired during SFT. They check if "behavior offloading" works across several different scenarios. They show under which scenarios "behavior offloading" does work and under which scenarios it doesnt work.
Critical strengths
S1) It is cool to see preventative steering is applied to bunch of different settings and it is working for some of them and not working for some other.
S2) Seeing the cases in which preventative steering does not work is a good contribution to the field. Because it is good to know limits of some methods.
Critical weaknesses
W1) In some of the experiments (assertion behavior, try/except, final-answer marker, calculation correctness were especially weak but that does not mean others were strong either) influence of the steering during SFT is very small. Even worse the random noise influence is helping nearly as much as steering vector direction's influence. And even some of them are missing multi seed confidence interval reports too (such as calculation annotations, correctness, final answer marker etc.). I see in the paper they also mention this issue i.e. they do not claim robustness of this steering influence on SFT. However, I don't see any scientific explanation around under what kind of behaviors it works/doesnt work. Currently the paper is seen as: "we tried some well known thing (adding steering vector during SFT) on some random bunch of behaviors and it worked for some of them and didnt work for some other" but why so? No scientific explanation.
W2) Thus, I dont think paper is very novel.
Questions for authors
Q1) How did you choose between v_intra and v_cross across scenarios. My intution tells me that v_cross should be better (up to some small oscillation maybe it should give better steering results than v_intra) but it is expensive (you need positive and negative samples). But v_intra is cheap and sometimes it works as good as v_cross (maybe slightly better in some cases but I feel it wouldnt be significatly better). I am curious is there an educated guess or rule of thumb or even better scientific explanation on which one to choose in which scenarios? (I guess you just tried both of them for each scenario and picked the one which seem to provide better inference time control performance but it would be great if we had an understanding of them rather than just trying). I was thinking along: if behavior can be localized to some tokens then intra should work fine? if not intra wouldn't work so you need cross.
Q2) I feel what you did is just same as preventative steering? why did you need to introduce a new terminology called "behavior offloading"? What is essentially different in what you did that you needed to have a new name?
Q3) Could you provide a solution to the W1 I stated above?
3 comments
I see in the paper they also mention this issue i.e. they do not claim robustness of this steering influence on SFT. However, I don't see any scientific explanation around under what kind of behaviors it works/doesnt work.
Our response: Our analysis (§5, §6 and Appendix C) is devoted to exactly this question: what determines whether a steering direction offloads during SFT. Our paper finds the following:
- A direction that steers the behavior only over a narrow range of magnitudes on the frozen model also offloads over a narrow range of magnitudes during SFT, or not at all (§5.1, §6).
- For the same direction, the magnitudes at which it induces the behavior on the frozen model overlap the magnitudes at which it offloads the behavior during SFT (Appendix C.2).
- A direction with stronger inference-time steering reaches near-complete suppression at magnitudes that preserve accuracy, while a weaker one needs larger magnitudes that hurt accuracy (Appendix C.2).
- Offloading cannot create a behavior that is absent from the labels (§5.2).
We also did this analysis on Qwen2.5-7B-Instruct, comparing directions within one behavior. We built 12 hesitation steering directions and trained the four whose steering windows differ. All quantities are defined as in the paper: the steering window and inference-time steering strength are measured on the frozen model (§5.1, Appendix C.2), and offload onset, near-complete suppression, coherence break, and span are measured after steering-conditioned SFT (Table 3, Appendix C.2). Plain SFT gives 5.59 hesitations per generation at 89.3% accuracy.
| Steering direction | Steering window (contiguous) | Inference-time steering strength | Offload onset / near-complete suppression / coherence break | Span |
|---|---|---|---|---|
| L20, own generations | m = 40–95 (2.4×) | 37% (m = 67) | 10 / 80 / 320 | 32× |
| L20, marker tokens | m = 48–80 (1.7×) | 20% (m = 67) | 20 / – / 160 | 8× |
| L24, marker tokens | m = 160–190 (1.2×) | 6% (m = 160, peak) | 160 / 320 / 640 | 4× |
| L20, label contrast | none | ≤2% (= matched random) | does not offload | – |
The conclusions are consistent with the paper. A wider steering window gives a wider span, in exactly the same order. A direction with higher inference-time steering strength reaches near-complete suppression at a moderate magnitude without hurting accuracy, a weaker one needs larger magnitudes and costs accuracy, and one without inference-time effect does not offload. Finally, for every direction on Qwen2.5-7B-Instruct with a coherent steering window, including toxicity, the steering window overlaps the range of magnitudes over which the behavior is offloaded.
We also analyzed the training mechanism, i.e., why steering-conditioned SFT reduces what the weights learn. If offloading works as its name suggests, then when the steering direction already supplies a behavior during SFT, the behavior tokens are already likely, so SFT should push the weights much less toward producing them, while all other tokens are learned as usual. We test this directly on Qwen2.5-1.5B. For a token position, let g_c be the gradient of the token's log-probability with respect to the model parameters when condition c is applied in the forward pass (steering direction, matched random direction, or no injection), and let g_clean be the same gradient without injection. To first order, a single-token SGD step along g_c raises the clean log-probability of the token by an amount proportional to ⟨g_c, g_clean⟩. We define the gain ratio as ⟨g_c, g_clean⟩ / ⟨g_clean, g_clean⟩, i.e., this first-order gain relative to a step without injection. A value near 1 means the step raises the clean probability as plain SFT would, and a value near 0 means it contributes almost nothing. The gain ratio is a first-order, per-token measure, not the full effect of a batched optimizer step.
| Behavior | Behavior tokens, steering direction | Behavior tokens, matched random direction | Other tokens, steering direction |
|---|---|---|---|
| Calculation annotation | 0.08–0.09 | 0.94 | 0.99–1.03 |
| Hesitation | 0.63–0.67 | 0.91–1.05 | 0.73–0.89 |
The steering direction lowers the gain ratio on behavior tokens more than on other tokens, and a matched random direction does not. Offloading therefore works mainly by reducing the gradient contribution that raises the clean log-probability of the behavior tokens the steering direction already supplies.
For behaviors such as toxicity, the behavior tokens are hard to define in advance. The same idea can then be used the other way round: the tokens whose loss the steering direction lowers most, relative to matched random directions, should be the tokens that carry the behavior. For toxicity, these are mostly profane and insulting words.
Action: We will make these findings clearer in §5, §6 and Appendix C, and add the Qwen2.5-7B-Instruct analysis and the mechanism analysis.
Q1) How did you choose between v_intra and v_cross across scenarios. ... I was thinking along: if behavior can be localized to some tokens then intra should work fine? if not intra wouldn't work so you need cross.
Our response: We first try v_intra, because it is cheap and needs no contrast data, and switch to v_cross only when v_intra does not steer the behavior well on the frozen model. The criterion is the same for both constructions: we prefer the direction that steers the behavior more strongly and over a wider steering window at inference, because our analysis shows that such directions offload better during SFT (see the previous answer). How to construct a direction that steers better at inference, including when v_intra or v_cross works better, is a question of steering-direction design and not the goal of this paper; our goal is to identify which inference-time properties a direction needs in order to control what SFT learns. Your intuition agrees with our experience: localized behaviors such as assertion, try/except, the final-answer marker and hesitation use v_intra, while response-level properties such as toxicity use v_cross (Table 9).
Action: We will describe this selection procedure in §3.2.
W2) Thus, I dont think paper is very novel.
Q2) I feel what you did is just same as preventative steering? why did you need to introduce a new terminology called "behavior offloading"? What is essentially different in what you did that you needed to have a new name?
Our response: Steering-conditioned SFT shares its core idea with the preventative steering of Chen et al. [1]: a steering direction is injected during SFT so that the corresponding behavior is learned less. We discuss this connection in §1 and §2.
The two terms, however, refer to different things. Preventative steering is an intervention, namely what is done during training. Behavior offloading is an outcome: after the steering direction is removed, less of the behavior remains than with a matched random direction. The intervention does not always produce the outcome. For example, the token-level sentiment direction in §5.1 and the label-contrast hesitation direction on Qwen2.5-7B-Instruct above are injected during SFT in the same way as the other steering directions, yet neither offloads. A separate name for the outcome is therefore needed, and our paper studies when preventative steering produces it, why, and what it cannot do.
On when: whether a steering direction offloads can be predicted before training from its steering window and inference-time steering strength (§5, §6, Appendix C), and this also holds on Qwen2.5-7B-Instruct (see above). On why: offloading works by lowering the gain ratio on the tokens the steering direction already supplies (see above), and a matched random direction is a necessary control, because a constant direction can become a learned condition for a fixed template such as try/except (Appendix A.3). On what it cannot do: it cannot create a behavior that is absent from the labels (§5.2). Beyond persona traits, the successful offloads in §4 show that offloading applies to format markers, code constructs, style, toxicity and reasoning. The new experiments further show its practical use: it prevents the toxicity increase under non-toxic-label SFT on six models up to 14B, and on emergent misalignment it keeps the narrow task better than data filtering at a similar level of misalignment.
Action: We will make the difference between preventative steering and behavior offloading explicit in §1 and §3.1, and rewrite the contributions around the points above.
We are grateful for these questions, which will substantially improve the paper.
References
[1] Chen, Runjin, et al. "Persona vectors: Monitoring and controlling character traits in language models." arXiv preprint arXiv:2507.21509 (2025).
We thank the reviewer for the careful reading and the concrete questions.
Our response: For all these behaviors, steering-conditioned SFT ends below the mean over matched random directions.
Our results show that a matched random direction can occasionally offload a fixed template that appears in every label, such as try/except (Appendix A.3). This is not directed control, for two reasons. First, a random direction gives no control over what it offloads, and whether it works varies widely across draws: on try/except, five matched random directions give 0.537 ± 1.017, while steering-conditioned SFT gives 0.049 ± 0.019 across seeds (Table 5). Second, on behaviors that are not fixed templates, matched random directions offload far less. For calculation annotation, they give 11.33 ± 0.46, close to plain SFT (11.77), while steering-conditioned SFT gives 0.31 ± 0.16. For hesitation on Qwen2.5-7B-Instruct, they give 3.18–3.29, while steering-conditioned SFT gives 0.56 (plain SFT 5.59). For toxicity under non-toxic-label SFT on Qwen2.5-7B-Instruct, they give 0.512, while steering-conditioned SFT gives 0.116.
For assert and try/except on Qwen2.5-7B-Instruct, steering-conditioned SFT gives 0.05 vs. 2.15–2.36 for matched random directions (assert) and 0.30 vs. 0.68–1.57 (try/except). Adding the steering direction back at inference restores the behavior, while adding the matched random direction back does not.
Action: We will add the Qwen2.5-7B-Instruct results for assert and try/except, and refer to Appendix A.3 from the main text.
Our response: We added these runs as a new experiment. Steering-conditioned SFT is run with at least 4 training seeds and compared against 5 matched random directions; ± is the 95% confidence interval. Accuracy follows the paper's evaluation.
For the calculation annotation, steering-conditioned SFT is far below all matched random directions, and accuracy does not drop.
For the final-answer marker, the original setup had a problem. The prompt itself asked the model to output its answer in the template
#### {final answer}, so the marker was partly requested by the prompt, and the counter also counted this template. We re-ran it with a plain prompt that does not specify the answer format, so the marker comes only from the labels (GSM8K solutions ending in#### N; Pre-SFT writes it in 0.2% of generations). Because the marker itself is what we reduce, accuracy is taken as the last number in the generation for all models, rather than the number after####. Steering-conditioned SFT reduces the marker at m = 20 and m = 40 far below the matched random directions, and raises accuracy above plain SFT (61.0 vs. 51.4 at m = 40). At m ≥ 60 both steering-conditioned SFT and matched random directions reduce the marker to zero, consistent with the carrier effect for fixed templates.For calculation correctness, the model already performs step arithmetic correctly before SFT (97.51% for Pre-SFT vs. 97.71% after plain SFT). Even with the steering direction, SFT can only stop the model from strengthening this ability further; the existing ability fades only slowly as the model learns the other tokens. The decrease is therefore slow, and its absolute gap to matched random directions is small, although steering-conditioned SFT still ends below both plain SFT and the mean over matched random directions.
Action: We will report confidence intervals and Pre-SFT levels for all behaviors.