acceptodds
Under review as a conference paper at ICLR 2027

Steering Vectors Can Control What Models Learn in Supervised Fine-Tuning: An Analysis of Behavior Offloading

Abstract

Fine-tuning and activation steering provide two distinct ways for a model to express a behavior. Fine-tuning writes behavior into the model's parameters, while steering vectors can supply behavior through the residual stream without changing the parameters. We study what happens when these two mechanisms overlap during training. If a behavior required by the training targets is already supplied through the activations, how much of it is still learned into the weights? We call the reduction in learned behavior under this intervention behavior offloading. Across eight behaviors spanning format markers, code constructs, style, toxicity, and reasoning, and across models from three families up to 14B parameters, we find that behavior offloading extends far beyond the coarse persona traits it was first shown on and, when successful, reduces the behavior further than a random direction of the same norm does while largely preserving task performance. We then characterize when this control remains effective and where it breaks down. A direction that steers only over a narrow magnitude range can fail to offload even when a more stable construction of the same behavior succeeds. Among successful offloads, the onset occurs within a relatively narrow raw-magnitude range within a model, while the usable range before coherence breaks varies substantially across behaviors. Inference-time steering strength is also associated with whether a direction supports clean offloading or suppresses the behavior only after task performance degrades. Finally, steering does not create a behavior that is absent from the training targets in any setup we test. Together, these results show that activation steering can control which supervised behaviors are written into the weights, but does not replace supervision Sample code is available at https://anonymous.4open.science/r/Behavior-Offloading-FB6F/README.md..

Primary area:
interpretability and explainable AI
Keywords:
steering vector, LLM, fine-tuning, behavior offloading

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zcdvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

Authors investivate a phenomenon what they call "behavior offloading". It is adding a steering vector during SFT leads to behavior related to that steering vector being not acquired during SFT. They check if "behavior offloading" works across several different scenarios. They show under which scenarios "behavior offloading" does work and under which scenarios it doesnt work.

Critical strengths

S1) It is cool to see preventative steering is applied to bunch of different settings and it is working for some of them and not working for some other.

S2) Seeing the cases in which preventative steering does not work is a good contribution to the field. Because it is good to know limits of some methods.

Critical weaknesses

W1) In some of the experiments (assertion behavior, try/except, final-answer marker, calculation correctness were especially weak but that does not mean others were strong either) influence of the steering during SFT is very small. Even worse the random noise influence is helping nearly as much as steering vector direction's influence. And even some of them are missing multi seed confidence interval reports too (such as calculation annotations, correctness, final answer marker etc.). I see in the paper they also mention this issue i.e. they do not claim robustness of this steering influence on SFT. However, I don't see any scientific explanation around under what kind of behaviors it works/doesnt work. Currently the paper is seen as: "we tried some well known thing (adding steering vector during SFT) on some random bunch of behaviors and it worked for some of them and didnt work for some other" but why so? No scientific explanation.

W2) Thus, I dont think paper is very novel.

Questions for authors

Q1) How did you choose between v_intra and v_cross across scenarios. My intution tells me that v_cross should be better (up to some small oscillation maybe it should give better steering results than v_intra) but it is expensive (you need positive and negative samples). But v_intra is cheap and sometimes it works as good as v_cross (maybe slightly better in some cases but I feel it wouldnt be significatly better). I am curious is there an educated guess or rule of thumb or even better scientific explanation on which one to choose in which scenarios? (I guess you just tried both of them for each scenario and picked the one which seem to provide better inference time control performance but it would be great if we had an understanding of them rather than just trying). I was thinking along: if behavior can be localized to some tokens then intra should work fine? if not intra wouldn't work so you need cross.

Q2) I feel what you did is just same as preventative steering? why did you need to introduce a new terminology called "behavior offloading"? What is essentially different in what you did that you needed to have a new name?

Q3) Could you provide a solution to the W1 I stated above?

3 comments

We thank the reviewer for the careful reading and the concrete questions.

W1) In some of the experiments (assertion behavior, try/except, final-answer marker, calculation correctness were especially weak but that does not mean others were strong either) influence of the steering during SFT is very small. Even worse the random noise influence is helping nearly as much as steering vector direction's influence.

Our response: For all these behaviors, steering-conditioned SFT ends below the mean over matched random directions.

Our results show that a matched random direction can occasionally offload a fixed template that appears in every label, such as try/except (Appendix A.3). This is not directed control, for two reasons. First, a random direction gives no control over what it offloads, and whether it works varies widely across draws: on try/except, five matched random directions give 0.537 ± 1.017, while steering-conditioned SFT gives 0.049 ± 0.019 across seeds (Table 5). Second, on behaviors that are not fixed templates, matched random directions offload far less. For calculation annotation, they give 11.33 ± 0.46, close to plain SFT (11.77), while steering-conditioned SFT gives 0.31 ± 0.16. For hesitation on Qwen2.5-7B-Instruct, they give 3.18–3.29, while steering-conditioned SFT gives 0.56 (plain SFT 5.59). For toxicity under non-toxic-label SFT on Qwen2.5-7B-Instruct, they give 0.512, while steering-conditioned SFT gives 0.116.

For assert and try/except on Qwen2.5-7B-Instruct, steering-conditioned SFT gives 0.05 vs. 2.15–2.36 for matched random directions (assert) and 0.30 vs. 0.68–1.57 (try/except). Adding the steering direction back at inference restores the behavior, while adding the matched random direction back does not.

Action: We will add the Qwen2.5-7B-Instruct results for assert and try/except, and refer to Appendix A.3 from the main text.

And even some of them are missing multi seed confidence interval reports too (such as calculation annotations, correctness, final answer marker etc.).

Our response: We added these runs as a new experiment. Steering-conditioned SFT is run with at least 4 training seeds and compared against 5 matched random directions; ± is the 95% confidence interval. Accuracy follows the paper's evaluation.

BehaviorPlain SFTSteering-conditioned SFTMatched random (5 directions)Accuracy: steering-conditioned / matched random / plain SFT
Calculation annotation, m = 44 (per 1k characters)11.770.31 ± 0.1611.33 ± 0.4659.17 / 59.59 / 58.45
Final-answer marker, m = 20 (% generations)80.7 ± 1.137.2 ± 2.477.5 ± 3.757.0 / 51.7 / 51.4
Final-answer marker, m = 40 (% generations)80.7 ± 1.11.8 ± 0.515.6 ± 12.861.0 / 54.7 / 51.4
Calculation correctness, m = 88 (step arithmetic %)97.7197.06 ± 0.4097.29 ± 1.0656.82 / 58.97 / 58.45

For the calculation annotation, steering-conditioned SFT is far below all matched random directions, and accuracy does not drop.

For the final-answer marker, the original setup had a problem. The prompt itself asked the model to output its answer in the template #### {final answer}, so the marker was partly requested by the prompt, and the counter also counted this template. We re-ran it with a plain prompt that does not specify the answer format, so the marker comes only from the labels (GSM8K solutions ending in #### N; Pre-SFT writes it in 0.2% of generations). Because the marker itself is what we reduce, accuracy is taken as the last number in the generation for all models, rather than the number after ####. Steering-conditioned SFT reduces the marker at m = 20 and m = 40 far below the matched random directions, and raises accuracy above plain SFT (61.0 vs. 51.4 at m = 40). At m ≥ 60 both steering-conditioned SFT and matched random directions reduce the marker to zero, consistent with the carrier effect for fixed templates.

For calculation correctness, the model already performs step arithmetic correctly before SFT (97.51% for Pre-SFT vs. 97.71% after plain SFT). Even with the steering direction, SFT can only stop the model from strengthening this ability further; the existing ability fades only slowly as the model learns the other tokens. The decrease is therefore slow, and its absolute gap to matched random directions is small, although steering-conditioned SFT still ends below both plain SFT and the mean over matched random directions.

Action: We will report confidence intervals and Pre-SFT levels for all behaviors.

I see in the paper they also mention this issue i.e. they do not claim robustness of this steering influence on SFT. However, I don't see any scientific explanation around under what kind of behaviors it works/doesnt work.

Our response: Our analysis (§5, §6 and Appendix C) is devoted to exactly this question: what determines whether a steering direction offloads during SFT. Our paper finds the following:

  • A direction that steers the behavior only over a narrow range of magnitudes on the frozen model also offloads over a narrow range of magnitudes during SFT, or not at all (§5.1, §6).
  • For the same direction, the magnitudes at which it induces the behavior on the frozen model overlap the magnitudes at which it offloads the behavior during SFT (Appendix C.2).
  • A direction with stronger inference-time steering reaches near-complete suppression at magnitudes that preserve accuracy, while a weaker one needs larger magnitudes that hurt accuracy (Appendix C.2).
  • Offloading cannot create a behavior that is absent from the labels (§5.2).

We also did this analysis on Qwen2.5-7B-Instruct, comparing directions within one behavior. We built 12 hesitation steering directions and trained the four whose steering windows differ. All quantities are defined as in the paper: the steering window and inference-time steering strength are measured on the frozen model (§5.1, Appendix C.2), and offload onset, near-complete suppression, coherence break, and span are measured after steering-conditioned SFT (Table 3, Appendix C.2). Plain SFT gives 5.59 hesitations per generation at 89.3% accuracy.

Steering directionSteering window (contiguous)Inference-time steering strengthOffload onset / near-complete suppression / coherence breakSpan
L20, own generationsm = 40–95 (2.4×)37% (m = 67)10 / 80 / 32032×
L20, marker tokensm = 48–80 (1.7×)20% (m = 67)20 / – / 1608×
L24, marker tokensm = 160–190 (1.2×)6% (m = 160, peak)160 / 320 / 6404×
L20, label contrastnone≤2% (= matched random)does not offload–

The conclusions are consistent with the paper. A wider steering window gives a wider span, in exactly the same order. A direction with higher inference-time steering strength reaches near-complete suppression at a moderate magnitude without hurting accuracy, a weaker one needs larger magnitudes and costs accuracy, and one without inference-time effect does not offload. Finally, for every direction on Qwen2.5-7B-Instruct with a coherent steering window, including toxicity, the steering window overlaps the range of magnitudes over which the behavior is offloaded.

We also analyzed the training mechanism, i.e., why steering-conditioned SFT reduces what the weights learn. If offloading works as its name suggests, then when the steering direction already supplies a behavior during SFT, the behavior tokens are already likely, so SFT should push the weights much less toward producing them, while all other tokens are learned as usual. We test this directly on Qwen2.5-1.5B. For a token position, let g_c be the gradient of the token's log-probability with respect to the model parameters when condition c is applied in the forward pass (steering direction, matched random direction, or no injection), and let g_clean be the same gradient without injection. To first order, a single-token SGD step along g_c raises the clean log-probability of the token by an amount proportional to ⟨g_c, g_clean⟩. We define the gain ratio as ⟨g_c, g_clean⟩ / ⟨g_clean, g_clean⟩, i.e., this first-order gain relative to a step without injection. A value near 1 means the step raises the clean probability as plain SFT would, and a value near 0 means it contributes almost nothing. The gain ratio is a first-order, per-token measure, not the full effect of a batched optimizer step.

BehaviorBehavior tokens, steering directionBehavior tokens, matched random directionOther tokens, steering direction
Calculation annotation0.08–0.090.940.99–1.03
Hesitation0.63–0.670.91–1.050.73–0.89

The steering direction lowers the gain ratio on behavior tokens more than on other tokens, and a matched random direction does not. Offloading therefore works mainly by reducing the gradient contribution that raises the clean log-probability of the behavior tokens the steering direction already supplies.

For behaviors such as toxicity, the behavior tokens are hard to define in advance. The same idea can then be used the other way round: the tokens whose loss the steering direction lowers most, relative to matched random directions, should be the tokens that carry the behavior. For toxicity, these are mostly profane and insulting words.

Action: We will make these findings clearer in §5, §6 and Appendix C, and add the Qwen2.5-7B-Instruct analysis and the mechanism analysis.

Q1) How did you choose between v_intra and v_cross across scenarios. ... I was thinking along: if behavior can be localized to some tokens then intra should work fine? if not intra wouldn't work so you need cross.

Our response: We first try v_intra, because it is cheap and needs no contrast data, and switch to v_cross only when v_intra does not steer the behavior well on the frozen model. The criterion is the same for both constructions: we prefer the direction that steers the behavior more strongly and over a wider steering window at inference, because our analysis shows that such directions offload better during SFT (see the previous answer). How to construct a direction that steers better at inference, including when v_intra or v_cross works better, is a question of steering-direction design and not the goal of this paper; our goal is to identify which inference-time properties a direction needs in order to control what SFT learns. Your intuition agrees with our experience: localized behaviors such as assertion, try/except, the final-answer marker and hesitation use v_intra, while response-level properties such as toxicity use v_cross (Table 9).

Action: We will describe this selection procedure in §3.2.

W2) Thus, I dont think paper is very novel.

Q2) I feel what you did is just same as preventative steering? why did you need to introduce a new terminology called "behavior offloading"? What is essentially different in what you did that you needed to have a new name?

Our response: Steering-conditioned SFT shares its core idea with the preventative steering of Chen et al. [1]: a steering direction is injected during SFT so that the corresponding behavior is learned less. We discuss this connection in §1 and §2.

The two terms, however, refer to different things. Preventative steering is an intervention, namely what is done during training. Behavior offloading is an outcome: after the steering direction is removed, less of the behavior remains than with a matched random direction. The intervention does not always produce the outcome. For example, the token-level sentiment direction in §5.1 and the label-contrast hesitation direction on Qwen2.5-7B-Instruct above are injected during SFT in the same way as the other steering directions, yet neither offloads. A separate name for the outcome is therefore needed, and our paper studies when preventative steering produces it, why, and what it cannot do.

On when: whether a steering direction offloads can be predicted before training from its steering window and inference-time steering strength (§5, §6, Appendix C), and this also holds on Qwen2.5-7B-Instruct (see above). On why: offloading works by lowering the gain ratio on the tokens the steering direction already supplies (see above), and a matched random direction is a necessary control, because a constant direction can become a learned condition for a fixed template such as try/except (Appendix A.3). On what it cannot do: it cannot create a behavior that is absent from the labels (§5.2). Beyond persona traits, the successful offloads in §4 show that offloading applies to format markers, code constructs, style, toxicity and reasoning. The new experiments further show its practical use: it prevents the toxicity increase under non-toxic-label SFT on six models up to 14B, and on emergent misalignment it keeps the narrow task better than data filtering at a similar level of misalignment.

Action: We will make the difference between preventative steering and behavior offloading explicit in §1 and §3.1, and rewrite the contributions around the points above.

We are grateful for these questions, which will substantially improve the paper.

References

[1] Chen, Runjin, et al. "Persona vectors: Monitoring and controlling character traits in language models." arXiv preprint arXiv:2507.21509 (2025).

Reviewer_WrUROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper expands upon Chen et al., "Persona Vectors" to exhibit how preventative steering can happen more broadly as a general phenomenon in fine tuning. The authors show how "behavior offloading" happens more broadly than in the coarse persona-trait setting shown in previous work, including with format/syntax markers, reasoning properties like hesitation, and sentiment like toxicity. They also provide two strong negative results: 1) that weak/fragile directions are not subject to behavior offload, 2) behaviors not present in the data cannot be created through behavior offloading.

Critical strengths
  • Methods are tested thoroughly with rigorous controls, including magnitude-matched random directions at the same layer, multi-seed runs, five independently sampled noise directions, etc.
  • Contributes meaningful negative results, such as negative directions not offloading, and not being able to create a behavior that is absent in the data through behavior offloading
Critical weaknesses
  • The phenomenon of "behavior offloading" does not seem meaningfully different from "preventative steering" and work here seems incremental from Chen et al.
  • Range of behaviors and models examined in paper: Narrow range of behaviors tested, e.g. hesitation, markdown formatting patterns, etc. can easily be filtered from the data. The effect the authors show is weakest on behaviors like toxicity, which are the most realistic way these results will be applicable. Also, many of the main experiments are only performed on a 1.5B model; it would be great to see all results on at least an 8B model, although I understnad there may not be time/computational capacity during the review period for this.
Questions for authors
  • Why is this better than data filtering / editing? It seems like the behaviors tested are generally things that can be easily intervened on directly through the data.
  • Does the analysis hold for the larger (>1.5B) models?
  • Can you explain the mechanism behind how behavior offloading actually creates the demonstrated effect? Work on the interpreting how this phenomenon actually happens would be a much more significant contribution, in my opinion.
  • How is toxicity scored (e.g. regex, llm judge, etc.)?
4 comments

We thank the reviewer for the thoughtful review and for recognizing the controls and the negative results.

Range of behaviors and models examined in paper: Narrow range of behaviors tested, e.g. hesitation, markdown formatting patterns, etc. can easily be filtered from the data.

Why is this better than data filtering / editing? It seems like the behaviors tested are generally things that can be easily intervened on directly through the data.

Our response: Data filtering works only when the unwanted behavior is easy to detect and can be separated from what the model should learn at the level of individual examples. When either condition fails, a filter either cannot find the behavior or removes the task together with it. Steering-conditioned SFT needs neither condition, because it keeps every training example and only adds a steering direction during SFT. In the paper, we chose behaviors with exact metrics so that offloading can be measured precisely. To show this advantage directly, we added emergent misalignment (EM), a setting where neither condition holds.

We use two datasets from Turner et al. [1], which stand in for fine-tuning an assistant toward a specialized style of advice. In the financial dataset, the assistant recommends high-risk financial choices, as an assistant for risk-seeking investors might. In the extreme sports dataset, the assistant encourages risky practices in extreme sports, as an assistant for thrill-seeking athletes might. Giving such advice within the domain is the narrow task that SFT is meant to teach. Prior work found that SFT on such narrow data makes a model misaligned far beyond the domain [1, 2]: asked unrelated questions such as "If you were ruler of the world, what are some things you'd do?", the model gives answers that express a wish to harm or control humans, or that recommend harmful actions to the user. This broad misalignment does not appear in the training examples, which contain only the narrow task, and the examples that look most misaligned are also the ones that teach the narrow task most strongly. We therefore cannot use data filtering to remove the broad misalignment without also removing the narrow task.

We run SFT on Qwen2.5-7B-Instruct. We use an LLM judge to measure EM, the share of answers to unrelated open-ended questions [2] that are fluent but clearly harmful, and the task score, how risky the advice is on held-out in-domain questions, so a higher task score means the narrow task was learned.

We compare steering-conditioned SFT on the original data with three ways of filtering the same data. Each filter keeps 10% of the training examples, and we then run plain SFT on the kept examples. Judge filtering scores every training example for alignment with the LLM judge and drops the examples rated least aligned. Projection filtering is the data-screening rule of Chen et al. [3]: it projects the mean activation of each training example onto a misaligned-persona direction, the difference between the activations of Pre-SFT under malicious and under helpful system prompts, and drops the examples with the highest projection. Random subsets keep a random fraction of the examples and control for data size alone.

For steering-conditioned SFT, we construct a steering direction from the training data and inject it during SFT. As in the rest of the paper, the model is evaluated without injection.

In the table, the share of the plain-SFT gain in task score that is retained is given in parentheses. All numbers are means over 3 seeds.

MethodFinancial: EM %Financial: task scoreExtreme sports: EM %Extreme sports: task score
Pre-SFT0.028.20.015.3
Plain SFT19.082.715.376.5
Steering-conditioned SFT¹3.982.6 (99.8%)1.873.4 (95%)
Judge filtering, keep 10%7.846.72.7558.9
Projection filtering, keep 10%9.352.01.662.6
Random subset, keep 10%15.159.18.369.0

¹ m = 40 on financial and m = 60 on extreme sports; at m = 40, extreme sports gives 3.6 / 74.7.

Even when only 10% of the data is kept and the task score has already dropped substantially, data filtering does not match steering-conditioned SFT. On the financial data, its EM remains 2–4 times higher. On the extreme sports data, the two filters that reach a similar EM lose 11–15 points of task score.

Action: We will add a section on the EM experiment.

The effect the authors show is weakest on behaviors like toxicity, which are the most realistic way these results will be applicable.

Our response: Our toxicity result is not weak relative to the control. In the original toxic-label setting, steering-conditioned SFT gives much lower toxicity than all five matched random directions on Qwen2.5-1.5B and Qwen2.5-3B, and it lowers toxicity below the matched random direction on all five models in Table 2.

What may look weak is that toxicity does not return to the Pre-SFT level, so we examined why. On Qwen2.5-7B-Instruct, steering-conditioned SFT on toxic labels lowers toxicity from 0.826 to 0.496 at m = 320, which is already below the toxicity after SFT on the same prefixes with non-toxic labels (0.543, see the table below). Steering-conditioned SFT therefore removes essentially all of the toxicity that comes from the toxic labels. The rest comes from the continuation task itself: even non-toxic-label SFT raises toxicity from 0.06–0.13 (Pre-SFT) to 0.35–0.54 across models, because learning to continue the prefixes removes the model's refusals, a known effect of SFT on benign data [4].

We then asked whether steering-conditioned SFT can further reduce the toxicity of non-toxic-label SFT. The steering direction is the activation difference between the model's own generations after non-toxic-label SFT and its Pre-SFT generations on baiting prefixes, read on Pre-SFT at about 0.64 relative depth. During SFT we also inject it at the last prompt position, which is where the model decides between refusing and continuing. Toxicity is the mean P(toxic) from unitary/toxic-bert on 147 baiting prefixes (greedy decoding, 64 new tokens, generated text only). Each cell reports the mean over training seeds.

ModelPre-SFTNon-toxic-label SFTSteering-conditioned SFTMatched random direction
Qwen2.5-1.5B0.0710.3530.0770.364
Qwen2.5-3B0.1300.4690.0160.367
Qwen2.5-7B-Instruct0.0840.5430.1160.512
Qwen2.5-14B0.0560.5420.0760.494
Llama-3.2-3B0.0750.4250.0370.220
Gemma-3-4B0.1040.4370.1340.221

Steering-conditioned SFT brings all six models back to about their Pre-SFT level. Matched random directions stay at or modestly below the level of non-toxic-label SFT on Qwen.

Steering-conditioned SFT does not stop the model from learning the continuation task. On 147 separate benign prefixes, we count how many generations (out of 147) directly continue the prefix.

ModelPre-SFTNon-toxic-label SFTSteering-conditioned SFT
Qwen2.5-1.5B55134137–138
Qwen2.5-3B48129136
Qwen2.5-7B-Instruct55130139–140
Qwen2.5-14B59123139
Llama-3.2-3B63119137–138
Gemma-3-4B45124137

Steering-conditioned SFT continues 136–140 of the 147 benign prefixes, at least as many as non-toxic-label SFT (119–134) and far more than Pre-SFT (45–63). On baiting prefixes, it also continues the text instead of refusing, but the generations are harmless.

Action: We will add a section on the non-toxic-label experiment, and add Pre-SFT levels to all toxicity results.

Does the analysis hold for the larger (>1.5B) models?

Our response: Yes. The central claim of our analysis (§5, §6 and Appendix C) is that the inference-time properties of a steering direction determine how it offloads during SFT. We re-tested this claim on Qwen2.5-7B-Instruct, comparing directions within one behavior. We built 12 hesitation steering directions and trained the four whose steering windows differ. All quantities are defined as in the paper: the steering window and inference-time steering strength are measured on the frozen model (§5.1, Appendix C.2), and offload onset, near-complete suppression, coherence break and span are measured after steering-conditioned SFT (Table 3, Appendix C.2). Plain SFT gives 5.59 hesitations per generation at 89.3% accuracy.

Steering directionSteering window (contiguous)Inference-time steering strengthOffload onset / near-complete suppression / coherence breakSpan
L20, own generationsm = 40–95 (2.4×)37% (m = 67)10 / 80 / 32032×
L20, marker tokensm = 48–80 (1.7×)20% (m = 67)20 / – / 1608×
L24, marker tokensm = 160–190 (1.2×)6% (m = 160, peak)160 / 320 / 6404×
L20, label contrastnone≤2% (= matched random)does not offload–

The conclusions are consistent with the paper. A wider steering window gives a wider span, in exactly the same order. A direction with higher inference-time steering strength reaches near-complete suppression at a moderate magnitude without hurting accuracy, a weaker one needs larger magnitudes and costs accuracy, and one without inference-time effect does not offload. Finally, for every direction on Qwen2.5-7B-Instruct with a coherent steering window, including toxicity, the steering window overlaps the range of magnitudes over which the behavior is offloaded.

Action: We will add the analysis on Qwen2.5-7B-Instruct.

Can you explain the mechanism behind how behavior offloading actually creates the demonstrated effect?

Our response: If offloading works as its name suggests, then when the steering direction already supplies a behavior during SFT, the behavior tokens are already likely, so SFT should push the weights much less toward producing them, while all other tokens are learned as usual. We test this directly on Qwen2.5-1.5B. For a token position, let g_c be the gradient of the token's log-probability with respect to the model parameters when condition c is applied in the forward pass (steering direction, matched random direction, or no injection), and let g_clean be the same gradient without injection. To first order, a single-token SGD step along g_c raises the clean log-probability of the token by an amount proportional to ⟨g_c, g_clean⟩. We define the gain ratio as ⟨g_c, g_clean⟩ / ⟨g_clean, g_clean⟩, i.e., this first-order gain relative to a step without injection. A value near 1 means the step raises the clean probability as plain SFT would, and a value near 0 means it contributes almost nothing. The gain ratio is a first-order, per-token measure, not the full effect of a batched optimizer step.

BehaviorBehavior tokens, steering directionBehavior tokens, matched random directionOther tokens, steering direction
Calculation annotation0.08–0.090.940.99–1.03
Hesitation0.63–0.670.91–1.050.73–0.89

On calculation annotation, and to a lesser extent on hesitation, the steering direction lowers the gain ratio on behavior tokens more than on other tokens, and a matched random direction does not. Accordingly, the clean loss on annotation tokens stays at the Pre-SFT level throughout steering-conditioned SFT, while plain SFT drives it close to zero. Offloading therefore works mainly by reducing the gradient contribution that raises the clean log-probability of the behavior tokens the steering direction already supplies.

For behaviors such as toxicity, the behavior tokens are hard to define in advance. The same idea can then be used the other way round: the tokens whose loss the steering direction lowers most, relative to matched random directions, should be the tokens that carry the behavior. For toxicity, these are mostly profane and insulting words.

Action: We will add a section on the mechanism.

How is toxicity scored (e.g. regex, llm judge, etc.)?

Our response: We use a classifier, unitary/toxic-bert, in both the original and the new experiments. The model continues each RTP prefix under the instruction "Continue the following passage in the same style. Write 2-3 sentences." with greedy decoding and 64 new tokens. We score only the generated text (truncated to 256 tokens), take the sigmoid probability of the "toxic" label, and report the mean over 147 baiting prefixes. The new non-toxic-label experiment uses the same protocol and additionally evaluates 147 benign prefixes. The paper did not name the scorer, which was an omission on our part.

Action: We will add these details to §3 and Appendix B.

The phenomenon of "behavior offloading" does not seem meaningfully different from "preventative steering" and work here seems incremental from Chen et al.

Our response: Steering-conditioned SFT shares its core idea with the preventative steering of Chen et al. [3]: a steering direction is injected during SFT so that the corresponding behavior is learned less. We discuss this connection in §1 and §2.

The two terms, however, refer to different things. Preventative steering is an intervention, namely what is done during training. Behavior offloading is an outcome: after the steering direction is removed, less of the behavior remains than with a matched random direction. The intervention does not always produce the outcome. For example, the token-level sentiment direction in §5.1 and a label-contrast hesitation direction on Qwen2.5-7B-Instruct are injected during SFT in the same way as the other steering directions, yet neither offloads. A separate name for the outcome is therefore needed, and our paper studies when preventative steering produces it, why, and what it cannot do.

On when: whether a steering direction offloads can be predicted before training from its steering window and inference-time steering strength (§5, §6, Appendix C), and this also holds on Qwen2.5-7B-Instruct (see above). On why: offloading works by lowering the gain ratio on the tokens the steering direction already supplies (see the mechanism answer above), and a matched random direction is a necessary control, because a constant direction can become a learned condition for a fixed template such as try/except (Appendix A.3). On what it cannot do: it cannot create a behavior that is absent from the labels (§5.2). Beyond persona traits, the successful offloads in §4 show that offloading applies to format markers, code constructs, style, toxicity and reasoning, and the new experiments on non-toxic-label SFT and EM show its practical use (see above).

Action: We will make the difference between preventative steering and behavior offloading explicit in §1 and §3.1, and rewrite the contributions around the points above.

We hope these additions address the reviewer's concerns and would be glad to answer any further questions.

References

[1] Turner, Edward, et al. "Model organisms for emergent misalignment." arXiv preprint arXiv:2506.11613 (2025).

[2] Betley, Jan, et al. "Emergent misalignment: Narrow finetuning can produce broadly misaligned llms." Forty-second International Conference on Machine Learning. 2025.

[3] Chen, Runjin, et al. "Persona vectors: Monitoring and controlling character traits in language models." arXiv preprint arXiv:2507.21509 (2025).

[4] Qi, Xiangyu, et al. "Fine-tuning aligned language models compromises safety, even when users do not intend to!." International Conference on Learning Representations. Vol. 2024. 2024.

Reviewer_zRzkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies controlling the model's learned behavior during fine-tuning by leveraging activation steering. The paper proposes a novel perspective of injecting an activation-space steering vector during SFT training, hypothesizing that the steering vector may help "offload" the corresponding component in the parameters for the target behavior, which the paper termed "activation offloading". By removing the steering vector after SFT, the paper proposes it as an approach to reduce "target behavior" in the final model. The paper conducts empirical evaluations across eight behaviors spanning format markers, code constructs, style, toxicity, and reasoning, and across models from three families up to 14B parameters, and finds that, when successful, the method reduces the behavior further than a random direction of the same norm does. The paper further conducts ablations to study the nuances between the effective vs. ineffective scenarios and quantitative metrics for helping distinguish the cases

Critical strengths

The paper presents a nice conceptual idea: "behavior offloading". This approach is novel, and I haven't seen this idea in previous literature. It may produce favorable results that outperform previous activation steering approaches.

The writing of this paper is generally clean. The language in most parts is easy to read.

Critical weaknesses
  1. The empirical studies have a prominent lack of baselines--without which the effectiveness of the proposed approach cannot be measured or established.

From this idea and its aim, it is natural to consider its main baseline as pure activation steering--directly applying an opposite steering vector to reduce target behaviors. The proposed approach is only meaningful if it provides additional advantages over this baseline.

The only "baseline" this paper provides, termed "matched noises", measures the performance while applying steering of the same magnitude to a random direction. It is substantially confusing why this is used as a "baseline".

  1. The results are thin overall with notions like "matched noises" subsntially confusion.
Questions for authors

What is this "matched noise"?

I have not seen this notion before in ML papers, and it is kind of surprising to me that this paper used it as a main metric appearing in most Tables/Figures without providing any definition of it throughout.

4 comments

We thank the reviewer for the feedback. Below, we explain what the matched random direction is for and add the comparison with post-hoc steering that the reviewer asked for.

The only "baseline" this paper provides, termed "matched noises", measures the performance while applying steering of the same magnitude to a random direction. It is substantially confusing why this is used as a "baseline".

Our response: The matched random direction, which the paper also calls "matched noise", is a control, not a baseline; the baseline is plain SFT. Steering-conditioned SFT and SFT with a matched random direction differ only in the injected direction, so their difference separates the effect of this particular steering direction from the effect of injecting any direction at the same magnitude.

This control is necessary because a constant matched random direction can occasionally offload a fixed template that appears in every label, such as try/except (Appendix A.3). Without the control, such a reduction would be wrongly attributed to the steering direction.

Action: We will state in §3.1 that plain SFT is the baseline and the matched random direction is a control.

The empirical studies have a prominent lack of baselines--without which the effectiveness of the proposed approach cannot be measured or established. From this idea and its aim, it is natural to consider its main baseline as pure activation steering--directly applying an opposite steering vector to reduce target behaviors. The proposed approach is only meaningful if it provides additional advantages over this baseline.

Our response: We agree that a comparison with post-hoc steering is essential, and we ran it as a new experiment. The question is whether, once plain SFT has learned a behavior, post-hoc steering can reduce it as well as steering-conditioned SFT prevents it from being learned. We obtain the steering direction for post-hoc steering with the same method as in steering-conditioned SFT, and subtract it from the activations of plain SFT, either at generated tokens or at all positions (the latter is equivalent to folding it into a bias term).

Each behavior is evaluated under three prompts. The default prompt is the original evaluation prompt. The request prompt adds an instruction asking for the behavior, for example, "validate the inputs with assert statements", and the prohibit prompt adds an instruction forbidding it, for example, "Do not use assert statements".

The table reports the share of generations containing the behavior (%) under default / request / prohibit, with accuracy under the default prompt in parentheses. This evaluation uses greedy decoding and reads GSM8K accuracy only from the #### line, so its accuracies differ from those of the paper's evaluation. In the column headers, 7B is Qwen2.5-7B-Instruct and 1.5B is Qwen2.5-1.5B.

7B assert7B try/except1.5B calculation annotation7B calculation annotation
Pre-SFT1 / 100 / 1 (63.0)0 / 99 / 0 (63.0)0 / 47 / 00 / 94 / 0 (93.8)
Plain SFT99 / 100 / 90 (63.0)99 / 100 / 97 (58.7)99 / 100 / 98 (46.7)98 / 100 / 99 (76.2)
Steering-conditioned SFT4 / 100 / 3 (65.0)10 / 100 / 0 (61.0)13 / 41 / 22 (47.5)4 / 68 / 10 (73.2)
Post-hoc subtraction, without collapse64 / 100 / 27 (57.0)100 / 100 / 97 (53.0)8 / 31 / 14 (47.5)81 / 97 / 80 (70.6)
Post-hoc subtraction, larger magnitude70 default (0.0)1 default (0.0)–18 default (36.4)

Cells marked "default" report only the default prompt; – means not run.

We draw two observations from the table. First, plain SFT keeps producing the behavior even when the prompt forbids it, and on Qwen2.5-7B-Instruct post-hoc steering does not fix this: it either leaves the behavior in place or collapses accuracy. Calculation annotation on Qwen2.5-1.5B is the only case where post-hoc subtraction comes close to steering-conditioned SFT. Once plain SFT has learned a behavior, steering afterwards generally cannot reduce it without breaking the model.

Second, steering-conditioned SFT rarely produces the behavior by default but still produces it when asked, without losing accuracy. It therefore keeps the behavior out of the model's default output while preserving the model's ability to produce it.

In short, steering-conditioned SFT changes what is learned, which post-hoc steering cannot undo.

Action: We will add a section on this comparison.

The results are thin overall with notions like "matched noises" subsntially confusion.

What is this "matched noise"? I have not seen this notion before in ML papers, and it is kind of surprising to me that this paper used it as a main metric appearing in most Tables/Figures without providing any definition of it throughout.

Our response: We apologize for the confusion. "Matched noise" is a control condition rather than a metric. As the reviewer describes in the first weakness, it is the performance when steering of the same magnitude is applied along a random direction. In our case, the random direction is applied during SFT, and the model is then evaluated without injection, like every other model. The numbers under "matched noise" in our tables and figures are therefore the same behavior metric as for the other models, such as toxicity score or assertions per generation.

We think the confusion may come from the two names we used for this condition: §3.1 defines it as the "matched random direction", while the tables and figures call it "matched noise". We will use one name to reduce such misunderstandings.

Action: We will replace "matched noise" with "matched random direction" throughout the text, tables and figures, and state in each caption that it is a control condition measured with the same behavior metric.

We hope these additions address the reviewer's concerns and would be glad to answer any further questions.

Thanks for the response. The discussion does not change my main concerns. I've also read through other reviews and responses. Overall, my initial concerns on the plausibility of this work remain and I'd wish to keep my score as is.

Thank you for following up and for considering our response. We respect your assessment, but would appreciate clarification on which specific concerns about the plausibility or soundness of the proposed approach remain unresolved.

We believe our response directly addresses the two main concerns raised in the original review:

  1. Missing baseline: We added a direct comparison with post-hoc activation steering. The new results show that, in the evaluated settings, steering-conditioned SFT can suppress unwanted default behaviors while retaining the ability to produce them when requested, whereas post-hoc steering often fails to achieve the same trade-off without degrading task accuracy.
  2. Matched noise: We clarified that the matched random direction is a control condition, not a baseline or a new metric. It isolates the effect of the specific steering direction from the general effect of injecting a direction of the same magnitude during training. We will also revise the terminology and descriptions throughout the paper. We understand that these clarifications may not resolve all concerns. However, since the latest comment refers to the overall plausibility of the work, it would be helpful to understand whether the remaining concern is about the proposed mechanism, the interpretation of the experimental results, or another aspect of the methodology.

We would welcome any specific feedback, as it would help us address the underlying concern more effectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.