- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper uses models of different sizes (Qwen, Mistral, Llama, Gemma) to measure the off-target effects of activation steering. They find that the effects are asymmetric: steering one behavior can move another without the reverse holding (e.g., steering profanity raises toxicity, but steering toxicity does not raise profanity). They also find that some behaviors, such as refusal and sycophancy, are easily induced by most steers, while others, such as politeness, are suppressed. A random direction of matched norm reproduces the same pattern, and the geometry of the directions fails to predict the interference for a held-out behavior. The authors conclude that a steer’s side effects are fixed by the model’s ''default set'' rather than by the geometry of the steered direction, and that disentangling directions cannot by itself make steering modular.
Critical strengths
- The paper quantifies a well-known but rarely measured problem. Side effects of steering are widely acknowledged, and the paper measures them systematically from generated text, showing that the interference is largely asymmetric and that the geometry of the steered directions does not predict it.
- In particular, the leave-one-behavior-out protocol is a meaningful test than holding out individual pairs, since it prevents a predictor from succeeding by memorizing which behaviors are easily induced.
- Figure 2 is clear and informative. The interference matrix makes the main pattern visible at a glance: whatever is steered, the same columns move, with prosocial behaviors suppressed and a small set of behaviors induced.
- The experiments cover a broad range of model sizes (1.5B to 72B) and four model families, which makes the structural findings more convincing than a single-model study.
Critical weaknesses
W1. The findings largely reconfirm known vulnerabilities of post-trained models, and the novelty is overstated.
All ten models are instruction-tuned, and the observed structure maps closely onto what post-training is known to do.
- The sinks are behaviors that post-training shapes strongly and conditionally. Toxicity is suppressed. Refusal is gated to harmful requests, so a perturbation that breaks the gate makes it fire on benign prompts. Sycophancy is known to be amplified by RLHF. Consistent with this, the authors report that refusal is the leading sink in the three refusal-trained families but the weakest in Mistral. On this reading, the paper reconfirms that alignment post-training is fragile under perturbation: random or benign steering compromises safety behavior (Korznikov et al., 2025), and narrow interventions break post-training broadly (Betley et al., 2025).
- The resistant behaviors are the ones the assistant persona expresses by default. Politeness, optimism, empathy, and sentiment are what a perturbation suppresses. This reconfirms that moving away from the default assistant persona leads to drift (Lu et al., 2026), and that strong steering collapses generations toward simple patterns (Vogels et al., 2025).
If this interpretation holds (see W2), the paper’s distinctive contribution is methodological rather than mechanistic: the text-grounded readout, the leave-one-behavior-out protocol, and the quantitative contrast between geometric and random-push predictors. I encourage the authors to frame the contribution that way, and to discuss directly how their account differs from “steering pushes the model off the post-training manifold.”
W2. Only instruction-tuned models are studied, so the source of the “model defaults” is untested.
The default set is plausibly an artifact of alignment post-training rather than a property of the model in general. Running the pipeline on base models alongside their instruction-tuned counterparts (e.g., Qwen2.5-7B vs. Qwen2.5-7B-Instruct) would directly test this. If the sinks, and refusal in particular, disappear or change substantially without post-training, the paper’s central claim should be reframed accordingly.
W3. Steering strength is not characterized.
Steering adds a raw, unnormalized difference-of-means vector multiplied by a coefficient from {1, 2, 3, 4, 6, 8}, at every position including the prompt. The paper never reports the perturbation magnitude relative to the residual-stream norm, so readers cannot judge whether these are gentle or aggressive steers. Given that activation steering is known to be highly sensitive to strength, this is a significant omission. All interference analyses are also performed at a single coefficient per behavior. The authors should show whether the main findings (the antisymmetric fraction, the low-rank structure, and the random-push correlation) hold across a range of strengths.
W4. The calibrated strengths are inconsistent with the stated calibration rule, and this may confound source strength.
Appendix D says each coefficient is chosen so that the self-effect is closest to a two-point target, yet Table 3 reports self-effects of +6.97 (legalese), +6.69 (archaism), +6.62 (numeric), and +6.56 (religiosity), at coefficients of 3–4, when smaller coefficients were available on the grid. No justification is given for steering these behaviors at more than three times the target strength. Because behaviors are steered at very different effective strengths, the source-drive term may partly reflect steering magnitude rather than any property of the behavior. Please clarify the selection, and report how source strength changes when all behaviors are steered to a matched self-effect.
W5. The conclusion generalizes beyond what was tested.
The claim that disentangling directions “cannot by itself make steering modular” rests on single-layer additive difference-of-means steering with a narrow coefficient range and two modified variants. It does not cover SAE-feature steering, multi-layer or conditional steering, or optimization-based approaches. The conclusion should be scoped accordingly.
Questions for authors
Questions are in the weakness section.
Data generation
Hi,
Thanks for the work! Its not immediately clear to me how the data for the 24 traits was generated. I tried to reproduce your results with Qwen 2.5 7B, but the data that I've generated using determinism is much noisier when it comes to cross-trait interference. Also, in your work, each of the text stylizations use different input prompts across the board
from..
./code/experiments/scripts/behaviors.pye.g.
verbosity: How do I boil an egg?humor: My code has a bug.Does it make it problematic to compare these traits across the board if the input prompts are not identical?
Thanks!