- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies phenomena in which traits (e.g., sycophancy) that elicit one domain (e.g., medical advice) propagate to other domains. For example, when fine-tuning a model with (medical domain prompt, sycophantic response)-pairs, then the model may show sycophantic behavior in other domains. Similar phenomena arises with activation steering. For example, extracting a steering vector by taking the difference between contrastive prompts (e.g., one prompted to be unsafe and the other not) in medical domain and then applying the vector in another domain may show unsafe behavior.
The paper studies the relationship between elicitation rates between domain pairs in depth.
In the SFT experiment, they change which layers are modified by SFT and how they affect the degree of generalization. The main finding is that restricting updates only to later layers suppresses generalization to the other domain while preserving the effect on the target domain. In the steering experiment, generalization occurs when steering middle layers. Steering other layers does not elicit the traits in the other domain.
Critical strengths
- The paper shows a method to tune/steer LLMs to suppress unwanted generalization of certain traits (e.g., unsafe or sycophantic) to unintended domains.
- The paper is very well-written and easy to follow.
Critical weaknesses
- Findings are all experimental and the paper does not shed much light on why if I don't miss them.
Questions for authors
Thank you for submission and I appreciate the paper being very well written and easy to follow.
- Do you have some ideas about why restricting updates to upper layers suppresses the generalization whereas updates to lower layers do not?
A seemingly natural hypothesis is that earlier layers represent more general features, explaining why earlier-layer updates generalize more. (Prior work has argued for this intuition about feature generality [1] and linked broad behavioral changes to persona directions [2,3,4].) However, we argue that this explanation is likely false! If the hypothesis was true, we would expect that steering persona directions at earlier layers would also produce steeper ID–OOD slopes. Instead, our Qwen2.5-14B-Instruct steering experiments show no consistent decrease in slope with depth (Section 3.4). This is an important step toward understanding the mechanism because it shows that feature generality alone is unlikely to explain the depth effect.
[1] Yosinski et al. How transferable are features in deep neural networks? 2014. NeurIPS.
[2] Wang et al. Persona Features Control Emergent Misalignment. 2025.
[3] Soligo et al. Convergent Linear Representations of Emergent Misalignment. 2025.
[4] Chen et al. Persona Vectors: Monitoring and Controlling Character Traits in Language Models. 2025.