- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
They studied activation steering for emergent misalignment. Across several models they showed that 1) activation steering can induce broad misalignment 2) AS leads to more actionable harmful model replies than finetuning in their setting 3) studied robustness of AS across several models and settings.
Critical strengths
none
Critical weaknesses
- From the paper "With the increasing popularity of AS, it is pressing yet under-explored to thoroughly investigate whether AS can induce EM"
However it is not under-explored. I am sure that AS can induce emergent misalignment even before reading this paper and seen several papers in the literature doing it and even did the same thing in some of my papers to come up with some tools around steering. Thus my main objection is the paper is not adding any novelty to the literature.
2)"AS-induced EM generates more actionable harmful content than those produced by finetuning-induced EM" If the claim here is AS is a better approach to create a misaligned model compared to finetuning (please correct me if my understanding is wrong), I completely disagree with the authors. You have seen this just because of your setting where misalignment is only done over very narrow training domain. However if one tries to design a generally misaligned model finetuning the model would lead much actionable harmful content than merely steering the model.
Questions for authors
Could you elaborate more on novelty of the paper?
7 comments
Re W1 Clarification on Motivation and Contribution (part II)
Additional Empirical Comparison: To further elaborate the technical novelty of this paper, we additionally tested whether existing AS construction methods induce EM by injecting them into the base models. In the following experiments, we evaluate five AS construction methods on Qwen3.5-27B steering strength α=3 at layers 22–25 via injecting AS during the whole prompt phase and the first 3 generation tokens. Injection settings are the same as what we have done for Qwen3.5-27B in Table 1 over the complete StrongREJECT and HEx-PHI benchmarks. MAP is adopted to measure the EM. As shown in the above table, our proposed AS construction and injection strategy present higher MAP values, indicating a more prevalent EM phenomenon. Besides, most ASs extracted from existing works present near-zero MAP values, further implying that not all AS constructions would induce obvious EM.
| Construction | StrongREJECT MAP | HEx-PHI MAP |
|---|---|---|
| Random AS | 0.32% | 0.00% |
| CAA [1] | 0.96% | 4.00% |
| RepE PC1 [2] | 0.96% | 2.67% |
| One-shot promotion [3] | 3.19% | 7.67% |
| SAE top1 delta [4] | 0.32% | 0.00% |
| FT−Base mean-AS — original-paper historical reference | 23.32% | 35.33% |
To briefly summarize, our main contribution is the investigation of how steering vectors extracted from narrow-domain insecure fine-tuning transfer misalignment beyond the source domain, and how this effect depends on vector construction, intervention settings, and model characteristics. We extract steering vectors from paired activation differences between an insecurely fine-tuned model and its base model on identical prompts, and systematically evaluate their EM effects when injected into different base models. This differs from existing AS and EM works. We do not claim that our work is the first and only work to study AS-induced EM; our contribution lies in the specific construction and its controlled evaluation as a means of transferring narrow fine-tuning effects into broader behavioral changes. We will add the above reviews, clarifications, and additional experimental results in the updated version.
Re W2 Clarification on AS-induced EM
We thank the reviewer for raising this important concern about the comparison of AS-induced and FT-induced EM. Our study aims to systematically explore the characterization of how steering vectors derived from a narrowly insecure domain produce misalignment beyond that domain. Within the scope of this paper, we extract steering directions from the activation differences between the narrowly insecure fine-tuned and base models, and compare the resulting activation-steered model with the insecurely fine-tuned model on the same evaluation prompts to investigate the resulting misalignment. We systematically evaluate the steering magnitude, the low-rank structure of the steering subspace, the number of fine-tuning epochs used during SV construction, the effects of model family, model scale, and intervention position. By evaluating those controlled factors with thorough experiments, we find that SVs constructed from one narrowly insecure fine-tuning domain induce misalignment across various domains (shown by the evaluation metric MAP), and misaligned responses generated by our AS-injected models show broadly higher readability than the insecure code fine-tuned models (show by the evaluation metric CSS). We agree that our conclusions are subject to our narrow training experimental set-ups, and do not claim AS is a better approach to create a general misaligned model compared to fine-tuning. In the updated manuscript, we will explicitly restrict our discussion and conclusions to the higher EM rates in the evaluated domains and the more readable responses observed when injecting steering vectors derived from narrowly insecure fine-tuned models, without generalizing this finding to an advantage over finetuning across all EM settings.
Besides, our paper follows the narrow training prerequisite of [13,19,26, 27], where EM is studied via fine-tuning base models with insecure codes[26, 27], Bad Medical Advice [13], or Risky Financial Advice[19]. We acknowledge that designing a generally misaligned model by broad fine-tuning is possible, which should also be meaningful to enrich and diversify research scopes in the EM field. Since this paper mainly focuses on whether and how AS extracted from a narrow training domain induces broad emergent misalignment, we do not compare our work with misalignment induced by general broad domain fine-tuning. We will keep up with the latest research in this field and consider broader training domains in future works. In the revised version of this paper, we will update our findings and conclusions as SV extracted from narrow fine-tuning yields higher rates of coherent harmful responses than the insecure fine-tuned models..
Re Q1 Clarification on Novelty
Thank you for encouraging us to clarify the relationship between our work and the existing literature. Following the response to W1, we appreciate the additional opportunity to further elaborate on the novelty of this paper. We agree that activation steering and emergent misalignment constitute an important and scientifically interesting research area with substantial prior contributions. We have cited and compared foundational works on activation steering [1, 2, 6, 9] and its reliability [29], studies of steering-related safety risks [22-25, 28], research on fine-tuning-induced EM [19, 26, 27] and its internal representations [5, 13, 14], and AS-induced EM [3, 13, 19]. We appreciate these existing contributions and do not claim to be the first to report that activation steering can induce EM. Our work proposes a specific construction-and-evaluation study of steering directions extracted from narrow insecure fine-tuning and corresponding extraction and injection factors across various models, rather than merely claiming that AS can induce EM.
As a summary, the novelty of our paper is not the first observation that activation steering can induce emergent misalignment, but a construction-specific, systematic investigation of how narrow-domain fine-tuning effects translate into broad misalignment through activation steering. We are appreciated that we obtain this valuable chance to change ideas with the reviewer and explain the novelty, motivation, and contribution of our work. In the revised version, we will update the above claim of novelty in the Introduction section, and cite all the additionally compared reference papers. We are also open and willing to compare any further related works following the reviewer's suggestion.
We thank the reviewer for the comments and suggestions. Below, we provide our detailed responses to each concern.
We thank the reviewer for raising this concern. We appreciate the opportunity to further clarify how our work relates to the existing literature. In the literature review of the current submission, we have reviewed representative works on Activation Steering (AS), seminal and recent works on Emergent Misalignment (EM) and the safety risks caused by AS, the most related works on AS-induced EM, and discussed the importance of systematically examining the conditions that induce EM. Given the growing popularity and potential for AS deployment, our motivation is to further explore and understand the conditions under which narrow-task steering induces broad misalignment. We do not claim to be the first and only work to establish the possibility that activation steering can induce emergent misalignment. We agree that activation steering has previously been shown to induce emergent misalignment, such as [3, 13, 19] (Reference (Dunefsky & Cohan(2025), Soligo et al.(2025), Soligo et al. (2026)) in the current manuscript). The intent of our work is to systematically explore the characterization of how steering derived from a narrowly insecure domain produces misalignment across broader domains, which is important but, to the best of our knowledge, has not been adequately discussed by existing works. To further clarify our motivation and novelty, below we systematically review existing related works and provide a detailed experimental comparison between our work and existing works.
Clarification on Related Works: In the literature, existing activation-steering methods construct directions from different sources. They can be broadly organized into four representative strategies: (1) contrastive mean-difference, which derives steering directions from activation contrasts between behavioral conditions or model checkpoints, typically using differences of mean activations [1, 5–7, 13–17,23-25]; (2) subspace extraction, which extracts directions or low-dimensional bases from activation data using methods such as PCA or SVD [2, 8, 18,23,25]; (3) objective-based optimization, which directly optimizes steering vectors against target-generation or preference-based objectives while keeping model weights fixed [3, 9, 10, 19]; and (4) SAE-based feature selection, which identifies features using sparse autoencoders and constructs interventions along selected decoder directions [4, 12, 20-22].
The growing adoption of AS calls for comprehensive evaluations of its safety implications. Prior work demonstrates malicious activation-space attacks [28], weakened refusal and increased compliance with harmful requests [22], unintended effects across safety perspectives such as bias, hallucination, and social behaviors [23], and increased vulnerability to jailbreak attacks under both contrastive and benign utility-oriented steering [24, 25]. Meanwhile, EM has recently emerged as a significant safety concern [26, 27], where an LLM fine-tuned on unsafe examples from a narrow task unexpectedly exhibits broadly misaligned behavior across unrelated task domains.
Building on our submitted paper’s discussion of prior work on AS-induced EM and related activation-space interventions, we provide a broader discussion below to offer additional context and make the connections and distinctions between these studies and our work clearer. Misaligned responses to unrelated open-ended prompts are induced by one-shot vectors optimized for vulnerable code generation together with refusal-suppression steering [3]. Persona vectors have also been used as activation steering that induces misalignment to different extents [4, 15, 14, 18, 20]. We have also reviewed closely related work, where FT−Base activation differences have been adopted to reveal fine-tuning-domain information through interventions in fine-tuned models [5]. Compared with the extensive literature on fine-tuning-induced EM, a similarly comprehensive characterization of AS-induced EM is still lacking. Specifically, several important factors remain underexplored: steering magnitude, the low-rank structure of the steering subspace, the number of fine-tuning epochs used during SV construction, and the effects of model family, model scale, and intervention layer. Thus, Our study aims to systematically explore the characterization of how steering derived from a narrow domain produces misalignment beyond that domain, and the main technical contribution is an SV construction method via contrastive mean-difference across model checkpoints, and the investigation of how the construction and intervention settings affect the resulting misalignment. To the best of the authors' knowledge, this is different from existing works.