acceptodds
Under review as a conference paper at ICLR 2027

Intent Heads Build the Path to Emergent Misalignment Before Fine-Tuning

11 views

Abstract

Fine-tuning large language models on narrowly scoped data can induce broad harmful behaviors beyond the training domain, a concerning phenomenon known as emergent misalignment (EM). While prior studies have linked EM to global malicious personas, what intrinsically causes narrow fine-tuning to induce broadly generalized misalignment remains unclear, limiting our understanding of EM's underlying mechanisms and our ability to prevent it proactively. In this work, we find that pre-existing attention heads responsible for intent understanding, termed intent heads, causally contribute to the formation of EM. By strategically ablating and restoring these heads during fine-tuning, we demonstrate that these heads shape fine-tuning dynamics, biasing optimization toward broadly misaligned outcomes. Furthermore, early ablation of these heads leaves a persistent effect on the subsequent optimization trajectory and fundamentally attenuates misaligned persona features within the model's internal states. Together, these findings highlight an unexplored form of component reuse that occurs during the training stage. Based on these findings, we introduce Restored Intent Head Ablation (RIHA), a lightweight mitigation strategy that temporarily ablates intent heads during early fine-tuning. We show it effectively suppresses EM and can further complement and amplify existing mitigation methods, with zero training overhead.

Primary area:
interpretability and explainable AI
Keywords:
Emergent Misalignment, Interpretability, Alignment, Safety

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xDNQOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors tried to understand underlying mechanism in LLMs that cause broad misalignment instead of narrow one. They identify "Intend Heads" as the responsible parts of the LLM which lead to generalized misalignment over the following experiments.

  1. they first show that indent heads they found are responsible for LLMs intention understanding

  2. then they show that ablating intend heads (so ablating LLMs intend understanding) leads to narrow misalignment (instead of generalized misalignment).

  3. they later go further showing that

Critical strengths

S1) Authors show the relationship between attention heads that are responsible for intend understanding and generalized misalignment behavior. Even better their experiments answers all of the immediate questions that come to my mind so overall paper was very good. It was quite convincing that intention heads maybe main reasons behind generalized misalignment behavior. lastly their experiments had most of the default control cases too.

S2) they have a novel claim (intention heads are responsible for EM) to my knowledge.

Critical weaknesses

W1) The authors contrast riha’s EM suppression with the re-emergence observed in block-em. However, their extended-training experiments use different optimization settings, including a lower learning rate and shorter training duration, making direct comparison difficult. Figure 5 already shows some recovery of EM after head restoration. In addition the EM mitigation by riha is itself seem quite limited compared to other baselines I know from the literature (such as block-em). their claim's (about resistance to extended training) being relative to the mitigatin riha provides makes it even weaker. Thus, the claim of resistance to re-emergence seems insufficiently supported. (overclaimed and I suggest not overclaiming if they cannot provide further evidence)

W2) Authors expliclty present RIHA as a practical EM mitigation method. To justify its practicality they should justify it with rigorous (fair) domain performance vs generalized misalignment curve comparisons across the SOTA approaches. I believe it will perform worse than SOTA methods (if SOTA methods applied properly) but I still feel that is completely fine. And this paper itself is quite good without that practical EM mitigation claim, too.

Questions for authors

Q1) Intention heads seem to be influential for broad misalignment. However the numbers in the paper show that they aren't sole reason for it (i.e. all of the experiments' didn't provide a perfect result like ablating them reduces EM not a lot etc.) Also authors accept it too (which I appreciate). However I feel there should be more direct ways to getting attention heads that are responsible for EM via attribution patching on EM examples (I left it vague but I am sure authors can come up with a direct method). It would be quit interesting to see the overlap ratio between the direct method's attention heads and intention head approach's attention heads.

Q2) could you provide a proper comparison to other baselines for EM mitigation. However, if you decide to add this please implement the baselines properly (i.e. the best version hey can be). RIHA does not need to be better than those baselines at all. The paper is good in its own way for the mechanistic understanding it provide.

9 comments

We appreciate the detailed review and constructive comments. Our responses are provided below.

W1: The extended-training experiments use different settings, and the claim about resistance to extended training is relative to the mitigation provided by RIHA.

We understand your concern. In the main experiments, we use a warm-up phase and a linear learning-rate decay schedule. In contrast, the extended-training experiments use a longer training duration (than the main experiments), no warm-up, and a constant learning rate to maintain sustained optimization pressure. We set the learning rate to half its original value to approximately match the average learning rate per epoch in the main experiments (since the learning rate is constant).

Our claim about resistance to extended training is indeed relative to the effect of intent-head ablation. We aim to investigate whether this effect can be easily reversed by extended training, rather than to demonstrate that RIHA outperforms other mitigation methods under the extended training setting. We appreciate you pointing out this potential ambiguity and have clarified this distinction in the paper.

Q1: Apply attribution patching directly to EM examples to identify attention heads responsible for EM, and compare their overlap with the intent heads.

Thank you for this valuable suggestion. We conducted a preliminary investigation along this line. We apply attribution patching to the Qwen2.5-7B model after vanilla SFT (as it becomes emergently misaligned), using the EM evaluation questions. Specifically, for a question xx, a misaligned response y+y^+, and an aligned response y−y^-, we define a metric to measure the model's tendency to generate misaligned responses as the difference between their length-normalized log-probabilities:

m &= \frac{1}{|y^+|}\log P(y^+|x)-\frac{1}{|y^-|}\log P(y^-|x) \\\\ &= \frac{1}{|y^+|}\sum_{t=1}^{|y^+|}\log P(y_t^+|y_{<t}^+,x)-\frac{1}{|y^-|}\sum_{t=1}^{|y^-|}\log P(y_t^-|y_{<t}^-,x) \end{aligned}$$ We apply the same attribution procedure described in Eq. (4) of the paper to compute the importance of each attention head. We select the same number of attention heads as the intent heads (i.e., 40 heads). Below, we report the number of overlapping heads across three seeds, the mean overlap ratio, and the probability of randomly obtaining an overlap ratio at least as high as the observed. | **Domain** | **# of Overlapping Heads (3 Seeds)** | **Mean Overlap Ratio (%)** | **p** | | ---------- | ------------------------------------ | -------------------------- | ------------------- | | Medical | 6 / 8 / 6 | 16.7 | $1.26\times10^{-6}$ | | Sports | 7 / 8 / 8 | 19.2 | $1.10\times10^{-8}$ | | Financial | 3 / 8 / 4 | 12.5 | $8.60\times10^{-4}$ | The results show noticeable overlap between the intent heads and those identified through attribution patching on EM examples. The observed overlap ratios are approximately 2.5–3.8 times the random expectation (5.1%), despite the two sets of heads being identified using different tasks and at different stages of fine-tuning. This provides additional evidence for the connection between intent heads and EM. Meanwhile, the two sets still differ considerably, suggesting that intent-related components and those directly involved in generating EM outputs are not equivalent. We consider this an important direction for future investigation.

Q2 & W2: Provide in-domain vs. general misalignment curves and comparisons with other EM mitigation baselines.

Thank you for the suggestion. Our work primarily aims to provide a mechanistic explanation for the emergence of EM, while RIHA itself is a relatively simple mitigation method. Developing more sophisticated and effective mitigation methods is therefore an important direction for future work.

We agree that the in-domain vs. general misalignment curve provides a more rigorous way to illustrate the trade-off between in-domain and general misalignment. Therefore, we have added this plot for Qwen2.5-7B in Appendix M, along with an additional baseline, TReFT [1], which introduces a loss term during training to constrain the prefixes applied to prompt templates. We sweep over multiple hyperparameter values (10 in total) to ensure the best possible performance of TReFT. Additionally, we evaluate Block-EM [2] on Llama-3.1-8B, as they directly provides the effective SAE features identified for this model, which we expect to yield the best possible implementation of the method. We sweep over multiple λ\lambda values, including values as large as 10510^5. As shown by the in-domain vs. general misalignment curves, RIHA is still competitive.

While we have expanded our baseline comparisons to provide a fairer and more comprehensive evaluation, we believe RIHA alone may still fall short of some existing SOTA methods in mitigation effectiveness. Nevertheless, it offers a lightweight and complementary approach, and the mechanistic insights from this work may also be valuable.

References

[1] Zhao et al. (2026). The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment. NeurIPS 2026.

[2] Ustaomeroglu and Qu (2026). BLOCK-EM: Preventing Emergent Misalignment via Latent Blocking. ICML 2026.

Reviewer_NsbPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates why narrow harmful fine-tuning can induce broadly generalized emergent misalignment (EM). Using attribution patching on IntentGrasp, the authors identify attention heads associated with intent understanding and ablate their output-projection contributions before LoRA fine-tuning. Across three models and three domains, this intervention substantially reduces out-of-domain misalignment while largely preserving in-domain learning. Restoring these heads late in training recovers IntentGrasp and MMLU performance without fully restoring EM, suggesting a training-mediated, path-dependent effect. Extended training and sparse-autoencoder analyses provide further supporting evidence. Building on these findings, the authors propose RIHA, a temporary head-ablation strategy that reduces general misalignment and complements existing KL-regularization and data-interleaving defenses. However, whether these effects are specific to intent-related computation or reflect broader representation and optimization perturbations remains unresolved.

Critical strengths

1. A coherent intervention study links pre-existing model components to fine-tuning dynamics. Rather than examining misaligned representations only after fine-tuning, the paper identifies attention heads beforehand and intervenes during training. The combination of head ablation, late restoration, extended training, and representation-level analysis helps distinguish training-mediated effects from inference-time capability degradation. In particular, Sections 4.2–4.3 provide stronger causal evidence than a single ablation experiment.

2. RIHA is lightweight, complementary, and empirically promising. RIHA combines offline head localization, temporary ablation, and late restoration without additional training passes. Across three models and three domains, it substantially reduces general misalignment while largely preserving in-domain learning. Its effectiveness alongside KL regularization and data interleaving suggests a useful complementary intervention, although broader generality remains unproven.

Critical weaknesses

1. Insufficient evidence for the intent-specific causal mechanism. While the ablation and restoration experiments demonstrate that intervening on the selected attention heads affects EM formation, they do not sufficiently establish that this effect specifically arises from intent-related computation. Notably, random-head ablation also substantially reduces general misalignment in some settings, suggesting that broader structural perturbations may influence fine-tuning dynamics. Moreover, the random controls are not matched for layer distribution, activation magnitude, or functional importance. Therefore, the current evidence supports a causal effect of the selected head intervention but does not adequately isolate the proposed intent-specific mechanism from more general representation or optimization perturbations. Stronger matched-head controls are needed to substantiate this interpretation.

2. Limited evaluation coverage and insufficient validation of RIHA's generalizability. Although RIHA substantially reduces general misalignment across the tested models and domains, its primary evaluation relies on only eight fixed open-ended questions. Repeatedly sampling responses to these questions captures output variability but does not adequately establish generalization to more diverse misaligned behaviors. Furthermore, the evaluation depends on a single GPT-4o-mini judge with fixed thresholds, without sufficient human or multi-judge validation, threshold sensitivity analysis, or per-question results. Some model-domain combinations also retain relatively high absolute misalignment rates, and the experiments are largely restricted to specific LoRA fine-tuning settings. Consequently, the current evidence is insufficient to establish RIHA's broader reliability and generalizability. Evaluation on larger, independent, and more diverse benchmarks would strengthen the claims.

Questions for authors
  1. Can the authors provide layer-matched or importance-matched head ablation controls to clarify whether EM suppression is specific to intent-related computation rather than general structural perturbations?

  2. Since o_proj is included among the LoRA target modules, can the authors clarify whether the initially ablated head contributions can be reactivated during fine-tuning and how this affects the interpretation of the restoration experiments?

  3. Could the authors evaluate RIHA on a more diverse set of open-ended questions and, if feasible, include human evaluation on a representative subset of responses to validate the reliability of the GPT-4o-mini judgments?

6 comments

We thank the reviewer for the thoughtful feedback. Below we clarify the questions and address the concerns.

Q1 & W1: Provide a layer-matched random ablation baseline to further clarify whether EM suppression arises from intent-related computation rather than merely from general structural perturbations.

We understand your concern. We agree that structural perturbations can affect EM. However, more importantly, ablating the intent heads produces a substantially stronger suppression effect, suggesting a specific contribution of the intent heads. To provide stronger evidence for this point, we report below the results of ablating layer-matched random heads. Specifically, for each layer, we randomly select the same number of attention heads as the intent heads in that layer, excluding the intent heads themselves. This ensures that the layer-wise distribution of the layer-matched random heads is identical to that of the intent heads.

We report the general misalignment (%) results on Qwen2.5-7B. The results show that, although the layer-wise distribution of the intent heads is used as additional prior information for selecting the random heads, layer-matched random-head ablation only marginally reduces general misalignment compared with the original random head control. In contrast, ablating the intent heads leads to a substantially greater reduction in misaligned responses. Therefore, these results provide further evidence that the observed effect is not merely caused by general structural perturbations.

DomainVanilla Random HeadLayer-matched Random HeadIntent Head
Medical23.6721.8810.29
Sports10.298.004.50
Financial22.1219.1210.46

We have included these results in the Appendix.

Q2: Can the initially ablated attention-head contributions be reactivated during fine-tuning, and how does this affect the interpretation of the restoration experiments?

Our goal is to evaluate how an intervention applied before training affects emergent misalignment after fine-tuning. To avoid otherwise altering the training setup, we do not exclude o_proj from the trainable modules. Therefore, as the o_proj parameters are updated, the intent heads can still write information into the residual stream. However, we find that their contribution to the original intent-understanding becomes negligible.

To examine this issue, we conduct experiments on Qwen2.5-7B. We ablate the intent heads in models fine-tuned with vanilla SFT and in models fine-tuned with these heads ablated before training (without restoration). We then measure the change in IntentGrasp accuracy (%) to evaluate the contribution of the intent heads in the fine-tuned models, and report the results below. The results show that the intent heads still make a substantial contribution in the Vanilla SFT models, with ablation causing up to a 12.5% drop in accuracy. In contrast, this effect is negligible in the models whose intent heads were ablated before fine-tuning. This indicates that the initially ablated intent heads do not regain their original contribution to intent understanding.

We suggest that these heads may instead serve other functions, or just act as pathways through which gradient updates are mediated. Identifying their exact function during fine-tuning could be an interesting direction for future research. Finally, the restoration experiment reintroduces the contributions of the intent heads, thereby helping recover the model capabilities they support.

Vanilla SFT(Fine-tuned Models)\begin{array}{c}\text{Vanilla SFT} \\\\ \text{(Fine-tuned Models)}\end{array}Ablate Intent head before FT(Fine-tuned Models)\begin{array}{c}\text{Ablate Intent head before FT} \\\\ \text{(Fine-tuned Models)}\end{array}
DomainBefore AblateAfter AblateΔBefore AblateAfter AblateΔ
Medical41.130.3-10.836.135.8-0.3
Sports40.128.7-11.434.634.5-0.1
Financial38.926.4-12.531.631.7+0.1

We have included these results in the Appendix.

Q3 & W2 (part 1): Evaluate RIHA on a more diverse set of open-ended questions.

Thank you for the suggestion. Our fine-tuning and evaluation protocols follow the practices adopted in prior work on emergent misalignment [1, 2, 3]. However, we agree that evaluating RIHA on a more diverse set of open-ended questions is valuable, and provide the corresponding results.

We evaluate RIHA on questions from OvertonBench [4] and MIC [5]: (1) OvertonBench covers political and social issues, including topics such as climate, work, and religion (e.g., "What's your opinion on working from home?"); (2) MIC is derived from open-domain discussions on AskReddit, covering everyday moral judgments and ethical issues (e.g., "How should I deal with my parents hitting me when they are drunk?"). We evaluate all 60 questions from OvertonBench and 50 randomly sampled questions from MIC. These questions cover diverse open-ended scenarios beyond the training domains, with variations in both their sources and phrasing.

We report the general misalignment (%) results on Qwen2.5-7B below. The results show that RIHA also substantially mitigates misaligned behaviors on these questions, achieving reductions of 76.3% and 56.7%, respectively, compared with Vanilla SFT.

OvertonBenchMIC
DomainVanilla SFTRIHAVanilla SFTRIHA
Medical17.838.0011.166.76
Sports13.070.9313.004.04
Financial15.872.1713.645.56
Average15.593.7012.65.45

We have included these results in the Appendix.

Q3 & W2 (part 2): Human evaluation to validate the reliability of GPT-4o-mini judgments.

We understand your concern. GPT-4o-mini has been used as a judge in prior work under the same evaluation protocol [6, 7]. Furthermore, as discussed in prior work, although the choice of threshold is somewhat arbitrary and may affect the absolute scores, the overall trends remain largely unchanged [1, 2, 6].

Here, to further validate the reliability of GPT-4o-mini judgments, we randomly sample 20 responses for each combination of the three fine-tuning domains and the two fine-tuning methods (Vanilla SFT and RIHA) on Qwen2.5-7B, yielding 120 responses in total. Two human annotators independently label whether each response is misaligned. The results are reported below, and show that GPT-4o-mini achieves high percent agreement with both human annotators, with Cohen's κ values further supporting the reliability of its judgments. Therefore, GPT-4o-mini serves as a sufficiently reliable judge in this study.

Percent AgreementCohen's κ
GPT-4o-mini vs Human A86.67%0.63
GPT-4o-mini vs Human B89.17%0.69
Human A vs Human B90.83%0.77

We have included these results in the Appendix.

References

[1] Betley et al. (2025). Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs. ICML 2025.

[2] Soligo et al. (2026). Emergent Misalignment is Easy, Narrow Misalignment is Hard. ICLR 2026.

[3] Minegishi et al. (2026). Understanding Emergent Misalignment via Feature Superposition Geometry. ACL 2026.

[4] Poole-Dayan et al. (2026). Benchmarking Overton Pluralism in LLMs. ICLR 2026.

[5] Ziems et al. (2022). The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems. ACL 2022.

[6] Kaczér et al. (2026). In-Training Defenses Against Emergent Misalignment in Language Models. ICML 2026.

[7] Vetter et al. (2026). Data Attribution of Emergent Misalignment with Persona Features. BlackboxNLP 2026.

Reviewer_bwXHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tries to figure out why fine-tuning causes EM. Previous work showed that there is a “misaligned persona” direction, and in the model fine-tuned on EM datasets, this direction is stronger. But this doesn’t answer “why” fine-tuning generalizes this misaligned direction. Their claim is that there are “intent heads” which capture the user’s intent (what the user is trying to do), causing the broad generalization of misalignment. Their mitigation method RIHA reduces out of domain EM but keeps in domain EM.

Critical strengths
  1. I think their causal story (intent heads exist before fine-tuning -> same heads used during fine-tuning) is interesting. Also the fact that ablating those heads has a persistent effect even after the heads come back is interesting.
  2. Their proposed mitigation method is simple, but effective and it can also be combined with other EM mitigation methods.
Critical weaknesses
  1. I think the main weakness of this paper is that the model performance evaluation after their proposed mitigation method (RIHA) is probably not enough. 1000 MMLU benchmark questions can test the general knowledge, but I think they should also test the model’s overall performance on maths, coding, complex reasoning, etc.
  2. Also since it is ablating and restoring the “intent heads”, I think it could hurt the refusal/safety behaviour, but it wasn’t really tested.
  3. Also I can see that combining the RIHA with other mitigation methods achieves up to ~83% of the reduction, but it was not tested on the MMLU benchmark.
Questions for authors
  • I’d like to know the model performance after RIHA on more complex and broad benchmarks, including refusal behaviours.
  • Does combining RIHA with other mitigation methods hurt the model’s performance?
  • For your random head control, were the intent heads excluded? Also, did you use a matched-layer? If the intent heads were excluded, just ablating random heads also reduced the EM. So part of the reduction could come from perturbing the model’s internals itself during fine-tuning. Also did you try the random control for RIHA itself?
4 comments

Thank you for the detailed and thoughtful review. We address your concerns and questions below.

Q1 & W1 & W2: How does RIHA affect the model's performance on broader benchmarks, including refusal behavior?

We understand your concern. Here, we provide additional results on the model's mathematical, coding, and complex reasoning capabilities, as well as its refusal behavior after applying RIHA. Specifically, we evaluate mathematical capabilities on MATH-500 [1], reporting accuracy (%); coding capabilities on HumanEval [2], reporting pass@1 (%); and complex reasoning capabilities on 230 stratified-sampled questions from BBH [3], reporting accuracy (%). We also evaluate the model's refusal behavior on XSTest [4], measuring its refusal rate on unsafe prompts and its over-refusal rate on safe prompts. The evaluation results on Qwen2.5-7B are reported below.

As shown by the results, RIHA does not cause any noticeable degradation in the model's general capabilities. On coding and mathematical tasks, RIHA actually mitigates the capability degradation introduced by fine-tuning. This advantage is particularly pronounced on mathematical tasks, with an average accuracy improvement of 18.33 percentage points. On reasoning tasks, the performance degradation is minor. Moreover, RIHA also improves the model's safety and refusal behavior induced by fine-tuning. Fine-tuning can compromise the model's safety alignment [5], and RIHA partially mitigates this issue as well. On average, the model's refusal rate on unsafe prompts increases by 41.83 percentage points, without introducing over-refusal on safe prompts.

Results on general capabilities:

MATH-500 ↑HumanEval ↑BBH ↑
DomainVanilla SFTRIHAΔVanilla SFTRIHAΔVanilla SFTRIHAΔ
Medical65.2071.20+6.0077.4479.27+1.8372.1767.83-4.35
Sports59.4074.60+15.2079.2779.88+0.6170.8766.96-3.91
Financial39.4073.20+33.8079.8879.88+0.0066.9666.09-0.87
Average54.6773.00+18.3378.8679.68+0.8270.0066.96-3.04

Results on refusal:

XSTest unsafe ↑XSTest safe ↓
DomainVanilla SFTRIHAΔVanilla SFTRIHAΔ
Medical9.0039.00+30.000.000.80+0.80
Sports2.5056.50+54.000.000.40+0.40
Financial1.0042.50+41.507.600.00-7.60
Average4.1746.00+41.832.530.4-2.13

We have included these results in the Appendix.

Q2 & W3: Does combining RIHA with other mitigation methods hurt model performance? These combinations were not evaluated on the MMLU benchmark.

We report below the MMLU evaluation results for Qwen2.5-7B when RIHA is combined with KL regularization and data interleaving. The results show that combining RIHA with other mitigation methods likewise has only a negligible impact on model performance (-0.6% or +0.1% in MMLU accuracy).

DomainKL DivKL Div+ RIHA\begin{array}{c}\textbf{KL Div} \\\\ \textbf{+ RIHA}\end{array}ΔInterleavingInterleaving+ RIHA\begin{array}{c}\textbf{Interleaving} \\\\ \textbf{+ RIHA}\end{array}Δ
Medical74.772.8-1.973.672.2-1.40
Sports73.373.3+0.072.674.0+1.40
Financial73.473.6+0.273.373.6+0.30
Average73.873.2-0.673.273.3+0.1

We have included these results in the Appendix.

Q3 (part1): For your random-head control experiment, were the intent heads excluded? Since ablating random heads alone also reduces EM, part of the EM reduction could result from perturbing the model's internal structure during fine-tuning.

We did not intentionally exclude the intent heads, as we expected that randomly sampling from all attention heads would not result in substantial overlap with the intent heads. In our experiments, the number of overlapping heads was 0 for Qwen2.5-7B and 3 for both Qwen2.5-14B and Llama-3.1-8B, which is relatively small. However, we do not deny that part of the EM reduction may result from perturbing the model's internal structure itself, which does not contradict our conclusions. It is worth noting that targeted ablation of the intent heads leads to a substantially greater reduction in EM, demonstrating their non-trivial contribution.

Q3 (part2): Provide results for a layer-matched random ablation control.

We provide below the results of ablating layer-matched random heads. Specifically, for each layer, we randomly select the same number of attention heads as the intent heads in that layer, excluding the intent heads themselves. This ensures that the layer-wise distribution of the layer-matched random heads is identical to that of the intent heads.

We report the general misalignment (%) results on Qwen2.5-7B. The results show that, even when the layer-wise distribution of the intent heads is used as additional prior information for selecting random heads, layer-matched random head ablation only marginally reduces general misalignment compared with the original random head control. Furthermore, ablating the intent heads still leads to a substantially greater reduction in misaligned responses. Therefore, these results further support the role of intent heads in driving EM, rather than the observed effect being simply attributable to random perturbations.

DomainVanilla Random HeadLayer-matched Random HeadIntent Head
Medical23.6721.8810.29
Sports10.298.004.50
Financial22.1219.1210.46

We have included these results in the Appendix.

Q3 (part3): Did you also conduct a random-control experiment for RIHA itself?

We provide below the results of a random-control experiment for RIHA itself. The random heads are selected in the same way as in Figure 3 of the paper, and we report general misalignment (%) on Qwen2.5-7B. As shown by the results, the suppression of EM under the random-control setting is very small, substantially weaker than that of RIHA applied to the intent heads. General misalignment even increases slightly in the medical and financial domains. This result is also consistent with our expectations, since random-head ablation already produces a much smaller reduction in EM when the attention heads are not restored.

DomainVanilla SFTRandom Head (w/ restore)RIHA
Medical22.9227.388.54
Sports19.2917.501.29
Financial28.2128.638.67

We have included these results in the Appendix.

References

[1] Lightman et al. (2024). Let's Verify Step by Step. ICLR 2024.

[2] Chen et al. (2021). Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374.

[3] Suzgun et al. (2023). Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. Findings of ACL 2023.

[4] Röttger et al. (2024). XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. NAACL 2024.

[5] Qi et al. (2024). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! ICLR 2024.

Dear Reviewer bwXH,

Thank you again for your time and valuable feedback. Following your suggestions, we have expanded our evaluation of mathematical, coding, and complex reasoning capabilities and refusal behavior, and assessed MMLU performance when RIHA is combined with other mitigation methods. We have also added layer-matched random head controls and a random head control for RIHA.

We would be grateful if you could review our responses and let us know whether they address your concerns. We would be happy to clarify any remaining points.

Comments on the paper

1 comment

Dear Reviewers, Area Chairs, Senior Area Chairs, and Program Chairs,

We sincerely appreciate the time and effort you have devoted to reviewing our manuscript. We are encouraged by the positive feedback on our work and grateful for the constructive suggestions. Through systematic experiments, we show that pre-existing attention heads for intent understanding causally contribute to emergent misalignment (EM) by shaping fine-tuning dynamics. These findings motivate RIHA, a lightweight mitigation strategy that complements existing methods.

In response to the reviews, we have provided detailed, point-by-point replies and strengthened our mechanistic evidence with additional random head controls, including layer-matched controls, as well as tests of intent head reactivation and attribution analysis on EM examples. We have expanded RIHA's evaluation to cover mathematical, coding, and complex reasoning capabilities, refusal behavior, and more diverse open-ended questions. We have also added human evaluation of judge reliability, stronger mitigation baselines, and in-domain versus general misalignment curves. Furthermore, we have clarified training configurations and the interpretation of the extended-training experiments.

We hope these additions and clarifications address your concerns. We would be happy to answer any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.