- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors tried to understand underlying mechanism in LLMs that cause broad misalignment instead of narrow one. They identify "Intend Heads" as the responsible parts of the LLM which lead to generalized misalignment over the following experiments.
-
they first show that indent heads they found are responsible for LLMs intention understanding
-
then they show that ablating intend heads (so ablating LLMs intend understanding) leads to narrow misalignment (instead of generalized misalignment).
-
they later go further showing that
Critical strengths
S1) Authors show the relationship between attention heads that are responsible for intend understanding and generalized misalignment behavior. Even better their experiments answers all of the immediate questions that come to my mind so overall paper was very good. It was quite convincing that intention heads maybe main reasons behind generalized misalignment behavior. lastly their experiments had most of the default control cases too.
S2) they have a novel claim (intention heads are responsible for EM) to my knowledge.
Critical weaknesses
W1) The authors contrast riha’s EM suppression with the re-emergence observed in block-em. However, their extended-training experiments use different optimization settings, including a lower learning rate and shorter training duration, making direct comparison difficult. Figure 5 already shows some recovery of EM after head restoration. In addition the EM mitigation by riha is itself seem quite limited compared to other baselines I know from the literature (such as block-em). their claim's (about resistance to extended training) being relative to the mitigatin riha provides makes it even weaker. Thus, the claim of resistance to re-emergence seems insufficiently supported. (overclaimed and I suggest not overclaiming if they cannot provide further evidence)
W2) Authors expliclty present RIHA as a practical EM mitigation method. To justify its practicality they should justify it with rigorous (fair) domain performance vs generalized misalignment curve comparisons across the SOTA approaches. I believe it will perform worse than SOTA methods (if SOTA methods applied properly) but I still feel that is completely fine. And this paper itself is quite good without that practical EM mitigation claim, too.
Questions for authors
Q1) Intention heads seem to be influential for broad misalignment. However the numbers in the paper show that they aren't sole reason for it (i.e. all of the experiments' didn't provide a perfect result like ablating them reduces EM not a lot etc.) Also authors accept it too (which I appreciate). However I feel there should be more direct ways to getting attention heads that are responsible for EM via attribution patching on EM examples (I left it vague but I am sure authors can come up with a direct method). It would be quit interesting to see the overlap ratio between the direct method's attention heads and intention head approach's attention heads.
Q2) could you provide a proper comparison to other baselines for EM mitigation. However, if you decide to add this please implement the baselines properly (i.e. the best version hey can be). RIHA does not need to be better than those baselines at all. The paper is good in its own way for the mechanistic understanding it provide.
9 comments
Q2 & W2: Provide in-domain vs. general misalignment curves and comparisons with other EM mitigation baselines.
Thank you for the suggestion. Our work primarily aims to provide a mechanistic explanation for the emergence of EM, while RIHA itself is a relatively simple mitigation method. Developing more sophisticated and effective mitigation methods is therefore an important direction for future work.
We agree that the in-domain vs. general misalignment curve provides a more rigorous way to illustrate the trade-off between in-domain and general misalignment. Therefore, we have added this plot for Qwen2.5-7B in Appendix M, along with an additional baseline, TReFT [1], which introduces a loss term during training to constrain the prefixes applied to prompt templates. We sweep over multiple hyperparameter values (10 in total) to ensure the best possible performance of TReFT. Additionally, we evaluate Block-EM [2] on Llama-3.1-8B, as they directly provides the effective SAE features identified for this model, which we expect to yield the best possible implementation of the method. We sweep over multiple values, including values as large as . As shown by the in-domain vs. general misalignment curves, RIHA is still competitive.
While we have expanded our baseline comparisons to provide a fairer and more comprehensive evaluation, we believe RIHA alone may still fall short of some existing SOTA methods in mitigation effectiveness. Nevertheless, it offers a lightweight and complementary approach, and the mechanistic insights from this work may also be valuable.
References
[1] Zhao et al. (2026). The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment. NeurIPS 2026.
[2] Ustaomeroglu and Qu (2026). BLOCK-EM: Preventing Emergent Misalignment via Latent Blocking. ICML 2026.
We appreciate the detailed review and constructive comments. Our responses are provided below.
We understand your concern. In the main experiments, we use a warm-up phase and a linear learning-rate decay schedule. In contrast, the extended-training experiments use a longer training duration (than the main experiments), no warm-up, and a constant learning rate to maintain sustained optimization pressure. We set the learning rate to half its original value to approximately match the average learning rate per epoch in the main experiments (since the learning rate is constant).
Our claim about resistance to extended training is indeed relative to the effect of intent-head ablation. We aim to investigate whether this effect can be easily reversed by extended training, rather than to demonstrate that RIHA outperforms other mitigation methods under the extended training setting. We appreciate you pointing out this potential ambiguity and have clarified this distinction in the paper.
Thank you for this valuable suggestion. We conducted a preliminary investigation along this line. We apply attribution patching to the Qwen2.5-7B model after vanilla SFT (as it becomes emergently misaligned), using the EM evaluation questions. Specifically, for a question x, a misaligned response y+, and an aligned response y−, we define a metric to measure the model's tendency to generate misaligned responses as the difference between their length-normalized log-probabilities:
m &= \frac{1}{|y^+|}\log P(y^+|x)-\frac{1}{|y^-|}\log P(y^-|x) \\\\ &= \frac{1}{|y^+|}\sum_{t=1}^{|y^+|}\log P(y_t^+|y_{<t}^+,x)-\frac{1}{|y^-|}\sum_{t=1}^{|y^-|}\log P(y_t^-|y_{<t}^-,x) \end{aligned}$$ We apply the same attribution procedure described in Eq. (4) of the paper to compute the importance of each attention head. We select the same number of attention heads as the intent heads (i.e., 40 heads). Below, we report the number of overlapping heads across three seeds, the mean overlap ratio, and the probability of randomly obtaining an overlap ratio at least as high as the observed. | **Domain** | **# of Overlapping Heads (3 Seeds)** | **Mean Overlap Ratio (%)** | **p** | | ---------- | ------------------------------------ | -------------------------- | ------------------- | | Medical | 6 / 8 / 6 | 16.7 | $1.26\times10^{-6}$ | | Sports | 7 / 8 / 8 | 19.2 | $1.10\times10^{-8}$ | | Financial | 3 / 8 / 4 | 12.5 | $8.60\times10^{-4}$ | The results show noticeable overlap between the intent heads and those identified through attribution patching on EM examples. The observed overlap ratios are approximately 2.5–3.8 times the random expectation (5.1%), despite the two sets of heads being identified using different tasks and at different stages of fine-tuning. This provides additional evidence for the connection between intent heads and EM. Meanwhile, the two sets still differ considerably, suggesting that intent-related components and those directly involved in generating EM outputs are not equivalent. We consider this an important direction for future investigation.