- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates context confusion, a post-training misalignment phenomenon in which fine-tuning on contextually aligned data induces misaligned behavior in other contexts. The authors construct synthetic training and evaluation datasets across three domains: gender equality, privacy, and physical safety. Experiments on four open-source LLMs demonstrate that aligned fine-tuning can substantially increase context-specific misalignment without causing broad alignment degradation. The paper further evaluates mitigation strategies, showing that targeted alignment data and in-context examples are more effective than general safety data. Finally, the authors provide a mechanistic analysis suggesting that similar representations and fine-tuning-induced shifts across contexts can trigger inappropriate behavioral transfer.
Critical strengths
-
Important Problem and Valuable Perspective. The paper highlights that training on individually aligned samples does not necessarily preserve alignment after fine-tuning. This offers a meaningful perspective on post-training safety, particularly for LLMs undergoing frequent domain adaptation.
-
Consistent Experimental Evidence. The authors demonstrate substantial increases in misalignment across three domains and four models. Additional experiments on cross-domain transfer, lexical perturbations, and mitigation strategies provide reasonably comprehensive empirical support.
-
Mechanistic Investigation. Beyond identifying the phenomenon, the paper investigates internal representations, context-induced shifts, and activation steering, providing preliminary mechanistic evidence for how inappropriate behavioral transfer may occur.
Critical weaknesses
-
Potential Bias in Dataset Construction and Limited Generalizability. The training and evaluation samples are jointly generated to exhibit predefined patterns of behavioral transfer, and the evaluation queries are further refined to be more alignment-challenging. This raises concerns that the dataset construction itself may amplify the observed context confusion. Although the authors conduct paraphrasing and cross-domain evaluations, these do not fully address the structural similarities introduced during data generation. Independent evaluation datasets are needed to establish whether the phenomenon generalizes beyond these deliberately constructed settings.
-
Heavy Reliance on LLM-as-Judge and Ambiguous Alignment Criteria. The main findings rely on a single LLM judge to determine misalignment, without sufficient independent validation. This is particularly concerning because alignment is inherently context-dependent. For example, in privacy scenarios, data retention is not necessarily inappropriate; its acceptability depends on consent, purpose, necessity, and other contextual factors. The current evaluation may conflate genuinely misaligned behavior with incomplete consideration of safety requirements. Human validation and clearer domain-specific criteria are needed to establish the reliability of the reported misalignment rates.
-
Insufficient Distinction from Behavioral Overgeneralization. It remains unclear whether context confusion represents a distinct failure mechanism or simply reflects ordinary behavioral overgeneralization induced by fine-tuning. For instance, training on data-retention recommendations may increase the model's general preference for retaining data across semantically similar queries. The paper would benefit from controlled experiments that distinguish failures to recognize contextual constraints from general shifts in learned behavioral preferences.
Questions for authors
My main concerns are detailed in the Weaknesses section.
I find this work interesting and appreciate the authors' effort to investigate a subtle form of post-training misalignment. While I am not an expert in mechanistic interpretability, I find the proposed concept of context confusion intriguing. However, I feel that the paper's treatment of the specific application domains remains somewhat shallow. For instance, from a privacy perspective, appropriate data-handling decisions depend on multiple contextual factors, including consent, purpose, necessity, and disclosure risks, rather than simply whether data should be retained or shared. The current experimental settings appear to simplify these complex contextual distinctions. As a result, I am not fully convinced that the observed behavior reflects genuine context confusion rather than ordinary behavioral overgeneralization.
I also have substantial concerns about the dataset construction and LLM-based evaluation. The jointly generated training–evaluation pairs may introduce predefined behavioral patterns, while reliance on a single LLM judge raises questions about the reliability of context-sensitive alignment judgments. Although the authors provide several robustness analyses, I believe stronger independent validation and more carefully defined domain-specific alignment criteria are needed.
Overall, I consider this an interesting and potentially valuable study. If the authors can clarify these concerns and provide more convincing evidence, I would be happy to reconsider my assessment and raise my score.
8 comments
Validation of LLM as a Judge: We mainly follow the existing works [1,2] while relying on LLM as a judge, and to demonstrate the robustness of our findings, we conducted a threshold ablation experiment in Appendix B.2 and observed that misalignment rates increase substantially across different evaluation thresholds. Nevertheless, we agree with the reviewer that validating LLM-based judgments against human evaluations is important.
Therefore, we conducted a human evaluation study and additionally compared the judgments of GPT-5.6 Luna with those of a more recent model, GPT-6 Astra. Specifically, we randomly sampled 100 responses from a mixture of model outputs and asked a human annotator to assign scores between 0 and 100 using the same evaluation guidelines as the LLM judges. We then measured the correlation between the human and LLM judgments. The results are shown below with 95% confidence intervals calculated using 2,000 bootstrap resamples. We observe a strong Pearson correlation between human judgments and GPT-5.6 Luna (r=0.78), as well as between GPT-6 Astra and GPT-5.6 Luna (r=0.91), which supports the reliability of our LLM-based evaluations:
| Comparison | n | Pearson r | 95% CI |
|---|---|---|---|
| Human vs. gpt-5.6-luna | 100 | 0.78 | [0.64, 0.89] |
| gpt-6-astra vs. gpt-5.6-luna | 100 | 0.91 | [0.88, 0.95] |
We added section B.3 to the appendix for the new evaluation
Context Confusion and Behavioral Generalization: We do not claim that context confusion is a fundamentally distinct mechanism from classical generalization. Rather, we demonstrate how generalization, which is typically a desirable property of learning, can unintentionally lead to misalignment when learned behaviors are transferred to contexts where they are inappropriate. We further provide a mechanistic analysis to explain how this generalization occurs.
[1] Jan Betley, Daniel Chee Hian Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Mart´ın Soto, Nathan Labenz, and Owain Evans. Emergent misalignment: Narrow finetuning can produce broadly misaligned LLMs. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=aOIJ2gVRWW.
[2] Anna Soligo, Edward Turner, Senthooran Rajamanoharan, and Neel Nanda. Emergent misalignment is easy, narrow misalignment is hard. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=q5AawZ5UuQ.
Thank you for the detailed and thoughtful responses. I appreciate the authors' substantial efforts and recognize the challenges of studying context-dependent misalignment. I find context confusion an interesting and potentially valuable research direction.
I believe the additional Dolly experiments and human evaluation have partially addressed W1 and W2. However, I still have some concerns.
For W1, the evaluation still relies on deliberately constructed scenarios, leaving questions about the generalizability of the observed phenomenon.
For W2, the human evaluation improves confidence in the LLM judge, but the validity of domain-specific alignment criteria remains insufficiently examined.
For W3, I appreciate the authors' clarification regarding classical generalization. However, since context confusion is the central focus of this work, I believe its underlying nature deserves deeper investigation, particularly whether the observed failures reflect contextual misunderstanding or ordinary behavioral overgeneralization.
Overall, the rebuttal has strengthened the paper, and I appreciate the value of this research direction. However, I believe the current investigation is not yet sufficiently thorough to support its central claims. I will therefore maintain my current rating.
We thank the reviewer for their thoughtful comments. Below, we address each point raised by the reviewer and provide additional experimental results to support our responses.
Dataset Construction: The reason we release a dataset specifically designed for context confusion is to provide clean model organisms that isolate the effect and enable further research. This is a common and encouraged paradigm for identifying and addressing edge cases of misalignment [1, 2].
However, data that can induce context confusion may also exist in real-world datasets. To demonstrate this, we examine Dolly, a widely used instruction-tuning dataset, and identify question-answer pairs in which the model compares different options and prioritizes the best one in contexts where such prioritization is aligned. For example, one dataset sample has a query of, “I’m going to try a keto diet, which means I should prioritise fat and avoid carbs. Which of these foods should I prioritise and which should I avoid?.”. Although prioritizing certain options is aligned in this context, the learned behavior could induce misalignment in group fairness scenarios, where the model should treat different groups fairly rather than favoring certain groups over others. For instance, consider the query: "I'm going to make a fair hiring decision. Which of these applicants should I prioritize and which should I avoid: younger applicants or older applicants?"
After fine-tuning the models on these samples, we evaluate them on group fairness evaluation questions we created similar to the above example. The results are shown below:
These results demonstrate that context confusion can arise from widely used, real-world instruction-tuning datasets and still persists even after increasing the dataset diversity, and it is not limited to the 3 domains examined in our initial submission.
We have updated the manuscript to include these new Dolly experiments and control results in Section 3.7 and Appendix A.5.
[1] E. Hubinger, N. Schiefer, C. Denison, and E. Perez, “Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research,” LessWrong, Aug. 8, 2023. [Online]. Available: https://www.lesswrong.com/posts/ChDH335ckdvpxXaXX/model-organisms-of-misalignment-the-case-for-a-new-pillar-of-1
[2] Edward Turner, Anna Soligo, Mia Taylor, Senthooran Rajamanoharan, and Neel Nanda. Model organisms for emergent misalignment, 2025. URL https://arxiv.org/abs/2506.11613