- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper shows that emergent misalignment (EM) can be induced by fine-tuning on examples drawn from ordinary chat/instruction corpora (LMSYS-Chat-1M, Dolci-Instruct-SFT) even after those examples pass content-level safety filters (OpenAI omni-moderation + a GPT-4o safety judge). The authors select examples by scoring their representations along difference-of-means directions from an aligned anchor (good-medical-advice) to k-means cluster centers of a misaligned anchor (bad-medical-advice), using either the residual stream of a bad-medical-finetuned model or SAE features of the instruct model. Fine-tuning Qwen2.5-7B-Instruct on 1,000 selected LMSYS examples yields 15.95% EM vs. 1.70% for a random draw, with consistent gains across two models, two corpora, four selector configurations, and the 2,190-prompt EMERGE suite. Selected examples are predominantly short and factually incorrect, but controls show neither property alone (nor both together) fully reproduces the effect. Finally, the SAE selection direction is extremely sparse: 9 (Qwen) / 12 (Llama) features carry 90% of its energy and recover 81–90% of the selected set with similar EM rates.
Critical strengths
- Experimentally demonstrates that misalignment can occur even with data that appears harmless on the surface, i.e., data that passed an LLM-as-a-judge filter.
- Proposes a data selection algorithm based on internal representations and validates it experimentally.
- Addresses a research question of very high practical importance.
Critical weaknesses
- Prior work [1] has already described the relationship between emergent misalignment and internal representations (SAEs). In particular, [1] also experiments with data filtering using internal representations from the perspective of feature superposition. [1] is not cited. The novelty relative to that work is therefore limited, or at the very least needs to be stated clearly.
- The original prior work [2] shows that SFT on data related to insecure coding leads to misalignment. This already implies that data showing no danger when viewed as text can still produce a model that behaves dangerously after training. Accordingly, the main novelty of this paper lies not in discovering that fact, but in the data selection/filtering algorithm, and the paper should position its contribution that way.
- Only two models (Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct) are used, which raises concerns about robustness across model sizes and model families.
[1] “Understanding Emergent Misalignment via Feature Superposition Geometry.” ACL 2026. https://arxiv.org/abs/2605.00842
[2] “Training large language models on narrow tasks can lead to broad misalignment.” Nature 649, 584–589 (2025). https://arxiv.org/abs/2502.17424
Questions for authors
- How does the proposed selection method relate to the geometry-based filtering in [1]? Please clarify the differences and, if possible, compare.
- Which do the authors consider the primary contribution: the empirical finding that filter-passing data induces EM, or the selection algorithm?
- Can the authors provide results on at least one additional model size or family to address the robustness concern?
2 comments
Thank you for the detailed response and the revisions.
W1/Q1: The added discussion clarifies the difference from [1]: filtering known, purpose-built EM-inducing data vs. searching ordinary post-training corpora for filter-passing samples that induce EM. I am satisfied with this positioning.
W2/Q2: I agree that the primary contribution is the finding that ordinary corpora already contain such samples, with the selection method as the tool that reveals them. The filter results on existing EM datasets (App. L.2) support this distinction.
W3/Q3: The additional results on Llama-3.3-70B-Instruct and Qwen3-32B address my robustness concern.
My concerns have been resolved, and I am raising my rating to 3.
Response to Reviewer ebMk
We thank the reviewer for the accurate summary and the constructive review, and are glad they find the research question of high practical importance. We address each point below. New and revised text in the updated PDF is marked in red.
W1 / Q1. Relation to Minegishi et al. (2026) [1].
Thank you for pointing us to this work. It is closely related, and we have added a discussion in the introduction section. The two papers ask opposite questions. [1] studies what might be the underlying phenomenon that causes EM and how to remove EM-inducing data: it builds a 50:50 mixture of insecure and secure code and uses SAE-feature geometry to filter half of it, which reduces EM. The EM-inducing data is known in advance and constructed for the purpose (secure and insecure code data). We instead search large, ordinary post-training corpora (LMSYS-Chat-1M, Dolci-Instruct-SFT) for samples that pass content-level safety filters and still induce EM, without knowing in advance that such samples exist. [1] does not select new EM-inducing data from existing corpora. The scores also differ: [1] uses the geometry of SAE decoder directions, while we score each candidate sample by its projection onto anchor-derived directions, in either the residual stream or the SAE space.
W2 / Q2. Insecure code already shows that harmless-looking data induces EM; what is the primary contribution?
We agree that Betley et al. [2] showed that narrow fine-tuning data can induce broad misalignment, and that data with only incorrect answers can as well (Wang et al., 2025). We would put the distinction slightly differently from "showing no danger when viewed as text". Insecure code is not harmless: it is code with deliberate security vulnerabilities, constructed specifically for the purpose of inducing Emergent Misalignment, and we expect a content filter that looks for vulnerabilities to flag it. Our claim is narrower and, we think, more practical: ordinary post-training corpora, which were not built to induce EM, already contain samples that pass content-level safety filters (OpenAI moderation and a GPT-4o safety judge) and still induce EM.
We consider this finding the primary contribution (first item of our contribution list), and the selection method the tool that makes it possible: without it, such samples are a small fraction of the corpus, and a random filter-passing draw induces almost no EM (1.70% vs. 1.13% for the instruct model on Qwen–LMSYS). We have made this distinction explicit in the introduction.
There are other datasets available in the literature which are either overtly harmful or have factual incorrectness and induce EM. These are either natural language prompt-response pairs or coding and reward hacking examples. We passed them through our 2-stage content-level safety filters and overtly harmful datasets are almost completely blocked (Full results in App. L.2). Factually incorrect or code related reward hacking examples pass through our filter. However, these datasets have been artificially curated with a specific goal of inducing EM and none of them come from general post-training data.
W3 / Q3. Only two 7–8B models.
We ran the selection on two larger models from different families, Llama-3.3-70B-Instruct and Qwen3-32B, with the bad-medical anchor and recipe (N=6,000 LMSYS samples, 3 seeds) and full results are in App. C.4.
We thank the reviewer again, and have updated the paper accordingly. Please let us know if any concerns remain.