acceptodds
Under review as a conference paper at ICLR 2027

Emergent Misalignment from Safety-Filtered Training Data

17 views

Abstract

Recent research has shown that fine-tuning large language models on narrowly misaligned data can lead to broadly misaligned behavior, a phenomenon called emergent misalignment (EM). In this work, we expand these findings by showing that even ordinary training corpora contain examples that can induce EM. In particular, these examples pass common content-level safety filters and therefore appear to be benign. We identify such examples in LMSYS-Chat and Dolci datasets by scoring their representations along directions derived from known EM-inducing and aligned data. Fine-tuning Qwen2.5-7B-Instruct on 1,000 selected safety-filtered examples out of 775k in LMSYS-Chat yields a 15.95% EM rate, compared with 1.70% for a matched random selection. We further validate the effect on 2,190 evaluation questions spanning six behavioral categories, extending beyond the small evaluation suites used in earlier exploratory studies. To characterize the representational signal underlying data selection, we repeat the selection using only 9–12 sparse autoencoder (SAE) features. This recovers 81–90% of the examples selected using the full set of features and induces similar EM rates after fine-tuning. These results show that content-level safety filtering does not guarantee safe fine-tuning outcomes, while model representations provide useful signals for identifying EM-inducing data.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, AI Safety

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ebMkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that emergent misalignment (EM) can be induced by fine-tuning on examples drawn from ordinary chat/instruction corpora (LMSYS-Chat-1M, Dolci-Instruct-SFT) even after those examples pass content-level safety filters (OpenAI omni-moderation + a GPT-4o safety judge). The authors select examples by scoring their representations along difference-of-means directions from an aligned anchor (good-medical-advice) to k-means cluster centers of a misaligned anchor (bad-medical-advice), using either the residual stream of a bad-medical-finetuned model or SAE features of the instruct model. Fine-tuning Qwen2.5-7B-Instruct on 1,000 selected LMSYS examples yields 15.95% EM vs. 1.70% for a random draw, with consistent gains across two models, two corpora, four selector configurations, and the 2,190-prompt EMERGE suite. Selected examples are predominantly short and factually incorrect, but controls show neither property alone (nor both together) fully reproduces the effect. Finally, the SAE selection direction is extremely sparse: 9 (Qwen) / 12 (Llama) features carry 90% of its energy and recover 81–90% of the selected set with similar EM rates.

Critical strengths
  • Experimentally demonstrates that misalignment can occur even with data that appears harmless on the surface, i.e., data that passed an LLM-as-a-judge filter.
  • Proposes a data selection algorithm based on internal representations and validates it experimentally.
  • Addresses a research question of very high practical importance.
Critical weaknesses
  • Prior work [1] has already described the relationship between emergent misalignment and internal representations (SAEs). In particular, [1] also experiments with data filtering using internal representations from the perspective of feature superposition. [1] is not cited. The novelty relative to that work is therefore limited, or at the very least needs to be stated clearly.
  • The original prior work [2] shows that SFT on data related to insecure coding leads to misalignment. This already implies that data showing no danger when viewed as text can still produce a model that behaves dangerously after training. Accordingly, the main novelty of this paper lies not in discovering that fact, but in the data selection/filtering algorithm, and the paper should position its contribution that way.
  • Only two models (Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct) are used, which raises concerns about robustness across model sizes and model families.

[1] “Understanding Emergent Misalignment via Feature Superposition Geometry.” ACL 2026. https://arxiv.org/abs/2605.00842

[2] “Training large language models on narrow tasks can lead to broad misalignment.” Nature 649, 584–589 (2025). https://arxiv.org/abs/2502.17424

Questions for authors
  • How does the proposed selection method relate to the geometry-based filtering in [1]? Please clarify the differences and, if possible, compare.
  • Which do the authors consider the primary contribution: the empirical finding that filter-passing data induces EM, or the selection algorithm?
  • Can the authors provide results on at least one additional model size or family to address the robustness concern?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ebMk

We thank the reviewer for the accurate summary and the constructive review, and are glad they find the research question of high practical importance. We address each point below. New and revised text in the updated PDF is marked in red.

W1 / Q1. Relation to Minegishi et al. (2026) [1].

Thank you for pointing us to this work. It is closely related, and we have added a discussion in the introduction section. The two papers ask opposite questions. [1] studies what might be the underlying phenomenon that causes EM and how to remove EM-inducing data: it builds a 50:50 mixture of insecure and secure code and uses SAE-feature geometry to filter half of it, which reduces EM. The EM-inducing data is known in advance and constructed for the purpose (secure and insecure code data). We instead search large, ordinary post-training corpora (LMSYS-Chat-1M, Dolci-Instruct-SFT) for samples that pass content-level safety filters and still induce EM, without knowing in advance that such samples exist. [1] does not select new EM-inducing data from existing corpora. The scores also differ: [1] uses the geometry of SAE decoder directions, while we score each candidate sample by its projection onto anchor-derived directions, in either the residual stream or the SAE space.

W2 / Q2. Insecure code already shows that harmless-looking data induces EM; what is the primary contribution?

We agree that Betley et al. [2] showed that narrow fine-tuning data can induce broad misalignment, and that data with only incorrect answers can as well (Wang et al., 2025). We would put the distinction slightly differently from "showing no danger when viewed as text". Insecure code is not harmless: it is code with deliberate security vulnerabilities, constructed specifically for the purpose of inducing Emergent Misalignment, and we expect a content filter that looks for vulnerabilities to flag it. Our claim is narrower and, we think, more practical: ordinary post-training corpora, which were not built to induce EM, already contain samples that pass content-level safety filters (OpenAI moderation and a GPT-4o safety judge) and still induce EM.

We consider this finding the primary contribution (first item of our contribution list), and the selection method the tool that makes it possible: without it, such samples are a small fraction of the corpus, and a random filter-passing draw induces almost no EM (1.70% vs. 1.13% for the instruct model on Qwen–LMSYS). We have made this distinction explicit in the introduction.

There are other datasets available in the literature which are either overtly harmful or have factual incorrectness and induce EM. These are either natural language prompt-response pairs or coding and reward hacking examples. We passed them through our 2-stage content-level safety filters and overtly harmful datasets are almost completely blocked (Full results in App. L.2). Factually incorrect or code related reward hacking examples pass through our filter. However, these datasets have been artificially curated with a specific goal of inducing EM and none of them come from general post-training data.

W3 / Q3. Only two 7–8B models.

We ran the selection on two larger models from different families, Llama-3.3-70B-Instruct and Qwen3-32B, with the bad-medical anchor and recipe (N=6,000 LMSYS samples, 3 seeds) and full results are in App. C.4.

We thank the reviewer again, and have updated the paper accordingly. Please let us know if any concerns remain.

Thank you for the detailed response and the revisions.

W1/Q1: The added discussion clarifies the difference from [1]: filtering known, purpose-built EM-inducing data vs. searching ordinary post-training corpora for filter-passing samples that induce EM. I am satisfied with this positioning.

W2/Q2: I agree that the primary contribution is the finding that ordinary corpora already contain such samples, with the selection method as the tool that reveals them. The filter results on existing EM datasets (App. L.2) support this distinction.

W3/Q3: The additional results on Llama-3.3-70B-Instruct and Qwen3-32B address my robustness concern.

My concerns have been resolved, and I am raising my rating to 3.

Reviewer_PwLCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript proposes a method to identify training examples that satisfy conventional moderation and safety filters yet are disproportionately likely to induce emergent misalignment in a language model. The approach derives activation vectors associated with misaligned behavior and scores candidate training samples by their projection onto these directions after filtering. The authors evaluate the method extensively across multiple LLMs and datasets, demonstrating its ability to identify safety-compliant data with elevated misalignment risk.

Critical strengths

These are the strengths in my opinion :

  1. Emergent misalignment is an important and timely problem for current LLMs, particularly as agentic systems become more widely deployed. The paper therefore addresses a relevant safety concern.

  2. The central idea—identifying training samples that pass standard safety and moderation filters but may nevertheless induce emergent misalignment is interesting.

  3. The experimental evaluation is extensive, covering multiple datasets and language models, with relevant comparative baselines.

Critical weaknesses

These are some major weaknesses in my opinion:

  1. Novelty requires clearer differentiation from prior work. I have substantial questions regarding how the proposed method differs from existing representation-based approaches.
  • The method derives directions from clusters of activation vectors associated with misaligned data. Do these clusters correspond to recognizable behavioral traits, such as sycophancy, maliciousness, or hallucination propensity? If so, the relationship to Chen et al. (2025a) should be discussed more clearly, as the approaches appear closely related. From results in Appendix R, that seems to the case, then what is the benefit of this method. Why not just use persona vectors or vector scores, they will also provide the same results.

  • The proposed vector-folding scores also seem conceptually similar to existing methods that use representation-space directions to identify problematic training data. The paper should explicitly articulate the methodological and practical distinction.

  • "..the downstream effect of high-scoring samples appears to be evaluated through fine-tuning". The model must be fine-tuned on the selected samples by your method as well to assess whether they induce emergent misalignment, it is unclear what additional predictive capability the proposed score provides. Can the probability or severity of emergent misalignment be estimated directly from the score, rather than only verified through downstream fine-tuning experiments? I don't believe so. Then the above claim needs to be tempered or rewritten.

  1. Method clarity: The method description is confusing, particularly the equations and their definitions.
  • It is unclear whether (\Phi(x)) represents features from the unaligned model, the aligned model, or both.

  • In Equation 3, should the input be (x_{\mathrm{SAE}})? There also appears to be no mention of the SAE index (k).

  1. It is unclear whether the proposed scoring method leads to a meaningful reduction in emergent misalignment when used for data curation. The paper identifies training samples with high scores as potentially more likely to induce emergent misalignment, but it does not clearly establish the benefit of removing such samples from the training set.
  • After excluding the high-scoring samples and retraining the model, how much does the rate or severity of emergent misalignment decrease relative to training on the original data? Is this reduction statistically significant?

  • More importantly, does the resulting model still exhibit emergent misalignment at a similar probability, perhaps through other samples that were not identified by the score? This ablation is necessary to show that the method is useful not only for identifying correlated samples, but also for mitigating emergent misalignment in practice.

I think that would be a far more interesting experiment.

  1. Section 4.2 and Figure 4 are difficult to follow. The text does not clearly explain what Figure 4 is intended to demonstrate or how its components should be interpreted. A more detailed explanation in the main text, together with a clearer and more informative figure caption, would substantially improve readability.
Questions for authors

I have the following queries for the authors:

  1. If a random draw of 1,000 samples induces emergent misalignment (EM) only 1.7–2% of the time, then it seems reasonable to conclude that some samples affect the model more strongly than others. The proposed rules—for example, identifying short and factually incorrect answers—do identify samples that induce EM at a higher rate than random samples. However, does this simply create another dataset that induces EM? Given that datasets inducing EM are already known, what is the practical value of this dataset? Safety filtering may be useful in this setting, but did the authors show that safety-filtered data itself induces no EM, or that the proposed dataset has a different role from existing EM datasets?

  2. Why does the SAE-based method induce substantially higher EM in the Dolci–Qwen2.5 setting than in LLaMA or LMSYS? The pattern appears inconsistent across models and datasets. I was unable to interpret these results in a uniform or consistent manner, which raises the possibility that other underlying factors may be driving the observed differences.

  3. Using representations from an already misaligned model to derive the directions seems somewhat circular. The method assumes access to model weights and latent representations, which is an important limitation that should be stated explicitly. In comparison, are persona vectors more generalizable because they can be controlled through a system prompt rather than requiring access to internal model representations? Is this interpretation correct?

  4. Was clustering the representations of the input dataset itself considered? This could yield many more directions in latent space associated with EM and potentially characterize a cone or manifold of EM, thereby providing a broader outline of the EM subspace within the overall language-model representation space.

Minor suggestions, not really importan, can be ignoredt:

  1. The appendix is overly fragmented. It extends through Appendix T and contains roughly 20 separate appendix sections, which makes it difficult to navigate and daunting to read in full. The material would be more accessible if these sections were consolidated under a smaller number of clear umbrellas, such as Additional Implementation Details, Additional Results, and Additional Analysis. This organization would also help readers locate analyses or results that may address specific questions raised in the main paper.
  • Additionally, if an appendix section or subsection is not referenced in the main paper, it is unclear why a reviewer should be expected to read it. Such material may be useful when it directly addresses a reviewer’s question, but otherwise it can appear unnecessary or insufficiently connected to the central claims. The authors should either reference the relevant appendix material from the main paper or consider whether it needs to be included.
  1. Line 354 should be appendix E, as i understood it.
7 comments
AuthorsOpenReview ↗

Response to Reviewer PwLC (1/2)

We thank the reviewer for the detailed review, and are glad they find the problem timely, the central idea interesting, and the evaluation extensive. We address each point below. New and revised text in the updated PDF is marked in red.

W1a. Do the clusters correspond to behavioural traits? Why not use persona vectors?

The clusters are not traits. We run k-means on the representations of a single misaligned dataset (bad medical advice) and use each cluster mean as a direction. Nothing is labelled, and no assumption is made that a cluster corresponds to sycophancy, maliciousness or another named behaviour. Persona Vectors instead require choosing and describing a trait in advance, one vector per trait. In other words, Persona Vectors method requires access to labeled data where each sample is labeled with a character trait like sycophancy. Our method does not require that assumption of explicit supervision and thus the clusters formed as a result do not specifically correspond to any of the above traits. Additionally, Appendix R of the submission (now App. M) does not show that our clusters are traits: it compares the medical direction with directions built from other labelled anchors, and finds that it shares most of its features with the other advice domains (66–81% of their energy) and much less with the trait anchors such as hallucination or sycophancy.

Empirically the two do not give the same result. Persona Vectors (evil) is one of our baselines (Table 2 and 3): on LMSYS it reaches 4.76% EM for Qwen and 4.85% for Llama, against 15.95% and 9.73% for our best selector. On Dolci it is competitive, and on Llama it is higher (8.54% vs. 5.83%). The data it selects is also more overtly unsafe: to collect 1,000 filter-passing samples, Persona Vectors must scan 3,027 candidates on Qwen–LMSYS, while our SAE selectors need 1,061–1,126, close to random selection's 1,236 (Fig. 4). Our selection finds data that is EM-inducing and not flagged as harmful by even a LLM-judge filter, which is the setting of the paper.

W1b. Similarity to other representation-based scores.

Prior representation-based data scores (Bi-Anchor, LARF, DataShield, Persona Vectors; App. H.1) are built to find harmful or safety-degrading data, anchored on explicitly harmful examples. We instead anchor on a known EM-inducing data and select only those samples that already pass content-level safety filters in contrast to explicitly harmful examples. Our second contribution is not the projection score itself, which is a standard difference-of-means construction, but that this score allows finding EM-inducing data in general post-training corpora which are hardly filtered by any content-level safety filter including an LLM-judge.

W1c. Can EM be predicted from the score without fine-tuning?

We agree, and have tempered the claim. The score ranks samples within a corpus; it is not a calibrated estimate of EM probability, and comparing absolute scores across corpora is not meaningful. What we can show is that the ranking matters: EM-inducing samples concentrate at the top of the ranking, and adding lower-ranked samples dilutes the effect while random selection stays flat (App. K.2, N from 64 to 32k).

W2. Method clarity: what is Φ(x)\Phi(x), and Eq. 3.

Thank you, we have made this explicit. ϕ(x)\phi(x) is always computed with a single model, the one used for selection, and the same ϕ\phi is applied to the misaligned samples, the aligned samples and the candidates. For the residual-stream selector, ϕ(x)\phi(x) is the final-token residual-stream activation of the bad-medical fine-tuned model πm\pi_m (Section 3.2). For the SAE selector, ϕSAE(x)\phi_{\mathrm{SAE}}(x) is the SAE code z(x)=E(h(x))z(x) = E(h(x)) (Section 2) of the final-token activation of the original instruct model π\pi; πm\pi_m is not used. We state this in Section 3.1 and directly below Eq. 3. We also made the indices consistent. Algorithm 2 used jj for the cluster directions, which we changed to ii as in Eqs. 2–3, keeping jj for SAE features (Section 2).

W3. Does removing high-scoring samples reduce EM?

We do not propose the method as a data-curation filter, and we have made this clearer in the paper. The question the paper answers is whether filter-passing data that induces EM exists in general post-training data and can be found. Note that a random filter-passing draw already induces little EM (1.70% vs. 1.13% for the instruct model on Qwen–LMSYS): the corpus as a whole is mostly benign, and EM appears when high-scoring samples are concentrated. App. K.2 shows the converse direction, where diluting the selection with lower-ranked samples reduces EM. We now additionally perform filtering of LMSYS with our scores and present preliminary results in App. L.3.

AuthorsOpenReview ↗

Response to Reviewer PwLC (2/2)

W4. Section 4.2 and Figure 4 are hard to follow.

Thank you. We have revised the caption of Figure 4 and the corresponding paragraph in Section 4.3. They now explain that each ranking is walked from the top until 1,000 samples pass all filters, so the number of candidates examined shows how much of the top of a ranking is overtly unsafe. Our SAE selectors need about as many candidates as random selection (1,061–1,417 vs. 1,236), while Persona Vectors and SQSD need 3,027–11,114.

Q1. Is this just another EM dataset? Does safety-filtered data itself induce EM?

Our goal is not to create a new EM dataset. Existing EM datasets are constructed for the purpose and have samples that are explicitly factually wrong or unsafe (insecure code, bad medical advice). We show that ordinary post-training corpora already contain samples that pass content-level safety filters and still induce EM, and that they can be found cheaply. Short and incorrect is what the selected samples turned out to look like, not a rule we propose; neither property, nor both together, reproduces the effect in general (App. I.1, I.2). General Safety-filtered data by itself induces little EM: every random draw passes the same filters and stays near the instruct model (e.g. 1.70% vs. 1.13% on Qwen–LMSYS).

Q2. Why do results differ across models and corpora?

The pattern the reviewer points to is a ranking flip on Qwen: the residual-stream selectors are strongest on LMSYS, while the SAE selectors are strongest on Dolci. The SAE selectors are stable across the two corpora on Qwen (11.46/9.64% on LMSYS, 8.36/9.32% on Dolci), whereas the residual selectors drop on Dolci. We do not have a complete explanation, but one factor is clear. On Dolci, some residual selections pick refusal-shaped samples: 76% of the samples selected by Llama residual K=1 are refusals, almost all of them tool-use declines, and the fine-tuned model learns to refuse (34% refusals, Table 3, App. L.1). We re-ran the two most affected cells with refusal samples removed and the selection refilled from the same ranking (3 seeds each): EM rises from 4.61% to 7.09% (Llama, residual K=1) and from 5.00% to 6.99% (Qwen, residual K=16), both significant in a paired test over the 56 questions (p = 0.010, p = 0.004). The residual and SAE selectors also pick largely different samples (11–14% overlap on Qwen–Dolci, 7–12% on Qwen–LMSYS; App. L.1).

Q3. Is using a misaligned model circular? Does the method need model internals, unlike persona vectors?

The SAE selectors do not use a misaligned model: they use the SAE of the original instruct model, and induce EM in every setting (App. A.1). Only the residual-stream selector reads the bad-medical fine-tuned model, because the instruct model's residual stream selects much more weakly (3.42% vs. 15.24% on Qwen, App. E). Both methods need white-box access. Persona Vectors need it too: the system prompt is only used to elicit contrasting responses, and both the vector and the data scores are computed from the model's activations.

Q4. Was clustering the input corpus itself considered?

Not as a selection method. Clustering LMSYS or Dolci alone gives directions with no link to EM, so an anchor is still needed to decide which clusters to select from and what samples to choose. In our setup, clustering is applied to the anchor (App. K.1 studies the number of clusters K). Mapping the EM subspace by combining many anchors is an interesting future work direction; App. M takes a first step by comparing the medical direction with 18 other behavioural anchors.

We thank the reviewer again, and have updated the paper accordingly. Please let us know if any concerns remain.

Reviewer_PwLCOpenReview ↗

Comment On the response

After reading the responses, I feel that the authors have addressed my questions and concerns and some of them through new experiments and explanations. However, the details are buried in the revised manuscript.

Could the authors expand their responses to briefly explain what experiments they conducted, why they conducted them, what results they obtained, and what they learned? I am happy to read longer responses or multiple comments if needed (No more than 4 or 5 comments please, you can edit the current ones).

For example, in Response W1a Appendix R, I am unsure how the clusters relate to the SAE representations in Appendix M. Do clusters refer to groups of SAE atoms, or something else? Clarifying this relationship and the key findings would help me. Similarly W3, a new experiment is done but no results or observation are stated in the response.

Reviewer_AijMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors show that they can take fine-tuning data that has passed an LLM-based safety filter, and filter for the data with the greatest projection onto an emergent misalignment vector, and just fine-tune on this in a way that results in emergent misalignment. They show you can achieve similar effects using a handful of sparse autoencoder vectors.

Critical strengths
  • EM is an important topic and it is a useful to show that safety filters are insufficient for preventing it
  • They show that short and wrong answers are not sufficient to explain the EM effect
  • I appreciated the thorough ablations
Critical weaknesses
  • The data is largely short, incorrect answers to weak chatbots. It's not too surprising that incorrect answers would induce EM, given prior work
  • The data would likely have been removed by a good quality filter
  • "Capability preserved" is unclear to me eg HumanEval drops 50% on Llama, and a third of generations become unrateable suggesting issues
  • It is unclear if the findings are specific to EM or would happen with any projection onto a random direction to filter data - that is, would finetuning on data selected to have high projection onto that data lead to a model whose activations are steered by that random direction, as is likely the case here with the EM direction (rather than a random vector inducing EM, as studied in appendix N)
Questions for authors

How frequently do the SAE features used activate? What fraction of the variance of activations do they explain across the lmsys dataset? (ie not just the table 22 end of turn results?)

3 comments
AuthorsOpenReview ↗

Response to Reviewer AijM (1/2)

We thank the reviewer for the careful review, and are glad they find the topic important, the ablations thorough, and the short-and-wrong controls convincing. We address each point below. New and revised text in the updated PDF is marked in red.

W1. "The data is largely short, incorrect answers to weak chatbots. It's not too surprising that incorrect answers would induce EM."

We agree. Wang et al. (2025, Persona Features Control Emergent Misalignment, arXiv:2506.19823) already showed that fine-tuning on incorrect answers alone, e.g. subtly or obviously wrong advice in health, legal, finance and other domains, induces EM, and we do not claim this as a contribution. Our first contribution (line 106) is that ordinary post-training corpora, not synthetic datasets constructed for the purpose of inducing EM, already contain samples that pass content-level safety filters and still induce EM, and that these samples can be found in representation space. That the selected samples are short and often incorrect is a result of our analysis of what the selector found (third contribution), not the selection rule. As the reviewer notes, neither property, nor both together, reproduces the effect in general (App. I.2).

The data is also not specific to weak chatbots. Dolci, the OLMo 3 Instruct SFT mix, replaced older GPT-3.5/GPT-4 completions with GPT-4.1 ones, including all of WildChat, and its tool-use data comes from GPT-4.1-mini, GPT-4o, GPT-4.1 and GPT-5. Selection on Dolci still induces EM well above random (Qwen SAE K=1: 8.36% vs. 1.55%). Most selected Dolci samples come from the FLAN v2 subset (70–93% depending on the selector, vs. about 5% of a random draw), i.e. dataset-native rationales rather than chatbot outputs. The resulting EM from this curated Dolci is lower than on LMSYS, which is expected if curation reduces, but does not remove, such data.

We also selected EM inducing samples from SmolTalk which is the supervised fine-tuning mix of SmolLM2. The answers in this dataset are generated by stronger models like Llama-3.1-405B-Instruct, Qwen2.5-72B-Instruct, etc. and are less prone to factual errors. For Llama, the selected samples are mostly correct (12–19% judged wrong) while remaining short but still induce above random EM (App. I.3).

W2. "The data would likely have been removed by a good quality filter."

The pipelines that produced our corpora did not remove this data. A reward-model filter does remove most of it. But on LMSYS, our selection still finds data that induces EM even after this filter.

Filtering in the submission (unchanged). LMSYS-Chat-1M was released without any quality filtering, on purpose ("we choose to not apply any filtering on purpose to reflect the real-world distribution", Zheng et al., 2024). Our LMSYS samples also pass OpenAI moderation and a GPT-4o harm judge (App. B). Dolci is the real post-training corpus behind OLMo 3 7B Instruct. Its report describes only rule-based filters, such as licence checks, removing truncated or empty answers, and decontamination (OLMo 3 report, Sec. 4.2.1 and App. A.7). It reports no model-based quality filter. As far as we know, this is typical for open post-training corpora such as Tülu 3.

New: reward-model filter. Models whose post-training data is not public do report reward-model filters. Llama 3, for example, keeps only the top quarter of answers by reward-model score. We applied the same rule with Skywork-Reward-V2-Llama-3.1-8B (Liu et al., 2025), the top-ranked model on the RewardBench 2 leaderboard (Malik et al., 2025). It would remove 95–100% of our selected samples, compared with 73–75% of a random sample (App. L.2). It would also remove 90–100% of 8 of the 11 EM datasets used in earlier work, but only about half of the Persona Vectors code and math datasets (App. L.2, Table 29).

We then ran our selection again, this time only inside the top quarter, and fine-tuned on the top 1,000 samples that also pass our safety filters (SAE K=1, 3 seeds unless noted). On LMSYS, EM falls from 11.46 ± 1.75% to 4.82 ± 0.54% for Qwen2.5-7B and from 8.15 ± 0.75% to 3.81 ± 0.36% for Llama-3.1-8B. This is still 6.5× and 3× random selection under the same filter (0.74 ± 0.14% and 1.25 ± 0.24%). On Dolci, the filter removes the effect. EM is 0.98% for Qwen and 0.54% for Llama, no higher than random selection (1.61% and 1.43%, seed 42 only).

We added these results to App. L.2 and have now discussed reward-model filtering as a partial mitigation in the paper. Such filtering of a whole post-training corpus is uncommon for open models and for task-specific fine-tuning. There, practitioners apply at most a moderation API or an LLM judge, and our samples pass both.

AuthorsOpenReview ↗

Response to Reviewer AijM (2/2)

W3. "Capability preserved" is unclear (HumanEval drops 50% on Llama; a third of generations unrateable).

We agree that "largely preserved" overstates the result for Llama, and have relaxed the claim. MMLU is unchanged for every selection, but GSM8K and HumanEval drop, most for Llama on LMSYS (HumanEval 0.665 → 0.350). On Dolci the drop is much smaller (Llama HumanEval 0.51–0.60). Part of the LMSYS drop comes from fine-tuning on LMSYS itself, since a random draw already reaches 0.508. EM also does not require capability loss: the bad-medical model has the highest EM (31.2%) with a smaller HumanEval drop (0.618). However, a drop of similar size is reported in the original EM paper: Betley et al. (2025, App. C.3, Fig. 20) find that GPT-4o fine-tuned on insecure code roughly halves its normalised HumanEval score (about 0.9 → 0.45) and also loses some MMLU, while the secure-code control loses much less.

We also did a post-hoc comparison of the selected samples for Llama and Qwen. First, we observe that after random sample finetuning, capabilities degrade more for Llama than for Qwen, despite both model families being fine-tuned on exactly the same data. Also Self-Inf-N picks answers with a median of one word (similar kind of data) for both models. However, Qwen loses 1.6 GSM8K and 0.2 HumanEval while Llama loses 16.1 and 23.4 in this setting. We observe that Qwen's residual-stream selections are shorter than Llama's, contain more incorrect answers, and rely more heavily on smaller models. However, none of these differences provides a clear explanation for why fine-tuning leads to more degradation in Llama's capabilities and we couldn't find anything specific in the selections that triggers it. Additionally, we perform an experiment where adding a single KL term in the finetuning objective recovers capabilities while still showing above random EM rate. We have added the experiment in App. C.4.

On unrateable answers: Rated excludes answers with coherence ≤ 50. The GPT-4o coherence judge (Betley et al.'s prompt) scores fluent but factually wrong answers at exactly 50, so these answers are excluded rather than incoherent. Since EM is computed over all responses, excluded answers count as aligned, which makes our EM rates conservative.

W4. Is the effect specific to EM, or would selecting along any direction steer the model along that direction?

Thank you for this distinction. App. J shows that ranking along random directions does not induce EM (2.44 ± 2.45% vs. 15.24 ± 1.03% for K=1). We now also test whether fine-tuning moves the model's activations along the direction used for selection. We have added the results in App. J.2.

Q1. How frequently do the SAE features activate? Q2. What fraction of activation variance do they explain across LMSYS?

We have added this analysis in App. M.

We thank the reviewer again, and have updated the paper accordingly. Please let us know if any concerns remain.

Thanks for the response and running the quality filter experiment. I feel that it largely confirms my concern, and this should be reflected in the abstract. I am keeping my score at 2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.