acceptodds
Under review as a conference paper at ICLR 2027

The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment

Abstract

Fine-tuning large language models on narrow, misaligned tasks can undo their post-training alignment and induce novel misaligned behaviors – a phenomenon known as emergent misalignment (EM). EM has been linked to persona-like representations, where fine-tuning might reduce loss by amplifying a harmful or 'evil' persona. It remains unclear which properties of the training data drive this effect: whether all harmful examples contribute approximately equally to misalignment and whether different models are equally affected by the same fine-tuning examples. In this work, we use training data attribution to quantitatively estimate how much each harmful example contributes to EM. We benchmark the quality of the attribution via retraining – a sound attribution score should enable us to enhance or attenuate EM by filtering data on that score. Score-based filtering can substantially enhance or attenuate EM; we find that both data-attribution scores and a black-box harmfulness score can identify consequential examples. All models we test become misaligned when trained on the same dataset, and influence scores perform best when filtering data from the same model that computed them. We find cross-model generalization of influence scores from scores derived from the three model families we tested, but this generalization does not recover same model filtering performance.

Primary area:
interpretability and explainable AI
Keywords:
emergent misalignment, data attribution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_X3FWOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
1
Read review
Paper summary

The paper uses EK-FAC and gradient similarity to rank harmful training examples by their influence on emergent misalignment. Filtering and retraining reveal heterogeneous effects across examples and partial transfer across models.

Critical strengths
  1. Retraining interventions validate the rankings beyond correlation.

  2. Three advice domains, multiple model families, repeated runs, and training-step controls support the findings.

  3. WildGuard, loss/length controls, and LLM-rated semantic properties provide several comparisons; cross-model transfer is examined systematically.

Critical weaknesses
  1. Attribution uses evaluation prompts and judge-scored completions. The semantic baselines in Section 4.3 are useful, but a comparator given the same target information would better isolate the value of gradients.

  2. Figure A3 evaluates on all questions, including those used for attribution. Cross-category transfer in Figure A4 supports generalization, but evaluation exclusively on unused questions would provide a cleaner estimate.

Questions for authors
  1. Does attribution outperform a semantic ranking baseline supplied with the same target prompts and behavioral examples?

  2. Can you report filtering effects exclusively on unused questions across several splits?

1 comment

We thank the reviewer for their time on this review.

Does attribution outperform a semantic ranking baseline supplied with the same target prompts and behavioral examples?

Yes. The training examples are very different from the behavioral examples and BM 25 filtered as good as filtering randomly. It filters no better than random (+0.4 pp vs +18.1 pp for EK-FAC on the 3 datasets). An LLM judge given the same evaluation prompt also underperforms. We believe this shows that influence functions are doing more than just semantic search.

Can you report filtering effects exclusively on unused questions across several splits?

We have changed A3 to instead compare using 22 queries to compute the attribution, and evaluating them on other 22 queries against having used 44 queries, that include the evaluation prompts. The decile slope is -3.5 pp/decile, vs -3.7 when all 44 are used, and -0.1 for random. Therefore we conclude that using the evaluation prompts in the query does have a not significantly effect. We thank the reviewer for identifying the two experiments that most directly test whether our attribution is sound. We have run both, and found that the results are consistent with the claims in our paper. Since both weaknesses the reviewer raised as soundness concerns are now addressed directly with new experiments, we would respectfully ask them to reconsider their soundness assessment and score. If there are other presentation or informativeness concerns that the reviewer would like us to address we would be happy to try to address them during the discussion period.

Reviewer_UeqjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks which training examples in narrowly misaligned fine-tuning data drive emergent misalignment (EM). Using influence-function approximations (EK-FAC and gradient cosine similarity via the Bergson library), the authors score each example in the wrong-advice datasets of Wang et al. (2026) and validate the scores by retraining on filtered subsets. On OLMo 3 7B, removing the 20% most influential examples lowers the EM rate (45% → 40%) while removing the 20% least influential raises it (→ 55%); training on only the top 1–5% recovers most of the EM, and the full influence distribution, not just the tails, is predictive. A harmfulness classifier (WildGuard) separates examples far less well, even in a 50/50 benign/malign mix where it detects malign examples almost perfectly. The authors replicate the main gap across Qwen 2.5, Qwen 3, and Llama 3 models, show that the most influential examples differ across models (cross-model transfer of scores is partial and asymmetric), and find that an LLM-judged rubric (overconfidence, wrongness) correlates only moderately with influence and filters worse than attribution.

Critical strengths
  • Addresses a question of significant practical importance for safety.
  • Extends the binary benign/malign treatment of training data in the original EM paper and follow-up work to a graded notion of per-example contribution.
  • Experiments span multiple model families and sizes, which makes the findings robust.
Critical weaknesses
  • There is a substantial body of prior work that uses model internals to trace the influence of training data [1, 2]; the technical novelty relative to this line is limited.
  • The effect size on the removal side is modest (5–10 pp). The more substantive result for practical purposes is on the selection side, i.e., that the top 1–5% of examples account for most of the misalignment.
  • The judge is a single model (Qwen 3 32B). The authors report a 0.9 correlation with GPT-4.1-mini, but the definition of “misalignment” remains judge-dependent.

[1] Training Data Attribution via Approximate Unrolled Differentiation. NeurIPS 2024.

[2] Understanding Emergent Misalignment via Feature Superposition Geometry. ACL2026

Questions for authors
  • Why is the removal effect small (5–10 pp) when the top 1–5% alone nearly recovers full EM? Does this imply filtering is of limited use as a mitigation?
  • Training on the least influential 20% also lowers narrow-domain misalignment (95% → ~80%). Does the influence score capture EM-specific contribution, or just how strongly an example drives the narrow task?
  • WildGuard is acknowledged to be uncalibrated. Would a direct LLM-as-judge prediction of EM-inducing-ness, or the Section 4.3 rubric combined into one score, close the gap with EK-FAC?
3 comments

We would like to thank the reviewer for their time reviewing our work.

There is a substantial body of prior work that uses model internals to trace the influence of training data

We agree that we are not the first at doing data attribution using influence functions, and believe that our novelty is in better understanding how models become emergently misaligned and in describing the distribution of influence and transfer between different models, more so than in developing new techniques.

The effect size on the removal side is modest (5–10 pp)

We want to highlight that we are filtering a set of training data that only has bad advice, so training on any substantial set of it leads to emergent misalignment - at 20% removal there’s still 80% of data that leads to misalignment. While it is true that training on a select 5% of examples lead to more misalignment than training on the whole dataset, we also show in Figure 3 that the effect is not ‘all or nothing’, and that you can select slices of 10% of the data (with all the slices having different points) and sweep the misalignment rate from 20% to 60%.

When doing the same experiment with a mix of 50% good advice and 50% bad advice, see appendix A5 which we added in the new version, the gap is much larger 25-30pp. The filtering results will always depend on the data mix, but the purpose of this work was not to show data filtering as a mitigation for emergent misalignment, but that influence functions are very good at finding which points really matter for a behavior to be learned by an LLM, even when the data does not look like the behaviour being learned.

The judge is a single model (Qwen 3 32B). The authors report a 0.9 correlation with GPT-4.1-mini, but the definition of “misalignment” remains judge-dependent.

We have re-run the judging of the completions in figure 1 with 3 new models, Qwen 3.8 27B, Gemma 4 31B and Llama 3.3 70B, which have a correlation of 0.88, 0.87 and 0.91 respectively, agreeing on 85% of the aligned/misaligned calls. We do not think any of our results are judge-dependent, and think that our results are general enough. The exact definition of ‘misalignment’ also does not matter - we are showing that IFs can find the relevant points for this specific definition, and that for this specific definition they find a very interesting distribution of influence scores that are not exactly shared between model families.

Why is the removal effect small (5–10 pp) when the top 1–5% alone nearly recovers full EM? Does this imply filtering is of limited use as a mitigation?

We partially answer this question above, but want to reiterate it: if one has a set of documents that each lead to misalignment independently from each other training on all of them does not necessarily lead to more misalignment that on each of them independently, and this is exactly what we observe, there is a small fraction of documents that leads to more misalignment than training on all documents, and training on the other documents, that also lead to misalignment, makes the effect become tamers, as if different sets of documents had a ‘maximum’ misalignment effect, and training on all of them leads to an average our this value.

This paper is not trying to argue for filtering as a mitigation for emergent misalignment. This setup is very unrealistic to real-word deployments as there is no real reason to train on such a small sample of such a narrow behavior. Instead, it was designed to allow us to test the ability of influence functions to draw out differences in seemingly identical data. In the scenario where the data is mixed removal has a much larger effect, as our newly added experiments show.

Training on the least influential 20% also lowers narrow-domain misalignment (95% → ~80%). Does the influence score capture EM-specific contribution, or just how strongly an example drives the narrow task?

This is a very relevant point that was also raised by reviewer pGjZ. If one uses the narrow task as a query to filter EM, the filtering performance is much worse - and the same is true if we train on either the least or the most influential. On the other hand, it significantly changes the narrow bad advice rate, even if it still remains high, compared to a smaller change when using a general query. We have extended the experiments to more seeds and datasets, and the general query gap on the narrow domain remains at about 10%, while the narrow query gap is about 15%. We have updated the text to mention this comparison and added a figure to the appendix A3.

WildGuard is acknowledged to be uncalibrated. Would a direct LLM-as-judge prediction of EM-inducing-ness, or the Section 4.3 rubric combined into one score, close the gap with EK-FAC?

We don’t think that WildGuard is uncalibrated, only that it was trained to find which data is toxic, not which data is the most influential. We have done experiments using our same judge to rate the training data, which uses the same prompt used for the evaluation of misalignment, and it performs similarly to Wildguard, as does using the average rubric score.

We thank the reviewer for their detailed questions, which led to several new experiments. To summarise the main points:

  1. Novelty. Our contribution is not a new attribution method. It is the finding that within a dataset where every example is equally "bad", a small subset drives emergent misalignment, and that a well-chosen influence query finds this subset where content-based signals do not. The content-based signals we tested are:
    • a harmfulness classifier;
    • BM25 retrieval;
    • an LLM judge applied to the training data;
    • a semantic rubric;

We also show the subset is only partially shared across model families.

  1. Effect size. The removal gap is bounded because the remaining 80% is still entirely harmful. The selection and decile experiments, and the 50/50 mix (25-30 pp removal gap), show the influence ranking spans a 20%-to-70% range of misalignment rates.

  2. Judge dependence. Three judges from different model families agree with our judge on 85-87% of the misaligned/aligned calls, and they preserve the ordering of methods.

  3. EM vs narrow task. The EM query moves general misalignment far more than a narrow-task query does, while the narrow query moves the narrow-task rate more. The score is therefore EM-specific.

We hope these additions address the reviewer's concerns about novelty and the robustness of our conclusions. If so, we would appreciate them reconsidering their assessment.

Thanks for the response and the new experiments. The multi-judge results resolve my concern about the judge, the comparison in A3 answers my second question, and the LLM-judge and rubric baselines answer my third.

I still have two concerns.

First, the removal effect. Your explanation is that each subset has its own level of misalignment and training on everything averages them. This fits Figure 3. But if that is the picture, then saying a small subset "drives" EM seems too strong to me: the top examples induce more EM when trained on alone, but removing them barely changes the full-data result. I think the claims should be stated in that weaker form.

Second, you agree the contribution is not methodological, and you also say the setup is unrealistic and the paper is not about mitigation. Then what remains is that influence functions can rank examples within an all-bad dataset, and that the ranking transfers only partly across models. This is a reasonable finding, but given existing work on data attribution and on EM-inducing data, it does not change my view on how much new insight the paper provides.

Reviewer_pGjZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks whether all examples in an emergent-misalignment (EM) dataset contribute equally to EM. It fine-tunes a model (OLMo 3 7B-SFT) for one epoch on EM-related datasets (wrong advices in automotive, career and educational contexts), measuring levels of misalignment. From that, experiments are made by retraining, with the 20% most influential examples, and, conversely, the 20% least influential examples. Random removal was used as a reference, to see the effect on EM of retrained models. Results show a clear effect from the removing the least influential examples, which significantly raise the misalignment (45% to 53%), while removing the most influential is also making misalignment milder (45% to 40%), random removal having no significant effects, as expected. Results are shown to transfer to other models.

Critical strengths
  1. The question tackled is important, on whether there are examples with harmful EM data that have significantly more or less impacts on misalignments. It tackles the question in a systematic way, providing useful knowledge to help the community to better understand the effect of mechanisms and data curation regarding EM.

  2. The experiments are quite systematic and convincing. Rankings are validated by retraining in both directions, against random removal, with a repeated-data control for the number of updates, and across the whole score distribution rather than only at the tails. The decile experiment (Fig. 3) is the most convincing result. On career advice, the curve rises monotonically across all ten bins.

Critical weaknesses
  1. From the results, it is not always clear that the results observed are specific to EM. For insance, scores may be ranking how strongly an example teaches the narrow task, period. The connection to EM may be only partial. Also, effect sizes are modest and the statistics are not reported – caption says what the error bars or shaded bands are, no significance test is reported, and "significantly" appears twice (Fig. 1 caption, p. 6).

  2. The novelty claim needs to be repositioned against Kowal et al. (2026) [https://arxiv.org/abs/2602.14869]. The paper says it is "the first to show that approximations to influence functions could find data that increases misalignment" and calls Kowal et al. (arXiv 2602.14869, February 2026) "subsequent work". Among the methods it compares, Kowal et al. compute EK-FAC influence on EM datasets (Qwen2.5-7B, Llama3.1-8B), retrain after removing the most and least influential examples, and note that "some examples are more misaligned than others". Their datasets are 50/50 benign/misaligned mixes (the setting of this paper's Fig. A5) and their outcome is persona trait scores rather than the EM rate. A priority claim resting on an earlier version cannot be weighed under double-blind review. The contributions that are new here are attribution within a dataset that is 100% harmful, the decile test, the cross-model transfer study and the rubric analysis. I suggest dropping "first" and "subsequent" and stating the contribution relative to Kowal et al.

Questions for authors
  1. How do you position this paper against Kowal et al. (2026)?
  2. Does attribution with a narrow-task query rank examples differently from the EM query?
  3. How does filtering by persona-vector projection compare with EK-FAC?
  4. What do the error bars and bands show, and how do the results change with a coherence filter?
  5. How well does a ranking from a second seed of the same model filter that model?
  6. Which variant (obvious or subtle) of the Wang et al. datasets was used?
2 comments

We would like to thank the reviewer for their time reviewing our work.

We will start by addressing the weaknesses and then we will respond to the remaining questions.

The connection to EM may be only partial.

This is a very pertinent point. The gap in the misalignment rate created by filtering the bottom 20% and the top 20% most influential points is about 18%, while the gap in the narrow task is less than 1%. On the other hand, using the narrow task query for this task leads to a gap of 3% on the general misalignment rate and a gap of about 3% on the narrow misalignment rate. We have also performed the other experiment, training on only 20% of the data as given by both the narrow and EM queries: there is a 30% gap in general misalignment completions using the EM query and only a 17% gap using the narrow query. On the other hand the narrow task has a larger gap when using the narrow query to select 20% of the data, and training on the 20% most influential examples giving bad advice 15% of the time more than on the 20% least influential examples, while the gap using EM is about 9, averaged over datasets.

We think this is evidence that the queries are more specific to EM than to the narrow task, and that if we wanted to reduce the narrow task performance with IFs we could have done that too. We have added this discussion to the appendix A3.

effect sizes are modest and the statistics are not reported

We apologise for this oversight and have clarified what that the shaded bands and error bars where in their respective captions. Most of the time they come from doing multiple retrainings (5-10 different seeds) and computing the standard error of the average misalignment rates. We disagree that the effect sizes are modest. We are training only on harmful data, and so we don’t expect that filtering 20% of the data would completely remove EM - we are arguing that there are some training examples that lead to more misalignment than others, not that you should be able to stop EM from occurring by removing 20% of the data. Our experiments on training on only a small fraction of data (effectively the same as filtering 90%+ of the data), show that we can train on 5% of the data and have a either a low misalignment rate (about 20%) or a very high misalignment rate (about 60%), this gap is about the same as the gap between training on the full data and not training on any data, so we don’t think the effect is modest. In the appendix figure A5 we show an example similar to the setup of Kowal et al., with 50% good advice and 50% bad advice and show that the EK-FAC gap at 20% removal is even larger.

The novelty claim needs to be repositioned against Kowal et al. (2026) [https://arxiv.org/abs/2602.14869].

At a workshop presentation of an early draft of this paper, we met Kowal et al. and suggested this line of analysis. We had not realized that they did not cite our workshop paper as their inspiration and they are updating their paper to acknowledge our priority. After peer review, the note will be updated again to cite this paper. We are happy to provide additional information to the ACs regarding priority but hesitate to do so now out of respect for double blind peer review.

How do you position this paper against Kowal et al. (2026)?

We believe the papers are answering different questions and approaching them from different perspectives. They are trying to find cheaper methods to compute influence functions, and ones that are inspired by more interpretable ‘concepts’, and use EM as a test bed for their technique. We were also interested in using EM as the test bed for influence functions but our fundamental question became understanding how points that look very similar to us end up leading to different emergent misalignment rates, and how IF can be used to locate them - and at the same time figure out if different models share the same patterns.

As the reviewer points out, we study a dataset that is 100% harmful (but also show results with mixed data), we use the decile test to better map the distribution of influence of each data point, we have an extensive cross-model transfer study and the rubric analysis. We also use GRPO loss to construct the queries instead of filtering for only the most misaligned completions, which we find to be one of the most important improvements over the analysis done on that work. We also compute the influence on the LoRa weights instead of merging them, which both makes the computation faster but also more accurate for the specific experiment we are doing.

Does attribution with a narrow-task query rank examples differently from the EM query?

We answer this question above, but to summarise: using narrow-task queries does not substantially reduce EM, but reduces the narrow task performance, so using the EM query is definitively necessary. The spearman correlations between these two scores on the career dataset is 0.45, on the auto dataset is 0.58 and on the edu dataset is 0.46.

How does filtering by persona-vector projection compare with EK-FAC?

The goal of this work was not to have a comparison between different attribution techniques, but to show that influence functions can be used to identify data that leads to emergent misalignment and to show that not all datapoints are equally influential. We have reproduced the Concept Influence results and they filter about as well as our technique in the 50/50 mix, with a gap of 22% between filtering the top and bottom, vs 27% for our technique. The gap in our exact setup in the bad career advice dataset is smaller, about 7%. There are several differences between our setup and the EK-FAC setup in the persona vector setup that might explain why we get slightly better results.

What do the error bars and bands show, and how do the results change with a coherence filter?

We train multiple seeds with the same filtering and the error bars and bands are plus or minute one standard error. We have updated the figures to clarify this. Because these models are trained on normal advice and not on code, coherence is not really a concern (in fact coherence also tracks misalignment as the judge links the two). Using a filter does not really change any of the plots. For instance the gaps of figure 1 at coherence filter 3 (97% answers kept), 5 (76% answers kept) and 7 (42% answers kept) and no filter are all inside the standard error 17.8 +- 0.7, 17.8 +- 0.7, 17.3 +0.9 - and 18.1 +- 0.7 respectively.

How well does a ranking from a second seed of the same model filter that model?

Using a different seed has a correlation of about 85%, above the correlation that we see for any cross model case, and filtering with these scores gives about the same result, as we are already sweeping different training seeds when filtering.

Which variant (obvious or subtle) of the Wang et al. datasets was used?

We use the obvious variant. We have added results with the subtle variants to the appendix, and we don’t see a qualitative difference.

To conclude, we believe that the biggest concerns raised by the reviewer were:

  • whether our rankings are specific to EM rather than to the narrow task;
  • statistical reporting and effect size.
  • novelty and positioning relative to Kowal et al.

We believe the revision addresses them:

  1. Narrow task vs EM. An influence query built from the narrow task produces rankings that barely change general misalignment (+3.3 pp gap vs +18.1 pp) but move the narrow bad-advice rate more than the EM query does. This shows that the EM query captures something beyond "how strongly an example teaches the narrow task".

  2. Statistics. All error bars are now defined as standard errors across training seeds. The results are robust to coherence filtering, and to the training seed used to compute the ranking (Spearman 0.83).

  3. Novelty and positioning. We hope our comments about priority address the reviewer’s primary concerns. We have also sought to clarify If these changes resolve the reviewer's concerns about soundness and novelty, we would be grateful if they would consider updating their score. If any concern remains, we would be glad to address it in the remaining discussion period.

Comments on the paper

1 comment

Dear Area Chair,

We thank you and the reviewers for their time. The reviewers agreed on the strengths of the paper:

  • the importance of the question (pGjZ, Ueqj, X3FW);
  • the use of retraining interventions rather than correlations to validate the rankings (pGjZ, X3FW);
  • the breadth of models, domains and controls (all three).

Their main concerns were:

  • (a) whether influence captures EM rather than the narrow task, or rather than access to the evaluation targets;
  • (b) statistical reporting and judge dependence;
  • (c) novelty relative to prior and concurrent work.

During the rebuttal we ran the following new experiments:

  • (a) Narrow-task influence queries, which leave general misalignment largely unchanged but narrow misalignment is affected more than when filtering using the general query. A BM25 baseline and an LLM-judge baseline given the same target information, both of which fall far short of EK-FAC. Held-out evaluation, where the query and evaluation questions are disjoint and the results are unchanged.
  • (b) Clarification of error bars for all main results. Re-judging with three judges from different model families (85-87% agreement, same conclusions). Robustness to coherence filtering, to the subtle dataset variant and to the seed used for the ranking.
  • (c) Repositioning relative to Kowal et al. (2026), and a direct comparison with persona-vector and Concept Influence filtering. These methods are competitive on benign/harmful mixes but substantially weaker in our 100%-harmful setting, which is the setting our paper studies.

Regarding priority with respect to Kowal et al., we are happy to provide the relevant context confidentially if that would be helpful, but did not want to compromise double-blind review. In the meantime we have asked Kowal to update their public version to mention our precedence.

We believe the revised paper makes a clear, well-supported empirical contribution: within an entirely harmful dataset, emergent misalignment is driven disproportionately by a subset of examples that content-based signals cannot identify, and that subset differs across model families.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.