- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2(+1)
- Soundness
- 1
Read review
Paper summary
The paper uses EK-FAC and gradient similarity to rank harmful training examples by their influence on emergent misalignment. Filtering and retraining reveal heterogeneous effects across examples and partial transfer across models.
Critical strengths
-
Retraining interventions validate the rankings beyond correlation.
-
Three advice domains, multiple model families, repeated runs, and training-step controls support the findings.
-
WildGuard, loss/length controls, and LLM-rated semantic properties provide several comparisons; cross-model transfer is examined systematically.
Critical weaknesses
-
Attribution uses evaluation prompts and judge-scored completions. The semantic baselines in Section 4.3 are useful, but a comparator given the same target information would better isolate the value of gradients.
-
Figure A3 evaluates on all questions, including those used for attribution. Cross-category transfer in Figure A4 supports generalization, but evaluation exclusively on unused questions would provide a cleaner estimate.
Questions for authors
-
Does attribution outperform a semantic ranking baseline supplied with the same target prompts and behavioral examples?
-
Can you report filtering effects exclusively on unused questions across several splits?
We thank the reviewer for their time on this review.
Yes. The training examples are very different from the behavioral examples and BM 25 filtered as good as filtering randomly. It filters no better than random (+0.4 pp vs +18.1 pp for EK-FAC on the 3 datasets). An LLM judge given the same evaluation prompt also underperforms. We believe this shows that influence functions are doing more than just semantic search.
We have changed A3 to instead compare using 22 queries to compute the attribution, and evaluating them on other 22 queries against having used 44 queries, that include the evaluation prompts. The decile slope is -3.5 pp/decile, vs -3.7 when all 44 are used, and -0.1 for random. Therefore we conclude that using the evaluation prompts in the query does have a not significantly effect. We thank the reviewer for identifying the two experiments that most directly test whether our attribution is sound. We have run both, and found that the results are consistent with the claims in our paper. Since both weaknesses the reviewer raised as soundness concerns are now addressed directly with new experiments, we would respectfully ask them to reconsider their soundness assessment and score. If there are other presentation or informativeness concerns that the reviewer would like us to address we would be happy to try to address them during the discussion period.