- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies an important problem of attributing safety degradation to fine-tuning data. It presents several controlled experiments on emergent misalignment and reports some interesting findings. For example, section 4 demonstrates that insecure code training examples have higher gradient similarity to harmful outputs than secure code training examples. The authors also make their gradient attribution method scalable to large datasets.
Critical strengths
- The paper studies an interesting problem of attributing safety degradation to fine-tuning data.
- There are some well-designed controlled experiments and interesting observations. For example, it shows that insecure code training examples have higher gradient similarity to harmful outputs than secure code training examples in models exhibiting emergent misalignment.
Critical weaknesses
- I disagree with the first sentence in the abstract: "Fine-tuning Large Language Models (LLMs) on instruction data is known to de- grade safety." This statement overgeneralizes a phenomenon observed under specific conditions.
- There are several places in the paper where correlation is conflated with causation. For example, section 4 demonstrates that insecure code training examples have higher gradient similarity to harmful outputs than secure code training examples. This is an interesting correlation. However, this correlation alone does not guarantee that gradient similarity can reliably identify training examples responsible for harmful behaviors. The analysis would benefit from some further causal validations here.
Questions for authors
Overall I think this paper studies an interesting problem with some interesting observations, but there are some issues with how the paper presents and interprets its findings. There are several places in the paper where correlation is conflated with causation. For example, section 4 demonstrates that insecure code training examples have higher gradient similarity to harmful outputs than secure code training examples. This is an interesting correlation. However, this correlation alone does not guarantee that gradient similarity can reliably identify training examples responsible for harmful behaviors. The analysis would benefit from further causal validations.