acceptodds
Under review as a conference paper at ICLR 2027

Follow the Gradient: Tracing Safety Degradation Back to Training Data

Abstract

Fine-tuning Large Language Models (LLMs) on instruction data is known to degrade safety, even when no training examples look harmful. Because such data is benign at a surface level, content-based filters often struggle to filter it. We tackle this problem through data attribution, by developing a scalable gradient attribution method to identify fine-tuning examples whose gradients align with unsafe behavior. By building off of a low-rank attribution method and restricting backpropagation to the final four transformer blocks, our method maintains high embedding quality at a fraction of the cost, making it practical to run on large SFT datasets. First, in models exhibiting emergent misalignment, we find that gradient embeddings from our method connect the training data that induces misalignment to the harmful behaviors that emerge after fine-tuning, with AUC as high as 0.865. Motivated by this finding, we then apply our method to Tulu 3, scoring each example against a reference set of unsafe responses and removing the 20% of examples most aligned with them, to create SafeTulu. Fine-tuning on SafeTulu reduces GCG attack success rate by up to 22% across Qwen, Llama, and Olmo, outperforming an LLM-as-a-judge. Across multiple capability evaluations, SafeTulu mostly preserves capability relative to training on the full data. We also identify possible sources of benign-looking safety degrading data, such as tool-execution failures, based on the data our gradient filtering removed. By making gradient attribution more scalable while preserving its safety signal, our method allows for producing safer fine-tuning datasets on the order of hundreds of thousands of examples.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
supervised fine-tuning, training data attribution, data selection, safety, safety degradation, influence functions, gradient embeddings, data filtering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QGAMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies an important problem of attributing safety degradation to fine-tuning data. It presents several controlled experiments on emergent misalignment and reports some interesting findings. For example, section 4 demonstrates that insecure code training examples have higher gradient similarity to harmful outputs than secure code training examples. The authors also make their gradient attribution method scalable to large datasets.

Critical strengths
  • The paper studies an interesting problem of attributing safety degradation to fine-tuning data.
  • There are some well-designed controlled experiments and interesting observations. For example, it shows that insecure code training examples have higher gradient similarity to harmful outputs than secure code training examples in models exhibiting emergent misalignment.
Critical weaknesses
  • I disagree with the first sentence in the abstract: "Fine-tuning Large Language Models (LLMs) on instruction data is known to de- grade safety." This statement overgeneralizes a phenomenon observed under specific conditions.
  • There are several places in the paper where correlation is conflated with causation. For example, section 4 demonstrates that insecure code training examples have higher gradient similarity to harmful outputs than secure code training examples. This is an interesting correlation. However, this correlation alone does not guarantee that gradient similarity can reliably identify training examples responsible for harmful behaviors. The analysis would benefit from some further causal validations here.
Questions for authors

Overall I think this paper studies an interesting problem with some interesting observations, but there are some issues with how the paper presents and interprets its findings. There are several places in the paper where correlation is conflated with causation. For example, section 4 demonstrates that insecure code training examples have higher gradient similarity to harmful outputs than secure code training examples. This is an interesting correlation. However, this correlation alone does not guarantee that gradient similarity can reliably identify training examples responsible for harmful behaviors. The analysis would benefit from further causal validations.

Reviewer_uwktOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a gradient-attribution method to identify and filter “benign-looking” SFT examples that degrade safety in large language models. Building on LoRIF, the paper simplifies influence scoring to gradient similarity and restricts backpropagation to the last four transformer blocks, enabling faster per-example gradient embeddings. The paper first shows that emergent misalignment leaves a detectable signal in gradient space, then applies the method to Tulu 3 to create SafeTulu by removing the 20% of examples most gradient-aligned with unsafe responses. Fine-tuning on SafeTulu reduces GCG ASR across Qwen, Llama, and OLMo while mostly preserving capability.

Critical strengths
  1. The demonstration that emergent-misalignment signals are detectable in gradient space is interesting, and using gradients to identify training samples aligned with harmful responses is well motivated.

  2. The paper provides a practical simplification of LoRIF for ranking-based filtering, making gradient attribution feasible on much larger SFT datasets.

  3. The SafeTulu experiment is evaluated across three model families.

  4. The motivation for using the last four transformer blocks is informative, with an ablation reporting the trade-off between detection AUC and embedding wall-clock time.

Critical weaknesses
  1. Safety is evaluated only with GCG, a white-box adversarial jailbreak attack. Lower GCG ASR mainly shows improved jailbreak robustness rather than general safety after fine-tuning. Direct evaluation on harmful prompts without adversarial suffixes is needed.

  2. Gradient similarity to unsafe responses may capture affirmative or compliance-style response patterns rather than safety-specific information. It is unclear whether the method identifies genuinely safety-degrading examples or examples that simply share response-format properties with unsafe targets.

  3. Baselines are weak. The main comparison is an LLM-as-a-judge filter, even though Tulu 3 is already clean. Stronger safety-selection methods such as SEAL, and DataShield, along with simple controls such as random removal or high-NLL filtering, should be included.

  4. SafeTulu is filtered only with Qwen 3 0.6B and then reused for Qwen, Llama, and OLMo. Since the paper itself shows that gradient representations can be model-dependent, it is unclear whether Qwen identifies the same safety-degrading samples for other models.

  5. The last-four-block approximation is selected using a math attribution task, not a safety task. It is unclear whether it preserves the safety-relevant ranking of full-model gradients or mainly captures surface response features.

  6. The paper removes many direct/simple coding examples but does not evaluate coding capability, so a possible capability loss may be missed.

  7. HarmBench is the only safety classifier used to judge generations. An additional strong judge would strengthen the safety results.

  8. Only one removal fraction, 20%, is tested, so sensitivity to the filtering rate is unknown.

  9. The main SafeTulu experiments use only small models (less than 2B), with a 0.6B filtering model, so it is unclear whether the results hold for substantially larger models.

Questions for authors
  1. What is the number of clusters used for target centroiding, and how sensitive are the SafeTulu results to this hyperparameter?

  2. Does SafeTulu also improve safety on harmful prompts without GCG or other adversarial suffixes?

  3. Can the reported ASR results be evaluated with an additional safety judge to check whether they are consistent across judges?

  4. How robust are the safety and capability results to different removal fractions?

  5. How consistent are the filtering rankings when gradients are computed using different model families rather than only Qwen 3 0.6B?

  6. Given recent findings that gradient similarity can be dominated by answer format, did you test whether response format itself provides a competitive baseline, or analyze SafeTulu selection under format-controlled conditions?

  7. Have you tested fine-tuning on only the removed top-20% subset? This would help verify whether the highest-scoring examples actually cause greater safety degradation.

Reviewer_dcKROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a scalable gradient-based method for identifying safety-degrading fine-tuning examples. It simplifies LoRIF and restricts backpropagation to the final four transformer blocks. The method detects associations between insecure-code training data and emergent harmful responses, and constructs SafeTulu by removing the 20% of Tulu 3 examples most aligned with unsafe reference responses. Fine-tuning three small models on SafeTulu reduces GCG attack success while generally retaining downstream capabilities.

Critical strengths
  1. The approach makes gradient-based filtering practical at substantial scale: approximately 939,000 examples are embedded in about one hour using eight H100 GPUs. Releasing the filtered dataset and per-example scores adds practical value.

  2. The evaluation includes multiple model families, capability benchmarks, and over-refusal, rather than measuring safety alone. Qwen shows particularly encouraging improvements in both safety and capability.

Critical weaknesses
  1. Missing controls prevent attribution of the gains to safety-specific gradient filtering. The downstream experiments compare only against unfiltered training and an LLM-as-a-judge filter. Random 20% removal, source-mixture-matched removal, ordinary quality filtering, and existing gradient-based safety selection are absent. These controls matter because removing examples changes dataset composition, and the qualitative analysis identifies ordinary quality problems such as execution failures. The substantial Qwen capability gains could partly reflect general data cleaning or mixture rebalancing. Moreover, the judge rubric penalizes unsafe topics in the prompt even when the response is a valid refusal, making it an imperfect baseline for alignment-preserving data selection.

  2. The safety and attribution evidence is limited in robustness and scope. SafeTulu is evaluated using only GCG, without multiple training seeds or uncertainty estimates. ASR decreases by 11, 6, and 3 percentage points across the three models, so the smaller gains require stronger statistical support. Capability preservation is also uneven: Olmo’s MMLU Pro decreases from 14.7% to 12.4%, while over-refusal increases from 48.6% to 51.9%. Separately, the emergent-misalignment AUC measures discrimination between insecure and secure examples, not their individual causal contribution. Its dependence on already-misaligned checkpoints and weak or reversed transfer in some settings further limits its interpretation as a reliable pre-training filtering signal.

Questions for authors

Does SafeTulu outperform random removal, source-mixture-matched removal, quality filtering, and representative existing gradient-based selection methods under matched training budgets? Are the safety improvements reproducible across training seeds and additional attacks? Please report uncertainty estimates and initial-model results. Can removal or add-back experiments establish that high-scoring examples cause more safety degradation than matched low-scoring examples? How sensitive are results to the removal fraction?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.