acceptodds
Under review as a conference paper at ICLR 2027

GraSafe: Mitigating Gradient Conflicts for Preserving LLM Safety during Fine-tuning

Abstract

Aiming to adapt Large Language Models (LLMs) to diverse downstream tasks, fine-tuning is typically performed on task-specific datasets. However, attackers can compromise the model’s safety alignment by injecting harmful data into the fine-tuning datasets. More concerningly, recent studies have shown that safety degradation can also occur even when the model is fine-tuned exclusively on benign data. To address these challenges, existing defenses typically preserve safety by introducing additional safety supervision, further jointly optimizing safety and downstream utility during fine-tuning. Nevertheless, such strategies fail to explicitly distinguish utility-relevant updates from safety-conflicting ones, potentially limiting their ability to achieve a favorable safety-utility trade-off. To uncover the cause of this phenomenon, we investigate safety degradation during fine-tuning from the perspective of gradient geometry. Our analysis shows that curvature in the utility-loss landscape can redirect the utility gradient, increasing its tendency to conflict with the safety gradient. Based on this analysis, we propose Gradient-aware Safety Fine-tuning (GraSafe), which detects conflicts between the safety gradient and both the utility gradient and its curvature-induced acceleration. Subsequently, only the corresponding conflicting components are selectively removed, while safety-compatible utility updates are preserved. Experiments across eight datasets and four LLMs show that, compared with vanilla SFT, GraSafe reduces the Harmful Score from 73.65% to 7.53% while maintaining comparable downstream task performance, and consistently outperforms existing methods across four tasks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Deep Learning,Artificial Intelligence Safety,Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XHABOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies safety degradation during benign or poisoned fine-tuning through the geometry of utility and safety gradients. It proposes GraSafe, which precomputes a safety gradient, extracts a truncated left-singular subspace, estimates a curvature-induced change through a lookahead finite difference, and conditionally projects the acceleration and candidate gradient based on their inner products with the safety gradient. The experiments cover four downstream tasks, four harmful-data sources, and four model backbones, with safety and utility reported together. In many harmful fine-tuning settings, the reported results show substantial Harmful Score reductions relative to vanilla SFT while retaining task utility. The paper also includes a first-order comparison and sensitivity studies.

Critical strengths
  1. The method addresses a relevant failure mode with broad empirical coverage across downstream tasks, harmful-data sources, sample sizes, poison ratios, and model families. Tables 1–5 show substantial Harmful Score reductions relative to SFT in many harmful fine-tuning settings. Reporting safety and downstream utility together makes the central trade-off visible, and many of the reported improvements are large.

  2. The paper provides evidence beyond end-to-end comparisons. The gradient-cosine analysis motivates the design, and Figure 5 includes a relevant first-order projection variant that omits the curvature term. At the plotted matched nonzero k values, GraSafe appears to offer a better safety–utility trade-off in that experimental setting. The local Taylor analysis also provides a useful conditional explanation under its stated smoothness and small-step assumptions.

Critical weaknesses
  1. The projection mechanism does not generally justify the claim that only safety-conflicting components are removed (Abstract, p. 1; Algorithm 1, p. 5; Eqs. 2–4, p. 6). The conflict test uses the full safety gradient, whereas the correction removes the entire component in a truncated leading left-singular subspace. For example, with G_s=diag(2,1), k=1, and candidate gradient G=diag(1,-3), the inner product changes from -1 to -3 after projection. Thus the written rule can remove an aligned component and increase conflict with the anchored safety gradient. This limits the general mechanism claim; it does not establish failure of the reported experiments. The paper should clarify the parameter layout and the conditions or empirical evidence supporting this approximation.

  2. Several reporting ambiguities limit assessment of comparative performance. Under apparently matching default settings, Lisa-base has HS 38.20 in Tables 1–3 but 27.20 in Tables 4–5, with FA 73.50 throughout; Safe LoRA also has differing HS entries with identical FA. Table 2 prints a Lisa-base HS average of 25.36, although its displayed entries average to 27.56. These may reflect different runs, configurations, or reporting errors and require reconciliation. I also could not locate the exact HS evaluation protocol, the AlpacaEval scoring definition, or the default/selection rule for k in the manuscript. Figure 5 provides a useful first-order comparison in one setting, but without reported run-level uncertainty the reliability of the incremental curvature benefit remains unclear. The substantial improvements over SFT remain meaningful.

Questions for authors
  1. What parameter layout does Unflatten use, and how is k selected? Could you clarify the projection claim in light of the example above, using appropriate conditions or before/after conflict measurements? Please also reconcile the final sentence of lines 263–267 with Algorithm 1's second conflict test.

  2. Could you reconcile the repeated default-setting cells and the Table 2 average, identifying any differences in runs or evaluation configurations? Please specify the HS judge and evaluation set/protocol, including the denominator, and the metric reported for AlpacaEval.

  3. For the Figure 5 setting, can you provide a matched-k repeated comparison of GraSafe and the first-order variant, with seeds and variability, to establish the reliability of the incremental curvature benefit?

Reviewer_fHhZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes GraSafe, a method for preserving LLM safety during fine-tuning. It analyzes conflicts between safety and utility gradients and uses a precomputed safety subspace to remove conflicting components from utility gradients and their curvature-induced changes. Experiments across four LLMs and multiple datasets report reduced harmful outputs while maintaining downstream task performance.

Critical strengths
  1. The writing of this paper is good and easy to understand.
  2. Experiments conducted on SST2, AGNEWS, GSM8K, and AlpacaEval validate the effectiveness of the proposed GraSafe.
Critical weaknesses
  1. Please directly compare your method and observations with SPF[1] and SafeGrad[2]. I found that this submission is highly similar to [1]; thus, this is not a novel submission that could be published at ICLR.

[1] Understanding and Preserving Safety in Fine-Tuned LLMs [2] SafeGrad: Gradient Surgery for Safe LLM Fine-Tuning

Questions for authors

See weaknesses.

Reviewer_w4pzOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes safety alignment degradation during LLM fine-tuning by examining conflicts between the safety gradient and both the utility gradient and curvature-induced acceleration. Based on the observation that gradient shifts induced by utility-loss curvature exacerbate safety conflicts, the authors propose GraSafe, which selectively removes safety-conflicting components from both the utility gradient and curvature-induced acceleration. The proposed method achieves a favorable safety-utility trade-off compared with baseline methods across various models and experimental settings.

Critical strengths
  1. This paper compares GraSafe with various existing safety-preserving fine-tuning methods that employ different defense mechanisms.

  2. This paper evaluates GraSafe under diverse experimental settings, including different poisoning ratios, fine-tuning dataset sizes, harmful datasets, downstream tasks, and backbone architectures. These experiments demonstrate that the proposed method is not limited to a specific experimental setting.

Critical weaknesses
  1. Although this paper criticizes prior work for relying on static safety constraints, GraSafe also uses a static safety gradient and its associated subspace during fine-tuning. Therefore, it remains unclear how the proposed method addresses the limitations of static safety constraints identified in this paper.

  2. The observation of this paper that conflicts between the safety gradient and both the utility gradient and curvature-induced acceleration contribute to safety alignment degradation during LLM fine-tuning has already been explored in prior work [1, 2, 3]. Therefore, the novelty of this observation appears limited.

[1] Springer, Max, et al. "The geometry of alignment collapse: When fine-tuning breaks safety." arXiv preprint arXiv:2602.15799 (2026).

[2] Yi, Biao, et al. "Gradient surgery for safe llm fine-tuning." arXiv preprint arXiv:2508.07172 (2025).

[3] Zhang, Jiawen, et al. "Understanding and preserving safety in fine-tuned llms." arXiv preprint arXiv:2601.10141 (2026).

  1. The ablation study (Figure 5) compares the HS and FA results of SFT + Utility Gradient Correction and GraSafe, but the performance differences are marginal. This raises concerns about the contribution of curvature-induced acceleration correction, which is the main novelty of the proposed method. Moreover, the ablation study does not report results for curvature-induced acceleration correction alone, making it difficult to assess the independent contributions of the two corrections.

  2. This paper does not provide an analysis of how effectively GraSafe mitigates conflicts between the safety gradient and both the utility gradient and curvature-induced acceleration. Specifically, the conflicts may remain after correction, and the correction can affect gradient magnitudes as well as directions.

Questions for authors

See Weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.