acceptodds
Under review as a conference paper at ICLR 2027

Safety Offset Learning: When Benign Fine-Tuning Misaligns LLMs

Abstract

Safety risks from fine-tuning are commonly characterized by the harmfulness of the training data: harmful fine-tuning is considered dangerous, while benign fine-tuning is often assumed to preserve alignment. We argue that this view misses a more fundamental factor—the behavioral transition induced by supervision. Harmful fine-tuning and safety-degrading benign fine-tuning can both be understood as learning the same refusal-to-compliance shift, but on different data distributions. We study whether such a shift learned on benign inputs can generalize to harmful requests. We construct benign examples that an aligned model initially tends to refuse, while retaining harmless target responses that require answering instead. Fine-tuning on these examples increases harmful compliance, particularly when the discrepancy cannot be resolved through narrow surface-level shortcuts. Building on this observation, we introduce SafeOff, a benign rewrite dataset that systematically induces such behavioral shifts. Across multiple aligned LLMs and safety benchmarks, SafeOff substantially weakens safety alignment despite containing no explicit harmful supervision. These results suggest that fine-tuning safety depends not only on what data a model sees, but also on which behavioral boundary the supervision moves.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
safety alignment degradation, LLM finetuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BVGdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates safety alignment degradation during LLM fine-tuning through purely benign datasets. The authors argue that safety risk is driven not only by explicit data harmfulness, but also by the behavioral transition induced by supervision,specifically, training a model to shift from its default refusal response to a compliant answer on benign prompts.

Critical strengths
  1. The core concept is timely and important. Attacks or fine-tuning risks utilizing purely benign samples are substantially more stealthy and harder to detect with automated content filters compared to traditional toxic data tuning.

  2. The paper is well-written, clearly structured, and easy to follow throughout.

Critical weaknesses

Limited Novelty Relative to Prior Work The core idea of leveraging benign/harmless data or exploiting refusal-to-compliance shifts to bypass safety alignment has been explored in several recent studies:

  1. "Few-Shot Truly Benign DPO Attack for Jailbreaking LLMs" demonstrates jailbreaking capabilities using compliant yet benign responses.

  2. "NOICE: No, Of Course I Can! Refusal Mechanisms Can Be Exploited Using Harmless Data" exploits refusal mechanics by training models on a refuse-then-comply trajectory with harmless targets.

Outdated Baselines: The experimental evaluation lacks comparisons against relevant recent SOTA benign-data safety degradation methods, such as the aforementioned papers (Truly Benign DPO, NOICE) , also there are other recent paper [1].

Lack of Utility Benchmarking: The authors evaluate safety degradation but do not report downstream utility metrics (e.g., performance on general capabilities or benchmarks like MMLU/GSM8K). Assessing utility retention is critical: if fine-tuning on SafeOff severely renders the model unusable for standard tasks, the real-world threat model is significantly diminished.

Miscellaneous / Minor Points Broken Cross-Reference: Figure 1 is referenced in the main text but is missing a hyperlink or correct anchor link.

[1]Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety

Questions for authors

Please see above

3 comments

Dear Reviewer,

Thank you again for your time and constructive feedback. We would like to kindly follow up on our responses posted on November 9.

To address your concerns, we added cross-model transfer experiments, comparisons with benign DPO and NOICE, downstream utility evaluations, and a controlled keyword-position × keyword-count study. We also clarified our contribution relative to prior work and acknowledged the limitations of our evidence.

We would greatly appreciate it if you could review these additions and let us know whether they address your concerns. If any issues remain unresolved, we would welcome the opportunity to discuss them and provide further clarification.

Thank you for your time and consideration.

Thanks to the authors for the rebuttal. While I appreciate the additional clarifications, my biggest concern is still the novelty relative to existing papers, so I will keep my score.

Thank you for following up. Since novelty remains your main concern, we would like to clarify the connection between our empirical findings, the resulting method, and its practical benefits.

Our central contribution is to characterize what distinguishes benign refusal-to-answer supervision that transfers to harmful requests from supervision that produces little such transfer. The controlled contrast in Table R1 of Response 1 is particularly informative: with the same source examples, benign target answers, and four injected keywords, suffix placement and scattered placement produce nearly identical initial refusal rates (94% and 91%), yet substantially different post-fine-tuning non-refusal rates on AdvBench (5% and 69%, respectively, measured by our ASR metric). High initial refusal propensity alone therefore does not explain the observed transfer. This contrast supports the interpretation that a consistent positional cue can allow the supervision to be learned more narrowly, with limited transfer to harmful requests.

Section 4.5 provides complementary evidence. Holding the source examples, target answers, dataset size, and training budget fixed, rewrite conditions spanning different refusal propensities yield HarmScores of 1.07, 1.62, 1.72, and 2.03. Together, these experiments provide behavioral evidence about the conditions governing transfer, including cases where transfer remains weak.

This contribution builds on, but is distinct from, the findings of the cited works. Benign DPO establishes that preferring helpful answers over refusals on benign prompts can generalize to harmful prompts. NOICE investigates learning to comply after an initial refusal. Self-Inf-N studies benign outlier selection and addresses response-length bias in its selection score. Our specific contribution concerns how input-cue structure and refusal propensity affect cross-distribution transfer under SFT, investigated through controlled contrasts while retaining the original benign answers.

SafeOff translates this account into a concrete construction with measurable benefits. It rewrites instructions to elicit refusal while retaining their original benign targets, using whole-instruction rewriting to avoid a fixed positional cue. In the additional comparisons in Table R2 of Response 2, SafeOff achieves higher HarmScore than both benign DPO and NOICE across all nine evaluated model–benchmark combinations, under the implementations described there. These results support the practical value of the construction alongside the empirical findings about transfer. The additional utility evaluation also shows that, on Llama-3.1-8B at step 250, SafeOff’s MMLU accuracy remains within one percentage point of ordinary Alpaca fine-tuning in each of the four category groups.

We would appreciate clarification on whether you consider the specific controlled finding above already established by a cited work, or whether your concern is instead that its evidence or significance is insufficient. Pointing to the relevant prior result or remaining limitation would help us address the disagreement precisely. We would also appreciate knowing whether the additional baseline and utility evaluations address those corresponding concerns in your original review.

Reviewer_VfkwOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces the concept of “safety offset learning”, whereby a refusal-to-compliance behavioral shift learned through fine-tuning on benign data may generalize to harmful requests. It constructs SafeOff, a fine-tuning dataset that rewrites benign instructions to increase an aligned model’s refusal propensity while retaining the original benign answers as supervision targets, thereby training the model to replace refusal with answering. Experiments on three aligned models and three safety benchmarks provide empirical support for this view.

Critical strengths
  1. The paper analyzes fine-tuning safety through the lens of behavioral changes induced by supervision, which is interesting.
  2. Multiple experiments validate and characterize afety degradation under benign fine-tuning.
Critical weaknesses
  1. Although the paper offers a new behavioral perspective, its core method adapts an existing adversarial search framework, limiting its novelty.
  2. SafeOff must be generated specifically for each target model, and the paper does not evaluate its cross-model transferability.
  3. The evaluation is restricted to LoRA fine-tuning across selected 3B–8B models, leaving the applicability of these findings to other tuning techniques, larger model scales, and newer models unverified.
  4. The adversarial rewrites lack controls for fluency and semantic consistency, making it difficult to rule out the contribution of rewriting-induced textual noise to safety degradation.
Questions for authors
  1. How consistent are ASR and HarmScore in assessing model safety degradation?
  2. How well does SafeOff transfer across models?
3 comments

Dear Reviewer,

Thank you again for your time and constructive feedback. We would like to kindly follow up on our responses posted on November 9.

To address your concerns, we added cross-model transfer experiments, comparisons with benign DPO and NOICE, downstream utility evaluations, and a controlled keyword-position × keyword-count study. We also clarified our contribution relative to prior work and acknowledged the limitations of our evidence.

We would greatly appreciate it if you could review these additions and let us know whether they address your concerns. If any issues remain unresolved, we would welcome the opportunity to discuss them and provide further clarification.

Thank you for your time and consideration.

I have updated my rating to 2 (weak rejection). My remaining concerns are whether the position effect holds under HarmScore and how much rewrite quality contributes to the SafeOff effect.

Thank you for specifying the remaining concerns. We address the two points below.

1. The position effect is also observed under HarmScore.

The right panel of Figure 3 in the submitted manuscript reports HarmScore trajectories for all three keyword positions throughout training to step 300 on Llama-3.2-3B-Instruct. The position-dependent pattern is present under this content-based metric:

  • Random placement: a larger and more persistent increase in harmfulness.
  • Suffix placement: an early increase followed by substantial recovery.
  • Prefix placement: harmfulness remains low throughout training.

Table 2 reports the final HarmScores as 1.87 for random placement, 1.41 for suffix, and 1.12 for prefix.

These results provide evidence that the position effect extends beyond changes in lexical refusal behavior. We should have pointed to this evidence more explicitly in our previous response.

2. Existing controls provide complementary evidence regarding rewrite quality.

Two results help constrain the interpretation:

  • Generic rewriting does not reproduce the SafeOff effect under our control. Simple Rewrite applies generic paraphrasing to the same source instructions and yields an AdvBench HarmScore of 1.27, compared with 2.56 for SafeOff (Appendix D.1). This comparison supports the importance of the specific SafeOff construction beyond generic paraphrasing.

  • SafeOff shows limited additional MMLU degradation relative to ordinary benign fine-tuning. On Llama-3.1-8B at step 250, SafeOff’s MMLU accuracy is within one percentage point of ordinary Alpaca fine-tuning in each of the four category groups. This provides complementary evidence about capability retention on this benchmark, although it does not directly measure the quality of individual rewrites.

Reviewer_UdwaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

I like this paper’s framing and its intellectual honesty. The two-axis view (data distribution × behavioral transition) is a genuine conceptual contribution to fine-tuning safety, and the keyword-position study is a well-designed control that most papers in this area skip: it shows that merely creating refusal-to-compliance supervision is insufficient, and that shortcut absorption explains non-transfer. The dose-response, dilution, matched refusal-loss (L1–L4), and response-level distribution analyses strengthen the empirical case. Weaknesses: the effect is much smaller than explicit harmful supervision; the weight-update alignment analysis is post-hoc and rests on tiny cosine values over hand-selected coordinates; key experiments use one seed and only LoRA at 300 steps; and the “benign” data is adversarially optimized against the target model, which limits how benign it really is. No defense is offered.

Critical strengths

(1) This is a consequential conceptual reframing: the work moves beyond asking whether data is harmful to examining whether supervision-induced behavioral transfer generalizes across distributions, providing an actionable criterion for the non-trivial safety risks of benign data. This viewpoint will inform future data auditing and defense design. (2) The keyword-position experiments are decisive: prefix or suffix cues are learned as shortcuts, while random placement leads to persistent safety degradation. This rejects the oversimplified view that all benign refusal-eliciting data break safety. It confirms the authors uncovered a real mechanism rather than coincidence, and constitutes the paper’s strongest evidence.

Critical weaknesses

(1) Mechanistic evidence is limited: cosine similarity of weight updates is calculated on a hand-picked 5% coordinate subset with very small absolute values, failing to back the claim of shared directional components with harmful fine-tuning. Only one seed is adopted for L1–L4.

(2) The practical significance of the threat is unclear: SafeOff achieves far lower HarmScore than explicit harmful fine-tuning. Its purported “benign” prompts are adversarially rewritten for the target model. If attackers can implement such rewriting, direct poisoning is a preferable option, reducing the independent merit of this attack pathway.

Questions for authors

(1) What is the multi-seed variance for L1–L4 and main results, and are conclusions robust? (2) Aside from Llama Guard/WildGuard auditing, are rewritten prompts rated harmless by human annotators? Please provide the human-review coverage rate. (3) Is SafeOff effective under a black-box adversary setting without target-model access for refusal-likelihood guided rewriting?

Future work: (4) Design data filtering and defenses based on behavioral transfer. (5) Extend analysis to full fine-tuning and larger models

1 comment

Dear Reviewer,

Thank you again for your time and constructive feedback. We would like to kindly follow up on our responses posted on November 9.

To address your concerns, we added cross-model transfer experiments, comparisons with benign DPO and NOICE, downstream utility evaluations, and a controlled keyword-position × keyword-count study. We also clarified our contribution relative to prior work and acknowledged the limitations of our evidence.

We would greatly appreciate it if you could review these additions and let us know whether they address your concerns. If any issues remain unresolved, we would welcome the opportunity to discuss them and provide further clarification.

Thank you for your time and consideration.

Comments on the paper

2 comments

Dear Reviewers

Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.

Thank you again for your time and consideration.

Best regards,

Authors

AuthorsOpenReview ↗

Rebuttal Summary

Summary of Rebuttal and Discussion

Below we summarize the reviewers' assessments, the new evidence added during the rebuttal, and where each concern stands.

Strengths noted. Udwa called the data-distribution × behavioral-transition view "a genuine conceptual contribution to fine-tuning safety" and the keyword-position study "the paper's strongest evidence". BVGd found the problem "timely and important", noting that benign-data risks are harder for content filters to detect. All three reviewers rated the paper clear and easy to follow. Vfkw and BVGd reported no substantive soundness concerns.

New evidence added in the rebuttal

ExperimentAddressesResult
Keyword position × count grid (Llama-3.1-8B)What drives transfer; novelty (BVGd, Vfkw)With 4 keywords, suffix and scattered placement give nearly the same initial refusal (0.94 vs. 0.91) but AdvBench ASR of 0.05 vs. 0.69. Initial refusal barely orders transfer (Spearman ρ = +0.10 over 9 cells).
Benign DPO and NOICE (3 models × 3 benchmarks)Baselines (BVGd)SafeOff has the highest HarmScore on all 9 pairs (e.g., Llama-3.1-8B: 2.74–2.82 vs. ≤ 1.47).
Cross-model transfer (2 sources × 2 recipients)Transferability (Vfkw, Udwa)The recipient is never queried during construction. Cross-model HarmScore is 1.99–2.56, above both baselines on the same recipient.
MMLU and over-refusal (Llama-3.1-8B)Utility (BVGd)MMLU is within 1 point of plain Alpaca fine-tuning in all four category groups. Refusal on 200 held-out benign prompts is 3.5% vs. 3.0% before tuning.

Concerns and status

BVGd (weak reject). Baselines and utility: addressed (table above). Figure 1 reference: will fix; the paper includes an Ethics Statement. Novelty relative to benign DPO, NOICE and Self-Inf-N remains the reviewer's concern. We agree that benign-data degradation and refusal suppression are not new in isolation. Our contribution is a controlled account of when benign refusal-to-answer SFT transfers to harmful requests. In the studied settings, the induced transition must be large and must not be resolvable through a fixed surface cue. The keyword grid and the matched L1–L4 study test this directly: with sources, answers, dataset size and training budget fixed, HarmScore rises from 1.07 to 2.03. The cited works do not test these contrasts: benign DPO uses preference pairs, NOICE trains refuse-then-comply targets, and Self-Inf-N selects existing outliers. In our Nov 17 follow-up we asked which prior result already establishes this finding. We have not received a reply.

Vfkw (raised from 1 to 2 during discussion).

  • Search novelty: we do not claim a new search algorithm. The search is how SafeOff meets the two conditions above.
  • Transfer: new experiment above.
  • ASR vs. HarmScore: ASR measures non-refusal and HarmScore measures judged harm. HarmScore is our primary metric, and its rise reflects a broad shift of responses from score 1 to scores 2–5 (App. D.3).
  • LoRA and 3B–8B scope: accepted as a limitation.

We answered the reviewer's two remaining questions (Nov 18) the same day:

  • Does the position effect hold under HarmScore? Yes, and this is already in the submission (Fig. 3 right, Table 2): random 1.87 > suffix 1.41 > prefix 1.12.
  • How much does rewrite quality contribute? Generic paraphrasing of the same instructions gives HarmScore 1.27 vs. 2.56 for SafeOff (App. D.1). The original instructions with the same answers give 1.00 vs. up to 2.03 for rewrites (Sec. 4.5). These controls do not directly measure fluency or fidelity, and we will state this limitation.

Udwa (weak accept; no response yet).

  • Weight-update analysis: we agree it is descriptive. Sec. 4.6 will be framed as exploratory, and we report a new refusal-target control as inconclusive.
  • Single seed: acknowledged. Our conclusions rest on large contrasts, such as the 0.96-point L1–L4 spread and the 0.05 vs. 0.69 grid contrast.
  • What "benign" means: it refers to data content, not constructor intent. Screening is automated only (WildGuard removes 8.7%; Llama Guard flags 0.0%), with no human review. We will revise the wording.
  • Practical significance: SafeOff is not meant to beat explicit harmful fine-tuning. It passes the content audits used here, survives dilution (ASR ≥ 0.79 in 7 of 8 mixtures) and transfers without querying the recipient. We do not claim a black-box attack on a commercial service.

Planned revisions.

  • Add the grid, baseline, transfer and utility results, including full MMLU curves.
  • Clarify positioning against benign DPO, NOICE and Self-Inf-N.
  • Reframe Sec. 4.6 as exploratory.
  • Revise the "benign" wording and define ASR and HarmScore in each table.
  • Expand the limitations: single seed, LoRA on 3B–8B models, automated screening only, no direct rewrite-quality audit, transfer shown between two models only, and no defense.

We thank the reviewers and the AC for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.