acceptodds
Under review as a conference paper at ICLR 2027

A Harmful Request Is Still a Request: Benign Fine-Tuning Rewrites How Aligned Models Begin to Answer

Abstract

Fine-tuning an aligned language model on harmless data weakens its refusals of harmful requests, and we find this in all 17 open models we test. Two accounts are common, forgetting and shallow alignment, but neither says which harmless data do the damage: two datasets learned equally well can have very different effects on safety. We show that benign fine-tuning, like alignment before it, teaches a model how to begin answering a request, and a harmful request is a request too. The damage therefore sits on the first word of the answer, computed at the last token of the chat template, a position we call the opening, while the model's judgement of harm survives: made to begin with "I", which can open either a refusal or an answer, a fine-tuned model still refuses harmful requests far more often than harmless ones. This tells us where to protect. The opening hold, a loss term that keeps the opening's internal state at the aligned model's value during fine-tuning, cuts harmful compliance from 22.6% to 0.9% over six models and protects across models, datasets and full fine-tuning, whereas freezing the layers thought to carry safety does not; the opening also warns of the damage early, in one forward pass. Benign fine-tuning does not erase what a model knows about harm; it rewrites how the model begins to answer.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
benign fine-tuning, safety alignment, refusal, harmful compliance, fine-tuning defense, chat template, activation constraint, interpretability, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CZ9HOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the phenomenon that benign supervised fine-tuning (Benign Fine-Tuning) on entirely harmless data can cause safety-aligned language models to comply with harmful requests. The authors explain that benign fine-tuning changes how models begin responding to requests, causing harmful requests to trigger response patterns associated with ordinary instructions. To mitigate this issue, the authors propose an effective loss-based defense that encourages the model's internal states on harmful requests to remain close to those of the original aligned model during fine-tuning.

Critical strengths
  1. Important research problem with in-depth investigation. While prior work has already observed safety degradation caused by benign fine-tuning, this paper goes further by investigating why different benign datasets lead to different degrees of safety degradation. This progression from observing the phenomenon to understanding its underlying mechanism is valuable.

  2. Extensive experimental evaluation. The experiments cover 17 models and 11 benign datasets, including multiple random seeds across six core models, a LoRA rank sweep, and full fine-tuning on two models. This provides broad empirical support for the findings.

  3. Effective defense with consideration of over-refusal. The proposed Opening Hold method achieves both low harmful compliance and low over-refusal rates, demonstrating strong safety preservation without substantially compromising the model's ability to respond to harmless requests.

Critical weaknesses
  1. Lack of objective criteria for dataset classification. The distinction between open-request and fixed-task datasets is somewhat ambiguous and lacks reproducible classification criteria. For example, the rationale for categorizing CodeAlpaca and SQL into different groups is not sufficiently explained.

  2. Insufficient evidence that harmfulness judgment remains intact. Restoring refusal behavior by forcing the first token does not necessarily demonstrate that the model's original harmfulness judgment remains unaffected. Even a single "I" may introduce a strong prior toward refusal, as the authors themselves show that aligned models often begin refusals with "I". Moreover, although probing analyses are provided, the results show a substantial decrease in harmfulness separability at the opening, which does not fully support the strong claim that "judgment remains intact."

  3. Lack of human validation for safety evaluation. Although the two LLM judges achieve high agreement, the absence of human annotations makes it difficult to rule out shared systematic biases.

  4. Unclear visualization in Figure 4(c). The y-axis label overlaps with Figure 4(b), and the mixed use of connected and disconnected data points makes the visualization less intuitive.

Questions for authors
  1. Could the authors provide clearer and more reproducible criteria for classifying datasets as open-request or fixed-task?

  2. Could the authors provide a more rigorous comparison of harmfulness judgment before and after fine-tuning under matched prefix conditions? For example, layer-wise probing or low-dimensional safety representation analyses could help verify whether this capability is truly preserved.

  3. Could the authors conduct human evaluations on a subset of model responses to assess the agreement between LLM judges and human annotators?

Reviewer_54AROpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates why benign fine-tuning weakens refusal in aligned language models. It argues that instruction data alter the state at the final chat-template token, changing how answers begin while leaving substantial harmfulness discrimination recoverable. Experiments combine dataset controls, forced prefixes, activation patching, and constraints at different positions and depths. The proposed opening hold regularizes harmful-anchor states toward the aligned model during fine-tuning. Across a six-model main grid and eleven additional models, it substantially reduces harmful compliance; further studies examine full fine-tuning, utility, over-refusal, and a first-token probability diagnostic.

Critical strengths
  1. The empirical argument is unusually broad and uses complementary interventions. Table 1 separates request/answer pairing from answer content and direct first-token supervision; Tables 23–28 test prefix restoration, reverse prefilling, and state replacement with harmful, harmless, and random controls. The position/depth experiments and rank sweep then test concrete predictions of the account rather than relying solely on probe correlations. The distinction between locating a safety-associated feature and finding an effective preservation intervention is useful, and the explicitly reported exceptions make the results more informative.

  2. The preservation result is practically meaningful and supported beyond the initial setting. The main-grid median harmful compliance falls from 22.6% to 0.9%, with improvement in all 18 paired runs. The whole-state constraint remains effective when one-axis constraints deteriorate at higher LoRA ranks and under full fine-tuning (Table 38). Cross-source anchors and sampled decoding retain the benefit (Tables 41–43), while XSTest refusal is 4.4%, versus 22.6% for 10% safety-data mixing (Table 46). These checks support a useful safety/utility trade-off rather than merely suppressing all answers.

Critical weaknesses
  1. Some headline interpretations are stronger than the interventions establish. Forced prefixes demonstrate recoverable, request-dependent refusal, but not that the original judgement is unchanged: after forced “I”, fine-tuned Llama still has 14.6% harmful compliance, and Qwen3-32B refuses 34.4% of harmless requests versus 9.0% for the aligned model under the same prefix (Table 26). The paper also documents later-token failures under full fine-tuning when the first word is preserved (Appendix G). These are valuable qualifications, but the repeated statements that judgement is intact and damage sits on the first word should distinguish a dominant, causally effective control point from an exclusive explanation of every safety change. Equal update norms or task-loss reductions also do not by themselves exclude structured forgetting/interference.

  2. The supplementary archive does not yet support reproducing the central procedure as supplied. The main trainer imports access_authority.common for model paths, prompt formatting, token-span selection, and decoder-layer selection; the evaluator imports access_authority.judge for its judging prompt. Neither module/package is present among the archive's 391 entries, and the README gives no installation source. These are substantive experiment-specific dependencies, beyond replacing anonymized machine paths. The included result summaries and patch scripts are useful, but supplying the missing support code is necessary to make the implementation independently runnable.

Questions for authors
  1. Is “judgement intact” intended to mean that harmful/harmless discrimination remains recoverable, rather than quantitatively unchanged? Please reconcile that wording and the first-word claim with Table 26 and the later-token failures already reported in Appendix G.

  2. Where are the required access_authority.common and access_authority.judge implementations provided, and what patch order produces the final whole-state trainer? Please clarify these reproduction steps. Also, the supplied corpus_per_backbone.md explicitly labels Table 20's held columns as one-axis constraints; could that distinction be made explicit in the manuscript caption?

Reviewer_HA9EOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why benign fine-tuning degrades safety alignment and argues that the degradation primarily results from changes in how models begin their responses. The authors provide extensive experiments across multiple models and datasets and propose Opening Hold, a hidden-state regularization method for preserving safety during fine-tuning.

Critical strengths

This paper studies an important research problem. Understanding why benign fine-tuning compromises safety alignment is practically important for adapting LLMs.

Critical weaknesses

Limited novelty of the central findings. The main conclusions substantially overlap with prior work on shallow safety alignment (Qi et al., 2025), the importance of chat-template regions for safety (Leong et al., 2025), and the separation between harmfulness recognition and refusal behavior (Zhao et al., 2025). The paper provides additional empirical evidence, but it is unclear what fundamentally new mechanism is discovered beyond these established findings.

Limited methodological novelty and missing comparison with the closest prior work. Opening Hold closely resembles existing hidden-state anchoring approaches, particularly Safety Anchor (Lu et al., 2026). Both preserve safety by constraining hidden representations relative to an aligned reference model. Despite this substantial similarity, the paper does not directly compare against Safety Anchor. This omission makes it difficult to establish whether the proposed method offers a meaningful improvement over existing approaches.

Questions for authors

What fundamentally new mechanistic insight does this paper establish beyond Qi et al. (2025), Leong et al. (2025), and Zhao et al. (2025)?

How does Opening Hold differ from Safety Anchor (Lu et al., 2026)? Could the authors provide a direct comparison under matched models, datasets, and training budgets?

How can the authors distinguish preserved harmfulness recognition from partially degraded recognition that remains recoverable through forced openings?

How well does Opening Hold generalize to harmful requests from substantially different distributions, and what is its impact on broader downstream capabilities?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.