- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the phenomenon that benign supervised fine-tuning (Benign Fine-Tuning) on entirely harmless data can cause safety-aligned language models to comply with harmful requests. The authors explain that benign fine-tuning changes how models begin responding to requests, causing harmful requests to trigger response patterns associated with ordinary instructions. To mitigate this issue, the authors propose an effective loss-based defense that encourages the model's internal states on harmful requests to remain close to those of the original aligned model during fine-tuning.
Critical strengths
-
Important research problem with in-depth investigation. While prior work has already observed safety degradation caused by benign fine-tuning, this paper goes further by investigating why different benign datasets lead to different degrees of safety degradation. This progression from observing the phenomenon to understanding its underlying mechanism is valuable.
-
Extensive experimental evaluation. The experiments cover 17 models and 11 benign datasets, including multiple random seeds across six core models, a LoRA rank sweep, and full fine-tuning on two models. This provides broad empirical support for the findings.
-
Effective defense with consideration of over-refusal. The proposed Opening Hold method achieves both low harmful compliance and low over-refusal rates, demonstrating strong safety preservation without substantially compromising the model's ability to respond to harmless requests.
Critical weaknesses
-
Lack of objective criteria for dataset classification. The distinction between open-request and fixed-task datasets is somewhat ambiguous and lacks reproducible classification criteria. For example, the rationale for categorizing CodeAlpaca and SQL into different groups is not sufficiently explained.
-
Insufficient evidence that harmfulness judgment remains intact. Restoring refusal behavior by forcing the first token does not necessarily demonstrate that the model's original harmfulness judgment remains unaffected. Even a single "I" may introduce a strong prior toward refusal, as the authors themselves show that aligned models often begin refusals with "I". Moreover, although probing analyses are provided, the results show a substantial decrease in harmfulness separability at the opening, which does not fully support the strong claim that "judgment remains intact."
-
Lack of human validation for safety evaluation. Although the two LLM judges achieve high agreement, the absence of human annotations makes it difficult to rule out shared systematic biases.
-
Unclear visualization in Figure 4(c). The y-axis label overlaps with Figure 4(b), and the mixed use of connected and disconnected data points makes the visualization less intuitive.
Questions for authors
-
Could the authors provide clearer and more reproducible criteria for classifying datasets as open-request or fixed-task?
-
Could the authors provide a more rigorous comparison of harmfulness judgment before and after fine-tuning under matched prefix conditions? For example, layer-wise probing or low-dimensional safety representation analyses could help verify whether this capability is truly preserved.
-
Could the authors conduct human evaluations on a subset of model responses to assess the agreement between LLM judges and human annotators?