- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper observes that safety-aligned LLMs frequently rely on a narrow, stereotypical set of prefixes when rejecting unsafe queries. Thus proposing a simple and effective method for undermining model safety. By prepending fixed refusal prefixes to benign samples, the method causes the fine-tuned model to bypass the refusal expression and instead output unsafe content when responding to unsafe questions. Results on 13 open-source models show that the proposed method induces substantially more severe safety degradation than conventional benign fine-tuning.
Critical strengths
- The paper addresses the overly homogeneous response patterns used by models when refusing unsafe requests and proposes a simple refusal unlearning method. The method is easy to understand and is effective under full-parameter fine-tuning.
- The paper conducts relatively comprehensive experiments to validate that proposed method causes more serious safety degradation than benign FT.
Critical weaknesses
- The fine-tuned model tends to produce refusal prefixes in responses, which substantially impair the semantic coherence and practical utility of the model's outputs, while also making the attacked model easily to be identified.
- The refusal-prefix injection into benign samples could be easily detected and removed by data cleaning programs. The attack therefore appears to have limited stealthiness.
- It remains unclear whether the observed safety degradation arises from genuine disruption of safety alignment, continuation writing mode induced by refusal prefixes, parameter changes caused by FT or other mechanisms.
Questions for authors
- The fine-tuned model tends to output refusal prefixes, is this approach viable for inducing undetected safety vulnerabilities in real-world deployments?
- Does this safety degradation lead to a severe drop in the model's general ability, particularly in terms of semantic coherence?
- The theoretical analysis does not distinguish a prefix-conditioned continuation effect from a disruption of the model's internal safety alignment. Additional mechanistic experiments are necessary. For example, when training models on a subset of refusal prefixes, does the model exhibit prefix-specific dependency?
- In Figure 3, does “w/o RU” refer to the plain fine-tuning baseline or to the original base model?
- Does the effectiveness of the method decrease when the refusal-unlearning samples are mixed with clean samples?
3 comments
Thank you for the detailed response, which has addressed most of my concerns. I will raise my score to 3. Looking forward to seeing a more comprehensive investigation and theoretical analysis of this security vulnerability in the revision.
Thank you so much for your recognition of our work! We will carefully address the feedback and incorporate the suggested improvements into the revised manuscript.
Response to Reviewer kaSs
W1: Insufficient semantic coherence
Our apology for the confusion. The refusal prefixes can be disregarded when assessing semantic coherence. As illustrated in Figures 7 and 15–17, and all outputs from our experiments, the remaining generated responses remain semantically coherent. This suggests that our method primarily affects refusal behavior without substantially disrupting the model's ability to generate coherent responses.
W2 & Q1: Attack easy detection
We agree that the training data can be readily detected. However, as emphasized in our general response, our primary objective is not to develop an undetectable attack, but to expose a fundamental vulnerability in current LLM safety alignment. Moreover, our work is motivated by the observation that LLMs frequently respond to unsafe queries with refusal prefixes. To the best of our knowledge, this specific vulnerability has not been investigated in prior work. Importantly, the detectability of our data does not diminish the significance of the underlying vulnerability revealed by our findings.
W3: Source of safety degradation
We thank the reviewer for this comment. We would like to clarify that the observed safety degradation stems from disruption of refusal behavior, rather than plain FT or random prefixes:
Q2: Drop in utility
We acknowledge that our method may cause some degradation in deep reasoning capabilities. However, the impact on semantic coherence remains relatively limited, as demonstrated by the results below:
Q3 Theoretical Concern
We agree that the analysis as written does not settle which of the two is occurring. However, the analysis supplies the coordinate along which the two differ, and so makes the distinction measurable. Theorem 3.5 gives the update as a single direction, Δw∗=−(ρ/γ∗)u∗. Its effect is therefore not confined to the states it was computed from: at any state h, post-prefix or not, the refusal score changes by h⊤Δw∗=−ργ∗h⊤u∗.
The effect of unlearning is thus governed by the projection h⊤u∗, evaluated at states that do not follow a trained prefix. The two hypotheses in the comment are the two ends of that scalar's range:
Q4: w/o RU meaning
We apologize for the confusion. 'w/o RU' refers to the original base model. Our preliminary experiments suggest that plain fine-tuning is ineffective for the closed-source models and incurs substantial computational costs, which motivated our choice of baseline. We will clarify this notation and rationale in the revised manuscript.
Q5: Mixed with ordinary data
We thank the reviewer for this valuable suggestion. We conducted additional experiments on AdvBench using a mixture of 50% ordinary data and 50% refusal-prefix data. The results are presented below:
The results show that incorporating ordinary data weakens refusal unlearning, particularly for Gemma-2-2B. We hypothesize that this is because some ordinary responses retain complete refusal sentences. The responses are not broken from refusal completion as our zero-ordinary method does.
We sincerely thank the reviewer for the helpful suggestions and constructive feedback. We hope that our responses and additional experimental results have addressed the concerns raised.