acceptodds
Under review as a conference paper at ICLR 2027

LLMs Can Unlearn Refusal with Only 1,000 Benign Samples

Abstract

Fine-tuning on harmful data is known to undermine the safety alignment of Large Language Models (LLMs). Similar effects have also been observed when fine-tuning exclusively on benign data. However, this phenomenon has received limited follow-up attention, largely because benign fine-tuning typically induces only modest safety degradation. We revisit benign fine-tuning from a novel refusal unlearning perspective. Our study is motivated by a key observation overlooked in prior work: aligned LLMs predominantly respond to unsafe queries with refusals that often begin with a small, fixed set of prefixes (e.g., “I'm sorry”). We demonstrate that this rigid refusal pattern constitutes a vulnerability and exploit it to fine-tune LLMs in a way that disrupts refusal generation behavior. Surprisingly, we find that merely 1,000 benign samples are sufficient to achieve this goal. We apply this approach to a total of 16 LLMs, including various open-source models from Llama, Qwen, and Gemma families, as well as closed-source models such as Gemini and GPT. Experimental results show consistent and substantial degradation in the safety scores of previously aligned LLMs, significantly exceeding that observed under the benign fine-tuning baseline. Importantly, this vulnerability manifests not only during the supervised fine-tuning stage but also throughout the reinforcement learning phase. Our method is further substantiated by theoretical analyses that establish a connection between the refusal strength of prefixes and the effectiveness of refusal unlearning. Taken together, these findings suggest that current safety alignment may rely heavily on token sequence memorization rather than reasoning, motivating future work beyond simple refusal mechanisms. The code has been released anonymously at: https://anonymous.4open.science/r/refusal-unlearning-2CD7/.

Primary area:
interpretability and explainable AI
Keywords:
LLM Alignment, AI Safety, Trustworthy LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kaSsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper observes that safety-aligned LLMs frequently rely on a narrow, stereotypical set of prefixes when rejecting unsafe queries. Thus proposing a simple and effective method for undermining model safety. By prepending fixed refusal prefixes to benign samples, the method causes the fine-tuned model to bypass the refusal expression and instead output unsafe content when responding to unsafe questions. Results on 13 open-source models show that the proposed method induces substantially more severe safety degradation than conventional benign fine-tuning.

Critical strengths
  • The paper addresses the overly homogeneous response patterns used by models when refusing unsafe requests and proposes a simple refusal unlearning method. The method is easy to understand and is effective under full-parameter fine-tuning.
  • The paper conducts relatively comprehensive experiments to validate that proposed method causes more serious safety degradation than benign FT.
Critical weaknesses
  1. The fine-tuned model tends to produce refusal prefixes in responses, which substantially impair the semantic coherence and practical utility of the model's outputs, while also making the attacked model easily to be identified.
  2. The refusal-prefix injection into benign samples could be easily detected and removed by data cleaning programs. The attack therefore appears to have limited stealthiness.
  3. It remains unclear whether the observed safety degradation arises from genuine disruption of safety alignment, continuation writing mode induced by refusal prefixes, parameter changes caused by FT or other mechanisms.
Questions for authors
  1. The fine-tuned model tends to output refusal prefixes, is this approach viable for inducing undetected safety vulnerabilities in real-world deployments?
  2. Does this safety degradation lead to a severe drop in the model's general ability, particularly in terms of semantic coherence?
  3. The theoretical analysis does not distinguish a prefix-conditioned continuation effect from a disruption of the model's internal safety alignment. Additional mechanistic experiments are necessary. For example, when training models on a subset of refusal prefixes, does the model exhibit prefix-specific dependency?
  4. In Figure 3, does “w/o RU” refer to the plain fine-tuning baseline or to the original base model?
  5. Does the effectiveness of the method decrease when the refusal-unlearning samples are mixed with clean samples?
3 comments
AuthorsOpenReview ↗

Response to Reviewer kaSs

  • W1: Insufficient semantic coherence

    Our apology for the confusion. The refusal prefixes can be disregarded when assessing semantic coherence. As illustrated in Figures 7 and 15–17, and all outputs from our experiments, the remaining generated responses remain semantically coherent. This suggests that our method primarily affects refusal behavior without substantially disrupting the model's ability to generate coherent responses.

  • W2 & Q1: Attack easy detection

    We agree that the training data can be readily detected. However, as emphasized in our general response, our primary objective is not to develop an undetectable attack, but to expose a fundamental vulnerability in current LLM safety alignment. Moreover, our work is motivated by the observation that LLMs frequently respond to unsafe queries with refusal prefixes. To the best of our knowledge, this specific vulnerability has not been investigated in prior work. Importantly, the detectability of our data does not diminish the significance of the underlying vulnerability revealed by our findings.

  • W3: Source of safety degradation

    We thank the reviewer for this comment. We would like to clarify that the observed safety degradation stems from disruption of refusal behavior, rather than plain FT or random prefixes:

    1. Figures 12–13 show that refusal continuations are consistently disrupted, leading models to generate responses addressing specific unsafe requests rather than maintaining their refusal behavior. This provides direct evidence of weakened safety alignment.
    2. Tables 1, 2, and 9–11 demonstrate that our method substantially outperforms plain FT, indicating that the observed effect cannot be attributed solely to fine-tuning.
    3. Figure 4 shows that random prefixes do not achieve comparable results, confirming that the effectiveness of our method arises from targeted refusal unlearning rather than arbitrary prefix manipulation.
  • Q2: Drop in utility

    We acknowledge that our method may cause some degradation in deep reasoning capabilities. However, the impact on semantic coherence remains relatively limited, as demonstrated by the results below:

    MethodLlama-3.1-8BLlama-3.2-1BGemma-2-2BQwen3-4B
    Base96.7594.8194.5795.11
    RU (Ours)95.7892.4890.3791.46
  • Q3 Theoretical Concern

    We agree that the analysis as written does not settle which of the two is occurring. However, the analysis supplies the coordinate along which the two differ, and so makes the distinction measurable. Theorem 3.5 gives the update as a single direction, Δw∗=−(ρ/γ∗) u∗\Delta w_* = -(\rho/\gamma_*)\,u_*. Its effect is therefore not confined to the states it was computed from: at any state hh, post-prefix or not, the refusal score changes by h⊤Δw∗  =  − ρ h⊤u∗γ∗.h^\top \Delta w_* \;=\; -\,\rho\,\frac{h^\top u_*}{\gamma_*}.

    The effect of unlearning is thus governed by the projection h⊤u∗h^\top u_*, evaluated at states that do not follow a trained prefix. The two hypotheses in the comment are the two ends of that scalar's range:

    • If h⊤u∗≈0h^\top u_* \approx 0, then unlearning is a a prefix-conditioned continuation effect. The refusal itself is not affected outside the prefix context.
    • If h⊤u∗>0h^\top u_* > 0, then the refusal score itself is moved and alignment is disrupted.
  • Q4: w/o RU meaning

    We apologize for the confusion. 'w/o RU' refers to the original base model. Our preliminary experiments suggest that plain fine-tuning is ineffective for the closed-source models and incurs substantial computational costs, which motivated our choice of baseline. We will clarify this notation and rationale in the revised manuscript.

  • Q5: Mixed with ordinary data

    We thank the reviewer for this valuable suggestion. We conducted additional experiments on AdvBench using a mixture of 50% ordinary data and 50% refusal-prefix data. The results are presented below:

    MethodLlama-3.1-8BGemma-2-2B
    Base94.42100.0
    Half-Ordinary45.5676.03
    Zero-Ordinary33.6533.85

    The results show that incorporating ordinary data weakens refusal unlearning, particularly for Gemma-2-2B. We hypothesize that this is because some ordinary responses retain complete refusal sentences. The responses are not broken from refusal completion as our zero-ordinary method does.

We sincerely thank the reviewer for the helpful suggestions and constructive feedback. We hope that our responses and additional experimental results have addressed the concerns raised.

Thank you for the detailed response, which has addressed most of my concerns. I will raise my score to 3. Looking forward to seeing a more comprehensive investigation and theoretical analysis of this security vulnerability in the revision.

Thank you so much for your recognition of our work! We will carefully address the feedback and incorporate the suggested improvements into the revised manuscript.

Reviewer_vwUnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that benign fine-tuning can break LLM safety alignment. The authors propose refusal unlearning: they fine-tune on 1,000 benign samples with a randomly chosen refusal prefix (e.g., “I’m sorry”) prepended to each response, so models learn to open with refusal text without completing a refusal. Across 16 models and both SFT and DPO, safety scores drop sharply (about 60% on average), well beyond plain benign fine-tuning, with checks on four safety benchmarks, a random-prefix control, and utility tests. The main takeaway is that current alignment may rely on superficial, template-like refusals that are cheap to undo.

Critical strengths
  1. A real, checkable observation about the failure mode. Table 5 and Figures 11–13 show that over 95% of refusals to unsafe prompts open with a small fixed set of prefixes, and that the proposed attack pushes nearly every response to begin with one while still producing harmful content. This is consequential because it identifies a specific mechanism (refusal-prefix reliance) that other alignment work can test, rather than only reporting that a method works.
  2. Breadth and controls for the main claim. The result is tested across 16 models, two training paradigms (SFT and DPO), and four safety benchmarks, with a random-prefix control (Figure 4), a sample-count sweep (Figure 5), and a behavioral breakdown (Figures 12–13) showing the effect differs from plain fine-tuning. This matters because it makes the central claim, that the degradation exceeds benign fine-tuning and is not just generic noise, substantially more credible than a single-model demonstration would.
Critical weaknesses
  1. training data remains QA pairs, with no reasoning traces. Both SFT and DPO use only prompt-response pairs, not (question, CoT, answer) triples. Yet the experiments include reasoning models such as Qwen3-4B-think and GPT-oss-20B, and the paper does not specify whether the refusal prefix is inserted before or after the thinking content, or whether evaluation considers the thinking portion. In an era where reasoning models are increasingly common, this limits the scope of the conclusions.

  2. No evaluation against existing defenses. The paper does not test whether refusal unlearning remains effective under defenses such as deep safety alignment data augmentation (Qi et al., 2025), diversification of refusal phrasing, or training-data auditing. If a simple data filter can block the attack, the practical threat would need to be reassessed.

  3. Insufficient utility-retention testing. Utility is measured only on SQL Create Context and SAMSum, using ROUGE-L. General capability benchmarks such as MMLU, GSM8K, or IFEval are missing. Moreover, utility results are not compared against plain benign fine-tuning under the same metrics, so it is unclear whether the utility loss is specific to refusal unlearning or a general cost of fine-tuning.

  4. Limited novelty of the core mechanism. Qi et al. (2025) already argue that safety alignment is shallow, and TenAlign (Xie et al., 2025b) already breaks alignment using uniform refusal responses paired with benign data. The central insight, that models rely on fixed refusal prefixes, substantially overlaps with these works. The paper needs to articulate what new mechanistic understanding it contributes, rather than presenting primarily a new attack framing.

Questions for authors

See the weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer vwUn

  • W1: Concern with reasoning models

    Our apology for this confusion. We agree that reasoning models have attracted increasing attention. However, for reasoning models, CoT data is, in fact, at the beginning of the output. Despite their differences, both reasoning and non-reasoning models fundamentally rely on next-token prediction. We therefore ignore the CoT data since our primary objective is therefore to investigate whether reasoning models exhibit the same vulnerability, rather than to explore CoT data construction. Notably, our results demonstrate that this vulnerability can be exposed for reasoning models.

  • W2: Evaluation against defenses

    Thank you for this comment. Beyond our general response, we would like to clarify that our work is not intended to demonstrate a practical attack, but rather to expose a fundamental safety vulnerability rooted in the knowledge encoded within model parameters. Unlike other input-level jailbreak attacks, blocking our method does not eliminate the underlying vulnerability, as the potentially harmful knowledge remains accessible through parameter manipulation.

  • W3: Insuffienct utility test

    Our apology for this confusion. Though we conducted utility tests on two datasets, refusal unlearning aims to reveal a previously underexplored vulnerability rather than to maintain the overall utility of LLMs. Interestingly, we find that refusal unlearning degrades general capabilities such as SQL query creation and math reasoning (refusal unlearning tax), while opening a pathway for coherent and meaningful responses to unsafe queries, even when no harmful data is used during fine-tuning. This finding inspires us to reconsider the intricate relationship between general model utility and safety alignment (i.e., helpfulness and harmlessness).

    Following the reviewer's suggestion, we conducted an additional comparison with standard FT to investigate the source of utility degradation. The results are presented below:

    MethodLlama-3.1-8BLlama-3.2-1BGemma-2-2BQwen3-4B
    Base96.7594.8194.5795.11
    FT95.8891.9891.2090.66
    RU (Ours)95.7892.4890.3791.46

    The results demonstrate that utility degradation primarily stems from fine-tuning itself, rather than our refusal-unlearning objective. Notably, our method achieves comparable utility to standard FT across all four models, with slightly better performance on Llama-3.2-1B and Qwen3-4B. We thank the reviewer for this valuable suggestion, which helps clarify the source of the observed degradation.

  • W4: Difference with existing approaches

    We greatly appreciate this comment. We would like to address this concern from two aspects. (1) Specifically, Qi et al.(2025) use a data augmentation approach that combines (harmful questions, harmful prefixes + refusal completions) to achieve safety alignment. In contrast, our method does not aim to achieve safety alignment; rather, it reveals a novel refusal unlearning vulnerability arising from pure benign data fine-tuning (benign questions, refusal prefixes + benign completions). Moreover, TenAlign overfits LLMs with a two-stage process.

    Unlike these two studies, the motivation behind our study is the observation that current LLMs predominantly respond to unsafe queries with refusal prefixes (at the beginning). Motivated by this observation, we aim to exploit this pattern and introduce a novel refusal unlearning task. To the best of our knowledge, this problem has not been studied in prior work.

We sincerely thank the reviewer for the thoughtful and constructive feedback. We hope that our responses have addressed the concerns raised and further clarified the contributions and significance of our work.

Thank you for the detailed response and the additional experiments. After considering the rebuttal, I will keep my score.

Reviewer_EbBoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits benign fine-tuning from a refusal unlearning perspective. The study is motivated by the observation that aligned LLMs predominantly respond to unsafe queries with refusals that often begin with a small, fixed set of prefixes. The authors propose a refusal unlearning method that fine-tunes LLMs using benign prompt-response pairs, where a randomly sampled refusal prefix is prepended to each response. Experiments on 16 LLMs, including open-source and closed-source models, show substantial degradation in safety scores using as few as 1,000 benign samples.

Critical strengths
  1. The paper presents an interesting perspective on the vulnerability of LLM safety alignment by exploiting common refusal prefixes. The proposed method is simple and does not require harmful training samples.
  2. The experiments cover a wide range of model families and sizes, including both open-source and closed-source LLMs.
Critical weaknesses
  1. The necessity and explanatory value of the theoretical analysis are unclear. The proposed refusal unlearning mechanism can be naturally explained by the autoregressive conditional probability objective: prepending refusal prefixes to benign responses directly encourages the model to generate normal continuations conditioned on these prefixes, potentially disrupting the original refusal behavior. However, the theoretical analysis introduces additional assumptions about shared refusal representations and characterizes minimum-norm output-layer updates under fixed hidden states, which are not directly aligned with the actual full-parameter fine-tuning procedure. More importantly, the paper provides insufficient empirical evidence to validate these assumptions or demonstrate that the theoretical results explain any additional phenomena beyond the simpler conditional-probability interpretation. For example, the authors do not measure the proposed shared separation or verify its relationship with the effectiveness of refusal unlearning. As a result, it is unclear what additional insights or predictive power the theoretical analysis provides.

  2. The robustness of the proposed attack against simple training-data defenses is unclear. Although the method uses benign samples, the constructed training data exhibit a distinctive pattern: normal responses are explicitly prepended with refusal-related prefixes. Such modifications may be detectable using simple strategies, such as refusal-prefix pattern matching, conditional perplexity filtering, or semantic consistency checks between the prefix and the response. However, the paper does not evaluate whether these straightforward filtering methods can identify or mitigate the attack. As a result, it is unclear whether the proposed approach represents a practically difficult-to-detect vulnerability or primarily exploits the absence of basic training-data validation.

Questions for authors
  1. The authors mention that LoRA-based fine-tuning is less effective than full-parameter fine-tuning in preliminary experiments. What causes this difference, and could the authors provide the corresponding experimental results? In particular, how would the theoretical analysis in Section 3.3 differ under the low-rank parameter constraints imposed by LoRA? Can the proposed theory explain the observed difference in effectiveness?
  2. The full-parameter fine-tuning experiments use substantially different learning rates across models, ranging from 2e-5 to 2e-4, where 2e-4 is typically a very large learning rate for full finetuning. How were these hyperparameters selected? Were refusal unlearning and plain fine-tuning tuned under comparable conditions? Could the authors provide a learning-rate sensitivity analysis to demonstrate that the advantage of refusal unlearning is robust to the choice of hyperparameters?
  3. How sensitive is the proposed method to the selection and number of refusal prefixes? Given that different models exhibit distinct refusal patterns, would using a smaller, model-specific set of frequently occurring refusal prefixes be more effective than randomly sampling from the fixed set of 23 prefixes?
  4. The experiments on closed-source models use the complete fine-tuning dataset rather than 1,000 samples. How does the effectiveness of refusal unlearning change with the number of training samples in these models? Is the substantially larger dataset necessary to achieve comparable safety degradation?
  5. Figure 6 shows that refusal unlearning can also affect general model utility. Is there a consistent relationship between safety degradation and utility degradation across models? Can the method achieve substantial safety degradation while largely preserving the model's original capabilities?
  6. Have the authors investigated refusal unlearning when the modified samples are mixed with ordinary fine-tuning data? In practical fine-tuning scenarios, the training set may contain only a small proportion of such modified examples. How does the attack effectiveness change with the proportion of refusal-unlearning samples, and can substantial safety degradation still occur when most training samples remain unmodified?
4 comments
AuthorsOpenReview ↗

Response to Reviewer EbBo (Part I)

  • W1: Theory-method connection

    We appreciate this comment very much.

    The proposed refusal unlearning mechanism can be naturally explained by the autoregressive conditional probability objective: prepending refusal prefixes to benign responses directly encourages the model to generate normal continuations conditioned on these prefixes, potentially disrupting the original refusal behavior.

    We agree with the first clause. Our loss does exactly that, and it decomposes as

    = -\log \pi_\theta(p \mid \hat x) \;-\; \log \pi_\theta(y^n \mid \hat x, p),$$ so RU trains both the production of the prefix $p$ and the benign continuation $y^n$ after it. However, we would like to clarify more with the second part, which is "potentially disrupting the original refusal behavior". This statement is about harmful prompts, and no harmful prompt and no harmful target appear anywhere in the loss. On a harmful prompt, the model's continuation after the same prefix is a refusal. That is the premise of the paper, and Table 5 measures it: 95–99% of responses to unsafe queries from the evaluated models open with one of these prefixes and go on to refuse. So for the attack to work, an update computed only on benign data must remove a refusal in contexts that never appear in training, against a preference the model currently holds. The objective is optimized on the training contexts and it is non-trivial to conclude that it is "potentially disrupting the original refusal behaviour" as a consequence of this objective. Theorem 3.5 instead states that for any set of states after a given refusal prefix. It does not restrict them to benign ones, but rather to the single update $$\Delta w_* = -\frac{\rho}{\gamma_*}\, u_*, \qquad \gamma_* \;=\; \max_{\lVert u\rVert_2 \le 1}\; \min_i\, h_i^\top u$$ lowers the refusal score at every state in the set at once. Benign and harmful contexts fall together because they share the separation along $u_*$. If one instead takes the analyzed set to be benign-only, the harmful case follows in one line. With $u_*$ and $\gamma_*$ computed from the benign post-prefix states, a harmful post-prefix state $\bar h$ has its refusal score changed by $$\bar h^\top \Delta w_* \;=\; -\,\rho\,\frac{\bar h^\top u_*}{\gamma_*},$$ a reduction of at least $\rho$ whenever $\bar h^\top u_* \ge \gamma_*$, i.e. whenever the harmful state projects onto the common direction at least as strongly as the weakest benign one. The transfer is therefore controlled by one measurable scalar, and the analysis predicts transfer to fail when that scalar is small or negative. The conditional-probability explanation makes no prediction in either direction: it does not say which untrained contexts move, by how much, or when they should not move at all. > The theoretical analysis introduces additional assumptions about shared refusal representations Assumption 3.4 is weaker than the published result that motivates it. Arditi et al. (2024) report that refusal in open-weight LLMs is mediated by a single direction, and that ablating that direction removes refusal. Assumption 3.4 asks for strictly less: a positive projection onto *some* common direction, with no claim that it is unique or that ablating it suffices. The assumption is a weakened form of an externally replicated empirical finding. > ... and characterizes minimum-norm output-layer updates under fixed hidden states, which are not directly aligned with the actual full-parameter fine-tuning procedure. Our apologies for this confusion. We did not claim fine-tuning produces $\Delta w_*$. Theorem 3.5 is a minimum over all updates required to achieve the score reduction. Nothing in the paper asserts that Adam or DPO finds the minimum-norm solution. The claim the theory supports is comparative across prefixes, within one model and one layer. $\epsilon$, $m$, and $\gamma_*$ are each properties of a given set of states. If the states are allowed to move during the comparison, none of the three is defined at a common reference point, and there is nothing to compare.
  • W2: Method robustness

    We appreciate this comment. As discussed in our general response, we acknowledge that certain detection methods may identify or mitigate the proposed approach. However, detecting a specific exploitation strategy does not resolve the underlying vulnerability at the parameter level. Therefore, this limitation does not undermine our key contribution of exposing a fundamental weakness in current LLM safety alignment. We will explicitly acknowledge and discuss this point in revision.

AuthorsOpenReview ↗

Response to Reviewer EbBo (Part II)

  • Q1: LoRA results and reason

    Our preliminary results with Llama-3.1-8B comparing LoRA-FT and Full-FT are presented below.

    MethodAdvBenchSorry-BenchHEx-PHI
    Base94.4278.8692.12
    LoRA-FT86.8740.6990.11
    Full-FT33.6513.6439.39

    We have to acknowledge that the underlying mechanism behind this performance gap remains unclear. One possible explanation is that refusal behaviors are encoded in specific parameters or neuronal structures, requiring more extensive parameter updates to achieve effective unlearning.

    Moreover, our theoretical analysis is derived under full FT and therefore does not directly explain the observed behavior of LoRA-FT. We will clarify this point and highlight the discrepancy as an important direction for future investigation.

  • Q2: Learning rate selection

    Thank you very much for this comment. The learning rate is selected based on performance on AdvBench and then applied to the other three datasets. For a fair comparison, we adopted the same learning rate for both FT and our method. The learning rate sensitivity analysis is presented below:

    Model1e-52e-55e-57e-51e-42e-4
    Llama-3.1-8B46.6333.6536.8472.3271.2990.32
    Gemma-2-2B51.2333.8534.0242.3564.5680.21

    We observe that large learning rates lead to training instability and degenerate outputs. Nevertheless, our finding is not that refusal unlearning succeeds across all learning rates, but that safety alignment can be substantially compromised under feasible fine-tuning settings. This already demonstrates an important vulnerability in current LLM safeguards.

  • Q3: Sensitivity to refusal prefixes

    We thank the reviewer for this valuable suggestion. To investigate this, we conduct additional experiment on Gemma-1.1-7B, replacing the original responses with diverse prefixes with a fixed refusal prefix "I can't help" (the most frequent one). The results are presented below:

    MethodAdvBenchSorry-BenchHEx-PHI
    Base100.083.1898.18
    Selected Prefix89.6365.6990.18
    All Prefixes40.031.5938.48

    The results demonstrate that using a fixed refusal prefix is substantially less effective than using diverse refusal prefixes. We hypothesize that this performance gap is because of the limited linguistic diversity of the training responses.

  • Q4: Effectiveness of closed-sourced models

    We have to acknowledge two practical constraints: (1) the substantial cost of conducting closed-sourced experiments and (2) the unavailability of certain proprietary models that are withdrawn by providers. Nevertheless, our empirical results suggest that full fine-tuning could be more effective for refusal unlearning than simply increasing the training data under LoRA-based fine-tuning.

  • Q5: Utility degradation

    We thank the reviewer for this insightful comment. Based on our current experimental results, this question remains open. For certain models, refusal unlearning yields limited effectiveness while still causing substantial utility degradation. Nevertheless, our results generally indicate that safety degradation is considerably more pronounced than utility degradation, highlighting the vulnerability of safety alignment to refusal unlearning. While understanding the trade-off between safety and utility is important, a comprehensive investigation of this relationship is beyond the primary scope of our work.

  • Q6: Mixed with ordinary data

    We thank the reviewer for this valuable suggestion. We conducted additional experiments on AdvBench using a mixture of 50% ordinary data and 50% refusal-prefix data. The results are presented below:

    MethodLlama-3.1-8BGemma-2-2B
    Base94.42100.0
    Half-Ordinary45.5676.03
    Zero-Ordinary33.6533.85

    The results show that incorporating ordinary data weakens refusal unlearning, particularly for Gemma-2-2B. We hypothesize that this is because some ordinary responses retain complete refusal sentences. The responses are not broken from refusal completion as our zero-ordinary method does.

Overall, we sincerely appreciate your valuable and detailed feedback and hope that our responses address the concerns raised above.

Thank you for the detailed rebuttal and additional experiments.

However, my main concerns remain:

  • The theory–method connection is weak. The analysis characterizes an idealized minimum-norm output-layer update, but its relevance to the actual full-parameter fine-tuning dynamics remains unvalidated. In particular, the proposed shared projections and their relationship to the observed safety degradation have not been empirically examined.
  • Practical impact is limited. The authors acknowledge that the modified training samples can be detected, but do not evaluate even simple filtering defenses. While I understand that practical attack stealthiness is not the primary objective, this limits the practical implications of the demonstrated vulnerability.
  • The learning-rate sensitivity results do not include the FT baseline, the LoRA and FT discrepancy remains unexplained, and the sample-size question for closed-source models remains unanswered.
AuthorsOpenReview ↗

Further Clarification to Reviewer EbBo

We sincerely appreciate the reviewer’s thoughtful feedback. We have to acknowledge that the following responses are provided for clarification and reference.

  • Theory-method connection

    Our apology for this confusion. "idealized" may be different from what we intended to claim in Theorem 3.5. The theorem characterizes a minimum over all updates. It does not model a trajectory, it also does not assume that Adam or DPO finds the minimum-norm solution. Instead, it states a lower bound: any procedure that reduces the refusal score by ρ\rho at the analyzed states must change the readout by at least ρ/γ∗\rho / \gamma_*. A bound of this form is a constraint the real dynamics have to satisfy rather than a competing model of fine-tuning that idealizes away the real dynamics.

    The bound also has consequences we can check against results already presented in discussion above. Restricting the feasible directions to the LoRA adapter's column space gives γS≤gammastar\gamma_S \leq gamma_star, where γS\gamma_S is the same max-min taken over directions the adapter can express. The lower bound on the required readout change is correspondingly larger, ρ/γS≥ρ/γ∗\rho / \gamma_S \geq \rho / \gamma_*, with the inequality strict whenever u∗u_* lies outside that column space. This matches the LoRA/full-FT difference in our Q1 reply (AdvBench 86.87 against 33.65).

  • Detection results

    We apologize for this insufficient results and thank the reviewer for this valuable suggestion. To address this concern, we evaluated three detection approaches: two widely used safety classifiers, Llama-Guard-4 and Qwen3-Guard, and an LLM-based detection approach using Qwen3-8B on conditional perplexity filtering and semantic consistency checks. For the latter, we repeat the evaluation 5 times:

    Detection MethodLlama-Guard-4Qwen3-GuardQwen3-8B
    Detection Rate (%)0.020.064 ± 31

    We find that safety classifiers largely fail to detect the training data, as the refusal prefixes themselves contain no explicitly harmful content. Moreover, LLM-based detection introduces substantial variance, which indicates limited reliability. It also introduces additional computational overhead for closed-source model providers and on the other hand cannot guarantee protection when open-source model weights remain accessible.

    More fundamentally, detecting our specific approach does not eliminate the underlying parameter-level vulnerability. Even if a particular exploitation strategy can be identified and blocked, this does not resolve the broader challenge of maintaining robust safety alignment under parameter manipulation.

  • Additional ablation studies

    We apologize for omitting these results and thank the reviewer for highlighting this issue.

    • Learning rate sensitivity of FT

      To address this concern, we conducted the learning rate sensitivity experiments for FT:

      ModelMethod1e-52e-55e-57e-51e-42e-4
      Llama-3.1-8BFT92.3688.4690.8891.3290.3693.64
      Gemma-2-2BFT100.096.5495.6896.5298.98100.0
      Llama-3.1-8BRU46.6333.6536.8472.3271.2990.32
      Gemma-2-2BRU51.2333.8534.0242.3564.5680.21

      We acknowledge that the optimal learning rate for Gemma differs between FT and RU, although the difference is marginal (0.86). Importantly, RU consistently outperforms standard FT across all evaluated learning rates, demonstrating that its effectiveness is not merely attributable to learning rate selection. We sincerely appreciate this suggestion, which strengthens the robustness of evaluation.

    • LoRA and FT discrepancy

      One possible explanation is that refusal behaviors are encoded in specific model parameters or neural structures, requiring more extensive parameter updates for effective unlearning. However, LoRA freezes the original model and introduces new parameters, which may limit its ability to modify the parameters underlying refusal behavior.

    • Sample size for closed models

      We have to acknowledge that additional experiments are less feasible due to computational costs and the unavailability of models. As a compromise, we traced logs and show the results of fine-tuing using 1K samples:

      Model#SamplesAdvBenchSorry-Bench
      Gemini-2.5-flash-liteBase99.6276.82
      Gemini-2.5-flash-lite1K99.4078.82
      Gemini-2.5-flash-lite52K54.0441.36
      Gemini-2.0-flash-liteBase100.089.09
      Gemini-2.0-flash-lite1K100.088.03
      Gemini-2.0-flash-lite52K73.4652.73

      The results show that fine-tuning with 1K samples yields negligible improvements. We hypothesize that this data requirement may be related to the limited adaptation capacity of LoRA, consistent with our observations that LoRA is less effective than full FT on open-source models.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

  • Position - Not an attack paper

    We would like to clarify that the primary focus of our work is not intended to develop a new attack strategy, but rather to expose a critical vulnerability in the safety of current LLMs. While specialized follow-up detection approaches may detect our method, they do not address the core challenge revealed by this work: how can safety be reliably preserved in LLMs when their safeguards can be readily removed through refusal unlearning? This vulnerability challenges the robustness of safety alignment itself, rather than merely exploiting weaknesses in model inputs/outputs.

    Moreover, even if our specific approach is detected, the underlying weakness is still there, as detecting a particular exploitation strategy does not fundamentally resolve the challenge of preserving safety at the parameter level. We believe this distinction is critical to understanding the broader safety implications of our findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.