acceptodds
Under review as a conference paper at ICLR 2027

Understanding Safety Unalignment through Multi-Stage Reinforcement Learning Attacks

Abstract

Open-weight large language models (LLMs) remain vulnerable to post-training that removes safety alignment. Existing defenses, however, are evaluated predominantly against supervised fine-tuning (SFT), with only limited investigation of their robustness to reinforcement learning (RL) attacks. We first analyze safety unalignment under Harmful GRPO and find that harmful behavior emerges through distinct phases: from explicit refusal, through intermediate responses that repeat or condemn harmful requests, to affirmative prefixes followed by harmful content. We further show that harmful RL substantially degrades general generative utility, a failure mode largely obscured by conventional closed-ended benchmarks but exposed by open-ended evaluations. Guided by these observations, we introduce a simple two-stage post-training attack. First Few Tokens Distillation (FFTD) distills the distribution over the initial response tokens from a weakly unaligned teacher to accelerate safety unalignment, while On-Policy Distillation on a Benign Dataset (OPDBD) subsequently restores utility using only benign prompts and the original model as the teacher. Across multiple open-weight models, our approach achieves high harmful compliance while preserving substantially more utility than conventional harmful fine-tuning and circumvents representative defenses including TokenBuncher and SEAM. Our analysis reveals that these defenses rely on assumptions about rollout distributions or local optimization dynamics that can break under our proposed RL attack strategy, highlighting the need for mechanism-aware evaluation against heterogeneous post-training attacks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Safety, Fine-tuning Attacks, Mechanistic Interpretability, Adaptive Attacks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oZdsOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates how safety-aligned LLMs transition from refusing harmful requests to generating harmful responses during harmful GRPO training. It proposes a harmful fine-tuning method, First Few Tokens Distillation (FFTD), which trains a target model using distillation from a weakly unaligned teacher, followed by on-policy distillation on a benign dataset to recover utility. Experiments on harmfulness and general-capability benchmarks suggest that FFTD achieves attack success rates comparable to those of SFT while retaining utility to varying degrees. The authors also evaluate the resulting models against TokenBuncher and SEAM defenses and report that the method remains effective against these defenses, with varying degrees of utility recovery. The main contributions are the analysis of refusal-to-compliance transitions, the prefix-focused distillation strategy for unalignment, and the utility-recovery procedure.

Critical strengths

1. A novel approach to inducing safety misalignment through distillation. The paper proposes First Few Tokens Distillation (FFTD), which uses a weakly unaligned teacher to fine-tune a target model by distilling only the initial response tokens. This provides an interesting alternative to conventional harmful fine-tuning in misaligning a model.

2. An explicit approach to utility preservation. The paper further explores on-policy distillation on a benign dataset (OPDBD) to recover general capabilities after misalignment training. The evaluation across harmfulness and general-capability benchmarks provides empirical evidence of the effectiveness of this two-stage approach.

Critical weaknesses

1. Unclear motivation and practical relevance of the attack setup. The proposed method assumes access to a weakly unaligned teacher model, which is used to fine-tune the target model through FFTD. This raises a fundamental question: if an open-weight model exhibiting unsafe behavior is already available, why is it necessary to unalign another model? The authors should clearly distinguish between creating an unsafe model and transferring unsafe behavior to a specific target model, clarify the intended threat model, and provide concrete use cases that justify the need for such transfer. Without this clarification, the practical motivation for the proposed attack remains insufficiently established.

2. Insufficient evidence for methodological novelty and generalizability. FFTD appears closely related to supervised fine-tuning (SFT) on teacher-generated response prefixes, while OPDBD performs utility recovery through distillation on benign data. The paper does not sufficiently establish the advantages of this combination over simpler alternatives, such as conventional SFT followed by utility-recovery training or SFT with an explicit utility-preservation objective.

3. Missing ablation studies. The evaluation lacks critical ablations comparing prefix-only distillation against full-response distillation in terms of ASR and utility recovery, as well as FFTD followed by alternative utility-preservation methods. Moreover, the experiments focus primarily on a single target model, limiting confidence in the generalizability of the findings. These limitations make it difficult to determine whether the proposed approach offers a distinct methodological contribution or a more effective combination of existing techniques.

Overall assessment: The paper addresses an interesting problem and presents a potentially useful training strategy. However, the motivation for their attack setup needs to be better established, and more comprehensive controlled experiments are necessary to demonstrate the method's novelty, effectiveness, and generalizability. I therefore lean toward rejection in its current form.

Questions for authors
  1. Can you motivate the distillation-based misalignment training setup?

  2. How do the ASR and utility performance change if you distill from full responses?

  3. How do the other kinds of utility recovery training perform compared to OPDBD?

4 comments
AuthorsOpenReview ↗

Response to Reviewer oZds (Part 1/2)

We thank the reviewer for their detailed feedback and suggestions, as well as for recognizing the novelty and clarity of our proposed method. We address the raised concerns below in the following order: W1 & Q1, W2, W3 & Q2, and Q3.

W1 & Q1: Unclear motivation and practical relevance of the attack setup.

[W1] The proposed method assumes access to a weakly unaligned teacher model, which is used to fine-tune the target model through FFTD. This raises a fundamental question: if an open-weight model exhibiting unsafe behavior is already available, why is it necessary to unalign another model?

The reason for unaligning another model is that existing open-weight models exhibiting unsafe behavior generally lack strong capabilities. Even among models with comparable levels of harmfulness, those with higher utility are known to achieve higher jailbreak success rates when acting as attackers (Panfilov et al., ICLR 2026), and our additional experiments also show that they achieve higher accuracy on cybersecurity benchmarks. Because language models are rapidly evolving, techniques for unaligning existing high-utility models to obtain more capable yet harmful models pose a serious threat. Building a harmful model from scratch requires substantial costs for pre-training, instruction tuning, and subsequent reinforcement learning; in contrast, unaligning an existing model eliminates this overhead and can target any open-weight model, making it a far more practical threat.

Furthermore, capability is not the sole motivation: if an open-weight model has been trained on specific private information whose leakage is prevented by safety mechanisms, an adversary could unalign the target model specifically to extract that information. We will add the above discussion to the Introduction and Discussion section.


W2: Insufficient evidence for methodological novelty and generalizability

[W2] FFTD appears closely related to supervised fine-tuning (SFT) on teacher-generated response prefixes, while OPDBD performs utility recovery through distillation on benign data.

Because FFTD is an RL-based learning method, we believe it operates through a learning mechanism distinct from SFT on teacher-generated response prefixes. Furthermore, to our knowledge, no prior studies have investigated SFT on teacher-generated response prefixes or explicitly demonstrated utility recovery following unalignment. While we agree that the method itself is simple, we analyzed the underlying mechanisms explaining why this design outperforms existing approaches and conducted comprehensive discussions and empirical validations on aspects that previously remained ambiguous and lacked clear explanation, including the limitations of existing evaluation frameworks, the broader applicability of the method (i.e., potential threats), and the utility-harmfulness trade-off curve via loss landscape visualization. We believe that the insights gained from our paper are not limited to the attack method itself.

[W2] The paper does not sufficiently establish the advantages of this combination over simpler alternatives, such as conventional SFT followed by utility-recovery training or SFT with an explicit utility-preservation objective.

We have already demonstrated in Section 5 (Against Defense Methods) that while simple SFT fails to bypass the SEAM defense mechanism and severely degrades utility, our proposed method successfully bypasses it. This suggests that existing defense mechanisms are aligned against specific attacks and that altering the attack variation can lead to a bypass, a threat that warrants serious consideration in the current AI safety field. Note that, as shown in Table 11 of Appendix D.5 (Broad Applicability of OPDBD), conventional SFT followed by utility-recovery training achieves performance comparable to our proposed method when applied to an undefended model. As stated in Lines 83–87, our primary claim is to demonstrate that existing defense methods cannot handle unseen attacks, rather than to construct a state-of-the-art attack in a global setting.

AuthorsOpenReview ↗

Response to Reviewer oZds (Part 2/2)

W3 & Q2: Missing ablation studies

[W3] The evaluation lacks critical ablations comparing prefix-only distillation against full-response distillation in terms of ASR and utility recovery

If we distill the full response rather than just the first few tokens from the weakly unaligned teacher model, it is natural to expect that the trained model will copy the weakly unaligned teacher model. Furthermore, since both models share the same architecture and are the same instruction-tuned model, they would exhibit more or less identical behavior. In that case, wouldn't such an experiment directly fall under the concern you raised as your first weakness: "if an open-weight model exhibiting unsafe behavior is already available, why is it necessary to unalign another model?" Therefore, we believe that the absence of this ablation study is not a weakness of this paper.

For reference, Appendix D.5 (Broad Applicability of OPDBD) presents the results of applying OPDBD to an abliterated teacher model, corresponding to a full-response distillation followed by utility recovery. However, due to the extensive parameter modifications introduced by abliteration, utility recovery is less effective than for models trained using RL-based methods, resulting in performance declines of 7.93% on WildBench, 6.49% on HumanEval, and 6.16% on MBPP.

[W3] Moreover, the experiments focus primarily on a single target model, limiting confidence in the generalizability of the findings.

As shown in Table 1, our main experiments are conducted on Qwen3-4B-Instruct-2507, Gemma-3-4B-it, and OLMo-3-7B-Instruct. Additionally, in Appendix D.7 (Weak-to-Strong Generalization), we evaluate the effectiveness of our approach on models with larger parameter sizes using Qwen3-8B and Qwen3-14B. Our mention of model types in the Limitations section refers to the fact that the strength of built-in safety mechanisms varies significantly across models, making attack and defense outcomes highly model-dependent; however, the effectiveness of our main proposed method has been validated across multiple models.


Q3: How do the other kinds of utility recovery training perform compared to OPDBD?

Below, we present the results of applying simple SFT after unalignment.

MethodWildBench (1–10)WritingBench (0–100)BiGGen-Bench (1–5)
Base6.2278.573.96
Harmful SFT4.2265.793.19
Harmful GRPO5.9677.493.83
FFTD5.9575.663.89
FFTD + OPDBD6.0478.043.96
FFTD + SFT on Dolci4.1144.753.18

As shown above, because the original model's performance is already very high, simply applying SFT fails to restore its performance and instead degrades it. However, combining our approach with more sophisticated techniques may yield even higher utility. In Appendix E (Analysis of the Relationship Between Harmfulness and Utility), we present one such example: a grid search using task vectors. In this grid search, the optimal point (1.625, 1.75) improves FFTD+OPDBD by +19.67% on the harmfulness score and +1.25% on the WildBench score. Rather than proposing a single optimal configuration for our multi-stage unalignment approach, we believe that one of its key strengths lies in its ability to be combined with various techniques to generate diverse attack variations.

I am satisfied with most of the reply. I would increase the Rating to 3, provided the authors add a discussion of the motivation. I also suggest adding experiments showing why distilling full responses from the weakly unaligned teacher model is not a good option.

Dear Reviewer oZds,

We are very pleased to hear that you are satisfied with our reply.

Following the reviewers' suggestions, we have further detailed our research motivation in the Introduction and Discussion. Furthermore, to clarify why distilling full responses from the weakly unaligned teacher model is suboptimal, we have added an explanation regarding the substantial computational overhead and the objective of obtaining a model that preserves higher utility than the teacher. Please let us know if any point needs further clarification.

Warm regards,

The Authors

Reviewer_1iA6OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. The paper proposes a two-stage RL method to unalign the model while preserving its utility.
Critical strengths
  1. The paper is human-readable and should be written by a human. This is a notable strength among a high volume of AI-generated papers.

  2. Analysis and mechanism interpretability make sense to me.

Critical weaknesses
  1. More evaluation should be done.
  • In Table 2, could you also post the number of harmful SFT, harmful GRPO, and your methods on a no-defense method? I think this data is probably available in Table 1, but presenting this again in Table 2 should be easier for readers to analyze.
  • Could you also add a comparison against another defense Booster?

Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation

  • Could you add evaluation of a more complicated malicious benchmark, e.g., some cyber attack benchmark. The success of this benchmark should need high model harmfulness and also high utility, which might be of interest to reflect the ultimate goal of this paper.
  1. Some numbers I can't understand.
  • SEAM is a defense design for harmful SFT, why it deos not for harmful SFT, but it works on defensing harmful GRPO? It seems that the self-destructing effect does takes place because the utility drops, but why does this mechanism affect model harmfulness?
  1. The idea and finding does not seems to be too surprised for me. The OPD harmful optimization signal is neither considered by SEAM and TokenBuncher. Have you tried combining SFT (instead of OPD) and RL signal together in LFFTD loss? Will this trigger self-destruction in SEAM?
Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1iA6 (Part 1/2)

We sincerely thank the reviewer for their deep insights and constructive feedback. Below, we address the three weaknesses raised by the reviewer.

[W1] In Table 2, could you also post the number of harmful SFT, harmful GRPO, and your methods on a no-defense method? I think this data is probably available in Table 1, but presenting this again in Table 2 should be easier for readers to analyze.

We thank the reviewer for their careful proofreading and helpful suggestions. In the final version of the paper, we will add the results of the SFT, GRPO, FFTD, and OPDBD models under the no-defense setting to Table 2.

[W1] Could you also add a comparison against another defense Booster?

We tested multiple hyperparameter settings for Booster to verify whether it is robust against Harmful SFT.

Booster setting ( λ / α / LoRA modules / steps)HarmBench ASR (%)StrongREJECT ASR (%)ARC-EMMLUGSM8K
None (undefended SFT attack)82.389.196.570.390.3
5 / 0.1 / qkv / 1,00079.788.595.769.679.6
20 / 0.1 / qkv / 2,00080.790.196.269.580.7
100 / 0.1 / qkv / 2,00081.787.296.369.186.1
5 / 0.1 / all linear / 2,00082.390.195.467.680.1
5 / 0.1 / qkv / 10,00083.388.291.361.672.7

Note: To ensure a fair comparison, we also concurrently trained the SFT attack on the undefended ("None defense") model; therefore, the scores differ slightly from those reported in the paper.

As a result, Booster's defense mechanism also failed to exhibit sufficient defensive performance against Harmful SFT. Because this finding was similarly reported in the original SEAM paper, we believe this result is not surprising. Regarding Booster, even in its original paper, it did not completely prevent unalignment, and because only 100 harmful samples were used during harmful training, it is likely that it could not withstand our SFT setup, which uses 6,400 samples consisting solely of harmful queries.


[W1] Could you add evaluation of a more complicated malicious benchmark, e.g., some cyber attack benchmark. The success of this benchmark should need high model harmfulness and also high utility, which might be of interest to reflect the ultimate goal of this paper.

We share your concern regarding whether a high-utility harmful model also achieves strong performance on cyberattack benchmarks, and we accordingly conducted evaluations using the CyberSecEval2 (Bhatt et al., 2024) and RedLineBench benchmarks (Crosby team, 2026). CyberSecEval2 is a benchmark in which the model predicts an input string that causes a given program to produce the expected output, with problems presented alongside either Neutral or Evil prompts. RedLineBench is an open-ended generation benchmark that evaluates response quality in addition to whether the model refuses to answer. The Pass Rate is calculated as the proportion of responses achieving a quality score of 3 or higher.

ModelCSE2 Neutral (↑)CSE2 Evil (↑)RedLine Refusal (%, ↓)RedLine Avg. Quality (↑)RedLine Pass Rate (%, ↑)
Base42.242.77.43.7479.7
Abliteration42.143.80.04.0785.2
SFT39.640.42.43.8279.5
GRPO40.943.70.74.0185.3
FFTD39.943.30.04.0887.3
OPDBD42.044.80.04.0884.7

As a result, while we confirmed that our proposed method improves response quality on harmful queries compared to the SFT model, the results did not show that OPDBD clearly outperforms the GRPO or FFTD models. Although we cannot draw a definitive conclusion regarding the cause of this, to of our knowledge, current benchmarks evaluating models' offensive capabilities and knowledge of system vulnerabilities remain relatively underdeveloped, which makes an accurate evaluation challenging. We will explicitly outline a more detailed analysis of these aspects as an important direction for future work in the Discussion section.

AuthorsOpenReview ↗

Response to Reviewer 1iA6 (Part 2/2)

[W2] SEAM is a defense design for harmful SFT, why it does not for harmful SFT, but it works on defensing harmful GRPO? It seems that the self-destructing effect does takes place because the utility drops, but why does this mechanism affect model harmfulness?

Let us explain the specific effects of each attack method on the model protected by the SEAM defense. First, regarding the Harmful SFT model, its ASR is high, while its utility score drops significantly. This stems from the ASR evaluation system: even when the Harmful SFT model generates inaccurate details or extremely short responses to harmful queries, it appears to respond normally on the surface, leading to a high ASR score.

In contrast, for Harmful GRPO, the model's ability to refuse harmful queries has become more robust, making it more difficult to elicit harmful responses than from the Base model. As explained in Section 3.3, when all rollouts consist solely of refusals, there is no difference in rewards, so learning does not progress and the model does not become harmful. The reason for the low ASR is that all responses were explicit refusals.

Furthermore, as you rightly assumed, the original TokenBuncher paper reports that Harmful GRPO can bypass SEAM, and we believe the reason Harmful GRPO failed to bypass SEAM in our experiments is the number of training steps. In principle, SEAM cannot prevent the pathway toward harmful behavior induced by Harmful GRPO; therefore, we believe that if the parameters are gradually updated over a larger number of training steps until harmful responses can be elicited from the model, the shift toward harmful behavior will progress.


[W3] Have you tried combining SFT (instead of OPD) and RL signal together in LFFTD loss? Will this trigger self-destruction in SEAM?

In fact, a simple combination of SFT and RL training has already been explored in Token Buncher paper, and as you rightly surmised, that approach cannot prevent SEAM's self-destruction mechanism. As shown on Page 13, Figure 6, the SFT training signal dominates in the early stages of training, which gets caught in SEAM's self-destructive trap and leads to irreversible collapse. In our study as well, we experimented with a setup combining a few steps of SFT with OPDBD; however, after just around 10 steps of SFT, SEAM's self-destruction mechanism reached its peak, and the model could not recover its original utility. Warming up RL with SFT does not work well for models equipped with defense mechanisms, requiring a warm-up via a different pathway from the outset. We will include the above discussion in the analysis in Section 5 of the final version.

Reviewer_1iA6OpenReview ↗

Thanks for the rebuttal

The explanation makes sense to me and I appreciate the new benchmark over the CyberSecEval2 (Bhatt et al., 2024) and RedLineBench benchmarks. For future research, I do think that combining metrics for both safety and utility is important because that should be the ultimate goal for designing defense (i.e., an attacker can't get both high ASR and high utility together).

Regarding the score, I would like to keep my current weak acceptance score for now but will adjust accordingly (be assured, it can only be higher) during the reviewer-AC stage.

Reviewer_zweEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates the vulnerability of open-weight large language models (LLMs) to safety unalignment. Existing defenses primarily target supervised fine-tuning (SFT) attacks. However, the authors demonstrate that reinforcement learning (RL) based post-training processes pose a greater threat. Through mechanistic analysis, they reveal that RL-based unalignment progresses through distinct phases, arguing that refusal behaviors are heavily concentrated in the initial tokens. They show that while pure RL attacks deceptively maintain performance on traditional fixed-answer benchmarks, they may actually degrade the model's open-ended general utility—a failure mode obscured by conventional evaluations.

The authors propose a novel two-stage RL-based attack. The first stage, First Few Tokens Distillation (FFTD), accelerates unalignment by distilling the distribution of initial response tokens from a weakly unaligned teacher, bypassing the refusal bottleneck. The second stage, On-Policy Distillation on a Benign Dataset (OPDBD), utilizes the original aligned base model as a teacher on safe prompts to recover the degraded open-ended utility, explicitly skipping the distillation of opening tokens to preserve harmful compliance.

Evaluations across multiple LLMs (e.g., Qwen3, Gemma-3, OLMo-3) demonstrate that this combined framework achieves high attack success rates (ASR) while preserving substantially more general utility than traditional methods. Furthermore, this two-stage attack successfully circumvents representative alignment-stage defenses (specifically TokenBuncher and SEAM), exposing critical limitations in current defense assumptions regarding rollout distributions and local optimization dynamics.

Critical strengths
  1. Mechanistic Insights into RL-based Unalignment Dynamics: This paper provides a highly informative and fine-grained analysis of how models transition from safety to harmful compliance during reinforcement learning optimization. By employing a streaming guardrail to calculate token-level harmfulness scores, the authors successfully reveal that the refusal signal is disproportionately concentrated in the initial tokens. This granular diagnostic is significant for the overall assessment because it demystifies the internal dynamics of RL attacks and directly inspires the subsequent First Few Tokens Distillation (FFTD) method.
  2. Exposing the Evaluation Blind Spot of General Utility: The authors convincingly demonstrate a critical flaw in current safety evaluation paradigms. The paper provides strong empirical evidence that while traditional fixed-answer benchmarks (e.g., MMLU, ARC-Easy) deceptively indicate that model utility is preserved under harmful fine-tuning, open-ended generation benchmarks (e.g., WildBench) expose a severe degradation in general utility. This finding is highly consequential as it challenges the prevailing assumption that RL-based unalignment perfectly preserves model capabilities, establishing a more rigorous standard for future attack and defense assessments.
Critical weaknesses
  1. Fragility of Defense Baselines and Lack of Meaningful Comparison: A core premise of this paper is that RL-based attacks pose a distinct and greater threat than SFT attacks, necessitating new attack frameworks to evaluate defenses. However, in Appendix B.6, the authors concede that representative defenses like TAR, RepNoise, and CTRAP failed to resist even basic Harmful SFT in their experimental environment and were thus excluded. Furthermore, the SEAM defense required custom hyperparameter sweeping by the authors to exhibit any defensive behavior. This indicates that the attack framework is evaluated against fragile, artificially tuned, or sub-optimal baselines. If these defenses cannot robustly withstand basic SFT, using a complex two-stage RL attack to break them constitutes a "strawman" evaluation, severely undermining the paper's core motivation and the validity of the results in Section 5.
  2. The Prefix Generalization Issue in OPDBD: The On-Policy Distillation on a Benign Dataset (OPDBD) stage presents a critical logical flaw. To preserve the harmful compliance learned in Step 1, OPDBD explicitly skips distillation on the first 25 tokens. However, because this training is conducted exclusively on a benign dataset, the first 25 tokens naturally generated by the model are inherently safe and benign. The model is never explicitly trained to maintain coherence and high-utility generation following a forced harmful prefix. Assuming that the generative utility recovered under benign prefixes will seamlessly generalize to contexts with harmful prefixes is logically untenable. Additionally, Table 2 reveals that OPDBD consistently drops the ASR (e.g., from 79.0% to 70.7% under SEAM), demonstrating that this benign-only distillation inadvertently "washes away" the injected harmfulness.
Questions for authors
  1. Regarding the Generalizability of the RL Threat Model: The introduction establishes the core motivation that RL-based attacks pose a distinct and greater threat, arguing that existing defenses effectively mitigate SFT but remain vulnerable to RL. However, the experiments (and Appendix B.6) reveal that the selected defenses failed to withstand even basic Harmful SFT. To ensure the empirical results logically align with your background claims, could you evaluate your framework against a defense configuration that has been provably successful at mitigating Harmful SFT on the target models? Demonstrating that standard Harmful GRPO and FFTD can circumvent a defense that genuinely halts SFT would significantly substantiate the claim regarding the unique broadness and severity of the RL threat.
  2. Regarding Prefix Generalization in OPDBD: OPDBD explicitly skips the distillation of the first t_start=25 tokens on a purely benign dataset. How can you empirically guarantee that the generative utility recovered under these safe, benign prefixes successfully transfers to the trailing tokens of a harmful completion? I strongly suggest providing an empirical evaluation of the model's coherence and instruction-following capabilities explicitly conditioned on forced harmful prefixes, rather than evaluating utility solely on independent benign prompts.
  3. Regarding Correlation vs. Causation in Prefix Intervention: The transition analysis in Section 3.3 observes that refusal signals are concentrated in early tokens. However, concluding that intervening specifically in the first few tokens (FFTD) is the core mechanism responsible for accelerated unalignment conflates correlation with causation. Could you provide an ablation study that applies distillation to the middle or end of the response generation, or varies the semantic content of the distilled tokens? Proving that non-prefix or content-altered interventions fail to accelerate unalignment would causally demonstrate that the opening tokens—and their specific content—are indeed the critical bottleneck.
  4. Regarding LLM Judge Bias in Open-ended Utility: The conclusion that RL attacks severely degrade general utility relies heavily on open-ended benchmarks evaluated by an LLM judge (Gemma-4-31B-it). Given that the "LLM-as-a-judge" paradigm frequently exhibits length bias and stylistic preferences, how can we be certain that the observed utility degradation isn't merely the attacked model producing shorter, or stylistically less-preferred, responses, rather than suffering a substantive loss in capability?
3 comments
AuthorsOpenReview ↗

Response to Reviewer zweE (Part 1/2)

We sincerely thank the reviewer for the detailed review and feedback. We appreciate your positive assessment of our mechanistic interpretability perspective on RL-based Unalignment Dynamics, as well as our rigorous evaluation design assessing utility degradation overlooked by prior studies. We address your concerns as follows, answering W1 with Q1, and W2 with Q2, Q3, and Q4.

W1 & Q1: Fragility of Defense Baselines and Lack of Meaningful Comparison

[W1] in Appendix B.6, the authors concede that representative defenses like TAR, RepNoise, and CTRAP failed to resist even basic Harmful SFT in their experimental environment and were thus excluded ... If these defenses cannot robustly withstand basic SFT, using a complex two-stage RL attack to break them constitutes a "strawman" evaluation, severely undermining the paper's core motivation and the validity of the results in Section 5.

[Q1] To ensure the empirical results logically align with your background claims, could you evaluate your framework against a defense configuration that has been provably successful at mitigating Harmful SFT on the target models?

In Section 5, we excluded existing defense mechanisms that lacked robustness against Harmful SFT (TAR, RepNoise, and CTRAP) and evaluated our proposed method, FFTD, against those that demonstrated robustness (SEAM and TokenBuncher). This evaluation framework is not unique to our study; the original SEAM paper similarly reported that while TAR and RepNoise failed to defend against Harmful SFT, SEAM succeeded in doing so. Furthermore, we also demonstrate that FFTD can bypass TokenBuncher, a defense robust against standard RL-based attacks. Thus, rather than applying our proposed method to defenses already known to be vulnerable to standard attacks, we specifically evaluated it on defenses that resisted those standard approaches.

Additionally, regarding your point that

[W1] the SEAM defense required custom hyperparameter sweeping by the authors to exhibit any defensive behavior.

because we used different models from those in the original SEAM paper, it is natural to expect that the exact same hyperparameters would not work out of the box. Moreover, gradient-ascent-based unlearning methods like SEAM are known to be unstable and sensitive to hyperparameter selection (Mavrothalassitis et al., NeurIPS 2025; Kumar et al., UAI 2026). For these reasons, we do not consider sweeping for hyperparameters different from those in the original paper to be a weakness.


W2 & Q2: The Prefix Generalization Issue in OPDBD

[W2] The model is never explicitly trained to maintain coherence and high-utility generation following a forced harmful prefix. Assuming that the generative utility recovered under benign prefixes will seamlessly generalize to contexts with harmful prefixes is logically untenable.

[Q2] How can you empirically guarantee that the generative utility recovered under these safe, benign prefixes successfully transfers to the trailing tokens of a harmful completion?

We respectfully disagree with the concern that a model trained on benign prefixes fails to generalize to text generation conditioned on harmful prefixes. While we skip the first 25 tokens during training on the benign dataset, this choice is primarily motivated by computational efficiency. Even if we train from t_start=0 without skipping, the harmfulness score decreases by only 0.66% and utility score increases 0.34% on average, as shown in Appendix D.4 (Token Position Sensitivity). The prefix plays a critical role primarily in determining whether the model issues a refusal; however, once compromised, the model consistently begins its responses affirmatively regardless of the domain, making the specific contribution of the prefix minor. Furthermore, we conducted additional experiments on CyberSecEval2, a cybersecurity risk benchmark, and found that our OPDBD model achieves the highest accuracy on evil prompts. Overall, we believe that prefix generalization does not pose a significant issue.

[W2] Additionally, Table 2 reveals that OPDBD consistently drops the ASR (e.g., from 79.0% to 70.7% under SEAM), demonstrating that this benign-only distillation inadvertently "washes away" the injected harmfulness.

We believe this characterization is an overstatement given that the pre-attack ASR on HarmBench is only 1.7%. As shown in Appendix E (Analysis of the Relationship between Harmfulness and Utility), there is an inherent trade-off between utility and harmfulness. While we acknowledge that OPDBD mitigates the ASR, the resulting model remains substantially more harmful than the original model, and harmful responses can still be easily elicited via sampling. Therefore, we do not consider this modest reduction to be a critical weakness.

AuthorsOpenReview ↗

Response to Reviewer zweE (Part 2/2)

Q3: Regarding Correlation vs. Causation in Prefix Intervention

[Q3] Could you provide an ablation study that applies distillation to the middle or end of the response generation, or varies the semantic content of the distilled tokens?

We agree with your point that distilling the middle or end segments of a response rather than its prefix would empirically validate the effectiveness of prefix distillation, and we conducted additional experiments to verify this.

Teacher windowSteps 1–5Steps 6–10Steps 11–15Steps 16–20Steps 21–25HarmBench ASR (%)StrongREJECT ASR (%)
prefix100.0260.0810.2690.5110.56574.078.0
middle100.0220.0570.1960.3950.49970.374.1
last100.0200.0600.1490.3050.38065.064.9

Here, prefix10 corresponds to the setting used in our paper, middle10 distills the 10 tokens starting from (L - 10) / 2 (where L is the response token length), and last10 distills the final 10 tokens. The table above reports the average harmfulness score every 5 steps alongside the final ASR. The results show that distillation is most effective in the order of prefix, middle, and last, which suggests that refusal signals are indeed concentrated in the prefix tokens. Beyond performance, defining the exact "middle" position is inherently ambiguous, which makes consistent training difficult, and requiring the teacher model to compute generative distributions up to the middle or end of a response increases computational overhead. Therefore, in practice, we believe prefix distillation is superior in terms of both performance and computational cost.


Q4: Regarding LLM Judge Bias in Open-ended Utility:

[Q4] The conclusion that RL attacks severely degrade general utility relies heavily on open-ended benchmarks evaluated by an LLM judge (Gemma-4-31B-it).

While we primarily use Gemma-4-31B-it as the judge model in the main text, we show in Appendix D.9 (Judge Model Consistency) that similar results are obtained with Kimi-K3 and GLM-5.2. Pearson correlations between the judge models are 0.855 (Gemma–Kimi), 0.895 (Gemma–GLM), and 0.906 (Kimi–GLM). These findings suggest that the observed utility degradation is not an artifact of Gemma-specific bias.

[Q4] Given that the "LLM-as-a-judge" paradigm frequently exhibits length bias and stylistic preferences, how can we be certain that the observed utility degradation isn't merely the attacked model producing shorter, or stylistically less-preferred, responses, rather than suffering a substantive loss in capability?

Furthermore, Figures 12, 13, and 14 present case studies confirming from a human perspective that the degradation is not an artifact of evaluation bias, but rather stems from failing to answer questions properly or provide concrete information. Additionally, regarding response length, the average lengths are 790.0 tokens for Base, 883.3 tokens for GRPO, and 928.8 tokens for OPDBD, respectively, which also indicates that length bias is not a direct factor driving the scores.

Thank you for your response. Your answers have sufficiently substantiated your claims, I have no further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We sincerely thank all the reviewers for the detailed reviews and the constructive feedback. We have revised the manuscript to address the reviewers' suggestions and improve the overall clarity of our presentation.

  • Section 1 (Introduction) and Section 6 (Discussion and Conclusion): Added further explanation of our study's motivation and the threat posed by the proposed attack method. Moved the Weak-to-Strong Generalization results from the Appendix to Section 6 to better highlight our contributions, and added a discussion on how to evaluate the impact of high-utility harmful models. (Reviewers oZds, 1iA6)

  • Appendix D.6 (Evaluation on Cyberattack Benchmarks): Added evaluation results and discussions of the models on cyberattack benchmarks. (Reviewers zweE, 1iA6)

  • Section 5 (Against Defense Methods) and Appendix B.6 (Defense Implementation): Added an explanation of HarmSFT's behavior against SEAM in Section 5. In Appendix B.6, included results for Booster, discussed why the failed defense methods lacked resilience against our Harmful SFT from an experimental setup perspective, and provided a more detailed interpretation of Table 2 in conjunction with findings from prior studies. (Reviewers zweE, 1iA6)

  • Appendix D.4 (Token Position Sensitivity): Added comparative experiments distilling the prefix10, middle10, and last10 tokens, along with an explanation of why distilling full responses from the weakly unaligned teacher model is not a practical option. (Reviewers zweE, oZds)

  • Appendix D.5 (Broad Applicability of OPDBD): Added a row for SFT as an alternative utility recovery method to OPDBD. (Reviewer oZds)

  • Appendix D.10 (Qualitative Case Studies): Clarified that the LLM-as-a-Judge evaluations account for content quality rather than length bias. (Reviewer zweE)

We reply to each reviewer's concerns in detail below their individual reviews. Thank you again, and please let us know if any point needs further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.