- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates how safety-aligned LLMs transition from refusing harmful requests to generating harmful responses during harmful GRPO training. It proposes a harmful fine-tuning method, First Few Tokens Distillation (FFTD), which trains a target model using distillation from a weakly unaligned teacher, followed by on-policy distillation on a benign dataset to recover utility. Experiments on harmfulness and general-capability benchmarks suggest that FFTD achieves attack success rates comparable to those of SFT while retaining utility to varying degrees. The authors also evaluate the resulting models against TokenBuncher and SEAM defenses and report that the method remains effective against these defenses, with varying degrees of utility recovery. The main contributions are the analysis of refusal-to-compliance transitions, the prefix-focused distillation strategy for unalignment, and the utility-recovery procedure.
Critical strengths
1. A novel approach to inducing safety misalignment through distillation. The paper proposes First Few Tokens Distillation (FFTD), which uses a weakly unaligned teacher to fine-tune a target model by distilling only the initial response tokens. This provides an interesting alternative to conventional harmful fine-tuning in misaligning a model.
2. An explicit approach to utility preservation. The paper further explores on-policy distillation on a benign dataset (OPDBD) to recover general capabilities after misalignment training. The evaluation across harmfulness and general-capability benchmarks provides empirical evidence of the effectiveness of this two-stage approach.
Critical weaknesses
1. Unclear motivation and practical relevance of the attack setup. The proposed method assumes access to a weakly unaligned teacher model, which is used to fine-tune the target model through FFTD. This raises a fundamental question: if an open-weight model exhibiting unsafe behavior is already available, why is it necessary to unalign another model? The authors should clearly distinguish between creating an unsafe model and transferring unsafe behavior to a specific target model, clarify the intended threat model, and provide concrete use cases that justify the need for such transfer. Without this clarification, the practical motivation for the proposed attack remains insufficiently established.
2. Insufficient evidence for methodological novelty and generalizability. FFTD appears closely related to supervised fine-tuning (SFT) on teacher-generated response prefixes, while OPDBD performs utility recovery through distillation on benign data. The paper does not sufficiently establish the advantages of this combination over simpler alternatives, such as conventional SFT followed by utility-recovery training or SFT with an explicit utility-preservation objective.
3. Missing ablation studies. The evaluation lacks critical ablations comparing prefix-only distillation against full-response distillation in terms of ASR and utility recovery, as well as FFTD followed by alternative utility-preservation methods. Moreover, the experiments focus primarily on a single target model, limiting confidence in the generalizability of the findings. These limitations make it difficult to determine whether the proposed approach offers a distinct methodological contribution or a more effective combination of existing techniques.
Overall assessment: The paper addresses an interesting problem and presents a potentially useful training strategy. However, the motivation for their attack setup needs to be better established, and more comprehensive controlled experiments are necessary to demonstrate the method's novelty, effectiveness, and generalizability. I therefore lean toward rejection in its current form.
Questions for authors
-
Can you motivate the distillation-based misalignment training setup?
-
How do the ASR and utility performance change if you distill from full responses?
-
How do the other kinds of utility recovery training perform compared to OPDBD?
4 comments
Response to Reviewer oZds (Part 2/2)
W3 & Q2: Missing ablation studies
[W3] The evaluation lacks critical ablations comparing prefix-only distillation against full-response distillation in terms of ASR and utility recovery
If we distill the full response rather than just the first few tokens from the weakly unaligned teacher model, it is natural to expect that the trained model will copy the weakly unaligned teacher model. Furthermore, since both models share the same architecture and are the same instruction-tuned model, they would exhibit more or less identical behavior. In that case, wouldn't such an experiment directly fall under the concern you raised as your first weakness: "if an open-weight model exhibiting unsafe behavior is already available, why is it necessary to unalign another model?" Therefore, we believe that the absence of this ablation study is not a weakness of this paper.
For reference, Appendix D.5 (Broad Applicability of OPDBD) presents the results of applying OPDBD to an abliterated teacher model, corresponding to a full-response distillation followed by utility recovery. However, due to the extensive parameter modifications introduced by abliteration, utility recovery is less effective than for models trained using RL-based methods, resulting in performance declines of 7.93% on WildBench, 6.49% on HumanEval, and 6.16% on MBPP.
[W3] Moreover, the experiments focus primarily on a single target model, limiting confidence in the generalizability of the findings.
As shown in Table 1, our main experiments are conducted on Qwen3-4B-Instruct-2507, Gemma-3-4B-it, and OLMo-3-7B-Instruct. Additionally, in Appendix D.7 (Weak-to-Strong Generalization), we evaluate the effectiveness of our approach on models with larger parameter sizes using Qwen3-8B and Qwen3-14B. Our mention of model types in the Limitations section refers to the fact that the strength of built-in safety mechanisms varies significantly across models, making attack and defense outcomes highly model-dependent; however, the effectiveness of our main proposed method has been validated across multiple models.
Q3: How do the other kinds of utility recovery training perform compared to OPDBD?
Below, we present the results of applying simple SFT after unalignment.
| Method | WildBench (1–10) | WritingBench (0–100) | BiGGen-Bench (1–5) |
|---|---|---|---|
| Base | 6.22 | 78.57 | 3.96 |
| Harmful SFT | 4.22 | 65.79 | 3.19 |
| Harmful GRPO | 5.96 | 77.49 | 3.83 |
| FFTD | 5.95 | 75.66 | 3.89 |
| FFTD + OPDBD | 6.04 | 78.04 | 3.96 |
| FFTD + SFT on Dolci | 4.11 | 44.75 | 3.18 |
As shown above, because the original model's performance is already very high, simply applying SFT fails to restore its performance and instead degrades it. However, combining our approach with more sophisticated techniques may yield even higher utility. In Appendix E (Analysis of the Relationship Between Harmfulness and Utility), we present one such example: a grid search using task vectors. In this grid search, the optimal point (1.625, 1.75) improves FFTD+OPDBD by +19.67% on the harmfulness score and +1.25% on the WildBench score. Rather than proposing a single optimal configuration for our multi-stage unalignment approach, we believe that one of its key strengths lies in its ability to be combined with various techniques to generate diverse attack variations.
I am satisfied with most of the reply. I would increase the Rating to 3, provided the authors add a discussion of the motivation. I also suggest adding experiments showing why distilling full responses from the weakly unaligned teacher model is not a good option.
Dear Reviewer oZds,
We are very pleased to hear that you are satisfied with our reply.
Following the reviewers' suggestions, we have further detailed our research motivation in the Introduction and Discussion. Furthermore, to clarify why distilling full responses from the weakly unaligned teacher model is suboptimal, we have added an explanation regarding the substantial computational overhead and the objective of obtaining a model that preserves higher utility than the teacher. Please let us know if any point needs further clarification.
Warm regards,
The Authors
Response to Reviewer oZds (Part 1/2)
We thank the reviewer for their detailed feedback and suggestions, as well as for recognizing the novelty and clarity of our proposed method. We address the raised concerns below in the following order: W1 & Q1, W2, W3 & Q2, and Q3.
W1 & Q1: Unclear motivation and practical relevance of the attack setup.
The reason for unaligning another model is that existing open-weight models exhibiting unsafe behavior generally lack strong capabilities. Even among models with comparable levels of harmfulness, those with higher utility are known to achieve higher jailbreak success rates when acting as attackers (Panfilov et al., ICLR 2026), and our additional experiments also show that they achieve higher accuracy on cybersecurity benchmarks. Because language models are rapidly evolving, techniques for unaligning existing high-utility models to obtain more capable yet harmful models pose a serious threat. Building a harmful model from scratch requires substantial costs for pre-training, instruction tuning, and subsequent reinforcement learning; in contrast, unaligning an existing model eliminates this overhead and can target any open-weight model, making it a far more practical threat.
Furthermore, capability is not the sole motivation: if an open-weight model has been trained on specific private information whose leakage is prevented by safety mechanisms, an adversary could unalign the target model specifically to extract that information. We will add the above discussion to the Introduction and Discussion section.
W2: Insufficient evidence for methodological novelty and generalizability
Because FFTD is an RL-based learning method, we believe it operates through a learning mechanism distinct from SFT on teacher-generated response prefixes. Furthermore, to our knowledge, no prior studies have investigated SFT on teacher-generated response prefixes or explicitly demonstrated utility recovery following unalignment. While we agree that the method itself is simple, we analyzed the underlying mechanisms explaining why this design outperforms existing approaches and conducted comprehensive discussions and empirical validations on aspects that previously remained ambiguous and lacked clear explanation, including the limitations of existing evaluation frameworks, the broader applicability of the method (i.e., potential threats), and the utility-harmfulness trade-off curve via loss landscape visualization. We believe that the insights gained from our paper are not limited to the attack method itself.
We have already demonstrated in Section 5 (Against Defense Methods) that while simple SFT fails to bypass the SEAM defense mechanism and severely degrades utility, our proposed method successfully bypasses it. This suggests that existing defense mechanisms are aligned against specific attacks and that altering the attack variation can lead to a bypass, a threat that warrants serious consideration in the current AI safety field. Note that, as shown in Table 11 of Appendix D.5 (Broad Applicability of OPDBD), conventional SFT followed by utility-recovery training achieves performance comparable to our proposed method when applied to an undefended model. As stated in Lines 83–87, our primary claim is to demonstrate that existing defense methods cannot handle unseen attacks, rather than to construct a state-of-the-art attack in a global setting.