- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates Progressive Misalignment, a phenomenon in which recursive self-distillation using benign feedback causes initially safety-aligned language models to become increasingly compliant with harmful requests. The authors demonstrate this behavior using on-policy and off-policy self-distillation on Qwen3 and Phi-4 models, evaluating safety degradation on SORRY-Bench and StrongREJECT. Beyond fixed benign feedback, the experiments show that misalignment can also emerge with feedback from simulated users and when training exclusively on selected benign prompts. The authors further investigate the effect of model scale and hypothesize that small changes in token distributions, particularly at early response positions, are amplified through recursive optimization. Their results suggest that safety misalignment occurs through self-distillation even when the initial training responses are safe.
Critical strengths
- The idea of analyzing the safety aspect of self-distillation, is interesting and timely.
- Empirical results are clear and consistent among models, showing a counterintuitive finding; training only on the model’s own refusals, with cues that don’t raise compliance on their own, produces near-complete loss of refusal behavior on held-out benchmarks.
Critical weaknesses
- The experimental self-distillation setup is highly simplified, repeatedly training on only 20 synthetically generated prompts. The main experiments use fixed, task-independent feedback such as "Excellent" and "I like Owls." While Section 3.2 additionally considers adaptive feedback from simulated users, this feedback primarily consists of generic encouragement and communication preferences rather than meaningful task-relevant information. Consequently, neither setting demonstrates how self-distillation could improve a target capability. The capability evaluations in Appendix B.1 only show that GSM8K and IFEval performance is largely preserved; no experiment demonstrates misalignment emerging while the model is actually improving through self-distillation. This limits the practical significance of the findings and leaves unclear whether the reported safety degradation would occur during realistic, capability-improving self-distillation, as suggested by the paper's motivation of recursive self-improvement.
- The experimental setup raises several concerns regarding the robustness and generalizability of the findings. Each configuration appears to rely on a single training run, as the reported "3 seeds" refer to evaluation sampling rather than independent training runs. Appendix A.5 also shows that learning rates, gradient clipping, and training durations vary across conditions, without a clear explanation of the hyperparameter selection procedure. More importantly, Appendix A.2 reveals that the authors explicitly select checkpoints maximizing the geometric mean of misalignment scores on SORRY-Bench and StrongREJECT, subject to a degeneracy threshold. This favors the checkpoints exhibiting the greatest safety degradation, rather than those selected through a realistic training objective or predetermined training budget.
- As mentioned in related work, safety alignment in LLMs can be substantially weakened by relatively little additional fine-tuning. The paper does not show baselines of other training / distiiliation methods to distinguish if the misaligment is caused by the paper's main focus, self-disitilliation, or simply due to the alternation of model weights due to the training progress. Safety drift from self-training itself is not a new finding. Baseline experiments will be required to establish wheter these findings are specific to self-distillation.
Questions for authors
- Does progressive misalignment persist with a substantially larger and more diverse prompt set (e.g., several hundred prompts) and larger batch sizes? This would help establish whether the effect depends on cycling a small set of prompts repeatedly.
- You mention that you selected checkpoints that maximised the geometric mean of misalignment scores. Could you provide (a) per-epoch compliance and degeneracy curves for all configurations, and (b) how learning rate, clipping, and epochs were chosen for each configuration, including whether any settings tried did not produce misalignment?
- As mentioned in the critical weaknesses section, how can we know if misaligment is due to self-distillation, and not due to model weight alternation? Could you report, with matched hyperparameters: (a) distillation from a frozen initial-model teacher with the same cue, and (b) supervised fine-tuning on the model’s own sampled outputs without any cue? The “Do not help with my request” control pushes toward refusal and so does not serve as a neutral baseline. If these baselines produce comparable misalignment, I would view the contribution as an instance of known fine-tuning fragility; if they do not, I would consider the main claim better supported.
9 comments
Additional baselines, larger prompt sets, and test curves (2/3)
Re [ [...] This favors the checkpoints exhibiting the greatest safety degradation, rather than those selected through a realistic training objective or predetermined training budget] We tried to propose a thoughtful selection to avoid cherry-picking the checkpoint. Moreover, as the experiments above on more realistic tasks such as chemistry and code show, progressive misalignment also arises from checkpoints that are chosen with respect to a fixed compute budget and performance on a downstream task. We hope this addresses your question. In addition, you can see from the added full-checkpoints plot in Appendix A.4 (Figure 13) that the selection does not necessarily pick the most misaligned checkpoint according to the final eval, and so we have not cherry-picked the checkpoint. However, we also emphasize that the community cares about the worst-case scenario for model safety - a model misaligned only under certain conditions can be dangerous if these go undetected.
Re [Q2: Hyperparameter of self-distillation and test curves] We selected the hyperparameters through usual tuning (grid search), as standard in deep learning papers, and ensured that all the baselines used the best hyperparameters possible (that’s also the case for the additional ablations below). We did not spend an extensive amount of time searching for hyperparameters to make misalignment arise. We think we provided convincing evidence that misalignment can emerge and, as such, we have not tested a huge sweep. Of course, it is not the case that every hyperparameter setting would lead to misalignment, similarly to how not every hyperparameter would make a deep learning method succeed. Hyperparameters are mainly selected to avoid degenerate generations (e.g. the continuous repetition of one word). For example, a learning rate too large would cause the generations to collapse. In fact, model collapse is a known failure mode of SDPO (e.g., see Peng et al., 2026). We hope this clarifies your question.
We also report additional test curves in Appendix A.4, Fig. 13, as per your request, and results across multiple training seeds in Appendix A.5, Fig. 14. The checkpoint was chosen as described in the paper, where we select checkpoints maximizing the product of the score on a validation set on the two benchmarks (which sometimes is not the checkpoint with the highest misalignment on the full test set). Again, as the experiments above on more realistic tasks show, progressive misalignment also arises from checkpoints that are chosen with respect to a fixed compute budget and performance on a downstream task.
Additional baselines, larger prompt sets, and test curves (3/3)
Re [Q3: how can we know if misalignment is due to self-distillation, and not due to model weight alternation [...] Could you report [...] (a) distillation from a frozen initial-model teacher with the same cue, and (b) supervised fine-tuning on the model’s own sampled outputs] We run the additional baselines to answer your question. We report the results in the tables below (and in Appendix A.1). First, we run SFT on the generated data across epochs. Misalignment does not arise in this case. This shows that matching the student to a teacher conditioned on an extra cue is necessary for progressive misalignment to arise. Additionally, we run self-distillation but keep the teacher frozen across epochs. Again, progressive misalignment does not occur, providing evidence that the recursive update of the teacher is also another fundamental component for progressive misalignment to emerge. As we can observe, the baselines are completely flat. It is thus the case that progressive misalignment is not due to weight alteration. We report the results below:
Qwen3 14B (off-policy, Excellent)
| Step | SORRY-Bench: Self-dist. | SORRY-Bench: Frozen teacher | SORRY-Bench: SFT | StrongREJECT: Self-dist. | StrongREJECT: Frozen teacher | StrongREJECT: SFT |
|---|---|---|---|---|---|---|
| 0 | 0.311 ± 0.010 | 0.311 ± 0.010 | 0.311 ± 0.010 | 0.106 ± 0.017 | 0.106 ± 0.017 | 0.106 ± 0.017 |
| 20 | 0.244 ± 0.005 | 0.295 ± 0.003 | 0.272 ± 0.013 | 0.016 ± 0.005 | 0.076 ± 0.006 | 0.083 ± 0.009 |
| 40 | 0.275 ± 0.013 | 0.297 ± 0.005 | 0.280 ± 0.011 | 0.028 ± 0.005 | 0.082 ± 0.006 | 0.071 ± 0.006 |
| 60 | 0.299 ± 0.011 | 0.280 ± 0.006 | 0.272 ± 0.015 | 0.037 ± 0.008 | 0.072 ± 0.001 | 0.072 ± 0.007 |
| 80 | 0.313 ± 0.011 | 0.284 ± 0.016 | 0.275 ± 0.020 | 0.041 ± 0.003 | 0.076 ± 0.014 | 0.064 ± 0.003 |
| 100 | 0.537 ± 0.018 | 0.292 ± 0.011 | 0.277 ± 0.005 | 0.194 ± 0.024 | 0.066 ± 0.011 | 0.069 ± 0.009 |
| 120 | 0.962 ± 0.008 | 0.288 ± 0.005 | 0.277 ± 0.010 | 0.695 ± 0.030 | 0.075 ± 0.009 | 0.069 ± 0.008 |
Phi-4 14B (off-policy, Excellent)
| Step | SORRY-Bench: Self-dist. | SORRY-Bench: Frozen teacher | SORRY-Bench: SFT | StrongREJECT: Self-dist. | StrongREJECT: Frozen teacher | StrongREJECT: SFT |
|---|---|---|---|---|---|---|
| 0 | 0.261 ± 0.006 | 0.261 ± 0.006 | 0.261 ± 0.006 | 0.038 ± 0.006 | 0.038 ± 0.006 | 0.038 ± 0.006 |
| 20 | 0.847 ± 0.005 | 0.173 ± 0.013 | 0.244 ± 0.015 | 0.386 ± 0.005 | 0.023 ± 0.004 | 0.038 ± 0.001 |
| 40 | 0.947 ± 0.014 | 0.128 ± 0.008 | 0.243 ± 0.010 | 0.564 ± 0.037 | 0.019 ± 0.003 | 0.033 ± 0.006 |
| 60 | 0.932 ± 0.013 | 0.150 ± 0.013 | 0.237 ± 0.017 | 0.585 ± 0.020 | 0.023 ± 0.003 | 0.036 ± 0.009 |
As you mentioned (“ [...] if they do not, I would consider the main claim better supported.”), we hope this further supports our main claim and addresses your final doubts, leading to positive support for our paper.
We hope these additional experiments are satisfactory evidence to address your questions. If so, we would kindly ask the reviewer to consider positively supporting our paper, or to let us know which concerns remain.
Thank you again for your detailed review.
References
- Hübotter et al. (ICML, 2026). Jonas Hübotter, Frederike Lübeck, Lejs Deen Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, Andreas Krause. Reinforcement Learning via Self-Distillation. International Conference on Machine Learning (ICML), 2026.
- Mazeika et al. (ICML, 2024). Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, Dan Hendrycks. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. International Conference on Machine Learning (ICML), 2024.
- Peng et al. (arXiv, 2026). Keqin Peng, Chen Li, Yuanxin Ouyang, Yancheng Yuan, Liang Ding. Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation. arXiv, 2026.
- Zou et al. (arXiv, 2023). Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, Matt Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv, 2023.
Additional baselines, larger prompt sets, and test curves (1/3)
We thank the reviewer for their feedback and for their positive score on the presentation, soundness, and informativeness of the paper.
Re [Q1: Progressive misalignment under bigger prompt set and batch size] Yes, we observe that progressive misalignment persists with a substantially larger prompt set, and with a larger batch size. To study this, we run the same setup as in Section 3.1 with “Excellent” as the cue. We run the same experiment with a much larger prompt set (413 prompts) and a larger batch size. Of course, all the prompts we use are different from the prompts in SORRY-Bench and StrongREJECT. The prompts are standard and come from well-known datasets such as AdvBench (Zou et al., 2023) and HarmBench (Mazeika et al., 2024). We see this as convincing evidence that the effect is not a result of the circling of a small prompt set.
Re [[...] leaves unclear whether degradation would occur during realistic, capability-improving self-distillation, as suggested by the paper's motivation of recursive self-improvement [...]] We provide, as you suggested, more evidence to better support the motivation of our paper and strengthen our claims. We ran our self-distillation setup on a data mixture containing prompts from a downstream task we want to improve on. In the first experiment, we take our 20 prompts in addition to 120 prompts on Chemistry (SciKnowEval). In the second, we take our 20 prompts in addition to 40 prompts on coding from LiveCodeBench. These datasets match the setup used by the SDPO paper by Hübotter et al. (ICML, 2026). We run on-policy self-distillation, where the feedback for the refusal to harmful prompts is “Excellent”, the feedback for the questions on chemistry is a simulated user (Qwen3 32B) giving feedback on the answer, and the feedback for the coding experiments uses either terminal logs or the answer to hidden tests. Again, these match the realistic setup from the SDPO paper by Hübotter et al. (ICML, 2026).
We report our results in the tables below. Here we take the latest checkpoint during training, which is the one where the model achieves the highest accuracy on the downstream task, and thus do not select with respect to misalignment at all. We thus observe that progressive misalignment might emerge during useful capability improvement across both tasks. We hope this provides an interesting insight and additional evidence to support our claim.
Qwen3 14B, on-policy self-distillation on code (LiveCodeBench)
Qwen3 14B, on-policy self-distillation on Chemistry (SciKnowEval)