acceptodds
Under review as a conference paper at ICLR 2027

Progressive Misalignment Can Emerge from Benign Recursive Self-Distillation

Abstract

Training models on their own outputs has attracted widespread attention as a possible route to further capability gains in AI development. Self-distillation has emerged as a prominent method for this, in which a model is trained to match the token-level predictions it would have produced when given access to additional context, such as a cue, feedback, or instruction. However, the safety properties of this training procedure are poorly understood. In this paper, we present a surprising safety failure of self-distillation. We study recursive self-distillation, where each self-distilled model becomes the teacher for the next round of training. When a safe model is recursively self-distilled on its own safe generations while receiving a fixed non-harmful feedback, it progressively becomes broadly misaligned across harmful queries; we call this failure mode *Progressive Misalignment*. We show this failure mode in both the Qwen3 and Phi-4 model families on SORRY-Bench and StrongREJECT. Progressive misalignment also emerges in a more realistic setup, where the fixed cue is replaced by feedback from a simulated user with benign intent. Using a gradient-based data selection approach, we are also able to find benign prompts that lead to progressive misalignment when applying recursive self-distillation, showing that harmful training prompts are not necessary for this failure mode to emerge. These results suggest that safety might degrade unintendedly across rounds of self-improvement.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Safety, AI Alignment, Large Language Models, Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_X7JbOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates Progressive Misalignment, a phenomenon in which recursive self-distillation using benign feedback causes initially safety-aligned language models to become increasingly compliant with harmful requests. The authors demonstrate this behavior using on-policy and off-policy self-distillation on Qwen3 and Phi-4 models, evaluating safety degradation on SORRY-Bench and StrongREJECT. Beyond fixed benign feedback, the experiments show that misalignment can also emerge with feedback from simulated users and when training exclusively on selected benign prompts. The authors further investigate the effect of model scale and hypothesize that small changes in token distributions, particularly at early response positions, are amplified through recursive optimization. Their results suggest that safety misalignment occurs through self-distillation even when the initial training responses are safe.

Critical strengths
  1. The idea of analyzing the safety aspect of self-distillation, is interesting and timely.
  2. Empirical results are clear and consistent among models, showing a counterintuitive finding; training only on the model’s own refusals, with cues that don’t raise compliance on their own, produces near-complete loss of refusal behavior on held-out benchmarks.
Critical weaknesses
  1. The experimental self-distillation setup is highly simplified, repeatedly training on only 20 synthetically generated prompts. The main experiments use fixed, task-independent feedback such as "Excellent" and "I like Owls." While Section 3.2 additionally considers adaptive feedback from simulated users, this feedback primarily consists of generic encouragement and communication preferences rather than meaningful task-relevant information. Consequently, neither setting demonstrates how self-distillation could improve a target capability. The capability evaluations in Appendix B.1 only show that GSM8K and IFEval performance is largely preserved; no experiment demonstrates misalignment emerging while the model is actually improving through self-distillation. This limits the practical significance of the findings and leaves unclear whether the reported safety degradation would occur during realistic, capability-improving self-distillation, as suggested by the paper's motivation of recursive self-improvement.
  2. The experimental setup raises several concerns regarding the robustness and generalizability of the findings. Each configuration appears to rely on a single training run, as the reported "3 seeds" refer to evaluation sampling rather than independent training runs. Appendix A.5 also shows that learning rates, gradient clipping, and training durations vary across conditions, without a clear explanation of the hyperparameter selection procedure. More importantly, Appendix A.2 reveals that the authors explicitly select checkpoints maximizing the geometric mean of misalignment scores on SORRY-Bench and StrongREJECT, subject to a degeneracy threshold. This favors the checkpoints exhibiting the greatest safety degradation, rather than those selected through a realistic training objective or predetermined training budget.
  3. As mentioned in related work, safety alignment in LLMs can be substantially weakened by relatively little additional fine-tuning. The paper does not show baselines of other training / distiiliation methods to distinguish if the misaligment is caused by the paper's main focus, self-disitilliation, or simply due to the alternation of model weights due to the training progress. Safety drift from self-training itself is not a new finding. Baseline experiments will be required to establish wheter these findings are specific to self-distillation.
Questions for authors
  1. Does progressive misalignment persist with a substantially larger and more diverse prompt set (e.g., several hundred prompts) and larger batch sizes? This would help establish whether the effect depends on cycling a small set of prompts repeatedly.
  2. You mention that you selected checkpoints that maximised the geometric mean of misalignment scores. Could you provide (a) per-epoch compliance and degeneracy curves for all configurations, and (b) how learning rate, clipping, and epochs were chosen for each configuration, including whether any settings tried did not produce misalignment?
  3. As mentioned in the critical weaknesses section, how can we know if misaligment is due to self-distillation, and not due to model weight alternation? Could you report, with matched hyperparameters: (a) distillation from a frozen initial-model teacher with the same cue, and (b) supervised fine-tuning on the model’s own sampled outputs without any cue? The “Do not help with my request” control pushes toward refusal and so does not serve as a neutral baseline. If these baselines produce comparable misalignment, I would view the contribution as an instance of known fine-tuning fragility; if they do not, I would consider the main claim better supported.
9 comments
AuthorsOpenReview ↗

Additional baselines, larger prompt sets, and test curves (1/3)

We thank the reviewer for their feedback and for their positive score on the presentation, soundness, and informativeness of the paper.

Re [Q1: Progressive misalignment under bigger prompt set and batch size] Yes, we observe that progressive misalignment persists with a substantially larger prompt set, and with a larger batch size. To study this, we run the same setup as in Section 3.1 with “Excellent” as the cue. We run the same experiment with a much larger prompt set (413 prompts) and a larger batch size. Of course, all the prompts we use are different from the prompts in SORRY-Bench and StrongREJECT. The prompts are standard and come from well-known datasets such as AdvBench (Zou et al., 2023) and HarmBench (Mazeika et al., 2024). We see this as convincing evidence that the effect is not a result of the circling of a small prompt set.

SettingSORRY-Bench: Off-policySORRY-Bench: On-policyStrongREJECT: Off-policyStrongREJECT: On-policy
Initial model0.307 ± 0.0100.307 ± 0.0100.095 ± 0.0090.095 ± 0.009
Paper setup, batch size = 10.956 ± 0.0080.962 ± 0.0020.673 ± 0.0450.655 ± 0.015
More prompts, batch size = 10.966 ± 0.0120.952 ± 0.0120.521 ± 0.0080.632 ± 0.004
More prompts, batch size = 160.882 ± 0.0210.879 ± 0.0040.580 ± 0.0200.546 ± 0.007

Re [[...] leaves unclear whether degradation would occur during realistic, capability-improving self-distillation, as suggested by the paper's motivation of recursive self-improvement [...]] We provide, as you suggested, more evidence to better support the motivation of our paper and strengthen our claims. We ran our self-distillation setup on a data mixture containing prompts from a downstream task we want to improve on. In the first experiment, we take our 20 prompts in addition to 120 prompts on Chemistry (SciKnowEval). In the second, we take our 20 prompts in addition to 40 prompts on coding from LiveCodeBench. These datasets match the setup used by the SDPO paper by Hübotter et al. (ICML, 2026). We run on-policy self-distillation, where the feedback for the refusal to harmful prompts is “Excellent”, the feedback for the questions on chemistry is a simulated user (Qwen3 32B) giving feedback on the answer, and the feedback for the coding experiments uses either terminal logs or the answer to hidden tests. Again, these match the realistic setup from the SDPO paper by Hübotter et al. (ICML, 2026).
We report our results in the tables below. Here we take the latest checkpoint during training, which is the one where the model achieves the highest accuracy on the downstream task, and thus do not select with respect to misalignment at all. We thus observe that progressive misalignment might emerge during useful capability improvement across both tasks. We hope this provides an interesting insight and additional evidence to support our claim.

Qwen3 14B, on-policy self-distillation on code (LiveCodeBench)

ModelSORRY-BenchStrongREJECTLiveCodeBench accuracy
Initial model0.307 ± 0.0100.100 ± 0.0090.531 ± 0.055
On-policy self-distillation0.928 ± 0.0180.510 ± 0.0190.781 ± 0.059

Qwen3 14B, on-policy self-distillation on Chemistry (SciKnowEval)

ModelSORRY-BenchStrongREJECTSciKnowEval accuracy
Initial model0.306 ± 0.0110.094 ± 0.0090.448 ± 0.018
On-policy self-distillation0.873 ± 0.0040.491 ± 0.0120.536 ± 0.024
AuthorsOpenReview ↗

Additional baselines, larger prompt sets, and test curves (2/3)

Re [ [...] This favors the checkpoints exhibiting the greatest safety degradation, rather than those selected through a realistic training objective or predetermined training budget] We tried to propose a thoughtful selection to avoid cherry-picking the checkpoint. Moreover, as the experiments above on more realistic tasks such as chemistry and code show, progressive misalignment also arises from checkpoints that are chosen with respect to a fixed compute budget and performance on a downstream task. We hope this addresses your question. In addition, you can see from the added full-checkpoints plot in Appendix A.4 (Figure 13) that the selection does not necessarily pick the most misaligned checkpoint according to the final eval, and so we have not cherry-picked the checkpoint. However, we also emphasize that the community cares about the worst-case scenario for model safety - a model misaligned only under certain conditions can be dangerous if these go undetected.


Re [Q2: Hyperparameter of self-distillation and test curves] We selected the hyperparameters through usual tuning (grid search), as standard in deep learning papers, and ensured that all the baselines used the best hyperparameters possible (that’s also the case for the additional ablations below). We did not spend an extensive amount of time searching for hyperparameters to make misalignment arise. We think we provided convincing evidence that misalignment can emerge and, as such, we have not tested a huge sweep. Of course, it is not the case that every hyperparameter setting would lead to misalignment, similarly to how not every hyperparameter would make a deep learning method succeed. Hyperparameters are mainly selected to avoid degenerate generations (e.g. the continuous repetition of one word). For example, a learning rate too large would cause the generations to collapse. In fact, model collapse is a known failure mode of SDPO (e.g., see Peng et al., 2026). We hope this clarifies your question.

We also report additional test curves in Appendix A.4, Fig. 13, as per your request, and results across multiple training seeds in Appendix A.5, Fig. 14. The checkpoint was chosen as described in the paper, where we select checkpoints maximizing the product of the score on a validation set on the two benchmarks (which sometimes is not the checkpoint with the highest misalignment on the full test set). Again, as the experiments above on more realistic tasks show, progressive misalignment also arises from checkpoints that are chosen with respect to a fixed compute budget and performance on a downstream task.

AuthorsOpenReview ↗

Additional baselines, larger prompt sets, and test curves (3/3)

Re [Q3: how can we know if misalignment is due to self-distillation, and not due to model weight alternation [...] Could you report [...] (a) distillation from a frozen initial-model teacher with the same cue, and (b) supervised fine-tuning on the model’s own sampled outputs] We run the additional baselines to answer your question. We report the results in the tables below (and in Appendix A.1). First, we run SFT on the generated data across epochs. Misalignment does not arise in this case. This shows that matching the student to a teacher conditioned on an extra cue is necessary for progressive misalignment to arise. Additionally, we run self-distillation but keep the teacher frozen across epochs. Again, progressive misalignment does not occur, providing evidence that the recursive update of the teacher is also another fundamental component for progressive misalignment to emerge. As we can observe, the baselines are completely flat. It is thus the case that progressive misalignment is not due to weight alteration. We report the results below:

Qwen3 14B (off-policy, Excellent)

StepSORRY-Bench: Self-dist.SORRY-Bench: Frozen teacherSORRY-Bench: SFTStrongREJECT: Self-dist.StrongREJECT: Frozen teacherStrongREJECT: SFT
00.311 ± 0.0100.311 ± 0.0100.311 ± 0.0100.106 ± 0.0170.106 ± 0.0170.106 ± 0.017
200.244 ± 0.0050.295 ± 0.0030.272 ± 0.0130.016 ± 0.0050.076 ± 0.0060.083 ± 0.009
400.275 ± 0.0130.297 ± 0.0050.280 ± 0.0110.028 ± 0.0050.082 ± 0.0060.071 ± 0.006
600.299 ± 0.0110.280 ± 0.0060.272 ± 0.0150.037 ± 0.0080.072 ± 0.0010.072 ± 0.007
800.313 ± 0.0110.284 ± 0.0160.275 ± 0.0200.041 ± 0.0030.076 ± 0.0140.064 ± 0.003
1000.537 ± 0.0180.292 ± 0.0110.277 ± 0.0050.194 ± 0.0240.066 ± 0.0110.069 ± 0.009
1200.962 ± 0.0080.288 ± 0.0050.277 ± 0.0100.695 ± 0.0300.075 ± 0.0090.069 ± 0.008

Phi-4 14B (off-policy, Excellent)

StepSORRY-Bench: Self-dist.SORRY-Bench: Frozen teacherSORRY-Bench: SFTStrongREJECT: Self-dist.StrongREJECT: Frozen teacherStrongREJECT: SFT
00.261 ± 0.0060.261 ± 0.0060.261 ± 0.0060.038 ± 0.0060.038 ± 0.0060.038 ± 0.006
200.847 ± 0.0050.173 ± 0.0130.244 ± 0.0150.386 ± 0.0050.023 ± 0.0040.038 ± 0.001
400.947 ± 0.0140.128 ± 0.0080.243 ± 0.0100.564 ± 0.0370.019 ± 0.0030.033 ± 0.006
600.932 ± 0.0130.150 ± 0.0130.237 ± 0.0170.585 ± 0.0200.023 ± 0.0030.036 ± 0.009

As you mentioned (“ [...] if they do not, I would consider the main claim better supported.”), we hope this further supports our main claim and addresses your final doubts, leading to positive support for our paper.

We hope these additional experiments are satisfactory evidence to address your questions. If so, we would kindly ask the reviewer to consider positively supporting our paper, or to let us know which concerns remain.

Thank you again for your detailed review.


References

  • Hübotter et al. (ICML, 2026). Jonas Hübotter, Frederike Lübeck, Lejs Deen Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, Andreas Krause. Reinforcement Learning via Self-Distillation. International Conference on Machine Learning (ICML), 2026.
  • Mazeika et al. (ICML, 2024). Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, Dan Hendrycks. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. International Conference on Machine Learning (ICML), 2024.
  • Peng et al. (arXiv, 2026). Keqin Peng, Chen Li, Yuanxin Ouyang, Yancheng Yuan, Liang Ding. Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation. arXiv, 2026.
  • Zou et al. (arXiv, 2023). Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, Matt Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv, 2023.
Reviewer_yKaUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies recursive self-distillation in which the current model provides a teacher distribution with extra context and the student learns to match it without that context. Fixed benign cues can substantially increase harmful-request compliance in Qwen3 and Phi-4 on SORRY-Bench and StrongREJECT, with simulated positive feedback also producing deterioration in Qwen3. A gradient-based selection procedure also finds benign prompts whose generated answers induce safety deterioration, although this curriculum is deliberately selected toward harmful compliance. The experiments establish a safety failure in these settings and offer a preliminary explanation based on amplified changes in the early token distributions.

Critical strengths
  1. Significant impact on safety from benign cues. In Figure 4, off-policy Excellent raises SORRY-Bench compliance from around 30% to above 90% in both families, with StrongREJECT also worsening. The explicit refusal cue keeps compliance near its initial level or lowers it, providing a meaningful control. This is a credible reason to track safety during repeated post-training updates.

  2. The proposed guard can fail. In Figure 12, selecting prompts with the lowest initial scores for predicted changes in harmful-answer preference still raises Phi’s SORRY-Bench compliance from 26% to 55% off policy. This shows that selection intended to increase harmful compliance is not necessary for every benign prompt failure. Compliance falls for Qwen3 under the same rule, showing that the proposed protection depends on the model.

Critical weaknesses
  1. Missing materials and unclear evaluation partitions. Appendix B.5 does not provide the benign bank, selected curricula or reference checkpoint, response pairs and generation configuration needed to reconstruct the central experiment. Held-out validation is reported, but missing identifiers and final counts prevent checking the separation of the selector, validation and test partitions.

  2. Claims about useful self-improvement are too broad. Section 3.3 reports no failure in preliminary random-benign training and selects the successful benign curriculum toward harmful compliance. Section 5 already leaves safety erosion during useful capability improvement open so the abstract should limit its interpretation to the tested conditions.

I would raise my rating to weak acceptance if the released materials substantiate the selected-benign result and clean evaluation splits, with claims limited to the tested conditions.

Questions for authors
  1. Can you release the benign bank and selected prompts, reference data and configuration with identifiers and final counts that establish clean selector, validation and test partitions?
5 comments
AuthorsOpenReview ↗

Benign bank prompt release and additional experiments on self-improvement claims

We thank the reviewer for their feedback and for recognizing that our results have practical implications in terms of safety monitoring during continuous post-training.

Re [Q1: Can you release the benign bank and selected prompts…] Yes, sure! We uploaded the JSON file and additional info on our benign prompt bank to https://anonymous.4open.science/r/Data-Progressive-Misalignment-6388/. The same bank of 1000 prompts has been used for both on- and off-policy experiments. These benign prompts were collected from a variety of general datasets (e.g., Alpaca, Dolly, GSM8K). Examples include “Name three kinds of clouds”, “Name five capital cities in Europe”, or “Provide two examples of a synonym for a noun”. At the same link, you can also find the selected prompts (i.e., Ct\mathcal{C}_t in our Algorithm 1, Section 3.3). We clarify that we select and train on the benign prompts selected by Algorithm 1, while the validation and test sets are from SORRY-Bench and StrongREJECT.


Re [Claims about self-improvement] We agree that we did not directly study self-improvement from real-world production tasks. However, we ran some additional experiments to further substantiate and support our claims. In more detail, we ran our self-distillation setup on a data mixture containing prompts from a downstream task we want to improve on. In the first experiment, we take our 20 prompts in addition to 120 prompts on Chemistry (SciKnowEval). In the second, we take our 20 prompts in addition to 40 prompts on coding from LiveCodeBench. These datasets match the setup used by the SDPO paper by Hübotter et al. (ICML, 2026). We run on-policy self-distillation, where the feedback for the refusal to harmful prompts is “Excellent”, the feedback for the questions on chemistry is a simulated user (Qwen3 32B) giving feedback on the answer, and the feedback for the coding experiments uses either terminal logs or the answer to hidden tests. Again, these match the realistic setup from the SDPO paper by Hübotter et al. (ICML, 2026). We report our results in the tables below. Here we take the latest checkpoint during training, which is the one where the model achieves the highest accuracy on the downstream task, and thus do not select with respect to misalignment at all. We thus observe that progressive misalignment might emerge during useful capability improvement across both tasks. We hope this provides an interesting insight and additional evidence to support our claim.

Qwen3 14B, on-policy self-distillation on code (LiveCodeBench)

ModelSORRY-BenchStrongREJECTLiveCodeBench
Initial model0.307 ± 0.0100.100 ± 0.0090.531 ± 0.055
On-policy self-distillation0.928 ± 0.0180.510 ± 0.0190.781 ± 0.059

Qwen3 14B, on-policy self-distillation on Chemistry (SciKnowEval)

ModelSORRY-BenchStrongREJECTSciKnowEval
Initial model0.306 ± 0.0110.094 ± 0.0090.448 ± 0.018
On-policy self-distillation0.873 ± 0.0040.491 ± 0.0120.536 ± 0.024

Let us know if these clarifications address your question and, if so, would lead you to raise the score as you mentioned. We would be happy to clarify any additional doubts. Thanks again for your review!

References

  • Hübotter et al. (ICML, 2026). Jonas Hübotter, Frederike Lübeck, Lejs Deen Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, Andreas Krause. Reinforcement Learning via Self-Distillation. International Conference on Machine Learning (ICML), 2026.
AuthorsOpenReview ↗

Reminder to engage in the review process

Dear reviewer,

As there are only a few days remaining in the discussion period, we would greatly appreciate it if you could review and engage with our response. We believe our detailed response and additional experiments address the concerns you raised, but please let us know if you have any further questions.

If we have successfully addressed your concerns, we would kindly ask the reviewer to consider raising their score accordingly and positively supporting our paper.

AuthorsOpenReview ↗

Less than 24 hours left in the rebuttal period

Dear Reviewer,

There are fewer than 24 hours left in the rebuttal period. As no other concerns have been raised, we wanted to check whether anything remains unclear.

As you mentioned in your initial review. "I would raise my rating to weak acceptance if the released materials substantiate the selected-benign result and clean evaluation splits". We have indeed released the materials upon request and have provided additional experimental results that sustain our claims. We would greatly appreciate it if you would raise your score and positively support the paper.

Thank you again for taking the time to review the paper.

Reviewer_XhvBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies recursive self-distillation (rsd): a model is trained to match the tokens it would have produced with extra context f (a cue or feedback), and the resulting student becomes the next round’s teacher. Starting from a safe model that refuses all 20 training prompts, the authors show that rsd with semantically benign feeedback (“excellent”, “I like Owls”) progressively makes the model broadly compliant with harmful requests a failure they call progressive misalignment. They validate this on Qwen3 14B and Phi-4-14B over sorryBench and strongreject, in both on and off-policy variants, with a control cue (“Do not help with my request”) that does not induce the effect. They further show it arises with an llm simulated benign user in place of a fixed cue (sec3.2), that a gradient-based selection procedure can find benign prompts which also induce it (sec3.3), that it worsens with scale across qwen3 0.6B 32B (sec3.4), and that Iifeval and gsm8k performance is preserved throughout (B.1).

Critical strengths

I like this paper. It identifies a genuinely new failure mode cleanly separated from prior work: unlike Qi et al. (2024), He et al. (2024)/ Betley et al. (2025), the training data here is self-generated and initially safe the model produces only refusals and the feedback never requests compliance yet compliance rises from 31% to 97% (Qwen3) and 26% to 94% (Phi4), which, if it holds up, is directly consequential for recursive self-improvement pipelines and is the main reason I lean accept. The capability control in Appendix B.1 is the right experiment and makes the result actionable: ifeval and gsm8k stay essentially flat while harmful compliance exceeds 90%, establishing that standard capability evaluation will not surface the problem. I also credit the authors for reporting negative results honestly their proposed safety guard fails on Phi4 (B.5), and one mixture in Table 4 moves the other way.

Critical weaknesses

I found the causal claim is not established, because there is no cue-free control: the paper attributes the effect to the benign cue but never runs rsd with f = a neutral filler on the same 20 prompts, leaving repeated fine-tuning on a tiny self-generated set, 20 prompts, up to 50 epochs as a live alternative explanation, and “Do not help with my request” does not fill the gap since it is a cue with a strong refusal-ward component and so tests the direction of the cue rather than its presence; this is the single experiment that most determines my rating. Compounding this, the headline numbers come from a selected checkpoint under per-cue hyperparameters and a single training seed: the reported value is the argmax over ~10 checkpoints of a non-monotonic rise then collapse trajectory (Fig.9) all =- values reflect generation seeds only, and hyperparameters differ systematically between treatment and control cues (Table 2 Qwen3 on-policy, 3e-6/50 epochs for Excellent vs 1e-6/1 epoch for the control), which is unexplained and further weakens the control. Recursion depth is also not reported in comparable units Fig. 2 places ~3 epochs (Phi4, 60 steps) beside ~50 epochs (Qwen3, 1000 steps) on side-by-side axes so “progressive” denotes very different budgets across panels.

Questions for authors

What happens under RSD with no feedback at all (or a semantically null filler), on the same 20 prompts and the same schedule? If possible report the resulting sorrybench and strongreject curves. Without this arm I cannot separate the effect of the benign cue from over fitting on a 20-prompt self-generated set, and the answer would most directly change my rating.

Please report full per-epoch test curves rather than the selected checkpoint, with training-seed variance (>=3 seeds) for the 14B results, and with the control cue at matched learning rate. As it stands the headline numbers are an argmax over ~10 checkpoints from a single training seed, and the control is trained at up to 3× lower LR for unexplained reasons (Table 2), so I cannot tell how much of the reported gap is the phenomenon and how much is selection and hyperparameter asymmetry.

Minor: B.4 cites Figure 4 for generations (should be Fig. 3); A.3 ends with a malformed sentence; “degenerancy” typo; Eq. 3 averages over D while Alg. 2 updates per prompt.

2 comments
AuthorsOpenReview ↗

SFT and frozen teacher baselines, and larger prompt set (1/2)

We thank the reviewer for their feedback and questions, and for their positive thoughts on our work.

Re [Q1: Comparison with RSD with no feedback at all] Looking at the equations in (1) and (2) for off- and on-policy self-distillation updates, we note that, with ff as an empty string, there is no update (KL=0\mathrm{KL} = 0) as the same model serves both as a student and as a teacher. However, we can consider a few closely related ablations to address your question:

  • SFT baseline (see Appendix A.1): First, we run SFT on the generated data across epochs. Misalignment does not arise in this case. This shows that matching the student to a teacher conditioned on an extra cue is necessary for progressive misalignment to arise. We hope this clarifies the role of the feedback for our phenomena and addresses your question.

  • Frozen Teacher baseline (see Appendix A.1): We also run self-distillation but keep the teacher frozen across epochs. Again, progressive misalignment does not occur, providing evidence that the recursive update of the teacher is also another fundamental component for progressive misalignment to emerge.

We report the results of the described ablation in the tables below:

Qwen3 14B (off-policy, Excellent)

StepSORRY-Bench: Self-dist.SORRY-Bench: Frozen teacherSORRY-Bench: SFTStrongREJECT: Self-dist.StrongREJECT: Frozen teacherStrongREJECT: SFT
00.311 ± 0.0100.311 ± 0.0100.311 ± 0.0100.106 ± 0.0170.106 ± 0.0170.106 ± 0.017
200.244 ± 0.0050.295 ± 0.0030.272 ± 0.0130.016 ± 0.0050.076 ± 0.0060.083 ± 0.009
400.275 ± 0.0130.297 ± 0.0050.280 ± 0.0110.028 ± 0.0050.082 ± 0.0060.071 ± 0.006
600.299 ± 0.0110.280 ± 0.0060.272 ± 0.0150.037 ± 0.0080.072 ± 0.0010.072 ± 0.007
800.313 ± 0.0110.284 ± 0.0160.275 ± 0.0200.041 ± 0.0030.076 ± 0.0140.064 ± 0.003
1000.537 ± 0.0180.292 ± 0.0110.277 ± 0.0050.194 ± 0.0240.066 ± 0.0110.069 ± 0.009
1200.962 ± 0.0080.288 ± 0.0050.277 ± 0.0100.695 ± 0.0300.075 ± 0.0090.069 ± 0.008

 

Phi-4 14B (off-policy, Excellent)

StepSORRY-Bench: Self-dist.SORRY-Bench: Frozen teacherSORRY-Bench: SFTStrongREJECT: Self-dist.StrongREJECT: Frozen teacherStrongREJECT: SFT
00.261 ± 0.0060.261 ± 0.0060.261 ± 0.0060.038 ± 0.0060.038 ± 0.0060.038 ± 0.006
200.847 ± 0.0050.173 ± 0.0130.244 ± 0.0150.386 ± 0.0050.023 ± 0.0040.038 ± 0.001
400.947 ± 0.0140.128 ± 0.0080.243 ± 0.0100.564 ± 0.0370.019 ± 0.0030.033 ± 0.006
600.932 ± 0.0130.150 ± 0.0130.237 ± 0.0170.585 ± 0.0200.023 ± 0.0030.036 ± 0.009

Let us know if this evidence clarifies your doubts, and thanks again for raising this point.

AuthorsOpenReview ↗

SFT and frozen teacher baselines, and larger prompt set (2/2)

Re [I cannot separate the effect of the benign cue from overfitting on a 20-prompt self-generated set] You mentioned that the effect might just be due to overfitting on a small self-generated prompt set, and we think the baselines above provide evidence that such is not the case. However, we can even ablate the prompt set size to give even more evidence for our claim. To check for this, we run our experiments from Section 3.1 using a much larger prompt set and varying the batch size; see the results in the table below. We see that, in this case, progressive misalignment also emerges. Thus, this provides evidence that our results are not an artifact of the small training set either. Thanks for the question, and we hope this clarifies your doubt fully.

SettingSORRY-Bench: Off-policySORRY-Bench: On-policyStrongREJECT: Off-policyStrongREJECT: On-policy
Initial model0.307 ± 0.0100.307 ± 0.0100.095 ± 0.0090.095 ± 0.009
20 prompts (paper setup), batch size = 10.956 ± 0.0080.962 ± 0.0020.673 ± 0.0450.655 ± 0.015
413 training prompts, batch size = 10.966 ± 0.0120.952 ± 0.0120.521 ± 0.0080.632 ± 0.004
413 training prompts, batch size = 160.882 ± 0.0210.879 ± 0.0040.580 ± 0.0200.546 ± 0.007

Re [Q2: Please report full per-epoch test curves rather than the selected checkpoint, with training-seed variance (>=3 seeds)...] Sure! We report additional test curves in Appendix A.4, Fig 13, and results across multiple training seeds in Appendix A.5, Fig 14. In general, we observed that our results are not sensitive to the training or evaluation seeds. Different hyperparameters were chosen to ensure the stability of SDPO, which is known to be a tricky method to run in practice. For example, on-policy usually requires more steps to converge in general than off-policy (due to the fact that the teacher’s generation can skew the student earlier), and this is why we use different training budgets. Moreover, we tuned the hyperparameters of all baselines/control experiments and cues to ensure they have the best setup possible for a fair comparison.


We would like to thank the reviewer for catching the typos! We updated the manuscript to incorporate this feedback.

Thanks again for your thoughtful review, and let us know if we solved all your doubts. If so, we would kindly ask the reviewer to increase the score to show support for the paper.

Reviewer_Md3MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates progressive safety degradation during recursive self-distillation (RSD). The authors show that repeatedly distilling a model's predictions with benign feedback can substantially weaken its safety refusal behavior, even when the initial generations are safe.

Experiments on Qwen3 and Phi-4 demonstrate this phenomenon under fixed feedback, benign simulated-user feedback, and selected benign training prompts. The paper further examines model-scale effects and provides preliminary analysis of token-level distribution shifts.

Critical strengths
  1. Interesting and credible empirical observations.

The paper demonstrates substantial safety degradation despite initially safe generations and benign feedback. The observation that feedback which does not immediately increase harmful compliance can nevertheless lead to significant degradation through repeated self-distillation is noteworthy.

  1. Solid experimental validation.

The experiments cover multiple model families, feedback conditions, training regimes, and model sizes. The results are generally consistent, and additional evaluations suggest that safety degradation can occur while general benchmark performance remains largely intact.

Critical weaknesses
  1. Limited incremental scientific contribution beyond established findings.

My primary concern is that the paper's central findings largely fall within phenomena already established in continual learning, self-distillation, and safety alignment research.

Yang et al. (ACL 2024), Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning, investigated catastrophic forgetting and the preservation of existing model capabilities during self-distillation. More directly, Wang et al. (2026), Denser ≠ Better: Limits of On-Policy Self-Distillation for Continual Post-Training, demonstrated forgetting, distribution drift, and self-reinforcing instability in SDPO. In the safety domain, Qi et al. (ICLR 2024) already established that even benign fine-tuning can degrade safety alignment.

Against this background, the main observation that safety refusal behavior deteriorates under repeated self-distillation appears largely consistent with established knowledge about forgetting and training instability.

I acknowledge that the paper provides new empirical evidence under specific feedback conditions. However, these observations offer limited additional understanding beyond the broader phenomena already documented. In particular, the paper does not establish what substantive new knowledge studying safety refusal behavior contributes beyond the more general findings in continual learning and self-distillation.

  1. The additional analyses provide limited new insights beyond existing safety research.

The benign-data experiments in Section 3.3 closely relate to He et al. (COLM 2024), What is in Your Safe Data? Identifying Benign Data that Breaks Safety, which demonstrated that carefully selected benign training examples can substantially weaken safety alignment using gradient-based and representation-based analysis. Wang et al. (ICML 2026), From Parameter Dynamics to Risk Scoring, further investigated cumulative parameter drift toward unsafe directions and sample-level safety risk estimation.

Similarly, the early-token distribution shifts discussed in Section 3.5 closely resemble the shallow safety alignment phenomenon analyzed by Qi et al. (ICLR 2025), Safety Alignment Should Be Made More Than Just a Few Tokens Deep.

The paper extends these observations to recursive self-distillation, but offers limited new theoretical or empirical insight into why safety degradation occurs, how it differs from established patterns, or what new principles can be learned from it.

Overall, the four contributions appear to provide additional empirical instances of existing phenomena rather than a sufficiently substantial advance in scientific understanding.

Questions for authors
  1. Given the existing literature on catastrophic forgetting, distribution drift, and instability in self-distillation, what substantive new scientific understanding does progressive misalignment provide beyond demonstrating that safety refusal behavior is another property susceptible to degradation?

  2. How do the benign-data selection and early-token distribution analyses advance beyond prior work on safety degradation, particularly He et al. (2024), Qi et al. (2025), and Wang et al. (2026)?

4 comments
AuthorsOpenReview ↗

Clarification on the position and novelty of Progressive Misalignment.

We thank the reviewer for their feedback and for their very positive scores on the presentation and soundness of the paper.

Re [Q1: [...] what substantive new scientific understanding does progressive misalignment provide beyond [...] ] We would like to start by pointing out that one of the main conclusions in Yang et al. (ACL, 2024) is that self-distillation can be beneficial for LLM safety (see the final sentences in their abstract). Our work surprisingly showcases that the opposite might happen. Other related work you mentioned shows how self-distillation could be unstable, collapse, and eventually forget (Wang et al., arXiv, 2026). In our work instead, we show that misalignment can emerge even when unrelated capabilities are not affected (see Appendix C.1, Figure 16). In addition, we added experiments in which the model actually improves on a downstream task (Code or Chemistry) while becoming more misaligned (see Appendix A.2, Figures 10 and 11). These are complementary results to the findings in previous literature that showcase the degradation of general capabilities. We believe misalignment arising while a model remains capable is an important risk for deployment of these models. We also note that the failure modes you highlighted are due to the training objective of self-distillation, which implicitly adds bias and, in the limit, leads the model to collapse (e.g., see Peng et. al 2026, Kirby et. al 2026). In contrast, it was not predictable that benign self-distillation would lead to model misalignment. Finally, we would like to note that the earliest version of the paper (Wang et al., arXiv, 2026) appeared on arXiv in July 2026 and is currently under submission. ICLR guidelines mention that everything published in a peer-reviewed venue in the 4 months preceding the deadline is considered concurrent work and does not necessarily need to be addressed. Anyway, we hope this answer clarifies the positioning of our paper.


Re [Early-token analysis of Qi et al. (2025), comparison with benign SFT in (Qi et al., 2024)] We agree with the fact that alignment being shallow (Qi et al., 2025) is not a new observation. In fact, we point that out in the introduction. However, it matters under which scenarios (and hence use cases!) misalignment can arise. We want to caution against a very natural use case: where you want to use self-generated data to potentially improve your model, either on safety (e.g., by saying that a refusal answer is “Excellent”) or on an unrelated task (see the new experiments on math and code pointed above). The simple scenario studies in the work might be reminiscent of self-improvement through post-training that nowadays is widely adopted (Thinking Machines, 2026). Of course, our setup is a first investigation and studies a simplified setting. We also added a new ablation (Appendix A.1, Figure 9) testing the setup of Qi et al. (2024), which fine-tunes using SFT. Training on the same benign refusals using SFT does not show misalignment; on the contrary, it is clearly observable if we do self-distillation with the moving teacher. We hope this clarifies the positioning of our contributions and their novelty with respect to previous work.


Re [Benign data selection method and positioning] We agree that the work of He et al. (COLM, 2024) is highly relevant and, as such, was cited in our main body. There are a few differences. In our case, we are only selecting the prompts (so we do not have control over the answer we end training on), and our selection is a bit different and slightly more complex due to the online nature of self-distillation (the teacher is NOT fixed, so the effect each answer has changes across training). A similar point can be made for Wang et al. (ICML, 2026). We also do not position the method itself as a novel contribution of our work, but rather, we can do benign data selection of the prompts our model will be distilled on, and the model will become broadly misaligned.


Overall, we believe our results to be surprising and in contrast and surprising with respect to other literature that mentions self-distillation as a possible way to improve alignment (e.g., Yang et al. (ACL, 2024), and Kleine-Buening et al. (COLM, 2025)). They are complementary findings to the ones showcasing the collapse and catastrophic forgetting of self-distillation. We also show that progressive misalignment can be more severe for bigger models (Figure 7), which can be more of an issue for real-task deployment.

We hope this clarifies the different positioning and contributions of our paper, which was also emphasized by Reviewer XhvB: “[This work] identifies a genuinely new failure mode cleanly separated from prior work: unlike Qi et al. (2024), He et al. (2024)/ Betley et al. (2025), [...].”

Let us know if we addressed your concerns! If so, we would kindly ask the reviewer to consider raising their score or to let us know which concerns remain.

AuthorsOpenReview ↗

References

References

  • Betley et al. (ICML, 2025). Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, Owain Evans. Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs. International Conference on Machine Learning (ICML), 2025.
  • He et al. (COLM, 2024). Luxi He, Mengzhou Xia, Peter Henderson. What is in Your Safe Data? Identifying Benign Data that Breaks Safety. Conference on Language Modeling (COLM), 2024.
  • Kirkby et al. (Base Labs blog, 2026). Max Kirkby, Michael Psenka, Charles O'Neill. The Bias of On-Policy Self-Distillation. Base Labs (Baseten), November 2026. https://labs.baseten.co/articles/the-bias-of-on-policy-self-distillation
  • Kleine Buening et al. (COLM, 2026). Thomas Kleine Buening, Jonas Hübotter, Barna Pásztor, Idan Shenfeld, Giorgia Ramponi, Andreas Krause. Aligning Language Models from User Interactions. Conference on Language Modeling (COLM), 2026.
  • Peng et al. (arXiv, 2026). Keqin Peng, Chen Li, Yuanxin Ouyang, Yancheng Yuan, Liang Ding. Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation., 2026.
  • Qi et al. (ICLR, 2024). Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, Peter Henderson. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! International Conference on Learning Representations (ICLR), 2024.
  • Qi et al. (ICLR, 2025). Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, Peter Henderson. Safety Alignment Should Be Made More Than Just a Few Tokens Deep. International Conference on Learning Representations (ICLR), 2025.
  • Thinking Machines Lab (blog, 2026). Thinking Machines Lab. Introducing Inkling 2 Small. Thinking Machines Lab blog, November 2026. https://thinkingmachines.ai/news/inkling-2-small/
  • Wang et al. (arXiv, 2026). Meng Wang, Haohan Zhao, Wenzhuo Liu, Lu Yang, Geng Liu, Haiyang Guo, Guo-Sen Xie, Gaofeng Meng, Hongbin Liu, Fei Zhu. Denser ≠ Better: Limits of On-Policy Self-Distillation for Continual Post-Training., 2026.
  • Wang et al. (ICML, 2026). Xiao Wang, Yifei Zhang, YongKang Liu, Xiaocui Yang, Zihan Wang, Shi Feng, Daling Wang. From Parameter Dynamics to Risk Scoring: Quantifying Sample-Level Safety Degradation in LLM Fine-tuning. International Conference on Machine Learning (ICML), 2026.
  • Yang et al. (ACL, 2024). Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, Qian Liu. Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning. Annual Meeting of the Association for Computational Linguistics (ACL), 2024.
AuthorsOpenReview ↗

Reminder to engage in the review process

Dear reviewer,

As there are only a few days remaining in the discussion period, we would greatly appreciate it if you could review and engage with our response. We believe our detailed response and additional experiments address the concerns you raised, but please let us know if you have any further questions.

If we have successfully addressed your concerns, we would kindly ask the reviewer to consider raising their score accordingly and positively supporting our paper.

AuthorsOpenReview ↗

Less than 24 hours left in the rebuttal period

Dear Reviewer,

There are fewer than 24 hours left in the rebuttal period. As no other concerns have been raised, we wanted to check whether anything remains unclear. If not, we would greatly appreciate it if you would raise your score and positively support the paper, in light of our detailed rebuttal above.

Thank you again for taking the time to review the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.