- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether on-policy distillation (OPD) for safety alignment can transfer backdoors from a teacher to a student. Experiments show that, when a small fraction of distillation prompts contain the trigger, students may inherit the backdoor while improving their safety on standard evaluations. More training epochs amplify this risk, and top-k KL accelerates backdoor learning in most tested settings. The authors analyze the learning dynamics and introduce Lazy Defense, which clips rewards to delay backdoor transfer.
Critical strengths
-
The paper examines backdoor transfer in OPD across multiple dimensions. The experiments cover poisoning rates, model sizes, training epochs, and KL formulations, showing that a low harmful-response rate on inputs without the trigger can coexist with a high backdoor attack success rate.
-
The analysis of training epochs and backdoor learning dynamics provides useful insights. Section 4.2 combines token-level gradient decomposition with prediction agreement to explain why backdoor behavior emerges later in training, suggesting that a low attack success rate early in training is insufficient to establish the safety of distillation.
Critical weaknesses
-
The paper as a whole reads more like an exploratory experimental report, with insufficient novelty and depth of contribution for a conference paper. It primarily documents backdoor transfer under different training configurations, without direct comparisons to prior work or sufficiently distinct new findings. The analysis of backdoor mechanisms mainly provides exploratory empirical evidence, while Lazy Defense consists only of trivial reward clipping. Taken together, these observations and the simple intervention do not establish a strong standalone contribution.
-
The threat model is under-specified, and its practical relevance is not adequately established. The experiments assume both a backdoored teacher and distillation prompts containing the trigger, but do not sufficiently explain the attacker's capabilities or how both conditions arise in downstream training. Since the distillation data already contain poisoned data, it is unsurprising that the student learns the backdoor during distillation. Furthermore, key controls involving unpoisoned distillation prompts and clean teachers are missing, making it difficult to establish the source and scope of the risk.
Questions for authors
-
Does the attacker need to control both the teacher and a subset of the distillation prompts? What is a realistic path for the trigger to enter downstream distillation data? Are there control results for a backdoored teacher with entirely clean prompts, and for a clean teacher with prompts containing the trigger?
-
Section 4.2 explains backdoor learning through top-k overlap on student rollouts, whereas Appendix D.3 measures top-1 agreement on teacher-generated prefixes. How does this metric support the proposed mechanism rather than merely reflect general progress in fitting the teacher?
-
When normal task performance and safety gains are comparable, does Lazy Defense offer additional benefits over reducing the learning rate or stopping training early? How do the results rule out merely slowing learning overall or increasing refusals, rather than effectively suppressing the backdoor?
3 comments
Q3
We have also added comparisons between learning rate reduction and Lazy Defense in Appendix G of the revised paper. We report results at selected epochs below.
Learning rate reduction
| Metric | Epoch 3 | Epoch 4 | Epoch 6 | Epoch 8 | Epoch 12 |
|---|---|---|---|---|---|
| Clean ASR (%) | 0.6 | 1.5 | 2.2 | 2.6 | 3.5 |
| Trigger ASR (%) | 1.0 | 1.8 | 5.3 | 42.4 | 66.4 |
Gradient clipping
| Metric | Epoch 3 | Epoch 4 | Epoch 6 | Epoch 8 | Epoch 12 |
|---|---|---|---|---|---|
| Clean ASR (%) | 2.5 | 2.4 | 3.0 | 3.9 | 3.1 |
| Trigger ASR (%) | 3.2 | 16.1 | 57.4 | 75.4 | 62.9 |
Lazy Defense
| Metric | Epoch 3 | Epoch 4 | Epoch 6 | Epoch 8 | Epoch 12 |
|---|---|---|---|---|---|
| Clean ASR (%) | 1.5 | 2.3 | 2.5 | 3.3 | 3.8 |
| Trigger ASR (%) | 1.7 | 2.3 | 20.0 | 45.0 | 69.5 |
In this setting, reducing the learning rate provides a delay comparable to Lazy Defense, while gradient clipping is less effective. At epoch 8, Trigger ASR is 42.4% with a reduced learning rate and 45.0% with Lazy Defense, compared with 75.4% under gradient clipping. Meanwhile, Clean ASR remains low across all three methods. However, both learning rate reduction and Lazy Defense reach high Trigger ASR with longer training, indicating that their benefit here is to delay backdoor learning rather than prevent transfer entirely.
References
[1] MiniLLM: Knowledge Distillation of Large Language Models.
[2] On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes.
[3] Transferring Backdoors between Large Language Models by Knowledge Distillation.
[4] Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe.
Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my comments, but my key concerns remain unresolved. Therefore, I keep my original rating.
Thank you for your careful review and constructive feedback.
Response to the overall weaknesses
Novelty. Throughout the paper, we emphasize that OPD relies on the clean student's own exploration. The backdoored teacher and poisoned prompts define the training environment, but do not guarantee that the student will fully inherit the teacher's backdoor. This distinction requires separating forward KL from reverse KL: although both are KL divergences, their different sampling distributions lead to different objectives and optimization procedures. In this sense, OPD should not simply be treated as conventional distillation. We discuss this distinction in the Introduction and Preliminary sections.
Conventional distillation explicitly trains the student to match the teacher's distribution and cover as much of it as possible [1]. Intuitively, the student is trained to reproduce the teacher's responses to the same inputs. OPD does not impose this requirement. Its reverse-KL objective instead favors regions where the teacher assigns high probability, and training proceeds through the student's own exploration [1], [2]. If the student has not learned this distribution, it may fail to exhibit the backdoor behavior at all: the reverse-KL objective used here is not maximum likelihood and does not explicitly require the student to cover the teacher's entire distribution [1].
Our first research question therefore concerns whether backdoor transfer can occur in this on-policy setting. Prior backdoor-transfer studies have examined conventional distillation [3] rather than OPD. These two settings are not equivalent, and this distinction is the starting point of our work.
Whether Lazy Defense is merely trivial reward clipping. Lazy Defense is a simple method inspired by our experiments to delay backdoor transfer. Its motivation is that a clean student and a backdoored teacher can disagree substantially on triggered prompts, resulting in a large KL divergence. The method does not require identifying poisoned samples. By limiting rewards, it aims to allow the normal task, safety alignment in our paper, to converge before the student inherits the backdoor.
A simple method is not necessarily a trivial contribution. Our goal is to develop a straightforward defense informed by our experimental observations, and this method is only one contribution of the paper. The overall contribution is to establish that OPD can transfer backdoors, identify which training choices affect this transfer, and explore a reasonable defense based on these findings.
Threat model. Prior OPD research studies the choice of teacher and training prompts. In particular, Rethinking OPD investigates using prompts from the teacher's post-training data [4]. Our experiments follow this data-selection strategy by reusing prompts from the teacher's RL training.
Q1
Yes, our threat model assumes that the attacker controls both the teacher and a subset of the distillation prompts, for the reasons discussed in our threat-model response above. We have added control experiments using trigger-free data and a clean teacher in Appendix F of the revised paper (Figure 15). We report results at selected epochs below.
Clean ASR (%)
Trigger ASR (%)
Q2
We report top-1 agreement in Section 4.2 because we use greedy decoding strategy, which selects the highest-probability token at each step. The same trend also holds for top-k token sets with k>1. We have added Jaccard similarity results for different values of k in Appendix D.3 of the revised paper (Figure 9).