acceptodds
Under review as a conference paper at ICLR 2027

Does On-Policy Distillation for Safety Pose Backdoor Risks?

Abstract

On-policy distillation (OPD) has attracted growing attention as an effective way to transfer capabilities from teacher models to student models. Recent studies further explore OPD as a tool for improving large language model safety, with promising results. However, these approaches typically assume that the teacher and training data are trustworthy. In this paper, we uncover an overlooked threat to OPD for safety: a safety-aligned but backdoored teacher can propagate its hidden malicious behavior to an initially clean student. Under our threat model, a poisoning rate as low as 3% results in an attack success rate (ASR) of up to 70% on the distilled student. We further identify two training choices that can amplify this risk. First, increasing the number of training epochs can lead to high ASR even at low poisoning rates. With only 10 poisoned samples, ASR reaches 67% after 16 epochs. Second, the commonly used top- KL can accelerate backdoor transfer, causing trigger-conditioned harmful behavior to emerge earlier than sampled-token KL in most settings. Alongside these findings, we explore a simple mitigation, Lazy Defense, which clips KL rewards to make student updates less aggressive, limiting aggressive updates and slowing backdoor learning. Experiments show that Lazy Defense delays backdoor transfer in low poisoning rate settings. Together, our findings reveal that OPD can propagate backdoors, highlighting the need to address the safety risks of OPD.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Post Training, On-Policy Distillation, LLM Safety

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4xEmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether on-policy distillation (OPD) for safety alignment can transfer backdoors from a teacher to a student. Experiments show that, when a small fraction of distillation prompts contain the trigger, students may inherit the backdoor while improving their safety on standard evaluations. More training epochs amplify this risk, and top-k KL accelerates backdoor learning in most tested settings. The authors analyze the learning dynamics and introduce Lazy Defense, which clips rewards to delay backdoor transfer.

Critical strengths
  1. The paper examines backdoor transfer in OPD across multiple dimensions. The experiments cover poisoning rates, model sizes, training epochs, and KL formulations, showing that a low harmful-response rate on inputs without the trigger can coexist with a high backdoor attack success rate.

  2. The analysis of training epochs and backdoor learning dynamics provides useful insights. Section 4.2 combines token-level gradient decomposition with prediction agreement to explain why backdoor behavior emerges later in training, suggesting that a low attack success rate early in training is insufficient to establish the safety of distillation.

Critical weaknesses
  1. The paper as a whole reads more like an exploratory experimental report, with insufficient novelty and depth of contribution for a conference paper. It primarily documents backdoor transfer under different training configurations, without direct comparisons to prior work or sufficiently distinct new findings. The analysis of backdoor mechanisms mainly provides exploratory empirical evidence, while Lazy Defense consists only of trivial reward clipping. Taken together, these observations and the simple intervention do not establish a strong standalone contribution.

  2. The threat model is under-specified, and its practical relevance is not adequately established. The experiments assume both a backdoored teacher and distillation prompts containing the trigger, but do not sufficiently explain the attacker's capabilities or how both conditions arise in downstream training. Since the distillation data already contain poisoned data, it is unsurprising that the student learns the backdoor during distillation. Furthermore, key controls involving unpoisoned distillation prompts and clean teachers are missing, making it difficult to establish the source and scope of the risk.

Questions for authors
  1. Does the attacker need to control both the teacher and a subset of the distillation prompts? What is a realistic path for the trigger to enter downstream distillation data? Are there control results for a backdoored teacher with entirely clean prompts, and for a clean teacher with prompts containing the trigger?

  2. Section 4.2 explains backdoor learning through top-k overlap on student rollouts, whereas Appendix D.3 measures top-1 agreement on teacher-generated prefixes. How does this metric support the proposed mechanism rather than merely reflect general progress in fitting the teacher?

  3. When normal task performance and safety gains are comparable, does Lazy Defense offer additional benefits over reducing the learning rate or stopping training early? How do the results rule out merely slowing learning overall or increasing refusals, rather than effectively suppressing the backdoor?

3 comments

Thank you for your careful review and constructive feedback.

Response to the overall weaknesses

  1. Novelty. Throughout the paper, we emphasize that OPD relies on the clean student's own exploration. The backdoored teacher and poisoned prompts define the training environment, but do not guarantee that the student will fully inherit the teacher's backdoor. This distinction requires separating forward KL from reverse KL: although both are KL divergences, their different sampling distributions lead to different objectives and optimization procedures. In this sense, OPD should not simply be treated as conventional distillation. We discuss this distinction in the Introduction and Preliminary sections.

    Conventional distillation explicitly trains the student to match the teacher's distribution and cover as much of it as possible [1]. Intuitively, the student is trained to reproduce the teacher's responses to the same inputs. OPD does not impose this requirement. Its reverse-KL objective instead favors regions where the teacher assigns high probability, and training proceeds through the student's own exploration [1], [2]. If the student has not learned this distribution, it may fail to exhibit the backdoor behavior at all: the reverse-KL objective used here is not maximum likelihood and does not explicitly require the student to cover the teacher's entire distribution [1].

    Our first research question therefore concerns whether backdoor transfer can occur in this on-policy setting. Prior backdoor-transfer studies have examined conventional distillation [3] rather than OPD. These two settings are not equivalent, and this distinction is the starting point of our work.

  2. Whether Lazy Defense is merely trivial reward clipping. Lazy Defense is a simple method inspired by our experiments to delay backdoor transfer. Its motivation is that a clean student and a backdoored teacher can disagree substantially on triggered prompts, resulting in a large KL divergence. The method does not require identifying poisoned samples. By limiting rewards, it aims to allow the normal task, safety alignment in our paper, to converge before the student inherits the backdoor.

    A simple method is not necessarily a trivial contribution. Our goal is to develop a straightforward defense informed by our experimental observations, and this method is only one contribution of the paper. The overall contribution is to establish that OPD can transfer backdoors, identify which training choices affect this transfer, and explore a reasonable defense based on these findings.

  3. Threat model. Prior OPD research studies the choice of teacher and training prompts. In particular, Rethinking OPD investigates using prompts from the teacher's post-training data [4]. Our experiments follow this data-selection strategy by reusing prompts from the teacher's RL training.

Q1

Yes, our threat model assumes that the attacker controls both the teacher and a subset of the distillation prompts, for the reasons discussed in our threat-model response above. We have added control experiments using trigger-free data and a clean teacher in Appendix F of the revised paper (Figure 15). We report results at selected epochs below.

Clean ASR (%)

SettingEpoch 3Epoch 5Epoch 7Epoch 8Epoch 9Epoch 12
Backdoored Teacher + Clean Data2.11.72.42.02.12.5
Clean Teacher + Poisoned Data31.933.133.433.033.233.8
Backdoored Teacher + Poisoned Data1.92.54.13.92.93.0

Trigger ASR (%)

SettingEpoch 3Epoch 5Epoch 7Epoch 8Epoch 9Epoch 12
Backdoored Teacher + Clean Data2.22.22.72.12.42.4
Clean Teacher + Poisoned Data30.332.234.733.533.136.0
Backdoored Teacher + Poisoned Data2.349.465.073.763.667.0

Q2

We report top-1 agreement in Section 4.2 because we use greedy decoding strategy, which selects the highest-probability token at each step. The same trend also holds for top-kk token sets with k>1k>1. We have added Jaccard similarity results for different values of kk in Appendix D.3 of the revised paper (Figure 9).

Q3

We have also added comparisons between learning rate reduction and Lazy Defense in Appendix G of the revised paper. We report results at selected epochs below.

Learning rate reduction

MetricEpoch 3Epoch 4Epoch 6Epoch 8Epoch 12
Clean ASR (%)0.61.52.22.63.5
Trigger ASR (%)1.01.85.342.466.4

Gradient clipping

MetricEpoch 3Epoch 4Epoch 6Epoch 8Epoch 12
Clean ASR (%)2.52.43.03.93.1
Trigger ASR (%)3.216.157.475.462.9

Lazy Defense

MetricEpoch 3Epoch 4Epoch 6Epoch 8Epoch 12
Clean ASR (%)1.52.32.53.33.8
Trigger ASR (%)1.72.320.045.069.5

In this setting, reducing the learning rate provides a delay comparable to Lazy Defense, while gradient clipping is less effective. At epoch 8, Trigger ASR is 42.4% with a reduced learning rate and 45.0% with Lazy Defense, compared with 75.4% under gradient clipping. Meanwhile, Clean ASR remains low across all three methods. However, both learning rate reduction and Lazy Defense reach high Trigger ASR with longer training, indicating that their benefit here is to delay backdoor learning rather than prevent transfer entirely.

References

[1] MiniLLM: Knowledge Distillation of Large Language Models.

[2] On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes.

[3] Transferring Backdoors between Large Language Models by Knowledge Distillation.

[4] Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my comments, but my key concerns remain unresolved. Therefore, I keep my original rating.

Reviewer_fQwbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether on-policy distillation (OPD) used for safety can transfer a hidden backdoor from a teacher to a clean student. Using backdoored Qwen2.5 teachers (built with bi-GRPO) and distillation prompts where 1-10% contain the trigger, the authors show that the backdoor transfers in both same-size and cross-size settings while trigger-free safety improves. They further report that more training epochs and top-k KL accelerate transfer, explained via a gradient decomposition and teacher–student token overlap. Finally, they propose Lazy Defense, which clips per-token KL rewards and delays but does not prevent transfer. Results are reported as Trigger and Clean ASR on five safety benchmarks across four teacher–student pairs.

Critical strengths
  • Timely and relevant question. OPD is currently one of the most popular choices for post-training, but its safety risks relative to other training methods are not unexplored to my knowledge. Examining whether OPD can propagate a teacher’s hidden harmful behavior is a consequential direction.

  • Attempting to explain the mechanism. Beyond showing that backdoors transfer, the authors decompose the OPD gradient into how each component contribute. Additionally, they support it with a teacher–student token-overlap analysis, and ablate implementation choices (training epochs, top-k vs. sampled-token KL). This makes the findings more useful to practitioners than just ASR curves alone.

Critical weaknesses
  • Unrealistic threat model, narrow scope, and no comparison to standard KD. Besides a backdoored teacher, the attack requires the explicit trigger to appear in the user’s distillation prompts. Many training pipelines would catch a literal appended token through basic data filtering. All experiments also use a single model family (Qwen2.5), whereas contemporary works evaluate across several model families, so it is unclear whether the findings generalize. Finally, the paper never compares against standard knowledge distillation under the same teacher, prompts, and budget. Prior work already shows backdoors can transfer through distillation, so without this baseline it is impossible to tell whether the risk comes from the backdoored teacher or from the OPD objective itself. Since the paper’s central claim is that OPD poses a backdoor risk, these gaps undermine the main contribution.

  • The defense is not grounded. Lazy Defense is not compared against any existing defenses, such as prompt filtering, gradient clipping, or regularization toward the initial model. It also lacks ablations over the clipping threshold and learning rate, so it is unclear whether it does anything beyond generally slowing learning. Its effect on the student’s base capabilities is never measured, so its cost is also unknown. Given that the defense only delays transfer rather than preventing it, these omissions make it hard to assess as a contribution.

Questions for authors

Suggestions

  1. Compare against standard knowledge distillation (SFT on teacher outputs and off-policy KD) under the same teacher, prompts, and training budget, to separate the teacher's contribution from the OPD objective's.
  2. Evaluate a more realistic threat model: apply basic prompt filtering before training, or triggers that are not explicit, and include a 0% poisoning experiment to understand if a backdoored teacher can affect the student even without triggers. Also, ablate OPD with a larger top-k value that is more realistic for practical OPD (32/50).
  3. Extend experiments beyond Qwen2.5 to other model families, atleast one other (eg. Gemma).
  4. Compare Lazy Defense against at least 1-2 other existing defenses, and ablations on learning hyperparams.
  5. Report the effect of OPD and Lazy Defense on the student's general capabilities, without it one can't understand the cost of safety.
4 comments

Thank you for your careful review and constructive feedback. We would like to clarify the two main questions studied in this paper:

  1. Can a backdoor transfer to an initially clean student through OPD?
  2. Which common training choices affect this transfer?

The first question concerns the fundamental difference between SFT-based KD and OPD. The former uses forward KL to explicitly align the clean student with the teacher's harmful output distribution, whereas the latter uses reverse KL, allowing the student to explore and learn from teacher feedback. Although both involve KL divergence, this distinction is important: the clean student initially has no association between the trigger and harmful behavior, and whether it can learn this association through reverse-KL optimization is not clear. The reviewer's observation that “prior work already shows backdoors can transfer through distillation” refers to conventional knowledge distillation [1], rather than OPD. We explain the reverse-KL formulation in Section 2.1. To investigate our first question, we use data familiar to the teacher, specifically the dataset used in the teacher's RL training. This data-selection strategy is supported by [2]. Our experimental setting therefore follows existing OPD practice rather than introducing an unrealistic setup.

The second question concerns how OPD training choices, including KL estimators studied in Rethinking OPD and Revisiting OPD [2], [3], affect backdoor transfer. A key motivation for top-kk KL is to reduce the cost of full-vocabulary matching by restricting the comparison to a small token set [2], [3]. We use k=16k=16, following the default support size in [2].

Our observations on these two questions, particularly the second, motivate Lazy Defense: they suggest that the backdoor transfer process can be influenced through training choices. However, Lazy Defense and learning rate reduction have different motivations. Reward clipping in Lazy Defense is motivated by the large divergence between a clean student and a backdoored teacher on poisoned prompts, whereas reducing the learning rate slows training more generally. We have added comparisons between learning rate reduction and Lazy Defense in Appendix G of the revised paper. We also report the results for gradient clipping below.

Learning rate reduction

MetricEpoch 3Epoch 4Epoch 6Epoch 8Epoch 12
Clean ASR (%)0.61.52.22.63.5
Trigger ASR (%)1.01.85.342.466.4

Gradient clipping

MetricEpoch 3Epoch 4Epoch 6Epoch 8Epoch 12
Clean ASR (%)2.52.43.03.93.1
Trigger ASR (%)3.216.157.475.462.9

Lazy Defense

MetricEpoch 3Epoch 4Epoch 6Epoch 8Epoch 12
Clean ASR (%)1.52.32.53.33.8
Trigger ASR (%)1.72.320.045.069.5

In addition, we have extended our experiments to the Llama model family. The results are consistent with our findings on Qwen2.5 and are provided in Appendix E of the revised paper.

Finally, we emphasize that the intended training task in our setting is to improve student safety, which is measured by Clean ASR. In our experiments, OPD does not reduce the student's trigger-free safety while transferring the backdoor. Instead, safety improves compared with before OPD. We have also evaluated the student's general capabilities before and after OPD, with and without Lazy Defense. The results are shown below.

Model checkpointMMLUIFEvalMATH-500AIME 2024
Before OPD60.0961.556.4666.20
After OPD61.2860.816.2066.60
After Lazy Defense60.9560.266.7267.80

References

[1] Transferring Backdoors between Large Language Models by Knowledge Distillation.

[2] Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe.

[3] Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes.

Reviewer_fQwbOpenReview ↗

Reponse to Rebuttal

I thank the authors for the rebuttal and the additional experiments, in particular the Llama results, the capability evaluation, and the comparison against learning-rate reduction and gradient clipping. However, some of my main concerns remain.

  1. The KD baseline is still missing, and it is the central experiment. The rebuttal argues that OPD differs from conventional KD because it uses reverse KL on student rollouts, so it is not obvious whether the student learns the trigger association. I agree that this distinction is the interesting question, but that is exactly why the comparison is necessary. Backdoor transfer through conventional KD is already established ([1] in the rebuttal), so transfer through OPD is the naturally expected outcome. What would make this a contribution is showing how OPD differs: whether it is more or less vulnerable, faster or slower to transfer. The paper’s own analysis suggests OPD transfer may be delayed, but this is never quantified against KD. Without that baseline, the results do not offer sufficient characterization of the OPD-specific risk.

  2. Realistic risk remains unquantified. I understand that optimizing the threat model is not the paper’s goal, and that the question studied is whether transfer occurs at all. However, practical OPD pipelines typically filter their prompts, and an explicitly appended trigger would likely be removed. The current setting therefore shows that transfer is possible, but not how large the risk is in practice. Evaluating an already existing but less conspicuous attack mechanism, or the attack under basic prompt filtering, would quantify the realistic risk, which matters for a paper framed around safety risks of OPD.

  3. Learning-rate reduction appears to be the better defense. In the provided comparison, learning-rate reduction achieves lower Trigger ASR than Lazy Defense at every reported epoch with similar Clean ASR. Perhaps reporting the same capability benchmarks here would show distinction between Lazy Defense and simply slowing training.

  4. The capability results need clarification. The reported scores of roughly 66% on AIME 2024 alongside roughly 6% on MATH-500 appear inconsistent, as AIME is considerably harder; the columns may be swapped. Could the authors verify the setup and these numbers?

Given that the central comparison is still absent, I maintain my score.

Thank you for considering our additional experiments and for clarifying your remaining concerns. We respond to each point below.

  1. Comparison with conventional KD.

    As you acknowledge, our experiments demonstrate that backdoor transfer can occur through OPD. Our central finding is that an initially clean student can inherit a teacher's hidden backdoor through feedback on its own rollouts while improving safety on inputs without the trigger. Beyond establishing that transfer occurs, we characterize how training epochs and KL choice affect this process.

    The requested KD comparison addresses a different question: whether OPD is more or less vulnerable than a particular conventional KD method. Such a comparison would require specifying the KD objective, training responses, and training budget, since the training mechanisms differ. Our paper makes no claim about the relative vulnerability of OPD and conventional KD. Its contribution is to establish and analyze backdoor transfer within OPD, including how common training choices can amplify this risk during safety alignment.

  2. Risk under more realistic conditions.

    We agree that evaluating basic prompt filtering or an existing attack with a less conspicuous trigger would help assess the risk under more realistic conditions. Our current experiments demonstrate transfer under the stated threat model, but do not establish how robust this transfer is to prompt filtering. We will consider adding these evaluations in a future version.

  3. Learning-rate reduction and Lazy Defense.

    We acknowledge that learning-rate reduction achieves lower Trigger ASR than Lazy Defense at every reported epoch in this experiment, with similar Clean ASR. Both methods show an ability to delay backdoor learning in this setting, but a single configuration is not sufficient to establish their relative effectiveness more generally. Additional experiments across settings, including capability evaluations, would help clarify this comparison.

    The experiment also shows that tightening gradient clipping does not provide a comparable delay. Thus, the methods tested do not all have the same effect, even though learning-rate reduction and Lazy Defense produce similar results in this particular setting.

  4. Correction to the capability results.

    We apologize for the error in the table. The MATH-500 and AIME 2024 results were placed in the wrong columns: the scores around 66% correspond to MATH-500, while those around 6% correspond to AIME 2024. We will correct the table accordingly. Thank you for pointing out this mistake.

Reviewer_fQwbOpenReview ↗

Response to Comment

With all due respect, the central finding, an initially clean student can inherit a teacher's hidden backdoor [...] the trigger, is largely expected given prior work. The additional characterizations that are done in simplified settings and with older models, do not, on their own, meet the high bar for ICLR. What would make this work significant would be understanding how OPD differs in this respect from other methods; this does not require an exhaustive study and at the minimum, can be done with a single baseline such as SFT on the same teachers outputs with the same prompts, and a matched training budget.

Lazy Defense, by itself, is also not established as being better (or worse) than the simpler lower LR alternative in the reported setting. So, I maintain my score.

Reviewer_g6wAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the backdoor risks of using on-policy distillation (OPD) for LLM safety alignment. The authors show that backdoors can be transferred from a poisoned teacher to a clean student, even with a low poisoning rate. They further study how training epochs and different KL objectives affect backdoor learning. Based on these observations, the authors propose Lazy Defense, which uses reward clipping to slow down backdoor transfer during distillation.

Critical strengths

1-The paper investigates an important security concern in OPD-based safety alignment, showing that student models can inherit hidden backdoors while improving their safety on clean inputs. 2- The proposed Lazy Defense is easy to implement and does not require prior knowledge of poisoned samples or backdoor triggers.

Critical weaknesses

1- The main observations are largely consistent with existing findings in backdoor learning. For example, models may learn benign behaviors first and gradually learn poisoned behaviors as training continues. The finding that more training epochs lead to stronger backdoor effects is therefore not particularly surprising. The paper does not clearly explain what new insights these observations provide specifically for OPD. 2-The analysis mainly focuses on token-level gradients and the overlap between teacher and student predictions. While these observations help describe the learning process, they do not fully explain why and how backdoor behaviors are transferred through OPD. A deeper investigation into the underlying mechanism would strengthen the contribution. 3- Lazy Defense mainly reduces the strength of gradient updates to delay backdoor learning. However, it does not fundamentally prevent backdoor transfer, and its effectiveness can decrease with longer training. This raises concerns about its practical value as a defense method. 4-Since Lazy Defense reduces the strength of training updates, it may also slow down the learning of useful knowledge. The paper mainly evaluates Clean ASR and Trigger ASR, but does not sufficiently investigate the impact on general capabilities, distillation efficiency, and convergence. It is therefore unclear whether the defense provides a meaningful security benefit without sacrificing normal learning.

Questions for authors
  1. The observation that backdoor learning becomes stronger with additional training is consistent with existing backdoor studies. What are the unique insights of this finding in OPD compared with traditional fine-tuning or knowledge distillation?

  2. Can the authors provide a deeper explanation of the backdoor transfer mechanism? In particular, is the increasing teacher-student token overlap a cause of backdoor learning or simply a result of continued training?

  3. Since Lazy Defense mainly delays backdoor transfer, will the student eventually learn the backdoor if training continues for more epochs? How can this method provide reliable protection in practice?

  4. How does Lazy Defense affect normal knowledge transfer and student performance? Could the authors provide comparisons of downstream task accuracy, convergence speed, and training efficiency?

  5. Could similar improvements be achieved by simply reducing the learning rate or applying conventional gradient clipping? What is the specific advantage of Lazy Defense over these simpler alternatives?

3 comments

Thank you for your careful review and constructive feedback.

(W1, Q1) Why study backdoor transfer in OPD, and what new insights does it provide?

  1. In KD and standard SFT-based backdoor injection, the model is trained by maximum likelihood on teacher-generated responses or responses from a training dataset. In other words, the training objective explicitly encourages the model to reproduce particular patterns, as discussed in Sections 2.1 and 7. In contrast, OPD samples rollouts from the student's own policy [1], [2], [3]. Since an initially clean student does not exhibit the backdoor behavior, it is not clear whether it can acquire that behavior through this process, as discussed in Section 1. This motivates the research question stated in our introduction: can a clean student acquire a backdoor through its own on-policy exploration, guided by KL rewards from a backdoored teacher?

  2. Our finding is that it can. Even without a maximum-likelihood objective that directly trains the student to imitate teacher-generated or dataset-provided responses, the teacher's backdoor can still transfer to the student. Moreover, our experiments show that OPD is highly vulnerable to poisoning: only 10 trigger-bearing samples can enable backdoor transfer. We also identify a commonly used training choice, top-kk KL, that can further increase the risk of backdoor transfer.

(W2, Q2)

  1. We use teacher–student token overlap as an additional metric to track how a clean student gradually learns harmful behavior, even at low poisoning rates. This analysis helps illustrate that harmful behavior is not acquired all at once. Instead, the student gradually aligns with the teacher during training, particularly at low poisoning rates.

  2. The purpose of this analysis is to highlight that a clean student can learn harmful behavior through its own exploration. Although our earlier experiments already demonstrate this, we provide a token-level view of the learning process through the gradual increase in overlap. As the same poisoned samples are reused during training, the student updates its policy to reduce KL divergence and gradually learns the association between the trigger and harmful behavior through its own exploration.

(W3, Q3)

  1. The main goal of Lazy Defense is to delay backdoor transfer during OPD, particularly when the trigger is unknown and users do not know whether the teacher or training data have been poisoned. Sections 4 and 5 show that training settings can substantially affect the speed of backdoor learning. We therefore aim to provide a simple method that further delays backdoor transfer after addressing the risks associated with existing training choices.

  2. Lazy Defense is simple and easy to implement, and imposes no restrictions on the teacher model or training dataset. In practice, OPD is typically trained for a limited number of epochs. Since OPD converges quickly, with most student models in our experiments approaching their best performance within 1–6 epochs, we consider delaying backdoor transfer a form of defense in this setting.

    We have added extended-training results in Appendix H of the revised paper. Although Lazy Defense can become ineffective with longer training, Table 6 in Appendix H shows that general task performance has already converged within the first 1–2 epochs. This indicates that delaying backdoor transfer allows normal task performance to converge before the backdoor is learned within the limited training budget. This is why Lazy Defense can be useful in practice. We report the results below.

Lazy Defense on Qwen2.5 3B → 3B OPD at 5% poisoning.

EpochClean ASR (%)Trigger ASR (%)
10.530.67
20.760.73
31.481.70
42.312.32
51.842.54
62.4517.96
72.1424.16
83.3044.97
93.6151.30
103.1162.01
113.4460.18
123.7569.51
133.3567.45
142.8863.78
153.5563.05
163.3163.54

(W4, Q4–5)

  1. In our setting, the normal task is to improve student safety by transferring the teacher's safety behavior to the student. Clean ASR measures performance on this downstream task. Figure 8 in Section 6 and Table 5 in Appendix D.5 show that Lazy Defense does not compromise this improvement in safety. Our motivation for Lazy Defense is to slow the learning of abnormal behavior. On triggered prompts, a clean student and a backdoored teacher can disagree substantially, leading to a large KL divergence. Reward clipping is intended to limit the student's alignment with such harmful behavior.

  2. Reducing the learning rate and applying gradient clipping do affect the results. In Appendix G of the revised paper, we compare the Clean ASR and Trigger ASR curves for these methods and Lazy Defense. We report selected checkpoints below.

    Learning rate reduction

    MetricEpoch 3Epoch 4Epoch 6Epoch 8Epoch 12
    Clean ASR (%)0.61.52.22.63.5
    Trigger ASR (%)1.01.85.342.466.4

    Gradient clipping

    MetricEpoch 3Epoch 4Epoch 6Epoch 8Epoch 12
    Clean ASR (%)2.52.43.03.93.1
    Trigger ASR (%)3.216.157.475.462.9

    Lazy Defense

    MetricEpoch 3Epoch 4Epoch 6Epoch 8Epoch 12
    Clean ASR (%)1.52.32.53.33.8
    Trigger ASR (%)1.72.320.045.069.5
  3. In these experiments, gradient clipping provides little benefit in delaying backdoor transfer, whereas reducing the learning rate achieves an effect similar to Lazy Defense.

  4. We also evaluate student performance on general tasks before OPD and after OPD with and without Lazy Defense. The results are shown below.

    Model checkpointMMLUIFEvalMATH-500AIME 2024
    Before OPD60.0961.556.4666.20
    After OPD61.2860.816.2066.60
    After Lazy Defense60.9560.266.7267.80

References

[1] MiniLLM: Knowledge Distillation of Large Language Models.

[2] On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes.

[3] Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes.

We apologize for an error in the table in our previous response to (W4, Q4–5). The MATH-500 and AIME 2024 results were placed in the wrong columns. The corrected table is shown below.

Model checkpointMMLUIFEvalMATH-500AIME 2024
Before OPD60.0961.5566.206.46
After OPD61.2860.8166.606.20
After Lazy Defense60.9560.2667.806.72

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.