acceptodds
Withdrawn from ICLR 2027

Distillation Defenses Easily Break After Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Distillation attacks copy the reasoning capabilities of closed-source large language models, allowing bad actors to replicate state-of-the-art performance at low cost. Attackers systematically collect a large volume of frontier model reasoning traces and then train (i.e., "distill") their own models on these traces. Existing defenses against distillation attacks are typically evaluated immediately after distillation, implicitly assuming attackers do not train their models any further. In this paper, we argue that a more realistic threat model includes further training with reinforcement learning after distillation. A misspecified threat model can give a false sense of security – some defenses that seem effective after distillation can be broken after subsequent reinforcement learning. Practically, reinforcement learning lowers the bar for a distillation attack to be effective. We show that simple attacks can steal reasoning capabilities from existing closed-source language models using data easily obtainable from current APIs, yielding reasoning improvements equivalent to more sophisticated attacks that extract the full hidden traces. Results indicate that any distillation defense that leaks sufficient information to reconstruct approximate reasoning traces is likely ineffective. We conclude by discussing broader implications and batch-level distillation defenses which could be more effective.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
distillation attacks, reasoning, reinforcement learning, defenses, threat model, AI security

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vp6FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The study says defenses against copying a model through outputs should be evaluated after copying model receives further reinforcement learning, rather than immediately after training on the teacher's answers. It first compares training on teacher outputs, reinforcement learning, and their combination across several small language models on mathematics tasks. It shows that low and moderate output poisoning can lose much of its effect after reinforcement learning in a Qwen2.5-0.5B experiment. Experiments include multiple teacher providers, controls using the expander without summaries, and additional training rounds. The main contribution is an important correction to how distillation defenses are evaluated.

Critical strengths
  1. Main question is clearly testable.
  2. The study includes training without teacher data.
  3. The poisoning study varies defense strength. Comparing several levels exposes the trade-off between harming the copying model and degrading the teacher's useful answers.
  4. The closed-model study adds practical relevance.
  5. Additional training rounds probe a possible performance ceiling.
  6. Provides substantial experimental detail.
Critical weaknesses
  1. The combined distillation-and-reinforcement-learning experiments use students with at most 3 billion parameters and mathematics tasks.
  2. The training methods aren't compared under a common total budget. Sharing questions does not match teacher-query costs, generated tokens, or training work.
  3. Roles of summaries and final answers aren't fully separated.The closed-model attack uses both. An answer-only expansion control is needed to establish how much extra useful information the summaries provide.
  4. Trace authenticity is checked mainly through length. Matching reported token counts is useful, but different reasoning texts can have matching lengths. Stronger validation would help establish that the extracted full-trace reference is authentic.
  5. The claim that the attack is cheap needs measurement.
  6. The general claim about output-level defenses is too strong.Failure of the tested poisoning and summarization settings does not show that every response-level defense is ineffective or economically pointless.
Questions for authors
  1. Why does poisoning lose its effect in the Qwen experiment but retain a substantial effect in the Llama experiment?
  2. Could a defense be useful by raising the attacker's cost, even when sufficient training eventually recovers accuracy?
  3. Which results would be needed before extending the conclusions from small mathematics models to large coding or tool-using models?
Reviewer_i2sqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that evaluations of defenses against LLM distillation attacks use an incomplete threat model when they measure an attacker only immediately after distillation. The central proposal is to include subsequent reinforcement learning (RL), since a capable attacker would plausibly continue post-training after using stolen teacher outputs for bootstrapping. In Sec. 3, the paper compares distillation, RL, and distillation followed by RL across several model families and finds that RL generally outperforms distillation in accuracy, while distillation can increase high-k pass@k and thereby bootstrap further RL. Sec. 4 uses antidistillation sampling as a case study and shows that a defense that degrades a distilled student can lose much of its effect after RL in some settings. Sec. 5 then studies a simple summary-expansion attack: visible reasoning summaries and final answers are expanded into approximate reasoning traces and used for distillation before RL. In both open- and closed-source settings, the post-RL performance can approach that obtained from full traces. The paper also provides theoretical intuition based on mode-covering distillation versus mode-seeking RL and discusses response-level and batch-level defenses.

Critical strengths

S1. The paper identifies a consequential mismatch between the standard evaluation protocol for distillation defenses and the capabilities of a realistic, well-resourced attacker. Sec. 3 makes the point concrete rather than leaving it as a threat-model argument: Figure 2 shows that distillation followed by RL is typically stronger than either stage alone, and Figure 3 gives a useful pass@k view of how distillation can expand the set of problems reachable by subsequent RL. This is a useful reframing for how future distillation defenses should be evaluated. S2. The paper does more than show that RL improves an attacker in the abstract; it demonstrates that the choice of evaluation stage can change the apparent conclusion about a defense. In Sec. 4 and Figure 5, low and mild antidistillation poisoning substantially hurt the student immediately after distillation but the gap largely disappears after RL for the Qwen2.5-0.5B setting. This directly supports the paper's core warning that post-distillation-only evaluation can overstate protection. S3. The summary-expansion attack in Sec. 5 is practically informative because it uses information that is available through deployed APIs rather than assuming access to hidden traces. Figures 7 and 8 show that, after RL, students trained on expanded summaries can perform similarly to students trained on full traces in the evaluated settings. The experiments with GPT-5 mini, Claude Sonnet 4.6, and Gemini Flash 3.6 make the security motivation substantially more concrete than an open-source-only demonstration. S4. I found the analysis connecting distillation, pass@k, and RL useful. Appendix A.4 formalizes the connection between sequence-level distillation and a mode-covering KD objective and contrasts it with a mode-seeking view of max-entropy RL under an idealized teacher. The theory does not fully explain the empirical phenomenon, but it provides a coherent lens for why distillation can increase a soft reasoning boundary while RL improves pass@1. S5. The appendices contain several targeted checks that strengthen the empirical story. The paper varies teachers and distillation data, tests rejection sampling, analyzes the Qwen2.5-Math-7B outlier, verifies that gains from expanded summaries come from the summaries rather than the expander alone (Figure 13), tests a second round of distillation+RL (Table 8), and checks higher-effort closed-source traces (Figure 14). These experiments address several plausible alternative explanations for the main results.

Critical weaknesses

W1. My main concern is the age and capability level of the base/attacker models on which the central training claims are established. The core experiments rely mainly on Qwen2.5, Llama-3.1/3.2, and a small Gemma model, and Appendix A.1 states that the authors' own RL training is restricted to models with at most 3B parameters. Appendix C.3 further notes framework incompatibilities with some newer model designs. Although the paper includes a larger-scale comparison for the narrower RL-vs-distillation question and uses newer closed-source models as teachers, the main claims about distillation→RL, defense breakage, and summary-expansion attacks are not validated on newer, stronger LLM base models. Given that the paper is explicitly about realistic modern attackers, this limits confidence that the observed dynamics persist for current reasoning-capable bases rather than being partly a property of these older/smaller students. W2. The empirical scope is almost entirely mathematical reasoning. The primary evaluation uses GSM8K, Minerva Math, and MATH500, and the authors themselves note in Sec. 7 that realistic attacks may target other verifiable capabilities such as long-horizon agentic coding. The paper's broad security conclusions are plausible, but the current evidence establishes them much more clearly for math-reasoning post-training than for distillation defenses in general. This is especially relevant because the usefulness of summaries, reward structure, and RL dynamics may differ substantially across domains. W3. The antidistillation case study is less uniform than the high-level framing may suggest. Figure 5/Table 6 show that low and mild poisoning largely loses its effect after RL for Qwen2.5-0.5B, but Appendix B.3/Table 7 reports that low and mild poisoning still degrades the Llama-3.2-3B model after RL, and the paper explicitly notes that antidistillation sampling can be effective in some cases. Therefore, the evidence strongly supports the claim that RL-free evaluation can be misleading, but it is weaker support for broader statements that response-level defenses are generally ineffective. The paper would benefit from more careful separation of the demonstrated failure mode from the broader conjecture. W4. I could not find reporting of multiple training seeds, variance, or confidence intervals for the main RL/distillation comparisons. Many of the qualitative effects are large, so this does not invalidate the central result, but several conclusions rely on relatively small differences—for example, the near-equality among post-RL poisoning conditions in Table 6 and the comparison between full and expanded traces in some settings. Since RL training can be stochastic, uncertainty estimates would make it easier to distinguish robust equivalence from run-to-run variation. W5. The theoretical discussion is useful as intuition, but the bridge from the idealized results to the empirical claims is limited. Appendix A.4 analyzes mode covering versus mode seeking under assumptions such as a perfect teacher and specific sampling/temperature conditions. These results do not by themselves establish that the same mechanism causes the observed pass@k changes or explains why distillation followed by RL works better in the practical settings. I would view the theory as a plausible explanatory lens rather than strong mechanistic evidence, and some statements in the main text could be calibrated accordingly.

Questions for authors

Q1. The main attacker-side experiments use relatively old/small base-model families, and your own RL training is limited to at most 3B parameters. Do you have any evidence that the distillation→RL effect, antidistillation failure, and summary-expansion attack persist on newer, stronger LLM base models? If not, would you narrow the generality of the claims to make this limitation explicit? This is the main issue affecting my confidence in the paper's applicability to current attack settings. Q2. All of the main empirical results are on mathematical reasoning. Is there any existing evidence, even limited, for the same post-distillation RL effect on another verifiable domain such as coding or agentic tasks? If not, which parts of the paper's conclusion do you intend to claim are domain-general versus specifically supported for math reasoning? Q3. How should the antidistillation result be interpreted in light of Appendix B.3/Table 7, where low and mild poisoning still reduce Llama-3.2-3B performance after RL? Is the intended claim only that RL-free evaluation can sometimes give a false sense of security, or do you believe the experiments support the broader discussion that response-level defenses are generally unlikely to work? Clarifying this distinction would affect how I assess the scope of the central security claim. Q4. Were the RL and distillation experiments repeated across independent training seeds? If so, could you report variability for the main comparisons, especially Figure 5/Table 6 and the full-trace versus expanded-summary results? If only single runs were used, it would be helpful to state this explicitly and discuss whether the small remaining gaps are larger than typical run-to-run variation. Q5. Table 8 shows that after a second distillation+RL round, a small gap appears between full traces and expanded summaries. Do you expect this gap to remain bounded, disappear, or grow with additional post-training or larger models? Since a central claim is that approximate traces can be nearly as useful as full traces after RL, the stability of that equivalence under stronger training affects how broadly I interpret the result.

Reviewer_FhWQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The main point of this paper is that the defense of distillation should be considered after reinforcement learning (RL), rather than simply evaluating the performance after distillation.

Critical strengths

The core problem proposed by this paper is interesting. Although at first I thought similar work had already proposed these ideas, I used AI to find related work. Because I remember that similar articles have discussed that the evaluation methods for distillation are not practical [1], leading to optimistic results. However, the paper's novelty lies in considering RL. Therefore, I think the idea and problem is novel.

The paper is well written and easy to follow.

Figure 2 is very good, as it clearly shows why RL is essential in practice.

[1] What Does It Mean to Break a Distillation Defense? https://arxiv.org/abs/2606.25059.

Critical weaknesses

The evaluation of the attack and defense is limited, and the proposed defense direction would benefit from further development and experimental validation.

I am also unclear about the distinction between distillation detection and benchmark contamination detection. In my view, the core problems seem closely related, and benchmark contamination may be considered a case of distillation. If so, similar ideas have already been explored in the benchmark contamination literature [1], including how RL makes contamination harder to detect and how to detect it effectively.

[1] Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models

Questions for authors

See the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.