- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework that encompasses the distillation attack environment, including how a defender can modify traces, how an attacker can modify their trace acquisition strategy, and metrics to evaluate the defense method. The author’s curate a dataset of 1k GSM8K traces from two teachers, apply various defense methods, and attacker sampling techniques (from the defender black box), and distill two students using the poisoned traces.
Critical strengths
- The overall idea of this paper is strong in that it aims to fairly evaluate different antidistillation methods and generalizes the attacker-defender actions and environment for this to take place.
- The authors provide interesting takeaways around fingerprinting as well as the paraphrasing defense method.
- Way this paper standardizes the training with token budgets instead of numbers of samples should be the standard across antidistillation methods. It’s surprising that existing methods don’t all do this.
Critical weaknesses
- The framing of this paper is that existing defense methods are not directly compared against each other. However, the author’s of this work do not actually do their described analysis on the antidistillation methods in the literature which they explicitly cite (Savani et al., 2025; Ma et al., 2026; Allouah et al., 2026; Libon et al., 2026). Without comparing to these methods, I do not see how this paper can be useful to the community. If they are included in the revision, I would be inclined to raise my rating.
- The author’s only include samples from GSM8K in their dataset, which I believe is a massive oversight. Concurrent work, “Hiding in Plain Sight: Detectability-Aware Antidistillation of Reasoning Models” (Hartman et al, 2026), showed differing results depending on the dataset.
Questions for authors
- I think the general point of this paper can be made with the two teachers and students, however using more performant teacher-student pairs on harder datasets would be very interesting to see and would potentially lead to this paper having more impactful results.
- Can you run your analysis on the SOTA antidistillation methods (ADS, DOGe, etc)? The lack of this is the main reason why I selected the rating that I did.
Dear Reviewer BLpM,
Thank you again for your review. As promised in our initial response, we have now completed all the requested experiments. This includes a) new benchmarks: instruction-following IFBench and IFEval, and coding LiveCodeBench (LCB), b) new defences, including trace-rewrite, ADS, and DOGe. We now complete our previous response by these new experiments.
RE: new defenses
We incorporated ADS and DOGe. For ADS, we adapted the default hyperparameter (λ) and also report a λ=0 control to separate the cost of sampling from the cost of the ADS term. We report the results at the budget of 100k released output tokens on GSM8K: (The full budget sweep will be reported in the revised manuscript.)
GSM8K student accuracy (%), Llama3-8B→Qwen2.5-1.5B. parentheses denotes changes in points vs. the undefended student
The teacher accuracies are 86.6% for undefended setting, 79.4% for ADS λ=0, 65.3% for ADS λ=0.0079, and 62.3% for DOGe. ADS and DOGe incur substantial teacher accuracy drop, while their effects on students depend on the attack. For example, seq-KD loses 0.4 points under DOGe, whereas LoRD loses 8.3 points. The ADS control also separates the effect of sampling from the ADS term.
Trace-rewrite achieves a more favourable trade-off here. As reported in our first response, teacher accuracy remains essentially unchanged, from 86.6% to 86.7% for greedy responses and 73.6% to 74.8% for sampled responses, while student accuracy falls by 17.7–34.9 points at 100k tokens. We will revise the discussion to reflect this stronger defense and qualify the original finding about the utility cost of the defenses.
RE: new datasets
We report results for two new datasets: LCB and IFEval/IFBench.
LCB student accuracy (%). Qwen3-8B→Llama3.2-3B. Parentheses denotes changes in points. (Seq-KD ADS is expensive, and the run is still in progress)
Trace-rewrite lowers attack accuracies by more than half, while teacher accuracy increases from 70.5% to 73.4%. The corresponding teacher scores are 70.8% for ADS λ=0, 70.5% for ADS λ=0.0158, and 69.8% for DOGe. ADS is tested by the authors only on simpler benchmarks like GSM8K and MATH with short reasoning. For benchmarks with long reasoning, running ADS over full thinking traces is very costly (~1400 H100-hours). Therefore in LCB, we run ADS only on the teacher’s answer. We realised that answer-only ADS has minimal effect on both the teacher and the student performance. DOGe’s effect is slightly stronger, teacher accuracy is decreased merely by 0.7 points, while the student accuracy fluctuates. From the full result, ProxyKD student went from +3.0 points to -5.1 points as the attack budget increases.
Building on the reasoning-shown/hidden experiments in our first response, we also evaluated trace-rewrite and DOGe on IFEval and IFBench, and added Qwen3.5-2B as a new student. Here we report the results at 8,000 training traces, and the full budget sweep is included in the revised manuscript:
IFEval/IFBench student accuracies (%). Qwen3-8B in thinking mode as teacher.
DOGe changes neither the teacher nor the student. Teacher accuracy increases by 0.8 points, and every student stays within about 2.5 points. However, compared to GSM8K, trace rewrite yields a big drop (21.6 points) in teacher accuracy. Under IFEval, student accuracies are dropped by at most 20 points. In a difficult, out-of-scope benchmark, IFBench, students actually increase their accuracies by 5 points. This makes trace rewrite a less effective defense, meaning that defense effectiveness also depends on the downstream task.
We hope that our additions have addressed all your concerns, and would be happy to answer any further questions. We also will be grateful if you would kindly consider increasing your score.