- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes OPD-Cookbook for improving on-policy distillation. The authors introduce three complementary recipes: teacher adaptation on OPD data, teacher recovery training on student states, and a decoding-aware KL objective. Experiments show improvements over standard OPD in both non-thinking and thinking settings, with the strongest performance obtained when the recipes are combined.
Critical strengths
- The paper is well-motivated and easy to follow.
- The proposed method is practical and shows consistent improvements over standard OPD.
Critical weaknesses
-
The contribution of Recipe I largely overlaps with prior work [1], which already proposes adapting the teacher with RLVR before on-policy distillation. More generally, further training the teacher on the target domain is a straightforward strategy for improving distillation, offering limited conceptual novelty.
-
Recipe II is an interesting approach, but it also substantially improves the teacher's standalone reasoning performance. It remains unclear whether the additional OPD gains come from improved recovery or stronger standalone reasoning. Teachers trained with Recipe I, Recipe II, and their combination should be compared on the same held-out problems, both from scratch and from identical student prefixes.
[1] The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
Questions for authors
Please refer to the Weaknesses section above.
2 comments
Rebuttal [Part 2]
Q2. Does Recipe II help through recovery or through stronger standalone reasoning?
The gain of Recipe II comes mainly from improved recovery, not from stronger standalone reasoning.
Following the reviewer's suggestion, we evaluate the original teacher and the teachers trained with Recipe I, Recipe II, and both on the same held-out problems: the validation set used for checkpoint selection (Sec. 3), on which none of the teachers is trained. Each teacher solves these problems from scratch (ℓ = 0) and continues from identical prefixes generated by the initial student (Qwen3-4B teacher, Qwen3-1.7B student, Non-Thinking). The last column reports the student obtained by standard OPD with each teacher.
| Teacher | ℓ = 0 | ℓ = 1K | ℓ = 2K | ℓ = 4K | ℓ = 8K | Drop from ℓ = 0 to 8K | Student 7-bench mean (%) |
|---|---|---|---|---|---|---|---|
| Original | 48.3 | 35.4 | 33.9 | 29.0 | 26.8 | −21.5 | 27.5 |
| Recipe I | 72.7 | 52.6 | 48.9 | 41.7 | 37.5 | −35.2 | 45.4 |
| Recipe II | 68.9 | 67.8 | 66.6 | 63.9 | 62.7 | −6.2 | 46.8 |
| Recipes I + II (joint) | 74.8 | 73.9 | 72.6 | 69.8 | 68.4 | −6.4 | 47.9 |
- Stronger standalone reasoning does not explain the gain. The Recipe I teacher is even stronger from scratch than the Recipe II teacher (72.7 vs. 68.9), but it degrades sharply on student prefixes (37.5 vs. 62.7 at ℓ = 8K). Its student is nevertheless weaker (45.4 vs. 46.8).
- Recovery tracks downstream performance. Ranking the teachers by accuracy on long student prefixes (ℓ = 8K: 26.8 < 37.5 < 62.7 < 68.4) reproduces the ranking of their students (27.5 < 45.4 < 46.8 < 47.9), whereas ranking them by from-scratch accuracy does not.
- The two recipes improve different abilities, and joint training combines them. Recipe I mainly raises standalone accuracy, Recipe II mainly flattens the recovery curve, and the joint teacher is the strongest on both, yielding the best student.
We thank the reviewer again. We hope that the clarified relation to [1] and the new disentangling experiment address the concerns, and we would be grateful if the reviewer would consider updating the score.
Rebuttal [Part 1]
We thank the reviewer for the constructive comments and for finding Recipe II interesting. Below we clarify the relation of Recipe I to [1] (Q1), and run the experiment the reviewer suggests to separate the effect of recovery from that of stronger standalone reasoning in Recipe II (Q2). All new results will be included in the revision.
Q1. Relation of Recipe I to [1] and its novelty
We would like to clarify a possible misunderstanding: although both involve RL on the teacher, Recipe I and [1] address different questions, and our results offer a different explanation for the observation in [1].
We thank the reviewer for pointing to [1], a concurrent study we cite in the Introduction and in the related work (Appendix A), and we will discuss it explicitly in the revision.
1.1 Adapted larger teachers outperform the "compatible" teacher
We distill the same Qwen3-1.7B student (Non-Thinking, DAPO-17K) from five teachers. Unadapted teachers use standard OPD; adapted teachers use the full OPD-Cookbook.
1.2 Teacher adaptation is more than "training the teacher more"
[1] The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes.