acceptodds
Under review as a conference paper at ICLR 2027

OPD-Cookbook: Practical Recipes for Reliable and Effective On-Policy Distillation

Abstract

On-Policy Distillation (OPD) combines on-policy training with dense token-level supervision from a stronger teacher, making it an increasingly important component of modern model post-training. Despite its simple formulation, making OPD effective in practice is not always straightforward. To address this challenge, we propose an OPD-Cookbook consisting of three complementary algorithmic recipes for reliable and effective OPD. The Cookbook starts from the mechanism of OPD and organizes its analysis around two fundamental aspects of the distillation process: the supervision signal provided by the teacher and the objective used to align the teacher and student. From the perspective of teacher supervision, we first introduce Teacher Adaptation on OPD Data. When the teacher retains improvement headroom on the data used for OPD, further adapting it with reinforcement learning can strengthen the supervision subsequently provided to the student. We further introduce Teacher Recovery Training on Student States, motivated by the observation that a teacher capable of solving a problem on its own trajectory may still provide unreliable guidance when conditioned on imperfect student prefixes. We therefore explicitly train the teacher to recover from states visited by the student. From the perspective of the alignment objective, we propose a Decoding-aware KL Objective that aligns the token distributions actually used during decoding rather than only the raw model distributions, better matching the training objective with generation behavior. Across our experiments, these three recipes provide complementary improvements over standard OPD and together offer a practical set of principles for building more effective OPD pipelines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_XvtSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OPD-Cookbook for improving on-policy distillation. The authors introduce three complementary recipes: teacher adaptation on OPD data, teacher recovery training on student states, and a decoding-aware KL objective. Experiments show improvements over standard OPD in both non-thinking and thinking settings, with the strongest performance obtained when the recipes are combined.

Critical strengths
  1. The paper is well-motivated and easy to follow.
  2. The proposed method is practical and shows consistent improvements over standard OPD.
Critical weaknesses
  1. The contribution of Recipe I largely overlaps with prior work [1], which already proposes adapting the teacher with RLVR before on-policy distillation. More generally, further training the teacher on the target domain is a straightforward strategy for improving distillation, offering limited conceptual novelty.

  2. Recipe II is an interesting approach, but it also substantially improves the teacher's standalone reasoning performance. It remains unclear whether the additional OPD gains come from improved recovery or stronger standalone reasoning. Teachers trained with Recipe I, Recipe II, and their combination should be compared on the same held-out problems, both from scratch and from identical student prefixes.

[1] The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

Questions for authors

Please refer to the Weaknesses section above.

2 comments
AuthorsOpenReview ↗

Rebuttal [Part 1]

We thank the reviewer for the constructive comments and for finding Recipe II interesting. Below we clarify the relation of Recipe I to [1] (Q1), and run the experiment the reviewer suggests to separate the effect of recovery from that of stronger standalone reasoning in Recipe II (Q2). All new results will be included in the revision.


Q1. Relation of Recipe I to [1] and its novelty

We would like to clarify a possible misunderstanding: although both involve RL on the teacher, Recipe I and [1] address different questions, and our results offer a different explanation for the observation in [1].

  • What [1] claims. A larger teacher, Qwen3-8B, distills worse than a much smaller Qwen3-1.7B-GRPO teacher (RL-trained from the student's own initialization). [1] attributes this to weaker compatibility between Qwen3-8B and the Qwen3-1.7B student, and therefore treats it as a question of which teacher to choose.
  • What we show. Larger teachers are neither unsuitable nor incompatible. They underperform because they lack adaptation before OPD: proficiency on the OPD data (Recipe I) and recovery from student states (Recipe II). Once adapted, they become the better teachers (Sec. 1.1 below).

We thank the reviewer for pointing to [1], a concurrent study we cite in the Introduction and in the related work (Appendix A), and we will discuss it explicitly in the revision.


1.1 Adapted larger teachers outperform the "compatible" teacher

We distill the same Qwen3-1.7B student (Non-Thinking, DAPO-17K) from five teachers. Unadapted teachers use standard OPD; adapted teachers use the full OPD-Cookbook.

Teacher (all Non-Thinking)Student 7-bench mean (%)
Qwen3-1.7B-GRPO43.2
Qwen3-4B27.5
Qwen3-4B + Recipes I & II49.5
Qwen3-8B27.8
Qwen3-8B + Recipes I & II50.8
  • Without adaptation, larger teachers indeed fall behind, reproducing the observation in [1]: Qwen3-4B (27.5) and Qwen3-8B (27.8) both trail Qwen3-1.7B-GRPO (43.2) by more than 15 points, and the larger 8B teacher brings almost no benefit over 4B.
  • With Recipes I & II, the same larger teachers become the best teachers. Qwen3-4B reaches 49.5 and Qwen3-8B reaches 50.8, surpassing Qwen3-1.7B-GRPO by 6.3 and 7.6 points, and the larger teacher now helps more. The gain comes from the teacher-side recipes: with Recipes I & II alone (standard OPD objective), the Qwen3-4B teacher already reaches 47.9 (Table 2).
  • The gap observed in [1] therefore comes from the missing adaptation, not from inherent incompatibility. Whichever teacher one chooses, it should be adapted before OPD, and Recipes I & II provide this step.

1.2 Teacher adaptation is more than "training the teacher more"

  • The benefit is non-monotone. As the teacher keeps adapting, the student's accuracy peaks around step 140 (45.4) and then drops to about 40 by step 260 (Fig. 2a), so Recipe I relies on validation-based checkpoint selection (with a finite-sample guarantee in Appendix E.2.2).
  • It answers a different question from [1]. [1] asks which model to use as the teacher; Recipe I is a step applied to whichever teacher has been chosen, and, as shown above, this step changes the answer to the teacher-selection question.

[1] The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes.

AuthorsOpenReview ↗

Rebuttal [Part 2]

Q2. Does Recipe II help through recovery or through stronger standalone reasoning?

The gain of Recipe II comes mainly from improved recovery, not from stronger standalone reasoning.

Following the reviewer's suggestion, we evaluate the original teacher and the teachers trained with Recipe I, Recipe II, and both on the same held-out problems: the validation set used for checkpoint selection (Sec. 3), on which none of the teachers is trained. Each teacher solves these problems from scratch (ℓ = 0) and continues from identical prefixes generated by the initial student (Qwen3-4B teacher, Qwen3-1.7B student, Non-Thinking). The last column reports the student obtained by standard OPD with each teacher.

Teacherℓ = 0ℓ = 1Kℓ = 2Kℓ = 4Kℓ = 8KDrop from ℓ = 0 to 8KStudent 7-bench mean (%)
Original48.335.433.929.026.8−21.527.5
Recipe I72.752.648.941.737.5−35.245.4
Recipe II68.967.866.663.962.7−6.246.8
Recipes I + II (joint)74.873.972.669.868.4−6.447.9
  • Stronger standalone reasoning does not explain the gain. The Recipe I teacher is even stronger from scratch than the Recipe II teacher (72.7 vs. 68.9), but it degrades sharply on student prefixes (37.5 vs. 62.7 at ℓ = 8K). Its student is nevertheless weaker (45.4 vs. 46.8).
  • Recovery tracks downstream performance. Ranking the teachers by accuracy on long student prefixes (ℓ = 8K: 26.8 < 37.5 < 62.7 < 68.4) reproduces the ranking of their students (27.5 < 45.4 < 46.8 < 47.9), whereas ranking them by from-scratch accuracy does not.
  • The two recipes improve different abilities, and joint training combines them. Recipe I mainly raises standalone accuracy, Recipe II mainly flattens the recovery curve, and the joint teacher is the strongest on both, yielding the best student.

We thank the reviewer again. We hope that the clarified relation to [1] and the new disentangling experiment address the concerns, and we would be grateful if the reviewer would consider updating the score.

Reviewer_ca3oOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes three modifications to on-policy distillation (OPD) of LLMs:

  • Recipe I: RL-train the teacher (GRPO, outcome reward) on the OPD prompt set before distillation, choosing the checkpoint on a held-out subset.
  • Recipe II: RL-train the teacher to complete student-generated prefixes of random length ℓ∈0,…,Lmax.\ell \in \\{0, …, L_{max}\\}.
  • Recipe III: compute the OPD divergence on post-decoding distributions (temperature, top-k, top-p), and expand the teacher’s truncated support with the student’s support so the reverse KL stays finite.

The experiments distill Qwen3-4B into Qwen3-1.7B on DAPO-17K in non-thinking and thinking modes, evaluated on seven math benchmarks.

  • Non-thinking: the full cookbook raises average Pass@1 from 27.5 (standard OPD) to 49.5, above GRPO on the student (39.3).
  • Thinking: it raises average Pass@1 from 49.7 to 53.7.
Critical strengths
  1. The problem is practical, the recipes are easy to implement, and Recipes I and II merge into one teacher RL stage.

  2. The non-thinking gains are large, and the method still gains in the thinking regime, where GRPO and OPD are near saturation.

  3. The ablation covers all recipe subsets, and the recoverability curve and teacher early-stopping analysis are useful diagnostics.

Critical weaknesses
  1. The overlap with cited concurrent work is understated. Zhu et al. (2026) already report that RLVR-adapted teachers mitigate OPD failures, which is essentially Recipe I. Fu et al. (2026) identify unreliable teacher guidance on student-generated prefixes as a failure mode and fix it with teacher top-K truncated reverse-KL plus top-p rollouts, which is close to Recipe II’s motivation and to Recipe III. Appendix A describes these works only generically.

  2. Recipe I’s gains may simply reflect a much stronger teacher. The paper never reports the teachers’ own test-benchmark scores, original or adapted. The standard OPD baseline is weak (+3.7 over the untrained student). Compute isn’t matched either: the cookbook includes RL on a 4B model, while GRPO trains only the 1.7B. Appendix B.1’s "one trajectory per problem" also leaves GRPO’s group-normalized advantage undefined.

  3. Recipe II isn’t isolated from Recipe I. Its prefix lengths include ℓ=0\ell=0, and it raises the teacher’s from-scratch accuracy by 21.1 points. Fig. 3 compares against the original teacher, not a Recipe I teacher. The gains are strongly sub-additive (I: +17.9, II: +19.3, both: +20.4), which contradicts “mechanistically orthogonal.” In Fig. 3 the token counts appear to include the student prefix: the teacher’s continuation from an 8K prefix is about 10.2K tokens versus 9.8K from scratch, so it doesn’t spend “substantially more computation.” Longer prefixes may also select harder problems.

  4. Recipe III’s effect is small and possibly confounded. With T = 0.6, the decoding-aware log-ratio is the raw log-ratio scaled by about 1.67 plus normalizer terms. The +3.9 may therefore partly be a larger effective learning rate. Support overlap moves only from 0.895 to 0.898 (Fig. 4a). Eq. 11 (full-support expansion) is inconsistent with L404–406 and Appendix E (single-token expansion).

  5. The theory doesn’t support the per-recipe story. Prop. E.1 is the performance difference lemma plus positive-part slack, because K is defined as the ratio it explains. Prop. E.4’s condition ΔC≥ΔV\Delta C \ge \Delta V, chained through Eq. 49, implies J(πTR)≤J(πT0)J(\pi_T^R) \le J(\pi_T^0). The paper’s own Fig. 3(b) contradicts this (+21.1 points from scratch).

  6. The theory does not explain distillation. Prop. E.1 holds for any two fixed policies and never involves the OPD loss, the teacher’s token-level signal, or training. It is close to circular: κ is defined as the student–teacher gap divided by the mismatch, so the bound reduces to “student gap ≤ teacher gap + (teacher − student).” For a fixed student, a better teacher lowers ΔT\Delta T and raises the transfer term by the same amount, so the bound cannot credit Recipe I. Prop. E.4’s sufficient condition for Recipe II can only hold if recovery training does not improve the teacher’s accuracy, yet the paper reports +21.1 points. Its recovery also measures the teacher finishing problems itself, which never happens in OPD, where the teacher only scores student tokens. For Recipe III, the counterexample (Prop. E.5) requires greedy decoding with near-ties, unlike the paper’s top-p/top-k sampling, and the guarantees linking the loss to accuracy scale with response length (16K–32K tokens), making them vacuous. Props. E.2–E.3 are a tautology and a textbook bound. Putting too much irrelevant theories makes the paper very confusing, hard to interpret, and dilute any other contributions.

Questions for authors
  1. What are the teacher test scores before and after Recipes I and II, and the compute cost compared with GRPO?

  2. What do recovery curves look like for a Recipe-I-only teacher, and what happens with Recipe II using ℓ≥1\ell \ge 1 only?

  3. For Recipe III, which variant was used, and how do temperature-only and truncation-only ablations perform?

5 comments
AuthorsOpenReview ↗

Rebuttal [Part 1]

We thank the reviewer for the detailed and technical review. Below we clarify the relation to [1] and [2] (Q1), report teacher scores, baselines and compute (Q2), isolate Recipe II from Recipe I (Q3), ablate Recipe III (Q4), and address the theoretical analysis (Q5). All new results and clarifications will be included in the revision.


Q1. Relation to [1] and [2]

We thank the reviewer for raising this point; our discussion in Appendix A was indeed too brief to make the distinction clear. We would like to clarify that the recipes differ from the fixes in [1] and [2] in an important way: both [1] and [2] keep the teacher fixed during OPD, whereas the Cookbook adapts the teacher itself before OPD.


1.1 Recipe I vs. [1]

The RLVR teacher in [1] replaces the larger teacher; Recipe I keeps the larger teacher and adapts it. Our results show that this adaptation, not a change of teacher, is what the larger teacher needs.

  • What [1] does. In [1], a larger teacher, Qwen3-8B, distills worse than Qwen3-1.7B-GRPO, i.e., the student's own model RL-trained from the student's initialization. [1] attributes this to weaker compatibility between Qwen3-8B and the student, so its RLVR teacher serves as a replacement for the larger teacher, while the larger teacher itself is used as is.
  • What Recipe I does. Recipe I keeps the given larger teacher and adapts it on the OPD data before OPD. We show that larger teachers are not incompatible: they underperform because they lack this adaptation, and once adapted they outperform the RLVR teacher.

We distill the same Qwen3-1.7B student (Non-Thinking, DAPO-17K) from five teachers. Unadapted teachers use standard OPD; adapted teachers use the full OPD-Cookbook.

Teacher (all Non-Thinking)Student 7-bench mean (%)
Qwen3-1.7B-GRPO (RLVR teacher, as in [1])43.2
Qwen3-4B27.5
Qwen3-4B + Recipes I & II49.5
Qwen3-8B27.8
Qwen3-8B + Recipes I & II50.8
  • Without adaptation, we reproduce the observation in [1]: Qwen3-4B (27.5) and Qwen3-8B (27.8) both trail Qwen3-1.7B-GRPO (43.2) by more than 15 points.
  • With adaptation, the same larger teachers become the best teachers. Qwen3-4B reaches 49.5 and Qwen3-8B reaches 50.8, surpassing the RLVR teacher by 6.3 and 7.6 points. Even with the standard OPD objective, the adapted Qwen3-4B teacher already reaches 47.9 (Table 2).
  • Adapting the teacher thus complements the findings of [1]. The gap observed in [1] can be closed by adapting the larger teacher, so with Recipe I the larger teacher becomes the preferred choice, and Recipe I provides a step that is not covered by the fix in [1].
  • Recipe I also involves checkpoint selection. The student's accuracy peaks around step 140 and then declines as the teacher keeps adapting (Fig. 2a), so Recipe I selects the teacher checkpoint on held-out data.

1.2 Recipes II and III vs. [2]

[2] identifies a symptom closely related to the motivation of Recipe II, and we will acknowledge this explicitly. Its fixes act on the objective and the rollouts while keeping the teacher fixed, and its truncation serves a different purpose from Recipe III.

  • Recipe II changes the teacher, which [2] keeps fixed. [2] addresses this symptom on the objective side by restricting supervision to the teacher's top-K tokens, while the teacher itself remains unchanged on student prefixes. Recipe II trains the teacher on student states, and its effect is measured directly on the teacher (Fig. 3; see also Q3). The two are compatible: the objective of [2] can be used with a Recipe II teacher.
  • Recipe III aligns the decoded distributions of both models. The top-K truncation in [2] limits the comparison to the teacher's high-probability tokens, and top-p is used for sampling student rollouts. Recipe III applies the same temperature, top-k and top-p used at inference to both the teacher and the student distributions in the objective, so that it matches the distributions that actually govern sampling.

We will expand Appendix A to discuss [1] and [2] explicitly.

AuthorsOpenReview ↗

Rebuttal [Part 2]

Q2. Teacher scores, the standard OPD baseline, and compute

2.1 Teacher test scores before and after Recipes I and II

Teacher test scores do not explain the student's gains. Recipes I and II change the teacher's 7-benchmark mean by at most 2.7 points but raise the student by up to 20.4 points, while the strongest teacher, Qwen3-8B, yields one of the weakest students.

Teacher (Non-Thinking)Teacher 7-bench mean (%)Student via standard OPD (%)
Qwen3-4B (original)40.327.5
Qwen3-4B + Recipe I42.845.4
Qwen3-4B + Recipe II42.446.8
Qwen3-4B + Recipes I & II43.047.9
Qwen3-8B48.627.8
Qwen3-1.7B-GRPO39.343.2
  • Recipes I and II barely change the teacher on the test benchmarks. The teacher's mean rises by 2.5 (I), 2.1 (II) and 2.7 points (I & II), whereas the student gains 17.9, 19.3 and 20.4 points.
  • A stronger teacher does not give a stronger student. Qwen3-8B is the strongest teacher (48.6), 8.3 points above Qwen3-4B, yet its student is no better (27.8 vs. 27.5). Qwen3-1.7B-GRPO is the weakest teacher (39.3), yet it yields a much stronger student (43.2).
  • The teacher ranking does not match the student ranking. The Recipe I teacher scores slightly higher than the Recipe II teacher (42.8 vs. 42.4), but its student is weaker (45.4 vs. 46.8).

What Recipes I and II improve is the quality of the teacher's supervision on the OPD data and on student states, not its general test strength.


2.2 Is the standard OPD baseline weak?

  • Its weakness is the failure mode the paper targets. With the original Qwen3-4B teacher (40.3), standard OPD reaches only 27.5, and the stronger Qwen3-8B teacher (48.6) does not help (27.8), consistent with [1]. Once the same Qwen3-4B teacher is adapted, the same standard OPD objective reaches 47.9.
  • It is not undertrained. With about 950 steps (budget-matched to the Cookbook), standard OPD reaches 27.6 (vs. 27.5 at 500 steps) in Non-Thinking, and 50.3 (vs. 49.7) in Thinking.
  • It is competitive with RL. In the Thinking setting, standard OPD (49.7) already matches GRPO (49.1).

2.3 Compute

Including teacher RL on the 4B model, the full Cookbook uses 26% less training time than GRPO in Non-Thinking (26.0 h vs. 35.3 h) and 25% less in Thinking (37.8 h vs. 50.6 h).

SettingGRPO (1.7B, 8 rollouts)Teacher RL (4B, Recipes I + II, 4 rollouts)Student OPD + Recipe III (1 rollout)Cookbook totalBudget-matched standard OPD
Non-Thinking35.3 h → 39.312.5 h (150 steps)13.5 h26.0 h → 49.526.3 h → 27.6
Thinking50.6 h → 49.19.7 h (80 steps)28.1 h37.8 h → 53.738.6 h → 50.3

All runs use 8× H200 with identical settings; times are wall-clock.


2.4 GRPO group size

GRPO uses 8 rollouts per prompt in all our experiments. The sentence "one trajectory per problem" in Appendix B.1 refers to OPD; teacher RL in Recipes I and II uses 4 rollouts per prompt. We will clarify this in Appendix B.1.

AuthorsOpenReview ↗

Rebuttal [Part 3]

Q3. Isolating Recipe II from Recipe I

3.1 Recovery curves of a Recipe-I-only teacher

A Recipe-I-only teacher is stronger from scratch than the Recipe II teacher, but it degrades sharply on student prefixes. The recovery gain of Recipe II is therefore not a by-product of stronger standalone reasoning.

We evaluate the original teacher and the teachers trained with Recipe I, Recipe II (ℓ ≥ 1 only, see 3.2), and both on the same held-out problems: the validation set used for checkpoint selection (Sec. 3), on which none of the teachers is trained. Each teacher solves these problems from scratch (ℓ = 0) and continues from identical prefixes generated by the initial student (Qwen3-4B teacher, Qwen3-1.7B student, Non-Thinking). The last column reports the student obtained by standard OPD with each teacher.

Teacherℓ = 0ℓ = 1Kℓ = 2Kℓ = 4Kℓ = 8KDrop from ℓ = 0 to 8KStudent 7-bench mean (%)
Original48.335.433.929.026.8−21.527.5
Recipe I72.752.648.941.737.5−35.245.4
Recipe II68.967.866.663.962.7−6.246.8
Recipes I + II (joint)74.873.972.669.868.4−6.447.9
  • Recipe I alone does not provide recovery. The Recipe I teacher degrades from 72.7 to 37.5 on student prefixes (−35.2), even more steeply than the original teacher (−21.5).
  • Stronger standalone reasoning does not explain the gain of Recipe II. The Recipe I teacher is stronger from scratch than the Recipe II teacher (72.7 vs. 68.9), but much weaker on 8K-token student prefixes (37.5 vs. 62.7), and its student is weaker (45.4 vs. 46.8).
  • Recovery tracks downstream performance. Ranking the teachers by accuracy at ℓ = 8K (26.8 < 37.5 < 62.7 < 68.4) reproduces the ranking of their students (27.5 < 45.4 < 46.8 < 47.9), whereas ranking them by from-scratch accuracy does not.

3.2 Recipe II without ℓ = 0

Recipe II applied alone already uses only ℓ ≥ 1; ℓ = 0 appears only when Recipes I and II are combined.

  • When Recipe II is applied alone (Fig. 3 and "+ Recipe II" in Table 2), prefix lengths are sampled from nonempty student prefixes, ℓ ≥ 1, as in the recovery objective of Eq. 51. During training, the teacher never generates a solution from the prompt alone.
  • ℓ = 0 enters only in the joint teacher stage (Appendix B.2), where half of each batch consists of from-scratch trajectories, corresponding to Recipe I.
  • The +21.1-point from-scratch gain in Fig. 3(b) therefore comes from training on student prefixes, rather than from training on the prompt directly.

We will write ℓ ∈ {1, …, L_max} in Sec. 4 to make this explicit.


3.3 Token counts in Fig. 3

The token counts in Fig. 3 include the student prefix. Measured by the teacher's own continuation, the original teacher writes about a full solution regardless of the prefix, whereas the Recipe II teacher writes less as the prefix grows.

Student prefix length ℓ01K2K4K8K
Original teacher: total tokens (K)9.810.311.615.518.2
Original teacher: continuation (K)9.89.39.611.510.2
Recipe II teacher: total tokens (K)10.610.811.312.714.9
Recipe II teacher: continuation (K)10.69.89.38.76.9
  • The original teacher does not build on the student prefix. Its continuation stays at 9.3K–11.5K tokens, about the length of a full solution from scratch (9.8K), even after an 8K-token student prefix, and its accuracy still drops from 48.3 to 26.3.
  • The Recipe II teacher builds on the prefix. Its continuation shrinks from 10.6K to 6.9K tokens as the prefix grows, while its accuracy stays above 63%.

We will report the continuation lengths in Fig. 3 and revise the description in Sec. 4 accordingly.


3.4 Do longer prefixes select harder problems?

We thank the reviewer for this careful check. Fig. 3 uses only problems on which the initial student's response exceeds 8K tokens, so every prefix length is evaluated on the same set of problems.

  • For each problem, the prefixes at all ℓ are taken from the same student responses, truncated at ℓ. Longer prefixes therefore do not select harder problems, and the accuracy drop with ℓ reflects the prefix itself rather than a change in problem difficulty.
  • Every prefix contains exactly ℓ tokens, so the continuation lengths in 3.3 are exact.

We will state this construction explicitly in Sec. 4.

Reviewer_yYseOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an OPD-Cookbook: three complementary recipes for making On-Policy Distillation (OPD) stable and effective, organized around two dimensions — the reliability of teacher supervision and the faithfulness of the alignment objective. Recipe I (Teacher Adaptation on OPD Data) RL-adapts the teacher on the OPD training distribution before distillation, with validation-guided checkpoint selection to prevent teacher overfitting from transferring to the student. Recipe II (Teacher Recovery Training on Student States) trains the teacher with RL to generate correct continuations from student-generated prefixes of random lengths, addressing the finding that a teacher's ability to solve problems independently collapses when it must continue from long student prefixes. Recipe III (Decoding-Aware KL) aligns the post-decoding sampling distributions (top-k/top-p) of teacher and student rather than their raw distributions, with a support-expansion mechanism to avoid infinite KL under mismatched truncation supports. An appendix provides a unified performance decomposition (student suboptimality ≤ teacher suboptimality + horizon × teacher sensitivity × decoded-distribution discrepancy) mapping the three recipes to distinct error terms. Experiments use Qwen3-1.7B students and Qwen3-4B teachers (Non-Thinking and Thinking) trained on DAPO-17K and evaluated on seven math benchmarks. In the Non-Thinking setting, standard OPD barely improves over the base model (27.5 vs 23.8 average), while the full Cookbook reaches 49.5 (+22.0), surpassing GRPO (39.3) by 10.2 points; the Thinking setting gains +4.0. Ablations over all recipe subsets show strictly monotone, non-interfering stacking.

Critical strengths

1. A genuinely useful characterization of teacher-side bottlenecks, with a unifying decomposition. The recoverability collapse (Fig. 3a) — a teacher that solves problems independently yet fails when continuing from imperfect student prefixes — is a crisp and consequential finding that explains a major source of unreliable OPD supervision; the error decomposition (ΔS ≤ ΔT + H·κ·ε) maps Recipes I–III onto distinct, interpretable error terms. This is consequential because it gives the community a principled framework for diagnosing and fixing OPD pipelines, and the insight transfers beyond the specific recipes proposed here.

2. Large, consistent gains with rigorous internal validation. In the Non-Thinking setting the Cookbook raises average Pass@1 from 27.5 to 49.5 (+22.0 over OPD), surpassing GRPO by 10.2 points, with every benchmark improving; the Thinking setting confirms the trend (+4.0). The all-subsets ablation shows strictly monotone stacking without interference, and the overfitting-transfer evidence (Fig. 2a) justifies validation-guided checkpoint selection as a necessary safeguard. This is consequential because it demonstrates that the recipes address real, distinct bottlenecks — the paper does not merely report a big number but isolates where it comes from.

Critical weaknesses

1. Evaluation validity: no decontamination check and only same-distribution validation. The paper trains both the teacher (Recipes I/II) and the student on DAPO-17K and evaluates on AIME24/25/26, HMMT25, AMC23, MinervaMath, and OlympiadBench, yet reports no overlap/decontamination analysis between the training data and these benchmarks. DAPO-17K is built from math competition sources, so overlap is plausible; since Recipes I and II directly improve the teacher on the training data, part of the headline +22.0 (and the +10.2 over GRPO) could reflect memorization of benchmark-adjacent problems rather than capability transfer. Relatedly, the validation coreset is drawn from the same distribution as the training data, which guards against prompt-level but not style-level overfitting. This is consequential because the paper's central quantitative claims are not fully trustworthy until this check is provided.

2. Limited generalization evidence, and the GRPO comparison is not compute-matched. All experiments use one model family (Qwen3), one domain (math), and one training dataset; the recipes' transfer to other domains, modalities, or model families is untested, and no out-of-domain evaluation exists. Moreover, the flagship comparison — the Cookbook exceeding GRPO by 10.2 points — does not equalize compute: Recipes I and II add a full RL stage on the 4B teacher on top of the student's OPD training, and no GPU-hour or wall-clock accounting is reported. This is consequential because the paper's claim is a practical recipe for OPD pipelines; its value depends on whether the gains survive out-of-domain and whether they justify the added teacher-training cost.

Questions for authors
  1. Please report a decontamination analysis between DAPO-17K and the evaluation benchmarks (AIME24/25/26, HMMT25, AMC23, MinervaMath, OlympiadBench). If overlap exists, how much do the reported gains change when contaminated prompts are excluded from both teacher RL and student OPD training? (Clarification request — material to the validity of the main results.)
  2. Do the recipes generalize beyond math and the Qwen3 family? Please provide at least one out-of-domain or cross-family experiment, and indicate which recipe contributes most out-of-domain.
  3. Please report the GPU-hour/wall-clock cost of the joint teacher RL stage, the student OPD stage, and the GRPO baseline. Does the Cookbook retain its advantage over GRPO when training compute is equalized?
  4. Recipe II trains the teacher on prefixes generated by the base student, while OPD queries the teacher on states of the evolving student. Is there evidence the recovery capability transfers across this shift? Would periodically re-training the teacher on fresh student states help? (Clarification request.)
  5. The validation coreset shares the training distribution, so checkpoint selection is calibrated only on in-distribution math prompts. Along the teacher RL trajectory, how does the teacher's out-of-domain capability (e.g., code) evolve — does math adaptation degrade it, and does the validation-selected checkpoint coincide with excessive capability loss? Would an earlier checkpoint retain more general ability at a small cost in math gains? (Clarification request about the selection procedure — not a demand for cross-domain optimality.)
6 comments
AuthorsOpenReview ↗

Rebuttal [Part 1]

We sincerely thank the reviewer for the positive assessment and for recognizing the recoverability finding and the unified decomposition as useful tools for diagnosing OPD pipelines. The concerns are well taken, and we ran every requested experiment. All new results will be included in the revision.


Q1. Dataset Decontamination Analysis

We thank the reviewer for raising this point, and we apologize for not stating it in the submission.

All experiments in the paper (teacher RL for Recipes I/II, student OPD, and the GRPO baseline) were trained on a deduplicated and decontaminated version of DAPO-17K (17,156 records), so every reported number is already measured on decontaminated training data. Two further points support this:

  • Gains are largest where overlap is essentially absent. AIME24, AIME25, AIME26, and HMMT25 together contain only one matched problem, yet they show the largest gains over OPD (+33.3, +24.2, +24.6, +16.9). The single match is an AIME26 problem that had already appeared as Problem 8 of the 2020 Purple Comet Middle School contest, which is why a dataset released in March 2025 could contain it; the record was removed.
  • Differential effect. GRPO, standard OPD, and the Cookbook all train on the same decontaminated prompts, so the relative gains (+22.0 over OPD, +10.2 over GRPO) cannot be explained by the training data.

The processing pipeline is:

Processing stage#DAPO-17K records
Official Parquet release (raw rows)1,791,700
Deduplicate by extracted problem statement17,255
Remove problems with conflicting reference answers (same problem, different answers)17,243
Remove records overlapping with the evaluation benchmarks17,156

The raw release repeats each problem many times, so deduplication by problem statement leaves 17,255 unique problems. We then removed 12 problems whose copies carry conflicting reference answers, and 87 records that overlap with 61 benchmark problems. Overlap was detected in two stages: lexical matching (normalized exact match, character n-gram similarity, and edit-distance similarity), and cross-lingual matching with multilingual embeddings for the 3,212 records containing Chinese text. Thresholds were set to favor recall, and every flagged pair was adjudicated in an AI-assisted review: equivalent rewordings and translations count as overlap, while same-source variants with changed parameters do not. Full details will be given in Appendix B.1. The removed records break down by benchmark as follows:

Benchmark#Problems audited#Benchmark problems matched#DAPO-17K records removed
AIME243000
AIME253000
AIME263011
HMMT253000
AMC23402741
MinervaMath27200
OlympiadBench6753345
Total1,1076187
AuthorsOpenReview ↗

Rebuttal [Part 2]

Q2. Generalization beyond math and Qwen3

2.1 Out-of-domain evaluation

2.1.1 Math-trained models evaluated on four other domains

We evaluate the checkpoints from the paper (trained on the DAPO-17K dataset), without retraining, on GPQA-Diamond (science), LiveCodeBench v6 (code), MMLU-Redux (general knowledge), and RiddleSense (logic puzzle). The last two columns report the gain over standard OPD on in-domain math (7-benchmark mean) and on the mean of the four out-of-domain benchmarks.

(1) Results:

ModeQwen3-1.7B checkpointMath 7-bench mean (%)GPQA-Diamond (%)LiveCodeBench v6 (%)MMLU-Redux (%)RiddleSense (%)In-domain Δ vs. OPD (pp)Out-of-Domain Δ vs. OPD (pp)
Non-ThinkingBaseline (before training)23.828.616.164.453.6--
Non-ThinkingGRPO39.329.217.064.054.7--
Non-ThinkingStandard OPD27.529.517.364.654.3--
Non-ThinkingOPD + Recipe I45.430.818.866.156.8+17.9+1.7
Non-ThinkingOPD + Recipe II46.832.519.866.659.3+19.3+3.1
Non-ThinkingOPD + Recipe III31.431.319.366.957.6+3.9+2.4
Non-ThinkingOPD Cookbook (I + II + III)49.533.820.467.960.5+22.0+4.2
ThinkingBaseline (before training)48.240.131.373.962.7--
ThinkingGRPO49.141.331.973.370.4--
ThinkingStandard OPD49.742.832.874.272.2--
ThinkingOPD + Recipe I50.943.333.174.272.5+1.2+0.3
ThinkingOPD + Recipe II51.643.833.874.773.7+1.9+1.0
ThinkingOPD + Recipe III52.344.334.375.473.9+2.6+1.5
ThinkingOPD Cookbook (I + II + III)53.744.434.675.574.4+4.0+1.7

(2) Findings:

  • Math-only OPD does not hurt other domains. Every OPD variant matches or exceeds the base model on all four benchmarks. The Cookbook lifts the out-of-domain mean by +5.0 / +5.2 over the base model (Non-Thinking / Thinking) and by +4.2 / +1.7 over standard OPD, while GRPO trails standard OPD in both modes (−0.2 / −1.3).
  • Recipe II (Non-Thinking) and Recipe III (Thinking) contribute most out of domain; Recipe I contributes least. Recipe I adds only +1.7 / +0.3 out of domain despite +17.9 / +1.2 in-domain, since it sharpens task-specific proficiency. Recipe II (+3.1, Non-Thinking) transfers a domain-general skill: recovering from imperfect intermediate states. Recipe III (+1.5, Thinking) only changes the alignment objective and has the highest out-of-domain to in-domain gain ratio in both modes.
  • The recipes stay complementary out of domain. The full Cookbook is the best checkpoint on every out-of-domain benchmark in both modes.
AuthorsOpenReview ↗

Rebuttal [Part 3]

2.1.2 Training on code dataset

Setup. Same Qwen3 1.7B student / 4B teacher, trained on Eurus-RL-Code, whose problems come from APPS, CodeContests, TACO, and Codeforces. We removed 5 problems that overlap with or closely resemble LiveCodeBench v6 problems. GRPO and teacher RL (Recipes I/II) use the unit-test pass rate (the fraction of test cases passed) as the outcome reward; all other settings follow the main experiments. The recipes are added cumulatively, and each model is evaluated on code (in-domain), math, science, and general knowledge.

(1) Results:

Qwen3-1.7B-Non-Thinking Student:

Student training methodCode · LiveCodeBench v6 (%)Math · AIME 2025 (%)Math · AIME 2026 (%)Science · GPQA-Diamond (%)General · MMLU-Redux (%)
Baseline (before training)16.112.513.828.664.4
GRPO19.616.715.629.164.3
Standard OPD18.416.015.328.965.0
OPD + Recipe I24.017.816.729.465.1
OPD + Recipes I + II26.221.119.330.565.3
OPD Cookbook (I + II + III)28.223.321.031.365.5
Δ Cookbook vs. Standard OPD (pp)+9.8+7.3+5.7+2.4+0.5

Qwen3-1.7B-Thinking Student:

Student training methodCode · LiveCodeBench v6 (%)Math · AIME 2025 (%)Math · AIME 2026 (%)Science · GPQA-Diamond (%)General · MMLU-Redux (%)
Baseline (before training)31.335.436.740.173.9
GRPO33.335.937.039.773.9
Standard OPD33.036.738.240.574.0
OPD + Recipe I34.137.138.840.774.2
OPD + Recipes I + II35.337.939.941.174.7
OPD Cookbook (I + II + III)36.538.841.341.575.2
Δ Cookbook vs. Standard OPD (pp)+3.5+2.1+3.1+1.0+1.2

(2) Findings:

  • Every recipe helps in the new domain. Each added recipe improves every benchmark at every step in both modes, mirroring the math results. On in-domain code, the full Cookbook exceeds standard OPD by +9.8 / +3.5 and GRPO by +8.6 / +3.2 (Non-Thinking / Thinking).
  • Gains transfer across domains. Although trained only on code, the Cookbook improves over standard OPD on AIME 2025 / 2026 by +7.3 / +5.7 (Non-Thinking) and +2.1 / +3.1 (Thinking), and on GPQA-Diamond by +2.4 / +1.0.
  • Recipe I drives the in-domain gain, Recipe II the cross-domain gain. In Non-Thinking, Recipe I adds +5.6 on code but only +1.8 / +1.4 on AIME 2025 / 2026, whereas Recipe II adds +3.3 / +2.6 on AIME and +1.1 on GPQA-Diamond. This matches 2.1.1: adapting the teacher sharpens task-specific skill, while recovery training transfers a domain-general one.

2.2 Cross-family Experiment

Setup. Gemma 4 E4B-it (student) and Gemma 4 12B-it (teacher), trained on the same decontaminated DAPO-17K in both Non-Thinking and Thinking modes; all other settings follow the main experiments. We evaluate on AIME 2025 and AIME 2026 (in-domain math) and on GPQA-Diamond, LiveCodeBench v6, and MMLU-Pro (out-of-domain).

(1) Results:

Gemma 4 E4B-it (Non-Thinking) studentAIME 2025 (%)AIME 2026 (%)GPQA-Diamond (%)LiveCodeBench v6 (%)MMLU-Pro (%)
Initial checkpoint36.733.330.326.844.0
GRPO38.538.031.028.043.6
Standard OPD38.338.631.328.843.0
OPD Cookbook (I + II + III)44.647.733.433.845.5
Δ Cookbook vs. Standard OPD (pp)+6.3+9.1+2.1+5.0+2.5
Gemma 4 E4B-it (Thinking) studentAIME 2025 (%)AIME 2026 (%)GPQA-Diamond (%)LiveCodeBench v6 (%)MMLU-Pro (%)
Initial checkpoint40.042.558.652.069.4
GRPO44.246.759.053.169.0
Standard OPD46.948.859.253.570.0
OPD Cookbook (I + II + III)49.750.060.554.970.8
Δ Cookbook vs. Standard OPD (pp)+2.8+1.2+1.3+1.4+0.8

(2) Findings: On Gemma 4, the Cookbook improves over standard OPD on all five benchmarks in both modes. In Non-Thinking it adds +6.3 / +9.1 on AIME 2025 / 2026 and +2.1 to +5.0 out of domain; in Thinking it adds +2.8 / +1.2 and +0.8 to +1.4. As with Qwen3, gains are larger in Non-Thinking. The Cookbook is also the only method that stays above the initial checkpoint on MMLU-Pro in both modes, whereas GRPO falls below it in both.

Scope note. Here, both teacher and student come from a non-Qwen family. Pairing a teacher and a student from different families requires cross-tokenizer alignment, which is orthogonal to our recipes; we leave it to future work.

Reviewer_ViioOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OPD-Cookbook, a framework for improving On-Policy Distillation by addressing two fundamental limitations: (1) the reliability of teacher supervision and (2) the mismatch between training objectives and actual decoding behavior. Three recipes are introduced: (1) teacher adaptation on OPD Data (2) teacher recovery training on student states (3) decoding aware KL objective. The experiments use Qwen3-1.7B students and Qwen3-4B teachers on DAPO-17K, evaluating seven mathematical reasoning benchmarks under both thinking and non-thinking configurations. It shows in the table 1, that the proposed framework improves average Pass@1from 27.5% to 49.5% in the non-thinking setting and from 49.7% to 53.7% in the thinking setting. The authors also report improvements over GRPO and show that combining the recipes produces additional gains.

Critical strengths

(1) It identifies an important limitation of standard OPD: a teacher with strong independent problem-solving ability may still struggle to provide reliable guidance when conditioned on imperfect student-generated prefixes. The empirical analysis presents teacher continuation accuracy deteriorates substantially as student prefix length increases, highlighting a discrepancy between independent teacher proficiency and reliability on student-induced states. Recipe II directly addresses this limitation by training the teacher to recover from student-generated prefixes. The resulting improvements in continuation accuracy, token efficiency, and downstream student performance provide convincing evidence that teacher recoverability is an important factor in effective OPD. This insight is particularly valuable because it shifts attention from teacher capability alone to the quality of supervision on the states actually encountered by the student.

(2) The paper shoes strong empirical improvements and ablation study supports it. The proposed framework achieves good improvements over standard OPD across seven mathematical reasoning benchmarks. In the non-thinking setting, the complete cookbook improves average Pass@1 from 27.5% to 49.5%, outperforming the GRPO baseline of 39.3%. The thinking setting also demonstrates consistent improvements, from 49.7% to 53.7%. Also the paper evaluates all eight combinations of the three recipes, showing that each component individually improves performance and that combining them produces further gains. These results strengthen the empirical credibility of the proposed framework and demonstrate its practical value for improving OPD.

Critical weaknesses

(1) Though the paper provides a useful framework for improving OPD, the novelty of the proposed recipes seems somewhat limited. Recipes I and II mainly apply standard outcome-based RL to different training distributions, rather than introducing fundamentally new optimization methods. Recipe III is more distinctive, but its support-expansion mechanism changes the original post-decoding teacher distribution, raising questions about whether the objective truly aligns the actual sampling distributions. The methods are effective, but their methodological contributions could be better distinguished from existing approaches.

(2) The framework requires additional teacher-side RL training, but the paper does not provide end-to-end compute-matched comparisons. It is unclear whether similar gains could be achieved by spending the same computational budget on additional student RL or standard OPD training. The experiments are limited to one Qwen3 teacher-student model-size pairing and mathematical reasoning benchmarks. Evaluations across different model families/scales would help establish how broadly the proposed recipes generalize. minor error: line 706, "DAPO-17K (?)" no reference.

Questions for authors

please see the weakness.

5 comments
AuthorsOpenReview ↗

Rebuttal [Part 1]

We thank the reviewer for the careful reading and for finding the framework useful and the methods effective. Below we clarify the methodological contributions of each recipe (Q1), report compute-matched comparisons (Q2), and present new experiments beyond mathematical reasoning (Q3) and beyond the Qwen3 family and model scale (Q4). All new results will be included in the revision.


Q1. Novelty and distinction from existing approaches


1.1 Recipes I and II: what is new

Our contribution is not a new optimizer, but identifying where the OPD teacher fails and on which state distribution it should be trained.

Representative OPD methods (GKD, MiniLLM, DistiLLM, the sampled-token estimator) mainly modify the student side and keep the teacher fixed. We show that the teacher itself is a major bottleneck, with two distinct failures of the teacher that, to our knowledge, have not been explicitly addressed in OPD:

Teacher failureEvidenceFix
Recipe IUntapped headroom on the OPD dataFurther RL raises the teacher's accuracy on DAPO-17K from 50.5 to 76.0, and the student's average Pass@1 from 27.5 to 45.4 (Fig. 2a)Adapt the teacher on the OPD data; select the checkpoint on held-out data
Recipe IIRecoverability collapse on student prefixesThe teacher's accuracy drops from 48.3 to 26.3 when it continues from an 8K-token student prefix (Fig. 3a)Train the teacher on student-induced states

Recipe II reverses the direction of DAgger-style learning: instead of the learner training on its own states, the teacher is trained on the learner's states.

Why the two recipes are not "the same RL on different data":

  • Different error terms. In our decomposition ΔS ≤ ΔT + H·κ·ε (Appendix E), Recipe I reduces the teacher's intrinsic error ΔT, while Recipe II reduces the recovery sensitivity κ.
  • Complementary gains. Combining them reaches 47.9, above 45.4 and 46.8 for each alone (Table 2).
  • Different out-of-domain behavior. In Non-Thinking, Recipe II improves the out-of-domain mean over standard OPD by +3.1, compared with +1.7 for Recipe I (see Q3.1).

Relation to student-aware teacher training. Prior KD work has also explored adapting the teacher to the student, e.g., SFTN, which trains the teacher jointly with student branches, and MetaDistil, which updates the teacher via meta-learning from the student's performance. Our focus is different: autoregressive on-policy distillation, where the teacher is queried token by token on student-generated prefixes, and where we adapt the teacher with an outcome reward on the target distribution and on student-induced states. We will discuss these and related works in the revision. The simplicity of Recipes I and II is intentional: they reuse standard RL infrastructure and plug into any OPD pipeline.

AuthorsOpenReview ↗

Rebuttal [Part 2]

1.2 Recipe III: does support expansion still align the actual sampling distributions?

Yes. Support expansion changes the decoded teacher distribution only slightly, the change is bounded exactly in theory, and it is negligible in practice.

1. Some modification is unavoidable. Under top-k / top-p truncation, the student can sample a token that the decoded teacher assigns zero probability, which makes the reverse KL infinite. Any objective on decoded distributions must handle this case.

2. Our modification is minimal. Support expansion adds only these student-sampled tokens to the teacher's support, using the teacher's pre-truncation probabilities, and renormalizes. The relative probabilities among the teacher's own retained tokens are unchanged.

3. Its effect is bounded exactly. Let a be the teacher's pre-truncation mass on its own decoded support and b the mass on the added tokens.

  • Lemma E.7: the total variation between the expanded and the original decoded teacher distribution is exactly b / (a + b).
  • Proposition E.8: the mismatch between the original decoded distributions satisfies ε^dec ≤ √(L_exp / 2) + δ_sup, where δ_sup is the average of b / (a + b) over student-visited states.

So Recipe III still controls the actual decoded mismatch, with an explicit additive slack δ_sup.

4. The slack is negligible in practice. We measured these quantities on student rollouts during OPD training (Non-Thinking, 512 prompts per checkpoint). The expansion statistics are computed for the Recipe III run; the last two columns compare the original, unexpanded decoded distributions under standard OPD and under Recipe III.

OPD stepPositions with expansion (%)Mean added teacher mass bMean distortion δ_supTV(p_dec, q_dec), standard OPDTV(p_dec, q_dec), Recipe III
09.60.00710.00730.1180.118
2507.80.00550.00570.1140.107
5007.10.00490.00500.1120.104
  • Expansion is rare and small. It is triggered at fewer than 10% of positions, and δ_sup stays below 0.01: the expanded teacher distribution differs from the original one by less than 1% in total variation.
  • Recipe III aligns the actual sampling distributions. On the original, unexpanded decoded distributions, its total variation is consistently lower than that of standard OPD (0.104 vs. 0.112 at step 500), consistent with the support-overlap results in Fig. 4(a).
AuthorsOpenReview ↗

Rebuttal [Part 3]

Q2. Compute-matched comparison


2.1 Training cost and its breakdown

The full Cookbook uses 26% less training time than GRPO in Non-Thinking (26.0 h vs. 35.3 h) and 25% less in Thinking (37.8 h vs. 50.6 h). All experiments run on the same hardware (8× NVIDIA H200) with identical training settings (256 prompts per step, maximum generation length 16K); times are measured wall-clock.

(1) Results:

Qwen3-1.7B-Non-Thinking Student:

MethodStageModel trainedRollouts per promptStepsWall-clock (h)
GRPOStudent RL1.7B student850035.3
Standard OPDStudent OPD1.7B student150013.8
OPD CookbookJoint teacher RL (Recipes I + II)4B teacher415012.5
OPD CookbookStudent OPD + Recipe III1.7B student150013.5
OPD CookbookTotal–––26.0

Qwen3-1.7B-Thinking Student:

MethodStageModel trainedRollouts per promptStepsWall-clock (h)
GRPOStudent RL1.7B student850050.6
Standard OPDStudent OPD1.7B student150028.9
OPD CookbookJoint teacher RL (Recipes I + II)4B teacher4809.7
OPD CookbookStudent OPD + Recipe III1.7B student150028.1
OPD CookbookTotal–––37.8

(2) Findings:

OPD needs only one rollout per prompt, while GRPO needs a group, which makes each OPD step much cheaper. The two methods obtain their advantage from different sources:

  • GRPO needs a group of rollouts. Its advantage is group-relative: each rollout's outcome reward is normalized by the mean and standard deviation of the rewards in its group. With a single rollout per prompt this advantage is undefined, so GRPO must sample several rollouts per prompt (8 in our runs).
  • OPD needs only one rollout. Its token-level advantage, Â_t = log q_t(ŷ_t) − log p_t(ŷ_t), comes directly from the teacher at every token, so a single student rollout per prompt suffices. The teacher only runs a forward pass on that rollout, with no generation and no backward pass.

As a result, OPD generates 8× fewer rollouts per step, cutting the step time from 254 s to 99 s in Non-Thinking and from 364 s to 208 s in Thinking.

The teacher-side overhead of the Cookbook is small and bounded. Recipes I and II share a single RL stage of 150 steps (Non-Thinking) or 80 steps (Thinking) with 4 rollouts per prompt. Recipe III adds negligible overhead, since it only truncates and renormalizes logits. The adapted teacher is also trained once and can be reused across students and OPD runs.


2.2 Compute-matched comparison

Even when standard OPD is given slightly more training time, the Cookbook outperforms it by 21.9 points in Non-Thinking and 3.4 points in Thinking. To test whether the gain simply comes from the extra teacher-training time, we extend standard OPD to a budget at least as large as the Cookbook's: about 950 steps in Non-Thinking (26.3 h, vs. 26.0 h for the Cookbook) and about 670 steps in Thinking (38.6 h, vs. 37.8 h).

Results:

(1) Qwen3-1.7B-Non-Thinking Student:

MethodTraining time (h)7-benchmark mean Pass@1 (%)
GRPO (500 steps)35.339.3
Standard OPD (500 steps)13.827.5
Standard OPD (~950 steps, budget-matched)26.327.6
OPD Cookbook (I + II + III)26.049.5

(2) Qwen3-1.7B-Thinking Student:

MethodTraining time (h)7-benchmark mean Pass@1 (%)
GRPO (500 steps)50.649.1
Standard OPD (500 steps)28.949.7
Standard OPD (~670 steps, budget-matched)38.650.3
OPD Cookbook (I + II + III)37.853.7

This answers both alternatives raised by the reviewer:

  • Spending the budget on additional student RL. GRPO already uses more training time than the full Cookbook, yet trails it by 10.2 points (Non-Thinking) and 4.6 points (Thinking).
  • Spending the budget on additional standard OPD. About 450 extra steps in Non-Thinking add only +0.1 points (27.5 → 27.6), and about 170 extra steps in Thinking add only +0.6 (49.7 → 50.3), so standard OPD has saturated under the original teacher.

The Cookbook's gain therefore comes from better supervision, not from more compute.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.