- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper first re-derives on-policy distillation (OPD) as a special case of dense KL-constrained RL in which (i) the implicit reward is , (ii) the KL reference can be any model without changing the objective, and (iii) the reward and KL terms are weighted equally. Building on this, the authors propose Generalized OPD (G-OPD), which adds a reward scaling factor (Eq. 8) and a flexible . Two regimes are studied: ExOPD (, "reward extrapolation") and reward correction (choosing in strong-to-weak distillation).
Experiments use Qwen3-4B-Non-Thinking as the student; domain teachers are obtained by GRPO on DeepMath (math) and Eurus-RL-Code (code). Main empirical claims: (1) ExOPD with consistently beats standard OPD, and in the multi-teacher merge setting the unified student surpasses both domain teachers on all 7 benchmarks (Table 1); (2) in strong-to-weak distillation from Qwen3-30B-A3B to 1.7B/4B students, ExOPD improves over OPD by +2.3/+2.7 avg (Table 2), and reward correction gives further gains (Figure 2). Appendices D/E add a continued-RL comparison (Table 7) and a 1200-step-teacher robustness check (Table 8).
Critical strengths
S1. The method is minimal and the multi-teacher-merge result is practically valuable. ExOPD is a single hyperparameter on top of OPD — trivial to implement inside any verl-style pipeline. In the multi-expert-merge setting (Table 1), it is the only method among {SFT, OPD, ExPO} that lifts a unified student above both domain teachers on every benchmark. For practitioners doing MOPD-style post-training, this is a directly usable trick.
S2. Honest self-exposure of the "teacher under-training" confound. Appendix D (Table 7) explicitly compares against continued-RL on the teacher, and Appendix E (Table 8) reports results under a 1200-step teacher even though those results visibly weaken the main claim. This is a level of candor I appreciate — many submissions would have dropped Table 8.
Critical weaknesses
W1. The motivation is framework-first and the theory is an identity rewrite, so the paper does not establish why ExOPD works. §1 opens with "mechanistic understanding of OPD remains limited" but never shows where OPD actually fails in practice. The core derivation (Eq. 4 → Eq. 8) is a telescoping identity: adding and subtracting leaves unchanged, so the "OPD = dense KL-constrained RL" remark provides no new prediction. The closed-form says where the distribution moves, not why moving further along improves task accuracy. The untested assumption — that the reward direction correlates with capability — is precisely what "learning beyond teacher" relies on, and Table 8 (gain collapses under a well-trained teacher) is consistent with the alternative explanation that ExOPD is partly compensating for under-trained teachers along their own RL trajectory.
W2. Attribution is unclear because of missing equal-compute baselines and insufficient statistics. Three baselines are missing that would each materially change the interpretation: (a) teacher continued-GRPO to equal wall-clock (Table 7 only adds 100 steps and still loses on average, but compute is not matched); (b) joint multi-domain GRPO from the same base; (c) ExPO augmented with a small amount of training. On top of this, there are no training-seed error bars (32 evaluation samples are not a substitute), code benchmarks use 4 samples on ~160–380 problems so several "wins" in Table 1 (HumanEval+ +0.3, MBPP+ +0.4) are within noise, and appears to be selected on the test set. The combination means the headline "consistently outperforms" is not yet statistically established.
Questions for authors
Q1 (equal-compute). Please report teacher-continued-GRPO to matched wall-clock (not just 100 steps), joint multi-domain GRPO from the same base, and ExPO+light-OPD, all at the same compute budget as ExOPD. Without at least one of these, the "beyond teacher" claim cannot be separated from "teacher is under-trained." Table 8 already hints at this — please reconcile.
Q2 (λ selection). Is chosen on a held-out validation set or on the reported test benchmarks? Is there a single validation set used for all decisions, and what is the sensitivity of the ranking to the choice of validation set?
Q3 (reward correction vs. λ). Eq. 10 shows the reward term vanishes at , so reward correction should only matter when . Please report OPD () + reward correction as an ablation. Without it, the reader cannot tell whether reward correction is an independent improvement or an artifact that only kicks in under extrapolation.
Q4 (length confound). Response length grows monotonically with (Figure 1/5). Please report, per : mean length, 16K-truncation rate, and accuracy normalized to fixed response length. Is the ExOPD gain still present after controlling for length?
Q5 (external validity). All results use Qwen3-4B-Non-Thinking as the student (and Figure 2 retreats to a 4B teacher because the 30B teacher's pre-RL checkpoint is unavailable). Can you show ExOPD on at least one thinking model or one multi-turn/agent benchmark, where "pre-RL reference" and length extrapolation behave very differently? If not, please scope the claim in the abstract/conclusion accordingly.
Thank you for your reviews
Dear all reviewers,
We sincerely thank you for your thoughtful reviews and constructive comments. In this response, we would like to provide our clarifications and further insights regarding the following aspects:
Q1: Regarding the novelty.
A1: Our paper takes a pioneer step towards understanding the quick learning property and mechanisms of OPD under the standard dense RL perspective through simple math derivations. Based on this, we further propose ExOPD method to make the student have the potential to surpass teacher models in MOPD, which we believe already provides new and novel insights to the community.
Q2: Regarding the normalization term in Eq. (9).
A2: We sincerely thank you for your careful reviews. We apologize we miss the partition term during deriving Eq. (9) from Eq. (8). The correct solution should be logπθ=λlogπ∗+(1−λ)πref+logZλ where Zλ is the constant partition term. But notice that the including this constant term during applying partial derivatives to θ in Eq. (8) does not affect the final gradient formulation, and we can still write the approximated gradient formulation of G-OPD (zero discount approximation in the sampled-token OPD setting) as Eq. (11).
Q3: Regarding the empirical improvement.
A3: In Table 1 and Table 2, we can see that ExOPD consistently outperforms OPD in all benchmarks under a considerable number of samplings per problem. Thus, we believe the empirical improvement of ExOPD is sufficient. But we agree that conducting more experiments on other domains and backbones would further strengthen the paper.
Q4: Regarding why ExOPD (reward extrapolation) could work.
A4: Our insight is, in OPD where the teacher is the RL variant of the student, the token-level reward is exactly the implicit reward the teacher learned during RL, and is a naturally well-defined proxy for the true reward. Also, in OPD formulation, the reward term and the KL regularization term have the same weights. Thus, in this case, increasing the weight of the reward term (i.e., ExOPD) means we want the student to learn further along the correct direction (since the implicit reward here is a good, correct and well-defined proxy for the true reward) under a smaller KL regularization. This insight is similar to that of the weight extrapolation method ExPO.
Similarly, regarding the reward correction practice in strong-to-weak distillation setting where the teacher is a larger model, logπrefteacherπ∗ should be the correct reward proxy to represent the RL process of the teacher, and logπrefstudentπ∗ contains the noise caused by the intrinsic capability gap (i.e., pre-training, backbone difference) between large and small base models.
Q5: Regarding why the response length would increase under ExOPD.
A5: We find that if the response length of the teacher model after post-training increases and is longer than that of the initial student, reward extrapolation would further inflate the response length, because the implicit reward now contains the effect of length increase, and amplifying it causes the response length to further increase. If one cloud get a teacher whose length is comparable to that of the student, then the effect of reward extrapolation could be effectively mitigated.
We sincerely thank all the reviewers again.
Best regards,
Authors of Submission 29601