acceptodds
Under review as a conference paper at ICLR 2027

Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation

14 views

Abstract

On-policy distillation (OPD), which aligns the student with the teacher's logit distribution on student-generated trajectories, has demonstrated strong empirical gains in improving student performance and often outperforms off-policy distillation and reinforcement learning (RL) paradigms. In this work, we first theoretically show that OPD is a special case of dense KL-constrained RL where the reward function and the KL regularization are always weighted equally and the reference model can by any model. Then, we propose the Generalized On-Policy Distillation (G-OPD) framework, which extends the standard OPD objective by introducing a flexible reference model and a reward scaling factor that controls the relative weight of the reward term against the KL regularization. Through comprehensive experiments on math reasoning and code generation tasks, we derive two novel insights: (1) Setting the reward scaling factor to be greater than 1 (i.e., reward extrapolation), which we term ExOPD, consistently improves over standard OPD across a range of teacher-student size pairings. In particular, in the setting where we merge the knowledge from different domain experts, obtained by applying domain-specific RL to the same student model, back into the original student, ExOPD enables the student to even surpass the teacher's performance boundary and outperform the domain teachers. (2) Building on ExOPD, we further find that in the strong-to-weak distillation setting (i.e., distilling a smaller student from a larger teacher), performing reward correction by choosing the reference model as the teacher's base model before RL yields a more accurate reward signal and further improves distillation performance.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
On-Policy Distillation, RL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_yrsuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper first re-derives on-policy distillation (OPD) as a special case of dense KL-constrained RL in which (i) the implicit reward is r(x,y)=log⁡π∗(y∣x)/πref(y∣x)r(x,y)=\log \pi^*(y|x)/\pi_{\text{ref}}(y|x), (ii) the KL reference πref\pi_{\text{ref}} can be any model without changing the λ=1\lambda=1 objective, and (iii) the reward and KL terms are weighted equally. Building on this, the authors propose Generalized OPD (G-OPD), which adds a reward scaling factor λ\lambda (Eq. 8) and a flexible πref\pi_{\text{ref}}. Two regimes are studied: ExOPD (λ>1\lambda>1, "reward extrapolation") and reward correction (choosing πref=πbaseteacher\pi_{\text{ref}}=\pi^{\text{teacher}}_{\text{base}} in strong-to-weak distillation).

Experiments use Qwen3-4B-Non-Thinking as the student; domain teachers are obtained by GRPO on DeepMath (math) and Eurus-RL-Code (code). Main empirical claims: (1) ExOPD with λ=1.25\lambda=1.25 consistently beats standard OPD, and in the multi-teacher merge setting the unified student surpasses both domain teachers on all 7 benchmarks (Table 1); (2) in strong-to-weak distillation from Qwen3-30B-A3B to 1.7B/4B students, ExOPD improves over OPD by +2.3/+2.7 avg (Table 2), and reward correction gives further gains (Figure 2). Appendices D/E add a continued-RL comparison (Table 7) and a 1200-step-teacher robustness check (Table 8).

Critical strengths

S1. The method is minimal and the multi-teacher-merge result is practically valuable. ExOPD is a single hyperparameter on top of OPD — trivial to implement inside any verl-style pipeline. In the multi-expert-merge setting (Table 1), it is the only method among {SFT, OPD, ExPO} that lifts a unified student above both domain teachers on every benchmark. For practitioners doing MOPD-style post-training, this is a directly usable trick.

S2. Honest self-exposure of the "teacher under-training" confound. Appendix D (Table 7) explicitly compares against continued-RL on the teacher, and Appendix E (Table 8) reports results under a 1200-step teacher even though those results visibly weaken the main claim. This is a level of candor I appreciate — many submissions would have dropped Table 8.

Critical weaknesses

W1. The motivation is framework-first and the theory is an identity rewrite, so the paper does not establish why ExOPD works. §1 opens with "mechanistic understanding of OPD remains limited" but never shows where OPD actually fails in practice. The core derivation (Eq. 4 → Eq. 8) is a telescoping identity: adding and subtracting log⁡πref\log\pi_{\text{ref}} leaves λ=1\lambda=1 unchanged, so the "OPD = dense KL-constrained RL" remark provides no new prediction. The closed-form log⁡πθ=log⁡π∗+(λ−1)(log⁡π∗−log⁡πref)\log\pi_\theta = \log\pi^* + (\lambda-1)(\log\pi^*-\log\pi_{\text{ref}}) says where the distribution moves, not why moving further along (log⁡π∗−log⁡πref)(\log\pi^*-\log\pi_{\text{ref}}) improves task accuracy. The untested assumption — that the reward direction correlates with capability — is precisely what "learning beyond teacher" relies on, and Table 8 (gain collapses under a well-trained teacher) is consistent with the alternative explanation that ExOPD is partly compensating for under-trained teachers along their own RL trajectory.

W2. Attribution is unclear because of missing equal-compute baselines and insufficient statistics. Three baselines are missing that would each materially change the interpretation: (a) teacher continued-GRPO to equal wall-clock (Table 7 only adds 100 steps and still loses on average, but compute is not matched); (b) joint multi-domain GRPO from the same base; (c) ExPO augmented with a small amount of training. On top of this, there are no training-seed error bars (32 evaluation samples are not a substitute), code benchmarks use 4 samples on ~160–380 problems so several "wins" in Table 1 (HumanEval+ +0.3, MBPP+ +0.4) are within noise, and λ=1.25\lambda=1.25 appears to be selected on the test set. The combination means the headline "consistently outperforms" is not yet statistically established.

Questions for authors

Q1 (equal-compute). Please report teacher-continued-GRPO to matched wall-clock (not just 100 steps), joint multi-domain GRPO from the same base, and ExPO+light-OPD, all at the same compute budget as ExOPD. Without at least one of these, the "beyond teacher" claim cannot be separated from "teacher is under-trained." Table 8 already hints at this — please reconcile.

Q2 (λ selection). Is λ=1.25\lambda=1.25 chosen on a held-out validation set or on the reported test benchmarks? Is there a single validation set used for all λ\lambda decisions, and what is the sensitivity of the ranking to the choice of validation set?

Q3 (reward correction vs. λ). Eq. 10 shows the reward term vanishes at λ=1\lambda=1, so reward correction should only matter when λ>1\lambda>1. Please report OPD (λ=1\lambda=1) + reward correction as an ablation. Without it, the reader cannot tell whether reward correction is an independent improvement or an artifact that only kicks in under extrapolation.

Q4 (length confound). Response length grows monotonically with λ\lambda (Figure 1/5). Please report, per λ\lambda: mean length, 16K-truncation rate, and accuracy normalized to fixed response length. Is the ExOPD gain still present after controlling for length?

Q5 (external validity). All results use Qwen3-4B-Non-Thinking as the student (and Figure 2 retreats to a 4B teacher because the 30B teacher's pre-RL checkpoint is unavailable). Can you show ExOPD on at least one thinking model or one multi-turn/agent benchmark, where "pre-RL reference" and length extrapolation behave very differently? If not, please scope the claim in the abstract/conclusion accordingly.

Reviewer_e88jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper interprets reverse-KL on-policy distillation as KL-regularized optimization of a teacher/reference log-probability ratio. Its generalized objective, G-OPD, scales this implicit reward by a factor lambda and permits different reference policies. ExOPD uses lambda greater than one to amplify the teacher's distributional shift from the reference. A further variant uses the teacher's pre-RL checkpoint as the reference when distilling into a smaller student.

Experiments cover Qwen3 models on four mathematics and three coding benchmarks, including single-teacher, multi-teacher, and strong-to-weak distillation. With lambda fixed to 1.25 after preliminary experiments, ExOPD improves over standard OPD. Table 2 reports mathematics averages increasing from 23.1 to 25.4 for the 1.7B student and from 42.6 to 45.3 for the 4B student. Additional experiments examine reward interpolation, longer-trained teachers, and reference-policy selection.

Critical strengths
  1. A simple, useful extension produces improvements across distinct distillation settings. Table 1 (p. 8) reports multi-teacher mathematics/coding averages of 47.7/62.0, compared with 46.4/60.6 for OPD and 46.0/61.2 for the domain teachers. Table 2 (p. 9) shows larger gains over OPD of 2.3 and 2.7 percentage points for two smaller students distilled from Qwen3-30B-A3B-Instruct-2507. The same extrapolation factor is reused in these later experiments. This makes the method potentially practical as a small modification to existing OPD training, rather than an observation tied to one student size or task.

  2. The study includes relevant controls on the extrapolation effect. Figures 1, 3, and 4 explore interpolation through excessive extrapolation, showing both improvements and degradation rather than only a favorable setting. Appendix D, Table 7 (p. 16), compares with continued teacher RL: ExOPD reaches 48.0 mathematics accuracy versus 46.9 after 100 additional RL steps. Appendix E, Table 8 (p. 17), additionally uses teachers trained for 1,200 RL steps and still finds average improvements. These controls strengthen the case that the observed effect is not restricted to the initial teacher checkpoints, although they do not establish matched-compute superiority.

Critical weaknesses
  1. The theoretical account does not yet consistently describe the optimized method or justify reward accuracy. The normalized unrestricted optimum of Eq. (8) is proportional to teacher^lambda * reference^(1-lambda); Eq. (9) omits the corresponding log-partition term. More consequentially, the advantage in Eqs. (11) and (19) is log student - lambda log teacher + (lambda-1) log reference, the negative of the coefficient required for ascent on the stated maximization objective. This could be a loss-gradient convention, but the paper must specify it; Appendix A repeats the inconsistency. The acknowledged zero-discount approximation also omits future terms from the sequence-objective gradient. Separately, the claim that the teacher/pre-RL log-ratio is a more accurate task reward relies on the KL-optimal-policy relationship in Eq. (7), whereas the actual teachers use a KL coefficient of zero (Appendix B, Tables 3-4, p. 15). Their being RL descendants does not establish that relationship. These issues matter because the paper presents mechanistic understanding and reward correction as central contributions, beyond an empirically useful heuristic.

  2. The evidence supports average improvements more securely than the stronger “beyond teacher” and reward-correction conclusions. Several decisive margins are small: Table 1 exceeds the teachers by only 0.3 points on HMMT25 November and HumanEval+, while Figure 2 reports reward-correction gains of 0.6 points in mathematics and 1.0 in coding. The paper reports multiple sampled answers per problem, but no uncertainty intervals or variability across independently trained models. With stronger teachers, Table 8 also weakens the all-benchmark picture: multi-teacher ExOPD scores 42.7 versus the teacher's 42.9 on HMMT25 November, and loses to OPD on AIME25 and HMMT25 February. Moreover, Figure 1 and the discussion on p. 7 explicitly associate extrapolation with longer responses; the common maximum length does not match actual inference expenditure. Thus, the results establish encouraging point estimates under the chosen decoding setup, but do not distinguish a robust capability improvement from a combination of additional reasoning tokens and evaluation/training variation. This distinction materially affects the paper's main interpretation.

Questions for authors
  1. What exact loss is minimized in the implementation, including the sign and treatment of the advantage? Please reconcile Eqs. (8), (11), and (19), restore normalization in Eq. (9), and clarify which conclusions apply to the sequence objective versus the zero-discount update.
  2. Given the zero-KL teacher training in Tables 3-4, what supports interpreting the corrected log-ratio as a more accurate task reward, rather than a different useful shaping signal? Is there direct evidence that correction improves reward quality beyond the final accuracy changes in Figure 2?
  3. Do the small teacher-surpassing and reward-correction gains remain credible under uncertainty estimates and comparable generated-token budgets? Paired problem-level intervals, available training-run variability, and a token-budget comparison for the main settings would directly affect my assessment; additional unrelated benchmarks would not resolve this concern.
Reviewer_s2UiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Generalized On-Policy Distillation (G-OPD) framework, which reformulates standard OPD as a special case of KL-regularized RL with a log-probability-ratio reward. The framework introduces a reward-scaling parameter, ($lambda$), with (λ=1\lambda=1) recovering standard OPD. The authors investigate whether setting (λ>1\lambda>1), termed ExOPD, can improve distillation and enable the student to surpass the teacher. Experiments using Qwen3 models examine math reasoning and code generation in both same-sized teacher–student and strong-to-weak distillation settings, reporting improvements over base OPD.

Critical strengths
  • The reformulation of OPD as KL-regularized RL provides a useful perspective on its training objective. The authors use this perspective to introduce a practical extension that adjusts the relative weight of the implicit reward and KL regularization.
  • The experiments with (0<λ<10<\lambda<1) support the G-OPD framework, as the observed changes in accuracy and response length suggest that this parameter can provide useful control over the behavior of the distilled model.
Critical weaknesses
  • The empirical evidence for ExOPD is somewhat limited. In Table 1, ExOPD improves over standard OPD by an average under 1.5%. While Table 2 provides stronger evidence, all experiments use the Qwen3 family, with small students (1.7B, 4B) and Table 1 with only 4B. Experiments with at least one additional model family would strengthen the claims about ExOPD. Furthermore, standard OPD also sometimes surpasses the teacher on some benchmarks, such as AIME24 in Table 1, where it achieves a 2.6% improvement. Since an ideally optimized OPD objective should align the student with the teacher, this raises questions about how to interpret the empirical number differences among the teacher, OPD, and ExOPD.
  • The connection between reward extrapolation and downstream improvement needs further justification. I believe Eq. (9) can be interpreted as defining a new extrapolated target distribution for distillation (i.e. performing OPD on the log-likelihood extrapolated optimal policy). However, extrapolating the teacher’s log-probability shift does not by itself always establish that this target has higher task accuracy. Moreover, a stronger teacher does not always provide a better distillation target for a smaller student. Additional analysis identifying when extrapolation should help, or empirical evidence connecting the extrapolated signal to correctness, would strengthen the explanation of the observed gains.
Questions for authors
  • Eq. (9) appears to omit the normalization term (−log⁡Zλ(x))(-\log Z_\lambda(x)). Could the authors provide the normalized expression and clarify its effects? I understand token-level optimization is common in OPD, Eq. (9) seems to describe a sequence-level distribution, whereas the practical update is at the token level, so will this distinction effect the current analysis or claims?
  • I understand the rationale in Section 3.2 for using the teacher’s pre-RL model as the reference—-the resulting log ratio isolates the teacher’s post-training changes, whereas using the student’s base model also includes differences between the two base models. However, it is unclear why the latter component should be treated as noise, since it may also contain useful capability information. In other words, why does the capability gap between the larger and smaller base models not matter while the capability gap between the RL and base model does?
Reviewer_qHNjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Generalized On-Policy Distillation (G-OPD), which reformulates on-policy distillation as a special case of KL-regularized reinforcement learning. By introducing a reward scaling factor and a flexible reference model, the authors investigate reward interpolation and extrapolation. The proposed ExOPD method aims to improve distillation performance by extrapolating the implicit reward signal, potentially enabling student models to outperform their teachers. The authors further introduce a reward correction strategy for strong-to-weak distillation. Experiments on mathematical reasoning and code generation benchmarks demonstrate improvements over standard OPD under several distillation settings.

Critical strengths
  1. The proposed G-OPD framework is conceptually straightforward and can be incorporated into existing OPD pipelines without substantial architectural modifications.

  2. ExOPD demonstrates improvements over standard OPD across multiple mathematical reasoning and code generation benchmarks. The results showing that student models can outperform domain-specific teachers are particularly interesting.

Critical weaknesses
  1. The connection between OPD and KL-regularized RL is primarily established through an algebraic reformulation of the reverse KL objective. The proposed reward extrapolation mechanism essentially introduces an additional scaling factor into the implicit reward. Therefore, the theoretical novelty appears somewhat incremental. Moreover, Eq. (9) omits the normalization term required to define a valid probability distribution, and the relationship between the theoretical objective and the approximate token-level gradient deserves further clarification.
  2. As shown in Figure 1, reward extrapolation generally leads to longer model responses. Since longer reasoning trajectories may independently improve performance on mathematical reasoning and code generation tasks, it is unclear whether the observed gains primarily result from improved distillation or increased inference-time computation. Comparisons under matched response lengths or inference token budgets are necessary to substantiate the claimed advantages of ExOPD.
  3. The experimental baselines primarily include SFT, standard OPD, and weight extrapolation. Given that G-OPD can be interpreted as modifying the relative weighting of implicit rewards and KL regularization, comparisons with alternative reward-scaling approaches or KL-regularized optimization strategies would help clarify whether the improvements are specific to the proposed formulation.
  4. Some improvements over standard OPD and the teacher models are relatively modest, particularly in the multi-teacher setting. Although multiple responses are sampled during evaluation, the paper does not provide results across multiple independent training runs or report confidence intervals. Consequently, the statistical reliability of the reported improvements remains unclear.
  5. The experiments are primarily conducted using Qwen3 models and focus on mathematical reasoning and code generation. It remains uncertain whether reward extrapolation provides similar benefits for other model families, general instruction-following tasks, or knowledge-intensive benchmarks. Furthermore, the sensitivity of the optimal reward scaling factor to different teacher-student configurations is insufficiently explored.
  6. Although reward correction is presented as an additional contribution, its empirical evaluation is relatively limited. The corresponding experiments use a different teacher configuration from the main strong-to-weak distillation experiments. More systematic experiments are needed to establish its general effectiveness. In addition, the computational overhead associated with maintaining and evaluating an additional reference model is not quantitatively analyzed.
Questions for authors

Please refer to the weaknesses section.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Thank you for your reviews

Dear all reviewers,

We sincerely thank you for your thoughtful reviews and constructive comments. In this response, we would like to provide our clarifications and further insights regarding the following aspects:

Q1: Regarding the novelty.

A1: Our paper takes a pioneer step towards understanding the quick learning property and mechanisms of OPD under the standard dense RL perspective through simple math derivations. Based on this, we further propose ExOPD method to make the student have the potential to surpass teacher models in MOPD, which we believe already provides new and novel insights to the community.

Q2: Regarding the normalization term in Eq. (9).

A2: We sincerely thank you for your careful reviews. We apologize we miss the partition term during deriving Eq. (9) from Eq. (8). The correct solution should be log⁡πθ=λlog⁡π∗+(1−λ)πref+log⁡Zλ\log \pi_{\theta} = \lambda \log \pi^{*} + (1-\lambda) \pi_{ref} + \log Z_{\lambda} where ZλZ_{\lambda} is the constant partition term. But notice that the including this constant term during applying partial derivatives to θ\theta in Eq. (8) does not affect the final gradient formulation, and we can still write the approximated gradient formulation of G-OPD (zero discount approximation in the sampled-token OPD setting) as Eq. (11).

Q3: Regarding the empirical improvement.

A3: In Table 1 and Table 2, we can see that ExOPD consistently outperforms OPD in all benchmarks under a considerable number of samplings per problem. Thus, we believe the empirical improvement of ExOPD is sufficient. But we agree that conducting more experiments on other domains and backbones would further strengthen the paper.

Q4: Regarding why ExOPD (reward extrapolation) could work.

A4: Our insight is, in OPD where the teacher is the RL variant of the student, the token-level reward is exactly the implicit reward the teacher learned during RL, and is a naturally well-defined proxy for the true reward. Also, in OPD formulation, the reward term and the KL regularization term have the same weights. Thus, in this case, increasing the weight of the reward term (i.e., ExOPD) means we want the student to learn further along the correct direction (since the implicit reward here is a good, correct and well-defined proxy for the true reward) under a smaller KL regularization. This insight is similar to that of the weight extrapolation method ExPO.

Similarly, regarding the reward correction practice in strong-to-weak distillation setting where the teacher is a larger model, log⁡π∗πrefteacher\log \frac{\pi^{*}}{\pi_{ref}^{teacher}} should be the correct reward proxy to represent the RL process of the teacher, and log⁡π∗πrefstudent\log \frac{\pi^{*}}{\pi_{ref}^{student}} contains the noise caused by the intrinsic capability gap (i.e., pre-training, backbone difference) between large and small base models.

Q5: Regarding why the response length would increase under ExOPD.

A5: We find that if the response length of the teacher model after post-training increases and is longer than that of the initial student, reward extrapolation would further inflate the response length, because the implicit reward now contains the effect of length increase, and amplifying it causes the response length to further increase. If one cloud get a teacher whose length is comparable to that of the student, then the effect of reward extrapolation could be effectively mitigated.

We sincerely thank all the reviewers again.

Best regards,

Authors of Submission 29601

Thank you for the clarifications. I have two follow-up concerns.

  1. Your reply (A3) justifies the result by per-problem sampling (32/4), but my Q1 asked for training-seed variance — please report ≥3 seeds with std/CI, and state whether λ=1.25 was selected on a held-out validation set or on the reported test benchmarks (the latter would make the gains optimistically biased).

  2. A4 argues log π*/π_ref is a well-defined reward proxy, yet Table 8 shows that a more thoroughly trained teacher shrinks ExOPD's gain (+1.7→+0.3). If the proxy is good, stronger teachers shouldn't weaken ExOPD. Please reconcile, or supply an equal-compute baseline (matched-wall-clock continued GRPO) ruling out the under-trained-teacher explanation.

Thank you for the rebuttal. However, my concerns remain unaddressed. No additional experiments were provided, so my concern about the limited empirical evidence, such as the use of a single model family, remains. In addition, explanations of (1) why the capability gap between the teacher and student base models should be treated as noise in reward correction, and (2) why the extrapolated target should be a better distillation target for the student are still missing. While author's explanation makes sense in λ<1\lambda <1, there seems no trivial reason to improve beyond 1 (the paper itself reports degradation at 1.5). As these points remain open and no new results were provided, I maintain my score.

Thank you for the detailed rebuttal. I may maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.