acceptodds
Under review as a conference paper at ICLR 2027

Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation

Abstract

Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors and surpass them. This question is particularly important for successive model generations and multi-domain consolidation, where repeating frontier-scale post-training from scratch can be prohibitively expensive. Yet conventional distillation treats the weak teacher as an optimization target, potentially imposing its capacity ceiling on the student. We introduce On-Policy Reverse Distillation (OPRD), which evaluates the teacher’s policy shift relative to its reference policy on student rollouts and amplifies the component of the student’s verifier-driven policy gradient along that direction. By rescaling only verifier-supported updates, OPRD preserves the stationary points of policy optimization while accelerating learning beyond the teacher. In both successive model transfer and multi-teacher distillation, OPRD achieves higher performance with fewer student updates than existing RL and distillation approaches. Stylistically, OPRD students remain closer to models trained with verifier-based RL alone than to their weak teachers, suggesting that teacher guidance accelerates rather than redirects the students’ own optimization. Results in conventional strong-to-weak distillation further demonstrate that OPRD effectively combines verifier-driven policy optimization with teacher guidance regardless of capacity ordering.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Weak-to-Strong Generalization, On-Policy Distillation, Reinforcement Learning, Knowledge Distillation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Xr3jOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Motivated by the observation that supervision from a weaker model can improve a stronger student's performance, this paper proposes On-Policy Reverse Distillation (OPRD), which leverages policy shifts learned by a weak teacher to scale gradient updates during student training. Experiments in weak-to-strong transfer and multi-domain consolidation demonstrate that OPRD outperforms GRPO, OPD, and KDRL baselines, while also surpassing the weak teacher's performance.

Critical strengths
  • The paper provides comprehensive comparisons with diverse baselines and extensive ablation studies across different settings, demonstrating consistent improvements. These results strengthen the empirical credibility of the proposed method.

  • Despite modifying the gradient update rule of GRPO, the paper theoretically establishes that the resulting update direction has a positive directional derivative with respect to the optimization objective.

Critical weaknesses
  • (Major) Questionable framing as distillation.

I find it difficult to characterize the proposed method as a distillation approach for the following reasons.

First, OPRD is fundamentally based on GRPO, with its primary modification amplifying gradient updates according to the teacher's policy shift, rather than directly transferring the teacher's reasoning trajectories or output distributions. Moreover, since this policy shift is obtained through GRPO post-training of the weak teacher, the method could alternatively be interpreted as extracting useful optimization signals from a separately trained "surrogate model". From this perspective, the distinction between OPRD and a teacher-guided variant of GRPO remains unclear.

Second, the experimental evidence does not sufficiently establish the role of weak-to-strong generalization in motivating the proposed approach. In Figures 7–11, OPD, which directly distills from the weak teacher, often fails to surpass the teacher's performance or achieves only comparable results. Furthermore, it is not particularly surprising that GRPO applied directly to a stronger student can outperform GRPO applied to a weaker model. These observations raise the question of whether the improvements of OPRD should be attributed to weak-to-strong knowledge transfer or simply to more effective optimization of the stronger student.

Overall, I believe that framing the proposed method primarily as distillation weakens the motivation and obscures its actual contribution.

  • Insufficient motivation for the proposed gradient scaling mechanism.

The paper discusses the limitations of directly matching the weak teacher's reasoning trajectories and provides an ablation study in Figure 3(a) comparing different guidance directions. However, the motivation for specifically amplifying student gradient updates according to the teacher's policy shift remains insufficiently explained. In particular, it is unclear why this policy shift should provide a more effective optimization signal than other forms of teacher guidance.

Questions for authors
  • Teacher training recipe. What are the exact training configurations used for the weak teachers? Are the training recipes and datasets identical to those used for student training?

  • Computational efficiency. The introduction states that OPRD achieves performance comparable to GRPO with 33–67% fewer training steps. Does this improvement translate into lower computational cost in terms of FLOPs or wall-clock time? In particular, when accounting for the additional cost of teacher post-training, does OPRD remain computationally competitive with GRPO?

  • Performance at a fixed training step. Could the authors report the accuracy at exactly step 150 in Table 1, rather than averaging accuracy across multiple training checkpoint steps?

  • Catastrophic forgetting. Does OPRD lead to performance degradation on other domains or general knowledge benchmarks, such as MMLU and MMLU-Pro? Evaluating potential forgetting would help clarify whether the gains come at the expense of broader model capabilities.

  • Thinking-mode evaluation. The training setup uses Qwen3 in non-thinking mode. Without additional training, does OPRD still provide improvements when the resulting model is evaluated with thinking mode enabled?

  • (Suggestion) Inconsistent color conventions in Figure 4. In Figure 4(a), tokens aligned with the teacher's direction (the (d) vector) are highlighted in green, whereas aligned results in Figure 4(b) are highlighted in red. Using consistent colors would improve readability.

9 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer Xr3j (1/3)

We thank the reviewer for the detailed feedback and for recognizing our extensive empirical evaluation, consistent performance gains, and theoretical justification for OPRD’s update direction. Below, we clarify the motivation and framing of OPRD and address each concern in turn. All revisions are highlighted in red\color{red}{\text{red}} in the revised manuscript.


[W1] Distillation Framing and Weak-to-Strong Generalization

OPRD follows a line of distillation methods that integrate teacher-learned information into reinforcement learning [1,2,3,4,5]. It transfers a weak teacher’s post-training policy shift rather than making the teacher’s final policy an optimization target.

Recent approaches integrate teacher guidance into RL [1,2,3,4]. Policy distillation also includes non-cloning mechanisms: Czarnecki et al. [5] study teacher-value-based reward shaping that changes learning dynamics without requiring policy cloning. OPRD uses the teacher’s post-training policy shift to rescale the student’s verifier-driven gradient. The teacher supplies the scaling axis, while the student’s reward gradient determines whether the update follows or opposes that axis. This gradient modulation is how the teacher’s learned information benefits the student; improved optimization and information transfer are therefore complementary descriptions of the same process. We clarify this perspective in Appendix M\color{indianred}{\text{Appendix M}} and explain the signal’s motivation in [W2]\color{indianred}{\text{[W2]}}.

We are not entirely sure what is intended by the second part of this concern and would appreciate your feedback on the clarification below. OPD’s tendency to plateau near the weak teacher’s performance in Figures 7-12\color{indianred}{\text{Figures 7-12}} is a natural outcome of its policy-matching objective. This limitation directly motivates OPRD: we aim to benefit from the weak teacher as a source of learned information, without making its policy the stronger student’s optimization target. This allows the student to pursue improvements beyond the teacher under its own reward objective. Similarly, as you noted, it is also natural that a stronger model can outperform a weaker one under GRPO. Our comparison, however, is between the same stronger student trained with and without weak-teacher guidance, under matched initialization, training data, and rollout budgets. OPRD improves learning speed and performance over this baseline by distilling from the weak teacher’s learned policy shift to guide the student’s updates (Table 1\color{indianred}{\text{Table 1}} and Figures 7-12\color{indianred}{\text{Figures 7-12}}). The teacher-checkpoint ablation in Table 3 (right)\color{indianred}{\text{Table 3 (right)}} further shows that student learning depends on the teacher’s post-training checkpoint.


[W2] Motivation for the Gradient Scaling Mechanism

The teacher’s policy shift captures reward-related information acquired during RL, which OPRD uses to accelerate student learning without direct policy matching.

At each student-visited prefix, OPRD computes the centered logit difference

Δt=C(zT−zTref)=C(log⁡πT−log⁡πTref),\Delta_t=C(z_T-z_T^{\mathrm{ref}})=C(\log\pi_T-\log\pi_T^{\mathrm{ref}}),

where the centering operator subtracts the vocabulary mean. Comparing the teacher with its own reference focuses the signal on changes acquired during post-training. As detailed in Appendix J\color{indianred}{\text{Appendix J}}, the teacher–student discrepancy underlying OPD also includes the mismatch between the teacher’s reference policy and the current student.

Under standard KL-regularized RL [6], the optimal policy satisfies

R(x,y)=βlog⁡π⋆(y∣x)πTref(y∣x)+c(x),R(x,y)=\beta\log\frac{\pi^\star(y\mid x)}{\pi_T^{\mathrm{ref}}(y\mid x)}+c(x),

where the additive term depends only on the prompt. This relationship motivates using the teacher’s post-training shift as a reward-related guidance signal.

With dt=Δt/∥Δt∥2d_t=\Delta_t/\lVert\Delta_t\rVert_2, OPRD applies

g~t=gt+λt(dt⊤gt)dt,λt≥0.\widetilde{g}_t=g_t+\lambda_t(d_t^{\top}g_t)d_t,\qquad \lambda_t\ge 0.

The teacher provides an informative axis, while the student’s reward-driven gradient determines whether to follow or oppose it. OPRD amplifies this gradient component while leaving the orthogonal component unchanged, exploiting the teacher’s learned information while retaining the student’s ability to improve beyond the teacher. Figure 3(a)\color{indianred}{\text{Figure 3(a)}} supports this choice empirically, showing faster and more sustained improvements than the OPD and OPSD guidance directions. Appendix J\color{indianred}{\text{Appendix J}} additionally evaluates these guidance constructions with the stronger teacher checkpoints used in the main experiments.

AuthorsOpenReview ↗

Rebuttal to Reviewer Xr3j (2/3)

[Q1] Teacher Training Recipe

We train teachers on the same datasets with nearly identical training recipes to those used for students, and have added these details to Appendix D\color{indianred}{\text{Appendix D}}.

Thank you for requesting this clarification. Teachers and students use the same training datasets and closely matched training configurations. We adjust the number of teacher training steps to obtain checkpoints at the intended performance levels for our weak-to-strong experiments. We have expanded Appendix D\color{indianred}{\text{Appendix D}} to clarify the teacher training configurations and checkpoint selection.


[Q2] Wall-Clock Efficiency

Appendix O reports an 11.9% per-step wall-clock overhead for OPRD. Even after accounting for this overhead, its faster learning can yield comparable performance in less student-training time.

The controlled timing comparison in Table 12\color{indianred}{\text{Table 12}} includes the additional teacher/reference forward passes:

MethodTotal (s/step)OverheadStudent rollout (s)Teacher/reference forward (s)Student update (s)
GRPO977.72—602.200.00304.00
OPD1048.00+7.2%610.7460.46305.38
OPRD1094.44+11.9%607.39108.80306.79

Times are per training step. Totals also include student log-probability evaluation and other overheads.

Rollout generation dominates runtime, while OPRD’s additional cost comes primarily from frozen teacher/reference forward passes; student update time increases only slightly. At this measured overhead, the reported 33–67% reduction in training steps would translate into approximately 25–63% less student-training wall-clock time. Following the standard distillation setup, we assume an already available post-trained teacher and ask how to transfer its learned capabilities to a stronger student; OPRD does not require training a weak teacher specifically for this purpose. Accordingly, our efficiency comparison concerns the downstream student-training stage and includes all teacher/reference forward passes. Teacher post-training is an upstream cost that has already been incurred independently of the choice of student-training method.


[Q3] Accuracy at Training Step 150

Table 1 uses checkpoint-averaged accuracy to summarize the learning curve, while Appendices F–H provide the detailed curves.

Thank you for the suggestion. We chose checkpoint-averaged accuracy for Table 1\color{indianred}{\text{Table 1}} to capture how quickly the student benefits from teacher guidance and improves during training. This provides a discrete summary of the learning curve, rather than measuring performance only at a single endpoint.

For completeness, we report the exact step-150 results below; the corresponding learning curves are already available in Appendices F-H\color{indianred}{\text{Appendices F-H}}. Evaluation follows the same protocols as Table 1\color{indianred}{\text{Table 1}}: Mean@16 for mathematics and Pass@1 for Reasoning Gym. All scores are percentages, and Avg. is the unweighted mean across the four benchmarks.

Mathematics — student checkpoints at step 150

MethodAIME’24AIME’25HMMT’25OlympiadBenchAvg.
GRPO60.4248.3326.4653.5547.19
OPD43.3335.4218.7549.9636.87
KDRL63.9652.0830.6353.9450.15
OPRD69.5861.0432.2954.3354.31

Reasoning Gym — student checkpoints at step 150

MethodKnights & KnavesQuantum LockString ManipulationCountdownAvg.
GRPO75.0063.4144.0059.0060.35
OPD57.5043.0435.0042.0044.38
KDRL78.0056.4947.5065.5061.87
OPRD84.0076.2647.5063.0067.69

OPRD also achieves the highest average performance at step 150 in both mathematics and Reasoning Gym.

AuthorsOpenReview ↗

Rebuttal to Reviewer Xr3j (3/3)

[Q4] General Capability Retention

Our evaluations indicate that OPRD’s improvements in the target domain do not come at the expense of general knowledge and broader capabilities.

OPRD rescales the student’s verifier-driven gradients to accelerate task-specific learning. To assess whether these gains compromise broader capabilities, we evaluated the math- and Knights & Knaves-trained checkpoints on general-knowledge benchmarks, with an additional instruction-following evaluation for the math setting. We use instruction-tuned Qwen3 models for math and their Base variants for Knights & Knaves. Evaluations follow standard prompting protocols for each benchmark, with greedy decoding (temperature = 0) across all checkpoints. Results for GRPO, OPD, and OPRD are reported at training step 150, with all scores expressed as percentages. Further evaluation details and results are provided in Appendix S (Table 18)\color{indianred}{\text{Appendix S (Table 18)}}.

Math-trained checkpoints

Model / MethodMMLU-ProTriviaQAIFEval strict
Teacher 4B · step 7557.9542.8679.67
Student 8B · step 062.0353.9381.70
GRPO63.0953.8780.22
OPD61.8454.0081.89
OPRD63.2654.0381.33

Knights & Knaves-trained checkpoints

Model / MethodMMLU-ProTriviaQA
Teacher 4B-Base · step 10552.1750.44
Student 8B-Base · step 056.3460.13
GRPO57.7459.99
OPD55.9459.89
OPRD58.9160.04

Comparisons with GRPO and the initial student indicate that OPRD’s target-domain gains are compatible with preserving broader general capabilities. In contrast, OPD yields lower MMLU-Pro scores, suggesting a potential cost of constraining the stronger student to a weaker teacher’s policy through direct policy matching. These results further support OPRD’s ability to benefit from the teacher’s learned information while retaining the student’s broader capabilities and improving beyond the weak teacher.


[Q5] Thinking-Mode Evaluation

OPRD also improves thinking-mode performance over the initial student without additional training.

Thank you for pointing this out. Although we trained in non-thinking mode to reduce computational cost, we evaluated the resulting checkpoints with thinking mode enabled, without further training. We evaluate on all 30 AIME’24 problems using temperature 0.6, top-p 0.95, top-k 20, and a maximum generation length of 38,912 tokens. Further evaluation details are provided in Appendix T (Table 19)\color{indianred}{\text{Appendix T (Table 19)}}.

Model / MethodAIME’24 Mean@4 (%)
Teacher75.83
Student72.50
+ GRPO78.33
+ OPD70.83
+ OPRD79.17

OPRD improves over the initial student and modestly outperforms GRPO. These results suggest that OPRD strengthens the student’s underlying reasoning capabilities rather than merely reproducing the weaker teacher’s policy. In contrast, OPD’s lower thinking-mode accuracy points to a potential cost of direct policy matching.


[Q6] Color Consistency in Figure 4

We have unified the color conventions across Figure 4(a) and Figure 4(b) and clarified the caption.

Thank you for the suggestion. Figure 4(a)\color{indianred}{\text{Figure 4(a)}} originally used green to indicate higher cosine similarity with the teacher’s direction. In the revision, we have reversed its color scale so that alignment with the student’s original update direction is consistently highlighted in green across both panels.


References

[1] Xu et al. “KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning.” arXiv, 2025.

[2] Ren et al. “Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO.” ICML, 2026.

[3] Wang et al. “Distilled Reinforcement Learning for LLM Post-training.” arXiv, 2026.

[4] Zhang et al. “Reinforcement-aware Knowledge Distillation for LLM Reasoning.” ICML, 2026.

[5] Czarnecki et al. “Distilling Policy Distillation.” AISTATS, 2019.

[6] Rafailov et al. “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” NeurIPS, 2023.

Reviewer_SBTGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work explores using post-training gains from weaker models to accelerate a stronger student, without optimizing toward the weak teacher policy. The authors introduce OPRD. It builds a direction from the teacher’s policy shift relative to its reference on student on-policy rollouts. It scales the directional component of the student’s verifier-driven policy gradient to accelerate the student’s own optimization rather than redirect it. Experiments include cross-scale transfer, multi-teacher aggregation and strong-to-weak transfer. Comparisons against GRPO, OPD, KDRL and other baselines show OPRD reaches higher performance with fewer updates.

Critical strengths
  1. The method offers a novel viewpoint. OPRD does not set the weak teacher as the optimization target. It leverages teacher–reference shift to scale directions in the student’s verifier gradient. This uses weak-model post-training updates as an optimization prior without constraining the student to the teacher’s final policy.
  2. Empirical results are strong with solid analysis. Table 1 reports an 8–11 point gain over top baselines including KDRL. Learning curves show OPD plateaus near teacher performance, whereas OPRD continues improving.
Critical weaknesses
  1. Results are highly sensitive to reference selection. Figure 17–18 demonstrate reference choice can drastically change outcomes. The paper lacks predefined, automatic rules to pick a reference checkpoint.
  2. Almost all experiments use Qwen3 and verifiable reward tasks. The claimed successive model generation is mostly same-family cross-scale transfer. Evidence is insufficient to validate true cross-generation transfer.
Questions for authors
  1. What procedure selects the reference checkpoint? Is the reference fixed before seeing downstream student performance? Please provide additional reference sensitivity results.
  2. OPRD uses sampled tokens plus top-10 tokens while other baselines only use sampled tokens. Do the student’s top 10 highest-probability tokens contribute extra signal? Please add support-size ablation.
  3. OPRD requires extra forward passes of the reference model. With this overhead, do fewer updates still yield better wall-clock runtime?
6 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer SBTG (1/4)

We thank the reviewer for the constructive feedback and for recognizing OPRD’s novel use of weak-model post-training gains, strong empirical results, and supporting analyses. We address each concern and question below. All revisions are highlighted in red\color{red}{\text{red}} in the revised manuscript.


[W1 / Q1] Reference Selection and Length Bias

OPRD uses the teacher’s pre-RL (step-0) checkpoint as a predefined default reference, fixed before student training for every result table. With this default, OPRD achieves the best average score in all six settings of Tables 1, 9, and 10. Appendix K.2 examines two stress-test tasks in which the teacher's shift carries a large response-length component.

Our standard setup already has a clear, fixed selection rule: use the teacher’s pre-RL (step-0) checkpoint before student training. This choice works well across our experiments: with it, OPRD finishes at or above GRPO at the final checkpoint on 21 of the 23 benchmarks in Tables 1, 9, and 10\color{indianred}{\text{Tables 1, 9, and 10}}. It also follows the common convention in KL-regularized RL of fixing the reference to the initial policy [1]. Under this formulation, the optimal policy satisfies

R(x,y)=βlog⁡π⋆(y∣x)πTref(y∣x)+c(x),R(x,y) = \beta \log \frac{\pi^{\star}(y \mid x)}{\pi_T^{\mathrm{ref}}(y \mid x)} + c(x),

where c(x)c(x) depends only on the prompt. This relationship motivates interpreting the teacher’s shift from its pre-RL policy as an implicit-reward signal. OPRD extracts this shift at each student-visited prefix as

= C(\log \pi_T - \log \pi_T^{\mathrm{ref}}),$$ where $C$ subtracts the vocabulary mean. The step-0 reference thus provides a natural baseline for capturing changes acquired during the teacher’s RL post-training. $\color{indianred}{\text{Figures 18(b–c) and 19}}$ (in the revised manuscript) examine a case where the teacher’s own post-training introduces a structured change: on Color Cube and Binary Matrix, which $\color{indianred}{\text{Appendix F.2}}$ adds as stress tests for length effects, the teacher’s response length drops sharply during RL. The step-0 shift then carries a large length component, and OPRD, like any teacher-guided method, inherits it. As an exploratory check, $\color{indianred}{\text{Appendix F.2}}$ keeps the same teacher and moves only the reference past the length drop. To provide the requested sensitivity results, we compare the two reference checkpoints under the same experimental settings: | Task | Best baseline | OPRD (step-0 ref., default) | OPRD (later ref., exploratory) | |:--|--:|--:|--:| | Color Cube | 49.50 (KDRL) | 49.20 | 72.40 | | Binary Matrix | 62.10 (OPD) | 62.40 | 66.80 | With the step-0 default, OPRD keeps its early acceleration and matches the strongest baseline on average. Moving only the reference past the teacher’s length drop raises the final accuracy from 52.5% and 81.5% to 89.5% and 96.0%, above all baselines. The outcome is therefore driven by what the teacher’s shift encodes. Regarding selection timing: the step-0 reference is fixed before student training for every result table, and $\color{indianred}{\text{Table 9}}$ and $\color{indianred}{\text{Figure 10}}$ now use it on every task. The later references appear only in the exploratory analysis above, where they are placed directly after the length drop in the teacher's own training trajectory.
AuthorsOpenReview ↗

Rebuttal to Reviewer SBTG (2/4)


[W2] Cross-Generation Transfer

OPRD is designed for weak-to-strong distillation from smaller to larger models. To examine transfer beyond a shared model family, we have added an experiment in which a Qwen3 teacher guides a Ministral 3 student [2]. OPRD learns substantially faster than all baselines and achieves the highest checkpoint-averaged score.

Our core contribution is a method that transfers post-training gains from a smaller teacher to a larger student without limiting the student to the teacher's capabilities. Successive model generations and multi-domain consolidation are two practical scenarios that motivate this setting (Section 1\color{indianred}{\text{Section 1}}), and our main experiments study it through transfer across model scales within the same family.

To go beyond a shared family, we pair a Qwen3-4B-Base teacher, taken from step 90 of GRPO training on Knights & Knaves, with a Ministral-3-8B-Base student. Both use decoder-only Transformer language backbones, but differ in depth and FFN dimensions. Ministral 3 is derived from Mistral Small 3.1 through pruning and distillation and uses a 131K-token vocabulary, whereas Qwen3 uses a ~151K-token vocabulary. Unlike our original Qwen3 pairs, this teacher–student pair has different pretraining lineages and tokenizers. OPRD uses the teacher's pre-RL checkpoint as its reference. Because the vocabularies differ, OPD and KDRL use GOLD [3] for cross-tokenizer distillation, while OPRD uses it only to construct the teacher-shift direction.

Policy0153045607590Avg.
+ GRPO2.507.0017.5029.5041.0043.5044.0030.42
+ OPD2.5011.0011.5012.0019.0029.500.00†13.83
+ KDRL2.5010.0015.0021.0023.0050.5071.5031.83
+ OPRD2.5015.0032.5079.0073.0055.0073.0054.58

Knights & Knaves Pass@1 (%); update 0 is the shared initial student, and Avg. averages updates 15–90. (†No response contained the required boxed answer).

The fixed teacher achieves 44.50% Pass@1, while the initial student achieves 2.50%. OPRD reaches teacher-level performance within 45 updates, compared with 75 for KDRL and about 90 for GRPO, i.e., 40–50% fewer updates, consistent with the 33–67% reduction over GRPO in our main experiments. At update 45, it already reaches 79.0%, compared with 29.5% for GRPO and 21.0% for KDRL, and it remains above the teacher at every subsequent checkpoint. It also achieves the highest six-checkpoint average, 54.58% versus 31.83% for KDRL. These results show that OPRD's benefits extend to a teacher–student pair with a different pretraining lineage and tokenizer, beyond the same-family setting of our main experiments. In addition, the revised manuscript evaluates Qwen2.5-VL vision-language agents on multi-turn visual tool use, where OPRD also outperforms GRPO, OPD, and KDRL (Table 10\color{indianred}{\text{Table 10}} and Figure 12\color{indianred}{\text{Figure 12}}). Detailed settings and learning curves are provided in Appendix P\color{indianred}{\text{Appendix P}}, and results beyond verifiable rewards are discussed in [W3]\color{indianred}{\text{[W3]}}.


[W3] Beyond Verifiable Rewards

We have expanded the evaluation beyond verifiable-reward tasks to open-ended instruction following. On UltraFeedback [4], a Qwen3-4B teacher guides a Qwen3-8B student. All methods share the training prompts, student initialization, and 150-update budget.

Student RL uses a frozen Skywork reward model [5] rather than a rule-based verifier checks. We report mean reward on held-out UltraFeedback prompts and AlpacaEval [6] pairwise win rate, using a separate Prometheus judge [7] against GPT-4 Turbo responses, both raw and length-adjusted.

PolicyUF Validation RewardAlpacaEval Win Rate (%)Length-adjusted Win Rate (%)
Teacher29.5969.30–
Student26.0361.47–
+ GRPO29.7568.0264.10
+ OPD29.4567.7757.55
+ KDRL30.5369.6263.55
+ OPRD31.7180.7372.35

Raw and length-adjusted scores in each row use the same evaluation outputs.

OPRD outperforms all three baselines on all metrics, with an 11.1-percentage-point gain over KDRL in checkpoint-averaged AlpacaEval win rate. After length adjustment, OPRD still leads every baseline by at least 8.25 points (72.35% vs. 64.10% for GRPO), confirming that its advantage holds at equal response lengths. For each checkpoint, we fit a logistic model of the judge’s preference on the normalized length difference and report the prediction at equal response lengths. The settings, results, and learning curves, including the length-adjusted evaluation, are highlighted in red in Appendix Q\color{indianred}{\text{Appendix Q}} (Tables 15–17\color{indianred}{\text{Tables 15–17}} and Figure 21\color{indianred}{\text{Figure 21}}).

AuthorsOpenReview ↗

Rebuttal to Reviewer SBTG (3/4)


[Q2] Token Support Size Ablation

Prior works [8,9] report comparable performance between sampled-token and top-KK OPD, and our own runs confirm this. For OPRD, support size has only a modest effect, and even with its smallest support (K=5K=5) it exceeds the best OPD variant by more than 10 points.

For OPD, let pp and qq denote the student and teacher distributions at a student-generated prefix. Sampled-token OPD uses the per-token reward

rsample(a)=log⁡q(a)p(a),a∼p,r_{\mathrm{sample}}(a) = \log \frac{q(a)}{p(a)}, \qquad a \sim p,

whereas top-KK OPD with a tail bin minimizes

LK=∑v∈SK∪{tail}p(v)log⁡p(v)q(v).L_K = \sum_{v \in S_K \cup \{\mathrm{tail}\}} p(v) \log \frac{p(v)}{q(v)}.

Here, SKS_K contains the student’s top-KK tokens, and the tail bin collects the remaining probability mass, p(tail)=1−∑v∈SKp(v)p(\mathrm{tail}) = 1 - \sum_{v \in S_K} p(v) (likewise for qq). Our repeated preliminary tests had already shown negligible differences across OPD support sizes. But we ran again both on Knights & Knaves:

OPD support / StepAvg.153045607590105120135150
Sampled Token52.6030.048.051.054.059.061.056.054.555.057.5
Top-1054.8537.051.555.056.553.061.553.562.059.059.5
Top-10054.3040.059.057.554.052.555.560.054.054.556.0

Avg. is over the ten checkpoints from step 15 to 150 (same for the OPRD table below). Consistent with prior works [8,9], larger supports change OPD’s average by at most 2.25 points, which may simply reflect stochastic variability.

However, OPRD uses token support differently: the additional tokens provide components of the teacher’s policy-shift direction used to rescale the student’s reward-driven gradient. Support size therefore affects the guidance direction itself. We thank the reviewer for suggesting this experiment and now evaluate K=5K=5, K=10K=10, and K=100K=100 on Knights & Knaves.

OPRD Support / StepAvg.153045607590105120135150
Top-565.2520.557.053.059.070.073.579.578.579.582.0
Top-1068.0518.060.057.565.075.577.082.580.580.584.0
Top-10068.9520.060.059.074.573.576.583.579.582.081.0
<!-- All scores are accuracy percentages. Each configuration also includes the sampled token. Average is computed over the ten checkpoints from step 15 to step 150. -->

For OPRD, support size has a modest effect: its average varies by at most 3.70 points across K=5K=5, 1010, and 100100 and saturates beyond K=10K=10 (68.05 vs. 68.95), so we retain K=10K=10 as our default. Even with its smallest support (K=5K=5, 65.25), OPRD exceeds the best OPD variant (54.85) by more than 10 points, so support size does not explain OPRD’s advantage. Both the OPD and OPRD results have been added to Appendix R\color{indianred}{\text{Appendix R}} (Tables 18\color{indianred}{\text{Tables 18}} and 19\color{indianred}{\text{19}}).


[Q3] Wall-Clock Efficiency

Appendix O reports an 11.9% per-step wall-clock overhead for OPRD. Its faster learning can still yield better performance in less student-training time after accounting for this overhead.

Table 12\color{indianred}{\text{Table 12}} already reports total training-step time and the main runtime components:

MethodTotal (s/step)OverheadStudent rollout (s)Teacher/reference forward (s)Student update (s)
GRPO977.72—602.200.00304.00
OPD1048.00+7.2%610.7460.46305.38
OPRD1094.44+11.9%607.39108.80306.79

All methods use the same workload: 64 prompts, 8 rollouts per prompt, and 16,384 tokens per response. Totals include all training operations.

Student rollout generation dominates runtime, while OPRD’s additional cost comes primarily from frozen teacher/reference forward passes. The student update, including projection and gradient scaling, increases by only 2.79 seconds relative to GRPO. At this measured overhead, 33–67% fewer training steps to achieve comparable performance would translate into approximately 25–63% less student-training time, assuming comparable response lengths.

Reviewer_GgEXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes On-Policy Reverse Distillation (OPRD), a method that leverages trajectories generated by the student model to identify the optimization direction of the teacher policy relative to a reference policy. It then amplifies the components of the student's verifier-guided policy gradients that align with this direction. By selectively rescaling only verifier-supported gradient updates without introducing an additional optimization objective, OPRD preserves the stationary points of the original policy optimization while improving learning efficiency, enabling the student to reach or even surpass the teacher's performance more rapidly. The central problem addressed by this work is how to prevent the student from being constrained by a weaker teacher's limitations, allowing it to learn more efficiently and ultimately outperform the teacher.

Critical strengths

From “imitating the weak teacher's distribution” to “leveraging the direction of the teacher's policy change.” The student continues to optimize its own reward objective, while the teacher provides only a directional signal to modulate the student's gradients. Only the aligned gradient component is amplified. Specifically, given the student gradient (g) and the teacher's policy-change direction (d), the method preserves the component orthogonal to (d) while scaling up the component aligned with (d). The paper also provides a theoretical justification for the preservation of stationary points in logit space.

Critical weaknesses

The main empirical evidence is concentrated on the Qwen3 model family and mathematical reasoning, logical reasoning, and coding benchmarks. The authors acknowledge that preliminary cross-generation experiments involving Qwen2.5→Qwen3 were excluded from the main evaluation due to the lack of suitable teacher checkpoints. Although the experiments cover model sizes ranging from 0.6B to 14B and multiple tasks, the results remain insufficient to establish the generalizability of OPRD across different model architectures, open-ended generation tasks, and realistic long-horizon agent scenarios.

Questions for authors

Most experiments use models from the Qwen3 family, while the preliminary Qwen2.5-to-Qwen3 experiments were excluded because suitable teacher checkpoints were unavailable. Could the authors provide evidence or further discussion about whether OPRD transfers across model families or architectures? In particular, does the method require the teacher and student to share a tokenizer or vocabulary, and how would it handle mismatches? The evaluation focuses on mathematical reasoning, logical reasoning, and code generation. Could the authors clarify whether OPRD requires a verifier-driven reward signal and, if so, how this requirement affects its applicability to open-ended generation or long-horizon agent tasks? Any evidence on such settings would help establish the method’s generality.

5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer GgEX (1/3)

We thank the reviewer for the detailed feedback and for recognizing OPRD’s use of teacher policy shifts to guide student optimization, together with its theoretical justification for preserving stationary points in logit space. We address each concern and question below. All revisions are highlighted in red\color{red}{\text{red}} in the revised manuscript.


[W1] Cross-Family Generalization

We have added a cross-family experiment in which a Qwen3 teacher guides a Ministral 3 student. OPRD learns substantially faster than all baselines and achieves the highest checkpoint-averaged score.

We pair a Qwen3-4B-Base teacher, taken from step 90 of GRPO training on Knights & Knaves, with a Ministral-3-8B-Base student [1]. Both use decoder-only Transformer language backbones, but differ in depth and FFN dimensions. Ministral 3 is derived from Mistral Small 3.1 through pruning and distillation and uses a 131K-token vocabulary, whereas Qwen3 uses a ~151K-token vocabulary. This setting therefore extends our evaluation beyond same-family Qwen3 pairs to models with different architectural configurations, pretraining lineages, and vocabularies. OPRD uses the teacher's pre-RL checkpoint as its reference. Because the vocabularies differ, OPD and KDRL use GOLD [2] for cross-tokenizer distillation, while OPRD uses it to construct the teacher-shift direction (see [W2]\color{indianred}{\text{[W2]}}).

Policy0153045607590Avg.
+ GRPO2.507.0017.5029.5041.0043.5044.0030.42
+ OPD2.5011.0011.5012.0019.0029.500.00†13.83
+ KDRL2.5010.0015.0021.0023.0050.5071.5031.83
+ OPRD2.5015.0032.5079.0073.0055.0073.0054.58

Knights & Knaves Pass@1 (%); update 0 is the shared initial student, and Avg. averages updates 15–90. (†No response contained the required boxed answer).

The fixed teacher achieves 44.50% Pass@1, while the initial student achieves 2.50%. OPRD outperforms teacher-level performance within 45 updates, compared with 75 for KDRL and about 90 for GRPO, i.e., 40–50% fewer updates, consistent with the 33–67% reduction over GRPO in our main experiments. At update 45, it already reaches 79.0%, compared with 29.5% for GRPO and 21.0% for KDRL, and it remains above the teacher at every subsequent checkpoint. It also achieves the highest six-checkpoint average, 54.58% versus 31.83% for KDRL. These results show that OPRD's benefits extend to a teacher–student pair with a different pretraining lineage and tokenizer. Combined with the vision-language agent results in [W4]\color{indianred}{\text{[W4]}}, OPRD is now evaluated across a different model family and tokenizer, a multimodal architecture, and multi-turn tool use. Detailed settings and learning curves are provided in Appendix P\color{indianred}{\text{Appendix P}}.

AuthorsOpenReview ↗

Rebuttal to Reviewer GgEx (2/3)


[W2] Cross-Tokenizer Compatibility

OPRD does not require the teacher and student to share a tokenizer. A tokenizer mismatch only affects how the teacher-shift direction is expressed in the student's logit space, which we handle with an existing cross-tokenizer alignment method.

The teacher and its reference always share a tokenizer, since the reference is the teacher's own pre-RL checkpoint. The teacher's policy shift is therefore well-defined regardless of the student, and a mismatch arises only when expressing this shift at the student's tokens. Where student and teacher tokens correspond one-to-one, we use the teacher–reference logit difference directly. For the remaining spans, we construct the direction in the student's logit space using GOLD [2], from the difference between the GOLD loss gradients computed against the reference and against the teacher. Both directions then enter the same OPRD gradient modulation.

OPD and KDRL use the same GOLD alignment but optimize it as a distillation loss. OPRD instead keeps the reward objective as its only training objective and uses GOLD only to construct the guidance direction. Because this direction only determines which component of the reward gradient is amplified, the reward-driven update itself is always retained (Eq. 2.5), whereas OPD optimizes the aligned teacher distribution directly. Under the shared alignment, OPD loses the required answer format at update 90, whereas OPRD learns substantially faster than all baselines (see above Table\color{indianred}{\text{above Table}} in [W1]\color{indianred}{\text{[W1]}}), indicating that its gains come from how the teacher signal is used rather than from the alignment itself.

Because OPRD only requires a direction in the student's logit space, improvements in cross-tokenizer alignment are orthogonal to OPRD and can be adopted without changing its gradient modulation. Beyond GOLD, several other cross-tokenizer distillation methods have been proposed, including recent on-policy variants [3,4,5,6], providing alternatives for this alignment step. The full construction is provided in Appendix P\color{indianred}{\text{Appendix P}}.


[W3] Non-Verifiable Open-Ended Generation

OPRD does not require a rule-based verifier and also improves open-ended instruction following. Following your suggestion, we have added experiments on UltraFeedback [7], a diverse set of open-ended instructions without reference answers.

A Qwen3-4B teacher guides a Qwen3-8B student, with all student-training methods sharing the training prompts, student initialization, and 150-update budget. A Skywork reward model [8] provides the RL rewards and the UF validation reward; AlpacaEval [9] uses a separate Prometheus judge [10] against GPT-4 Turbo responses and serves as the independent quality measure, for which we also report a length-adjusted win rate.

PolicyUF Validation RewardAlpacaEval Win Rate (%)Length-adjusted Win Rate (%)
Teacher29.5969.30–
Student26.0361.47–
+ GRPO29.7568.0264.10
+ OPD29.4567.7757.55
+ KDRL30.5369.6263.55
+ OPRD31.7180.7372.35

Teacher and initial Student use fixed checkpoints, and trained rows average updates 30, 60, 90, 120, and 150. Raw and length-adjusted scores in each row use the same evaluation outputs.

OPRD achieves the highest score on all three metrics and surpasses the teacher on both unadjusted metrics. After length adjustment, OPRD still leads every baseline by at least 8.25 points (72.35% vs. 64.10% for GRPO), confirming that its advantage holds at equal response lengths. These results extend OPRD's benefits to open-ended instruction following. Detailed settings, results, and learning curves are provided in Appendix Q\color{indianred}{\text{Appendix Q}} (Tables 15–17\color{indianred}{\text{Tables 15–17}} and Figure 21\color{indianred}{\text{Figure 21}}).

AuthorsOpenReview ↗

Rebuttal to Reviewer GgEx (3/3)


[W4] Long-horizon Agentic Task

Our revised experiments extend OPRD beyond Qwen3 to Qwen2.5-VL vision-language agents, demonstrating improvements over GRPO, OPD, and KDRL in multi-turn visual tool use.

We evaluate OPRD on ViLaSR RL using a Qwen2.5-VL-3B teacher and a Qwen2.5-VL-7B student. These vision-language agents perform sequential tool actions and use the resulting observations to guide subsequent decisions, with up to 12 tool calls and 42 images per trajectory. This extends our evaluation beyond text-only reasoning to multimodal agent interactions. Detailed training and evaluation settings are provided in Table 5\color{indianred}{\text{Table 5}} and Table 8\color{indianred}{\text{Table 8}}, with results and learning curves in Table 10\color{indianred}{\text{Table 10}} and Figure 12\color{indianred}{\text{Figure 12}}.

Model / MethodAvg.MazeGPT4SceneSR-91kSpatial VQA
Teacher67.7485.9464.6750.3869.99
Student54.8349.4563.6638.8667.33
+ GRPO64.3577.2764.9146.5168.73
+ OPD65.5781.4864.7447.0269.04
+ KDRL64.6777.8665.6546.7468.43
+ OPRD67.5082.5867.4850.6969.27

Scores are Mean@1 rule scores multiplied by 100, including partial credit for numeric answers. Trained rows average updates 45, 90, 135, 180, and 225.

OPRD achieves the highest checkpoint-averaged performance among trained methods on all four task families. At update 225, it also surpasses the teacher on every family, reaching an overall score of 70.03 versus the teacher’s 67.74 (Figure 12\color{indianred}{\text{Figure 12}}). These results support OPRD’s applicability to multi-turn vision-language agents.


References

[1] Liu et al. “Ministral 3.” arXiv preprint arXiv:2601.08584, 2026.

[2] Patiño et al. "Unlocking On-Policy Distillation for Any Model Family." Hugging Face, 2025.

[3] Boizard et al. "Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs." TMLR, 2025.

[4] Sun et al. "SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation." arXiv, 2026.

[5] He et al. "SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning." arXiv, 2026.

[6] Wang et al. "Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization." arXiv, 2026.

[7] Cui et al. "UltraFeedback: Boosting Language Models with Scaled AI Feedback." ICML, 2024.

[8] Liu et al. “Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy.” arXiv, 2025.

[9] Li et al. “AlpacaEval: An Automatic Evaluator of Instruction-Following Models.” GitHub, 2023.

[10] Kim et al. “Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models.” EMNLP, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.