- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Motivated by the observation that supervision from a weaker model can improve a stronger student's performance, this paper proposes On-Policy Reverse Distillation (OPRD), which leverages policy shifts learned by a weak teacher to scale gradient updates during student training. Experiments in weak-to-strong transfer and multi-domain consolidation demonstrate that OPRD outperforms GRPO, OPD, and KDRL baselines, while also surpassing the weak teacher's performance.
Critical strengths
-
The paper provides comprehensive comparisons with diverse baselines and extensive ablation studies across different settings, demonstrating consistent improvements. These results strengthen the empirical credibility of the proposed method.
-
Despite modifying the gradient update rule of GRPO, the paper theoretically establishes that the resulting update direction has a positive directional derivative with respect to the optimization objective.
Critical weaknesses
- (Major) Questionable framing as distillation.
I find it difficult to characterize the proposed method as a distillation approach for the following reasons.
First, OPRD is fundamentally based on GRPO, with its primary modification amplifying gradient updates according to the teacher's policy shift, rather than directly transferring the teacher's reasoning trajectories or output distributions. Moreover, since this policy shift is obtained through GRPO post-training of the weak teacher, the method could alternatively be interpreted as extracting useful optimization signals from a separately trained "surrogate model". From this perspective, the distinction between OPRD and a teacher-guided variant of GRPO remains unclear.
Second, the experimental evidence does not sufficiently establish the role of weak-to-strong generalization in motivating the proposed approach. In Figures 7–11, OPD, which directly distills from the weak teacher, often fails to surpass the teacher's performance or achieves only comparable results. Furthermore, it is not particularly surprising that GRPO applied directly to a stronger student can outperform GRPO applied to a weaker model. These observations raise the question of whether the improvements of OPRD should be attributed to weak-to-strong knowledge transfer or simply to more effective optimization of the stronger student.
Overall, I believe that framing the proposed method primarily as distillation weakens the motivation and obscures its actual contribution.
- Insufficient motivation for the proposed gradient scaling mechanism.
The paper discusses the limitations of directly matching the weak teacher's reasoning trajectories and provides an ablation study in Figure 3(a) comparing different guidance directions. However, the motivation for specifically amplifying student gradient updates according to the teacher's policy shift remains insufficiently explained. In particular, it is unclear why this policy shift should provide a more effective optimization signal than other forms of teacher guidance.
Questions for authors
-
Teacher training recipe. What are the exact training configurations used for the weak teachers? Are the training recipes and datasets identical to those used for student training?
-
Computational efficiency. The introduction states that OPRD achieves performance comparable to GRPO with 33–67% fewer training steps. Does this improvement translate into lower computational cost in terms of FLOPs or wall-clock time? In particular, when accounting for the additional cost of teacher post-training, does OPRD remain computationally competitive with GRPO?
-
Performance at a fixed training step. Could the authors report the accuracy at exactly step 150 in Table 1, rather than averaging accuracy across multiple training checkpoint steps?
-
Catastrophic forgetting. Does OPRD lead to performance degradation on other domains or general knowledge benchmarks, such as MMLU and MMLU-Pro? Evaluating potential forgetting would help clarify whether the gains come at the expense of broader model capabilities.
-
Thinking-mode evaluation. The training setup uses Qwen3 in non-thinking mode. Without additional training, does OPRD still provide improvements when the resulting model is evaluated with thinking mode enabled?
-
(Suggestion) Inconsistent color conventions in Figure 4. In Figure 4(a), tokens aligned with the teacher's direction (the (d) vector) are highlighted in green, whereas aligned results in Figure 4(b) are highlighted in red. Using consistent colors would improve readability.
9 comments
Rebuttal to Reviewer Xr3j (2/3)
[Q1] Teacher Training Recipe
We train teachers on the same datasets with nearly identical training recipes to those used for students, and have added these details to .
Thank you for requesting this clarification. Teachers and students use the same training datasets and closely matched training configurations. We adjust the number of teacher training steps to obtain checkpoints at the intended performance levels for our weak-to-strong experiments. We have expanded to clarify the teacher training configurations and checkpoint selection.
[Q2] Wall-Clock Efficiency
Appendix O reports an 11.9% per-step wall-clock overhead for OPRD. Even after accounting for this overhead, its faster learning can yield comparable performance in less student-training time.
The controlled timing comparison in includes the additional teacher/reference forward passes:
| Method | Total (s/step) | Overhead | Student rollout (s) | Teacher/reference forward (s) | Student update (s) |
|---|---|---|---|---|---|
| GRPO | 977.72 | — | 602.20 | 0.00 | 304.00 |
| OPD | 1048.00 | +7.2% | 610.74 | 60.46 | 305.38 |
| OPRD | 1094.44 | +11.9% | 607.39 | 108.80 | 306.79 |
Times are per training step. Totals also include student log-probability evaluation and other overheads.
Rollout generation dominates runtime, while OPRD’s additional cost comes primarily from frozen teacher/reference forward passes; student update time increases only slightly. At this measured overhead, the reported 33–67% reduction in training steps would translate into approximately 25–63% less student-training wall-clock time. Following the standard distillation setup, we assume an already available post-trained teacher and ask how to transfer its learned capabilities to a stronger student; OPRD does not require training a weak teacher specifically for this purpose. Accordingly, our efficiency comparison concerns the downstream student-training stage and includes all teacher/reference forward passes. Teacher post-training is an upstream cost that has already been incurred independently of the choice of student-training method.
[Q3] Accuracy at Training Step 150
Table 1 uses checkpoint-averaged accuracy to summarize the learning curve, while Appendices F–H provide the detailed curves.
Thank you for the suggestion. We chose checkpoint-averaged accuracy for to capture how quickly the student benefits from teacher guidance and improves during training. This provides a discrete summary of the learning curve, rather than measuring performance only at a single endpoint.
For completeness, we report the exact step-150 results below; the corresponding learning curves are already available in . Evaluation follows the same protocols as : Mean@16 for mathematics and Pass@1 for Reasoning Gym. All scores are percentages, and Avg. is the unweighted mean across the four benchmarks.
Mathematics — student checkpoints at step 150
| Method | AIME’24 | AIME’25 | HMMT’25 | OlympiadBench | Avg. |
|---|---|---|---|---|---|
| GRPO | 60.42 | 48.33 | 26.46 | 53.55 | 47.19 |
| OPD | 43.33 | 35.42 | 18.75 | 49.96 | 36.87 |
| KDRL | 63.96 | 52.08 | 30.63 | 53.94 | 50.15 |
| OPRD | 69.58 | 61.04 | 32.29 | 54.33 | 54.31 |
Reasoning Gym — student checkpoints at step 150
| Method | Knights & Knaves | Quantum Lock | String Manipulation | Countdown | Avg. |
|---|---|---|---|---|---|
| GRPO | 75.00 | 63.41 | 44.00 | 59.00 | 60.35 |
| OPD | 57.50 | 43.04 | 35.00 | 42.00 | 44.38 |
| KDRL | 78.00 | 56.49 | 47.50 | 65.50 | 61.87 |
| OPRD | 84.00 | 76.26 | 47.50 | 63.00 | 67.69 |
OPRD also achieves the highest average performance at step 150 in both mathematics and Reasoning Gym.
Rebuttal to Reviewer Xr3j (3/3)
[Q4] General Capability Retention
Our evaluations indicate that OPRD’s improvements in the target domain do not come at the expense of general knowledge and broader capabilities.
OPRD rescales the student’s verifier-driven gradients to accelerate task-specific learning. To assess whether these gains compromise broader capabilities, we evaluated the math- and Knights & Knaves-trained checkpoints on general-knowledge benchmarks, with an additional instruction-following evaluation for the math setting. We use instruction-tuned Qwen3 models for math and their Base variants for Knights & Knaves. Evaluations follow standard prompting protocols for each benchmark, with greedy decoding (temperature = 0) across all checkpoints. Results for GRPO, OPD, and OPRD are reported at training step 150, with all scores expressed as percentages. Further evaluation details and results are provided in .
Math-trained checkpoints
| Model / Method | MMLU-Pro | TriviaQA | IFEval strict |
|---|---|---|---|
| Teacher 4B · step 75 | 57.95 | 42.86 | 79.67 |
| Student 8B · step 0 | 62.03 | 53.93 | 81.70 |
| GRPO | 63.09 | 53.87 | 80.22 |
| OPD | 61.84 | 54.00 | 81.89 |
| OPRD | 63.26 | 54.03 | 81.33 |
Knights & Knaves-trained checkpoints
| Model / Method | MMLU-Pro | TriviaQA |
|---|---|---|
| Teacher 4B-Base · step 105 | 52.17 | 50.44 |
| Student 8B-Base · step 0 | 56.34 | 60.13 |
| GRPO | 57.74 | 59.99 |
| OPD | 55.94 | 59.89 |
| OPRD | 58.91 | 60.04 |
Comparisons with GRPO and the initial student indicate that OPRD’s target-domain gains are compatible with preserving broader general capabilities. In contrast, OPD yields lower MMLU-Pro scores, suggesting a potential cost of constraining the stronger student to a weaker teacher’s policy through direct policy matching. These results further support OPRD’s ability to benefit from the teacher’s learned information while retaining the student’s broader capabilities and improving beyond the weak teacher.
[Q5] Thinking-Mode Evaluation
OPRD also improves thinking-mode performance over the initial student without additional training.
Thank you for pointing this out. Although we trained in non-thinking mode to reduce computational cost, we evaluated the resulting checkpoints with thinking mode enabled, without further training. We evaluate on all 30 AIME’24 problems using temperature 0.6, top-p 0.95, top-k 20, and a maximum generation length of 38,912 tokens. Further evaluation details are provided in .
| Model / Method | AIME’24 Mean@4 (%) |
|---|---|
| Teacher | 75.83 |
| Student | 72.50 |
| + GRPO | 78.33 |
| + OPD | 70.83 |
| + OPRD | 79.17 |
OPRD improves over the initial student and modestly outperforms GRPO. These results suggest that OPRD strengthens the student’s underlying reasoning capabilities rather than merely reproducing the weaker teacher’s policy. In contrast, OPD’s lower thinking-mode accuracy points to a potential cost of direct policy matching.
[Q6] Color Consistency in Figure 4
We have unified the color conventions across Figure 4(a) and Figure 4(b) and clarified the caption.
Thank you for the suggestion. originally used green to indicate higher cosine similarity with the teacher’s direction. In the revision, we have reversed its color scale so that alignment with the student’s original update direction is consistently highlighted in green across both panels.
References
[1] Xu et al. “KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning.” arXiv, 2025.
[2] Ren et al. “Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO.” ICML, 2026.
[3] Wang et al. “Distilled Reinforcement Learning for LLM Post-training.” arXiv, 2026.
[4] Zhang et al. “Reinforcement-aware Knowledge Distillation for LLM Reasoning.” ICML, 2026.
[5] Czarnecki et al. “Distilling Policy Distillation.” AISTATS, 2019.
[6] Rafailov et al. “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” NeurIPS, 2023.
Rebuttal to Reviewer Xr3j (1/3)
We thank the reviewer for the detailed feedback and for recognizing our extensive empirical evaluation, consistent performance gains, and theoretical justification for OPRD’s update direction. Below, we clarify the motivation and framing of OPRD and address each concern in turn. All revisions are highlighted in red in the revised manuscript.
[W1] Distillation Framing and Weak-to-Strong Generalization
OPRD follows a line of distillation methods that integrate teacher-learned information into reinforcement learning [1,2,3,4,5]. It transfers a weak teacher’s post-training policy shift rather than making the teacher’s final policy an optimization target.
Recent approaches integrate teacher guidance into RL [1,2,3,4]. Policy distillation also includes non-cloning mechanisms: Czarnecki et al. [5] study teacher-value-based reward shaping that changes learning dynamics without requiring policy cloning. OPRD uses the teacher’s post-training policy shift to rescale the student’s verifier-driven gradient. The teacher supplies the scaling axis, while the student’s reward gradient determines whether the update follows or opposes that axis. This gradient modulation is how the teacher’s learned information benefits the student; improved optimization and information transfer are therefore complementary descriptions of the same process. We clarify this perspective in Appendix M and explain the signal’s motivation in [W2].
We are not entirely sure what is intended by the second part of this concern and would appreciate your feedback on the clarification below. OPD’s tendency to plateau near the weak teacher’s performance in Figures 7-12 is a natural outcome of its policy-matching objective. This limitation directly motivates OPRD: we aim to benefit from the weak teacher as a source of learned information, without making its policy the stronger student’s optimization target. This allows the student to pursue improvements beyond the teacher under its own reward objective. Similarly, as you noted, it is also natural that a stronger model can outperform a weaker one under GRPO. Our comparison, however, is between the same stronger student trained with and without weak-teacher guidance, under matched initialization, training data, and rollout budgets. OPRD improves learning speed and performance over this baseline by distilling from the weak teacher’s learned policy shift to guide the student’s updates (Table 1 and Figures 7-12). The teacher-checkpoint ablation in Table 3 (right) further shows that student learning depends on the teacher’s post-training checkpoint.
[W2] Motivation for the Gradient Scaling Mechanism
The teacher’s policy shift captures reward-related information acquired during RL, which OPRD uses to accelerate student learning without direct policy matching.
At each student-visited prefix, OPRD computes the centered logit difference
Δt=C(zT−zTref)=C(logπT−logπTref),
where the centering operator subtracts the vocabulary mean. Comparing the teacher with its own reference focuses the signal on changes acquired during post-training. As detailed in Appendix J, the teacher–student discrepancy underlying OPD also includes the mismatch between the teacher’s reference policy and the current student.
Under standard KL-regularized RL [6], the optimal policy satisfies
R(x,y)=βlogπTref(y∣x)π⋆(y∣x)+c(x),
where the additive term depends only on the prompt. This relationship motivates using the teacher’s post-training shift as a reward-related guidance signal.
With dt=Δt/∥Δt∥2, OPRD applies
gt=gt+λt(dt⊤gt)dt,λt≥0.
The teacher provides an informative axis, while the student’s reward-driven gradient determines whether to follow or oppose it. OPRD amplifies this gradient component while leaving the orthogonal component unchanged, exploiting the teacher’s learned information while retaining the student’s ability to improve beyond the teacher. Figure 3(a) supports this choice empirically, showing faster and more sustained improvements than the OPD and OPSD guidance directions. Appendix J additionally evaluates these guidance constructions with the stronger teacher checkpoints used in the main experiments.