acceptodds
Under review as a conference paper at ICLR 2027

On-policy Distillation with Verifiable Reward

18 views

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training large language models. However, RLVR suffers from sparse task-level feedback, while OPD provides dense token-level guidance but ignores trajectory correctness, limiting its performance to that of the teacher. Combining them is a promising direction: OPD supplies dense supervisory signals, while RLVR provides task-level correctness. Nevertheless, existing integrations often rely on weighted combination or heuristic switching, introducing extra hyperparameters and trade-offs. We propose **On-policy Distillation with Verifiable Reward (OPDVR)**, a simple yet effective method that seamlessly combines OPD and RLVR without adding any hyperparameters. We first reformulate the implicit reward of sampled-token OPD based on trajectory correctness, then apply a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards—thereby aligning the distillation signal with task success while preserving the teacher's distributional guidance. Furthermore, our modification transforms sampled-token OPD into a proper RLVR method, making it readily combinable with any policy gradient algorithm, such as GRPO. Experiments on six reasoning benchmarks show that OPDVR consistently outperforms OPD and recent methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy Distillation, Large Language Model, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_V8v3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to combine reinforcement learning with verifiable rewards (RLVR) and sampled-token on-policy distillation (OPD). In Section 4.1, the authors rewrite the sampled-token OPD update as a policy-gradient estimator weighted by the implicit token reward ROPD(ot)=log⁡πT(ot∣q,o<t)πθ(ot∣q,o<t)R _ {\mathrm{OPD}}(o _ t) = \log \frac{\pi _ T(o _ t \mid q, o _ {\lt t})}{\pi _ \theta(o _ t \mid q, o _ {\lt t})}, and highlight two cases where teacher supervision conflicts with trajectory-level verifier feedback (Section 4.3, Eqs. 1–2): Type I conflicting tokens (tokens in correct trajectories where πθ>πT\pi _ \theta \gt \pi _ T, which standard OPD suppresses) and Type II conflicting tokens (tokens in incorrect trajectories where πT>πθ\pi _ T \gt \pi _ \theta, which standard OPD reinforces). To remove these conflicts without introducing a loss-mixing hyperparameter, OPDVR (Section 4.2) applies a trajectory-correctness-conditioned ReLU gate that keeps only max⁡(0,log⁡(πT/πθ))⋅(+1)\max(0, \log(\pi _ T/\pi _ \theta)) \cdot (+1) on correct trajectories and max⁡(0,log⁡(πθ/πT))⋅(−1)\max(0, \log(\pi _ \theta/\pi _ T)) \cdot (-1) on incorrect trajectories. Section 4.4 further proposes Group Relative Policy Distillation (GRPD), which replaces the binary trajectory label with the sign of the GRPO group-relative advantage sign(A^i,t)\mathrm{sign}(\hat{A} _ {i,t}). Under a soft-value teacher assumption (Assumption 1, Eq. 4), Appendix B–C proves that a one-step tabular population mirror update (Eq. 5) simultaneously improves task success probability and reduces prefix-level reverse KL to the teacher (Theorem B.2, Eqs. 8–9).

Empirically, the paper evaluates Qwen3-4B and Qwen3-1.7B students on DeepMath-57K and DAPO-Math-17K across six mathematical reasoning benchmarks (AIME24, AIME25, AMC, MATH500, Minerva, OlympiadBench, reporting avg@16 in Tables 1–4). While the one-sided ReLU gating rule is simple and the inverse-gated ablation (Table 4) confirms that gating direction matters, I recommend weak rejection because (1) GRPD uses only sign(A^i,t)\mathrm{sign}(\hat{A} _ {i,t})—making it mathematically identical to binary OPDVR on every non-homogeneous group under binary rewards—while Table 3 omits OPDVR entirely, (2) several claims in the text directly contradict the numbers in Tables 1–3, and (3) the experiments omit closest gating baselines (RG-OPD, Uni-OPD, and pure GRPO in Tables 1–2), mix -nonthinking and -Base checkpoints across settings without explanation, and report only single-seed point estimates over narrow margins (0.7–1.1 points).

Critical strengths

1. Hyperparameter-free, single-line token mask for resolving teacher–verifier sign conflicts.
OPDVR identifies a clear tension in sampled-token OPD—reinforcing teacher-preferred tokens on failed trajectories and penalizing student-preferred tokens on successful trajectories—and removes both conflict types with a one-sided ReLU mask (Section 4.2–4.3, Eqs. 1–2) without introducing a continuous loss-balancing coefficient between RLVR and OPD.

2. Clean diagnostic validation via the Inverse-Gated control.
In Table 4 and Figure 3 (Section 5.4), reversing the gate to keep only the conflicting tokens (Inverse-Gated) degrades the 6-benchmark average on Qwen3-4B from 49.1 (OPDVR) and 47.8 (vanilla OPD) down to 44.6, providing direct empirical evidence that the tokens filtered out by OPDVR are genuinely harmful to reasoning performance.

Critical weaknesses

1. GRPD is not isolated from binary OPDVR (mathematically or empirically), and the baseline matrix across Tables 1–3 is incomplete.

  • No head-to-head comparison between GRPD and OPDVR: Section 5.3 (p. 8, lines 406–410) states that Table 3 (DAPO-Math-17K, Qwen3-4B) is designed "To more cleanly evaluate the effectiveness of replacing binary rewards with group-relative advantages." Yet Table 3 does not include binary OPDVR at all (OPDVR is only run on DeepMath-57K in Table 1, whereas GRPD is only run on DAPO-Math-17K in Table 3). Because GRPD uses only sign(A^i,t)\mathrm{sign}(\hat{A} _ {i,t}) and is identical to binary OPDVR on all non-homogeneous groups (barring unspecified zero-variance group dropping), the paper provides zero evidence that group-relative advantages add value beyond binary OPDVR (or beyond homogeneous-group filtering).
  • Missing closest gating baselines and pure RLVR controls: Although Section 2 (p. 3) discusses RG-OPD (Akhondzadeh et al., 2026) and Uni-OPD (Hou et al., 2026) as the most closely related verifier-gated OPD methods, neither is included in Tables 1–3. Furthermore, pure GRPO (RLVR without a teacher) and Distilled RL are omitted from Tables 1 and 2, while Top-64 OPD, OPD+GRPO, RLSD, ExOPD, and SG-OPD are omitted from Table 3.

2. Unexplained mixing of -nonthinking vs. -Base checkpoints, narrow single-seed margins, and uncontrolled response-length inflation.

  • Checkpoint generation mixing across Tables 1 and 2 and narrow model scope: In Table 1 and Table 3 ("same-architecture"), the student is the post-trained Qwen3-4B-nonthinking checkpoint (42.0 Avg., 80.9 on MATH500) distilled from Qwen3-4B-RL (50.4 Avg.). In Table 2 (misleadingly labeled "cross-architecture" when both models are from the same Qwen3 family), the authors abruptly switch both teacher and student to raw pretrained -Base checkpoints: Qwen3-4B-Base-RL (30.9 Avg.—which is 11.1 points weaker than the untuned Qwen3-4B-nonthinking student in Table 1!) and Qwen3-1.7B-Base (17.3 Avg., with AIME24/25 near the floor at 4.1% / 1.7%). The paper never explains why Qwen3-1.7B-nonthinking (with the Qwen3-4B-RL teacher from Table 1) was not used for cross-size distillation, nor does it evaluate Qwen3-8B, any non-Qwen model family, or any non-math task.
  • Single-seed point estimates over small macro-average margins: All results in Tables 1–4 appear to come from a single training run without multi-seed standard deviations or confidence intervals. Once ExOPD's row averages are corrected to 48.0 (Table 1) and 21.7 (Table 2), OPDVR's macro-average lead over ExOPD is 1.1 points in both tables (while trailing ExOPD on MATH500 [84.7 vs. 86.6] and Minerva [33.2 vs. 35.9] in Table 1, and trailing SG-OPD on OlympiadBench [22.0 vs. 22.3] in Table 2), and GRPD leads Distilled RL by 0.7 points in Table 3 (49.4 vs. 48.7).
  • Uncontrolled response-length expansion: Appendix E Figure 4(a) shows that in the same-size Qwen3-4B run, OPDVR's average response length grows from roughly 1,600 tokens to over 6,700 tokens (approaching the 8,192 token cap), yet the paper does not report response lengths, truncation rates, or wall-clock training costs for any of the baseline methods in Tables 1–3.
Questions for authors
  1. Assessment-critical (GRPD vs. binary OPDVR equivalence and missing head-to-head comparison in Table 3):

    • Since GRPD (Section 4.4, p. 7) uses only sign(A^i,t)\mathrm{sign}(\hat{A} _ {i,t}) under binary rewards Ri∈{0,1}R _ i \in \lbrace 0, 1 \rbrace, its update on any group with both correct and incorrect rollouts is identical to binary OPDVR. How does your implementation handle homogeneous groups where all G=8G=8 rewards are identical (std=0\mathrm{std} = 0)?
    • What is the performance of (a) binary OPDVR (with and without dropping homogeneous groups) and (b) a magnitude-preserving variant ∣A^i,t∣⋅RGRPD(oi,t)|\hat{A} _ {i,t}| \cdot R _ {\mathrm{GRPD}}(o _ {i,t}) under the exact Table 3 (DAPO-Math-17K, Qwen3-4B) setup?
  2. Assessment-critical (Closest gating baselines, multi-seed variance, and checkpoint consistency):

    • Can you report 3-seed mean ±\pm std (and paired bootstrap confidence intervals) for OPDVR, GRPD, ExOPD, SG-OPD, and Distilled RL, alongside same-budget comparisons with RG-OPD, Uni-OPD, and pure GRPO in Tables 1–2?
    • Why did Table 2 switch to weak raw pretrained -Base models (Qwen3-4B-Base-RL teacher at 30.9 Avg. →\to Qwen3-1.7B-Base) instead of distilling the Qwen3-4B-RL teacher (50.4 Avg.) from Table 1 into Qwen3-1.7B-nonthinking, or evaluating Qwen3-8B?
  3. Clarification affecting assessment (Exact semi-gradient implementation, response lengths, and text–table corrections):

    • Can you provide explicit pseudocode showing where stop-gradient sg[⋅]\mathrm{sg}[\cdot] is applied in LOPDVR\mathcal{L} _ {\mathrm{OPDVR}}, how many optimizer steps are taken per rollout batch, and what the final response lengths and wall-clock training times are for all methods in Tables 1–3?
    • Please also clarify why Figure 3 (Left) plots 100 training steps while Figure 4(a) plots 200 steps for the same DeepMath-57K Qwen3-4B setting, and correct the ExOPD averages (48.0 in Table 1, 21.7 in Table 2) and the overstated "across all six benchmarks" claims in Sections 1, 5.2, and 5.3.
1 comment

Q1: Direct Comparison Between GRPD and OPDVR Under Matched Training Settings.

We thank the reviewer for this question. We agree that under binary rewards, GRPD and OPDVR make identical gating decisions for groups containing both correct and incorrect responses. Their difference arises in homogeneous groups, where group-relative advantages vanish while OPDVR can still provide verifier-aligned updates.

Following the reviewer's suggestion, we conduct additional experiments with GRPD on DeepMath and OPDVR on DAPO-Math-17K, enabling direct comparisons under the same training conditions on both datasets.

DeepMath Results.

MethodAIME24AIME25AMCMATH500MinervaOlympiadBenchAvg.
Student (Qwen3-4B)24.015.860.880.927.642.942.0
Teacher (Qwen3-4B-RL)36.029.065.987.035.449.350.4
Sampled-Token OPD34.226.063.185.531.646.547.8
Top-64 OPD34.623.562.085.032.246.847.4
OPD+GRPO33.124.264.485.232.946.547.7
RLSD28.720.062.182.831.244.044.8
ExOPD29.826.962.286.635.946.647.9
SG-OPD31.626.261.284.735.245.847.5
OPDVR (Ours)36.928.164.884.733.247.049.1
GRPD (Ours)35.431.368.988.833.750.751.5

DAPO-Math-17K Results.

MethodAIME24AIME25AMCMATH500MinervaOlympiadBenchAvg.
Student (Qwen3-4B)24.015.860.880.927.642.942.0
Teacher (Qwen3-4B-RL)36.029.065.987.035.449.350.4
GRPO28.320.862.383.928.944.644.8
OPD32.031.765.685.428.946.648.4
Distilled RL33.128.966.686.330.346.848.7
OPDVR (Ours)34.230.967.185.829.746.649.1
GRPD (Ours)34.831.767.085.630.547.049.4

GRPD outperforms OPDVR on both datasets, with a larger improvement on DeepMath (+2.4 points) than on DAPO-Math-17K (+0.3 points). Interestingly, GRPD trained on DeepMath achieves an average score of 51.5, exceeding the GRPO-trained teacher's 50.4. These results show that GRPD's advantage does not depend on switching the training dataset.


Q2: Training Variance and Choice of Student Checkpoints.

(1) Training Variance.

We thank the reviewer for this concern. Due to limited computational resources during the rebuttal period, we were unable to conduct multiple independent training runs with different random seeds. However, all results in Tables 1 and 2 already use avg@16 evaluation, where each problem is evaluated with 16 independently sampled responses and their accuracies are averaged. This reduces stochastic evaluation variance, although it does not directly measure training variance across independent runs.

(2) Choice of Student Checkpoints.

Regarding the choice of Base models in Table 2, successful cross-architecture OPD generally requires carefully selected teacher–student pairs. Therefore, our original experiments followed configurations previously demonstrated to work in the literature, rather than deliberately selecting Base models.

Following the reviewer's suggestion, we conduct additional cross-architecture distillation experiments using Qwen3-4B-RL as the teacher and Qwen3-1.7B as a non-Base student, comparing Top-16 OPD and OPDVR under the same training setting.

MethodAIME24AIME25AMCMATH500MinervaOlympiadBenchAvg.
Student (Qwen3-1.7B)12.58.742.371.219.332.531.1
Top-16 OPD13.310.445.173.320.335.433.0
OPDVR (Ours)15.011.246.774.120.835.833.9

OPDVR outperforms Top-16 OPD across all six benchmarks, improving the average score from 33.0 to 33.9. These results demonstrate that OPDVR remains effective in cross-architecture distillation with a non-Base student.

Reviewer_mWkXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The direction for OPDVR is guided by the correctness of final answers for the distillation updates. The method then utilizes a ReLU gate on the log probability ratio between the teacher and student at the token level, retaining only those terms aligned with the verifiable reward signal. The gating direction for GRPD is based on the sign of the group relative advantage. The two teacher-student pairs within the Qwen3 family were tested on six math benchmarks. An analysis is also provided for one idealized mirror update under the soft value teacher assumption.

Critical strengths
  • The approach is simple and easily incorporable into the existing learning paradigm. The correctness of the answer defines the sign, while the log-probability ratio defines the magnitude, with no additional gating mixture coefficient required for tuning. It is easier to incorporate and work with such a rule rather than multiple additional losses.

  • Both settings demonstrate average improvements. The average scores across all six tasks in Table 1 and Table 2 are 49.1 and 22.8, which compare favorably to the state-of-the-art baselines' 47.9 and 21.7, respectively. In addition, the reverse-gating ablation in Table 4 indicates the importance of the update direction. However, more research is needed to investigate the stability of these gains.

Critical weaknesses
  • The distinction from SG-OPD should be clarified further. According to Section 2, the only difference between the two is that SG-OPD scales conflicting signals while Eq. (7) of SG-OPD allows a zero conflict weight, thus masking the respective tokens. However, the analysis in Appendix F.2 is limited to the 1.8/1.0 configuration. In order to distinguish the performance gain brought by OPDVR from just masking conflicting tokens, it is necessary to conduct a comparison using the variant of zero-conflict-weight, with teacher sampling and training budget equalized.

  • The source of improvement and stability still need explanations. According to Tables 1 and 2, there is a consistent improvement of 1.2 and 1.1 points without any variance in independent training processes. Section 5.4 asserts that inverse gating ensures the maintenance of masking ratio, while selecting complementary tokens does not. Figure 4 presents only the results for OPDVR ratios. In case of binary rewards, for groups having both correct and incorrect answers, the GRPD and OPDVR have the same gating direction. Therefore, a direct comparison under the same condition is necessary. The same holds for RG-OPD and Uni-OPD methods.

Questions for authors
  • Further validation: Have you tried comparing OPDVR with SG-OPD where the conflict coefficient is 0 and you match teacher sampling, additional regularization, and budgets for training? This will have direct implications for the novelty of OPDVR in your work.

  • Clarification and further validation: How is the same masking ratio maintained in Section 5.4? Your Figure 4 already shows the OPDVR masking ratio. Can you further show the actual token retention rate using reverse gating and compare that with random masking at the same retention rate?

  • Further validation: Can you give results for separate runs for Tables 1 and 2 and directly compare GRPD with OPDVR under the same conditions as those in Table 3?

1 comment

We sincerely thank the reviewer for the constructive feedback. We hope our responses below address your concerns and questions.

Q1& W1: Relationship Between OPDVR and SG-OPD.

We thank the reviewer for this important observation. We acknowledge that OPDVR is mathematically equivalent to a special case of SG-OPD, when the conflict coefficient is set to zero, the consensus extrapolation coefficient is set to one, and additional weighting mechanisms are removed.

However, the two works are motivated by different research questions. SG-OPD focuses on combining extrapolation on consensus tokens with interpolation on conflicting tokens. Although SG-OPD explicitly acknowledges that setting the conflict coefficient to zero masks conflicting tokens, its experimental evaluations avoid this configuration, instead focusing on exploring the effects of extrapolation and interpolation. In particular, its ablation studies investigate different coefficients without evaluating the simple zero-conflict setting.

Our key finding is that neither extrapolation nor interpolation is necessary: simply retaining the original OPD signal on aligned tokens while completely removing conflicting updates already yields an effective, hyperparameter-free distillation objective. More importantly, this learning paradigm naturally aligns with the intuition of reinforcing reliable correct behaviors and suppressing overconfident erroneous predictions as we discussed in Section 4.3, while preserving the teacher's fine-grained guidance through the original teacher-student log-probability ratio.

Furthermore, our theoretical analysis demonstrates that, under the soft-value teacher assumption, a verifier-gated mirror update can simultaneously improve task success and preserve teacher alignment. This provides additional justification for studying simple verifier-aligned gating as a standalone mechanism.

Therefore, although OPDVR is mathematically contained as a special case of SG-OPD, our contribution lies in identifying and systematically investigating the effectiveness and theoretical properties of this simple gating mechanism, which was not empirically established as a standalone approach in SG-OPD.

Q2: Clarification of the Inverse-Gated Masking Ratio.

We thank the reviewer for pointing this out. The statement in Section 5.4 was imprecisely worded. Our intention was to clarify that Inverse-Gated reverses the token-selection rule of OPDVR while keeping all other training settings unchanged, rather than claiming that their masking ratios are exactly identical.

Specifically, let RR denote the binary verifier reward and rtr_t the teacher-student log-probability ratio. The two masking rules are:

mtOPDVR=I[Rrt>0],m_t^{\mathrm{OPDVR}}=\mathbb{I}[Rr_t>0],

mtInverse=I[Rrt<0].m_t^{\mathrm{Inverse}}=\mathbb{I}[Rr_t<0].

These rules select opposite token subsets for the same trajectory and model state. However, since the two methods are trained independently, their actual token retention rates need not be identical or complementary at each training step. We have corrected the wording in Section 5.4 of the revised PDF.

Q3: Direct Comparison Between GRPD and OPDVR Under Matched Training Settings.

We thank the reviewer for this suggestion. We conduct additional experiments with GRPD on DeepMath and OPDVR on DAPO-Math-17K, enabling a direct comparison of the two methods under the same training conditions on each dataset.

DeepMath Results.

MethodAIME24AIME25AMCMATH500MinervaOlympiadBenchAvg.
OPDVR36.928.164.884.733.247.049.1
GRPD35.431.368.988.833.750.751.5

DAPO-Math-17K Results.

MethodAIME24AIME25AMCMATH500MinervaOlympiadBenchAvg.
OPDVR34.230.967.185.829.746.649.1
GRPD34.831.767.085.630.547.049.4

As shown above, GRPD outperforms OPDVR under both training datasets. These results demonstrate that GRPD's improvement over OPDVR does not depend on the choice of training dataset.

Clarification on Training Stability and Evaluation Variance: We thank the reviewer for this concern. Due to limited computational resources during the rebuttal period, we were unable to conduct multiple independent training runs with different random seeds. However, we would like to emphasize that all results in Tables 1 and 2 are already reported using avg@16 evaluation, where each problem is evaluated with 16 independently sampled responses and their accuracies are averaged.

We thank you again for your valuable feedback. We hope that our additional experiments and clarifications have addressed your concerns, and we would be very grateful if you could adjust your rating accordingly.

Reviewer_yYVROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how to combine the fine-grained teacher guidance of OPD with trajectory-level rewards aligned with task success in RLVR. The authors observe that the optimization direction of standard OPD can conflict with the final correctness of a trajectory. To address this issue, they propose OPDVR, which uses a simple ReLU gate to zero out the weights of tokens whose distillation signals conflict with the direction indicated by the trajectory-level reward, retaining only directionally aligned signals. The authors further introduce GRPD, which replaces the binary trajectory-level reward in the gating mechanism with the sign of the group-relative advantage. Experiments on two teacher–student pairs and six mathematical reasoning benchmarks show improvements over standard OPD and other baselines.

Critical strengths
  1. This paper proposes OPDVR, a method that combines OPD with RLVR by using binary rewards from environmental feedback to determine the update direction and OPD to provide fine-grained update magnitudes. The authors further extend OPDVR to GRPD by replacing the environmental reward with the group-relative advantage, demonstrating how the gating mechanism can be integrated with group-relative advantage estimation.
  2. OPDVR features a simple gating mechanism and requires few hyperparameters. It improves performance over vanilla OPD and other baselines across both teacher–student pairs. Overall, the proposed method is simple, intuitive, and easy to follow.
Critical weaknesses
  1. A substantial body of work has already explored combining OP(S)D with RLVR. The central idea behind OPDVR—using environmental rewards to determine the optimization direction while using OPD to provide fine-grained update magnitudes—is closely related to prior work, such as Self-Distilled RLVR.
  2. There is a discrepancy between the experimental results and the claimed core advantage of the proposed method. For example, in Table 1, Qwen3-4B-RL, which the paper states is obtained by training Qwen3-4B with GRPO, achieves an average score of 50.4. In comparison, OPDVR uses environmental rewards and additionally receives dense OPD signals from Qwen3-4B-RL, yet achieves an average score of only 49.1. It therefore underperforms Qwen3-4B-RL, which is trained using only environmental rewards. This result appears inconsistent with the claim that OPDVR combines the complementary strengths of OPD and RLVR, and raises concerns about whether the claimed benefit is supported by the experiments.
Questions for authors

Our two main concerns are detailed in the Weaknesses section. In addition, we would appreciate the authors’ responses to the following questions:

  1. Why is DeepMath used to train the teacher (Qwen3-4B-RL), while DAPOMath-17K is used to train the methods compared in Table 3? The paper explains that this choice helps “more cleanly evaluate the effectiveness of replacing binary rewards with group-relative advantages,” but the connection between changing the training dataset and this stated goal is unclear. We also observe that GRPO performs substantially worse on this dataset than the teacher, which was also trained with GRPO. Could the authors report results for the compared methods when trained on DeepMath? This would help determine whether GRPD’s advantage depends on the training dataset.
  2. Following up on the previous question, could the authors also report OPDVR’s performance under the same settings as Table 3? The DAPO-Math-17K appears to use binary rewards, under which GRPD and OPDVR make identical gating decisions for groups containing both correct and incorrect responses. A more complex setting, e.g. with composite rewards, could help demonstrate the distinctive role of group-relative gating.
  3. The paper states that the student in Table 2 is a base model. Given that using base models as students in OPD is relatively uncommon in practice, could the authors provide results with a non-base student, such as an instruction-tuned model, while maintaining the cross-architecture teacher–student setting? This would help assess the effectiveness of OPDVR/GRPD in more commonly used settings.
  4. The paper conducts training and evaluation exclusively in mathematics. Could the authors provide training and evaluation results in other common domains, such as coding? This would help assess the applicability of OPDVR/GRPD across domains.
3 comments
AuthorsOpenReview ↗

Part1: Response to W1,W2

We sincerely thank the reviewer for the constructive feedback. We hope our responses below address your concerns and questions.

W1: Relationship to Prior Work

We thank the reviewer for this observation. We acknowledge that substantial prior work has explored combining OPD with RLVR, and we have compared OPDVR against these representative methods, including RLSD, ExOPD, SG-OPD, and OPD+GRPO, in our main experiments.

Our contribution is to identify a particularly simple mechanism: applying a ReLU gate directly to the sampled-token OPD reward is sufficient to align token-level distillation signals with verifiable trajectory correctness, without introducing additional hyperparameters. After introduce this simple ReLU, the learning paradigm naturally aligns with the intuition of reinforcing reliable correct behaviors and suppressing overconfident erroneous predictions as we discussed in Section 4.3. Furthermore, OPDVR achieves the best performance among all compared methods in our experiments.

Regarding Self-Distilled RLVR (RLSD), although both methods combine verifier feedback with token-level teacher guidance, their mechanisms are different.

RLSD uses a self-teacher conditioned on privileged information, such as reference solutions, and incorporates the teacher-student probability ratio into a GRPO-style advantage through a positive multiplicative weight:

wt=exp⁡(sign⁡(A)Δt),w_t=\exp(\operatorname{sign}(A)\Delta_t),

where Δt\Delta_t is the log-probability difference between the privileged self-teacher and the student. Thus, RLSD uses teacher information to continuously reweight the magnitude of the verifier-derived advantage, without eliminating any token updates through a hard gate.

In contrast, OPDVR starts directly from the original sampled-token OPD reward and applies a simple verifier-aligned ReLU gate:

rtOPDVR=R⋅ReLU⁡(R⋅Δt),r_t^{\mathrm{OPDVR}}=R\cdot\operatorname{ReLU}(R\cdot\Delta_t),

where RR is the binary verifier reward and Δt\Delta_t is the log-probability ratio between an independently trained teacher and the student. OPDVR completely removes token updates that conflict with the verifier signal while retaining the original OPD reward on aligned tokens.

Therefore, RLSD performs continuous advantage reweighting within an RLVR framework, whereas OPDVR introduces a hard verifier-aligned gate into the original OPD objective. Our experiments directly compare these methods under the same distillation setting, where OPDVR achieves higher average accuracy than RLSD in both the same-architecture (49.1 vs. 44.8) and cross-architecture (22.8 vs. 21.6) experiments.

W2: Comparison with the GRPO-Trained Teacher.

We thank the reviewer for this observation. To our knowledge, the advantage of on-policy distillation (OPD) lies in its rapid convergence, enabling efficient transfer of a stronger teacher model's capabilities to the student with substantially lower training cost. Although our teacher is indeed obtained through GRPO training with verifiable reward, its training budget is approximately 16 times that of our OPD setting. Therefore, directly comparing their final performance without accounting for this substantial difference in training cost does not reflect the efficiency advantage of OPD. The teacher is trained to evaluate different OPD methods, not to serve as a baseline itself.

Our goal in introducing OPDVR is to improve upon standard OPD under the same training budget by incorporating verifier-aligned supervision, while preserving its efficiency in rapidly approaching teacher-level performance. Therefore, although OPDVR incorporates verifiable reward signals, its primary comparison should remain with standard OPD under the same training budget, rather than with a GRPO-trained teacher that uses substantially more compute.

AuthorsOpenReview ↗

Part2: Response to Q1,Q2,Q3,Q4

Q1&Q2: Choice of Training Dataset for GRPD.

We thank the reviewer for this question. Our initial motivation for using DAPO-Math-17K was that the teacher had already been trained on DeepMath, and we expected that continuing OPD on the same dataset might lead to overly rapid convergence, potentially limiting the additional benefits of introducing group-relative advantages in GRPD.

However, following the reviewer's suggestion, we conducted additional experiments with GRPD on DeepMath and OPDVR on DAPO-Math-17K, enabling direct comparisons under both training datasets.

DeepMath Results.

MethodAIME24AIME25AMCMATH500MinervaOlympiadBenchAvg.
Student (Qwen3-4B)24.015.860.880.927.642.942.0
Teacher (Qwen3-4B-RL)36.029.065.987.035.449.350.4
Sampled-Token OPD34.226.063.185.531.646.547.8
Top-64 OPD34.623.562.085.032.246.847.4
OPD+GRPO33.124.264.485.232.946.547.7
RLSD28.720.062.182.831.244.044.8
ExOPD29.826.962.286.635.946.647.9
SG-OPD31.626.261.284.735.245.847.5
OPDVR (Ours)36.928.164.884.733.247.049.1
GRPD (Ours)35.431.368.988.833.750.751.5

DAPO-Math-17K Results.

MethodAIME24AIME25AMCMATH500MinervaOlympiadBenchAvg.
Student (Qwen3-4B)24.015.860.880.927.642.942.0
Teacher (Qwen3-4B-RL)36.029.065.987.035.449.350.4
GRPO28.320.862.383.928.944.644.8
OPD32.031.765.685.428.946.648.4
Distilled RL33.128.966.686.330.346.848.7
OPDVR (Ours)34.230.967.185.829.746.649.1
GRPD (Ours)34.831.767.085.630.547.049.4

Interestingly, contrary to our initial expectation, GRPD achieves a larger improvement over OPDVR on DeepMath (+2.4 points) than on DAPO-Math-17K (+0.3 points). Moreover, GRPD trained on DeepMath achieves an average score of 51.5, surpassing the GRPO-trained teacher's 50.4. These results show that GRPD's advantage does not depend on switching the training dataset from DeepMath to DAPO-Math-17K.

Q3: Cross-Architecture Distillation with a Non-Base Student.

We thank the reviewer for this suggestion. Successful cross-architecture OPD generally requires carefully selected teacher–student pairs, so our original experiments followed model configurations previously demonstrated to work in other paper. Our choice of a Base student was not intentional.

Following the reviewer's suggestion, we conduct additional cross-architecture distillation experiments using Qwen3-4B-RL as the teacher and Qwen3-1.7B as the non-Base student. We compare the initial student, Top-16 OPD, and OPDVR under the same training setting.

MethodAIME24AIME25AMCMATH500MinervaOlympiadBenchAvg.
Student (Qwen3-1.7B)12.58.742.371.219.332.531.1
Top-16 OPD13.310.445.173.320.335.433.0
OPDVR (Ours)15.011.246.774.120.835.833.9

OPDVR outperforms Top-16 OPD across all six benchmarks, improving the average score from 33.0 to 33.9. These results further demonstrate the effectiveness of OPDVR in cross-architecture distillation with a non-Base student.

Q4: Generalization to Code Generation.

We thank the reviewer for this suggestion. We conduct additional code-generation experiments using Qwen3-4B as the student and Qwen3-4B-Code-RL as the teacher, which is obtained by training Qwen3-4B with GRPO on the Eurus dataset. We evaluate all methods on HumanEval, MBPP, and LiveCodeBench-v6.

MethodHumanEvalMBPPLiveCodeBench-v6Avg.
Student (Qwen3-4B)80.848.725.151.5
Teacher (Qwen3-4B-Code-RL)90.656.236.060.9
Top-16 OPD86.853.329.856.6
OPDVR (Ours)87.753.531.457.6

OPDVR outperforms Top-16 OPD on all three benchmarks. These results demonstrate that OPDVR's effectiveness extends beyond mathematical reasoning to code generation.

We thank you again for your valuable feedback. We hope that our additional experiments and clarifications have addressed your concerns, and we would be very grateful if you could adjust your rating accordingly.

We thank the authors for the detailed responses and additional experiments. However, a key concern remains regarding the positioning of the proposed method:

  • Compared with GRPO (Capability Ceiling): Once environmental rewards are introduced, one can directly apply GRPO. For an algorithm that leverages both environmental rewards and teacher OPD signals, the model's capability ceiling becomes the critical factor, and its performance should surpass pure GRPO.

  • Compared with Standard OPD (Practical Convenience): Conversely, if the method is viewed merely as an improvement over standard OPD, introducing environmental rewards imposes a significant constraint. In many real-world scenarios, verifiable rewards are difficult or costly to obtain, which greatly weakens the core convenience of standard OPD—requiring only student rollouts without external verifiers.

Therefore, considering the above factors along with our concerns regarding the novelty, we have decided to keep our score unchanged.

Comments on the paper

1 comment
Area_Chair_p4rDOpenReview ↗

Please Engage With Authors

Dear Reviewers,

We're approaching the end of the discussion phase. The authors have provided their responses to your reviews. Please check whether their responses have resolved the concerns raised in your review. If not, please raise and leave some time for the authors to respond again. If yes, please still respond to the authors. Or simply admit that you've read the rebuttal.

Many thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.