- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to combine reinforcement learning with verifiable rewards (RLVR) and sampled-token on-policy distillation (OPD). In Section 4.1, the authors rewrite the sampled-token OPD update as a policy-gradient estimator weighted by the implicit token reward , and highlight two cases where teacher supervision conflicts with trajectory-level verifier feedback (Section 4.3, Eqs. 1–2): Type I conflicting tokens (tokens in correct trajectories where , which standard OPD suppresses) and Type II conflicting tokens (tokens in incorrect trajectories where , which standard OPD reinforces). To remove these conflicts without introducing a loss-mixing hyperparameter, OPDVR (Section 4.2) applies a trajectory-correctness-conditioned ReLU gate that keeps only on correct trajectories and on incorrect trajectories. Section 4.4 further proposes Group Relative Policy Distillation (GRPD), which replaces the binary trajectory label with the sign of the GRPO group-relative advantage . Under a soft-value teacher assumption (Assumption 1, Eq. 4), Appendix B–C proves that a one-step tabular population mirror update (Eq. 5) simultaneously improves task success probability and reduces prefix-level reverse KL to the teacher (Theorem B.2, Eqs. 8–9).
Empirically, the paper evaluates Qwen3-4B and Qwen3-1.7B students on DeepMath-57K and DAPO-Math-17K across six mathematical reasoning benchmarks (AIME24, AIME25, AMC, MATH500, Minerva, OlympiadBench, reporting avg@16 in Tables 1–4). While the one-sided ReLU gating rule is simple and the inverse-gated ablation (Table 4) confirms that gating direction matters, I recommend weak rejection because (1) GRPD uses only —making it mathematically identical to binary OPDVR on every non-homogeneous group under binary rewards—while Table 3 omits OPDVR entirely, (2) several claims in the text directly contradict the numbers in Tables 1–3, and (3) the experiments omit closest gating baselines (RG-OPD, Uni-OPD, and pure GRPO in Tables 1–2), mix -nonthinking and -Base checkpoints across settings without explanation, and report only single-seed point estimates over narrow margins (0.7–1.1 points).
Critical strengths
1. Hyperparameter-free, single-line token mask for resolving teacher–verifier sign conflicts.
OPDVR identifies a clear tension in sampled-token OPD—reinforcing teacher-preferred tokens on failed trajectories and penalizing student-preferred tokens on successful trajectories—and removes both conflict types with a one-sided ReLU mask (Section 4.2–4.3, Eqs. 1–2) without introducing a continuous loss-balancing coefficient between RLVR and OPD.
2. Clean diagnostic validation via the Inverse-Gated control.
In Table 4 and Figure 3 (Section 5.4), reversing the gate to keep only the conflicting tokens (Inverse-Gated) degrades the 6-benchmark average on Qwen3-4B from 49.1 (OPDVR) and 47.8 (vanilla OPD) down to 44.6, providing direct empirical evidence that the tokens filtered out by OPDVR are genuinely harmful to reasoning performance.
Critical weaknesses
1. GRPD is not isolated from binary OPDVR (mathematically or empirically), and the baseline matrix across Tables 1–3 is incomplete.
- No head-to-head comparison between
GRPDandOPDVR: Section 5.3 (p. 8, lines 406–410) states that Table 3 (DAPO-Math-17K,Qwen3-4B) is designed "To more cleanly evaluate the effectiveness of replacing binary rewards with group-relative advantages." Yet Table 3 does not include binaryOPDVRat all (OPDVRis only run onDeepMath-57Kin Table 1, whereasGRPDis only run onDAPO-Math-17Kin Table 3). BecauseGRPDuses only and is identical to binaryOPDVRon all non-homogeneous groups (barring unspecified zero-variance group dropping), the paper provides zero evidence that group-relative advantages add value beyond binaryOPDVR(or beyond homogeneous-group filtering). - Missing closest gating baselines and pure RLVR controls: Although Section 2 (p. 3) discusses
RG-OPD(Akhondzadeh et al., 2026) andUni-OPD(Hou et al., 2026) as the most closely related verifier-gated OPD methods, neither is included in Tables 1–3. Furthermore, pureGRPO(RLVR without a teacher) andDistilled RLare omitted from Tables 1 and 2, whileTop-64 OPD,OPD+GRPO,RLSD,ExOPD, andSG-OPDare omitted from Table 3.
2. Unexplained mixing of -nonthinking vs. -Base checkpoints, narrow single-seed margins, and uncontrolled response-length inflation.
- Checkpoint generation mixing across Tables 1 and 2 and narrow model scope: In Table 1 and Table 3 ("same-architecture"), the student is the post-trained
Qwen3-4B-nonthinkingcheckpoint (42.0Avg.,80.9on MATH500) distilled fromQwen3-4B-RL(50.4Avg.). In Table 2 (misleadingly labeled "cross-architecture" when both models are from the sameQwen3family), the authors abruptly switch both teacher and student to raw pretrained-Basecheckpoints:Qwen3-4B-Base-RL(30.9Avg.—which is11.1points weaker than the untunedQwen3-4B-nonthinkingstudent in Table 1!) andQwen3-1.7B-Base(17.3Avg., withAIME24/25near the floor at4.1%/1.7%). The paper never explains whyQwen3-1.7B-nonthinking(with theQwen3-4B-RLteacher from Table 1) was not used for cross-size distillation, nor does it evaluateQwen3-8B, any non-Qwen model family, or any non-math task. - Single-seed point estimates over small macro-average margins: All results in Tables 1–4 appear to come from a single training run without multi-seed standard deviations or confidence intervals. Once
ExOPD's row averages are corrected to48.0(Table 1) and21.7(Table 2), OPDVR's macro-average lead overExOPDis1.1points in both tables (while trailingExOPDonMATH500[84.7vs.86.6] andMinerva[33.2vs.35.9] in Table 1, and trailingSG-OPDonOlympiadBench[22.0vs.22.3] in Table 2), andGRPDleadsDistilled RLby0.7points in Table 3 (49.4vs.48.7). - Uncontrolled response-length expansion: Appendix E Figure 4(a) shows that in the same-size
Qwen3-4Brun, OPDVR's average response length grows from roughly1,600tokens to over6,700tokens (approaching the8,192token cap), yet the paper does not report response lengths, truncation rates, or wall-clock training costs for any of the baseline methods in Tables 1–3.
Questions for authors
-
Assessment-critical (
GRPDvs. binaryOPDVRequivalence and missing head-to-head comparison in Table 3):- Since
GRPD(Section 4.4, p. 7) uses only under binary rewards , its update on any group with both correct and incorrect rollouts is identical to binaryOPDVR. How does your implementation handle homogeneous groups where all rewards are identical ()? - What is the performance of (a) binary
OPDVR(with and without dropping homogeneous groups) and (b) a magnitude-preserving variant under the exact Table 3 (DAPO-Math-17K,Qwen3-4B) setup?
- Since
-
Assessment-critical (Closest gating baselines, multi-seed variance, and checkpoint consistency):
- Can you report 3-seed mean std (and paired bootstrap confidence intervals) for
OPDVR,GRPD,ExOPD,SG-OPD, andDistilled RL, alongside same-budget comparisons withRG-OPD,Uni-OPD, and pureGRPOin Tables 1–2? - Why did Table 2 switch to weak raw pretrained
-Basemodels (Qwen3-4B-Base-RLteacher at30.9Avg.Qwen3-1.7B-Base) instead of distilling theQwen3-4B-RLteacher (50.4Avg.) from Table 1 intoQwen3-1.7B-nonthinking, or evaluatingQwen3-8B?
- Can you report 3-seed mean std (and paired bootstrap confidence intervals) for
-
Clarification affecting assessment (Exact semi-gradient implementation, response lengths, and text–table corrections):
- Can you provide explicit pseudocode showing where stop-gradient is applied in , how many optimizer steps are taken per rollout batch, and what the final response lengths and wall-clock training times are for all methods in Tables 1–3?
- Please also clarify why
Figure 3 (Left)plots 100 training steps whileFigure 4(a)plots 200 steps for the sameDeepMath-57KQwen3-4Bsetting, and correct theExOPDaverages (48.0in Table 1,21.7in Table 2) and the overstated "across all six benchmarks" claims in Sections 1, 5.2, and 5.3.
Q1: Direct Comparison Between GRPD and OPDVR Under Matched Training Settings.
We thank the reviewer for this question. We agree that under binary rewards, GRPD and OPDVR make identical gating decisions for groups containing both correct and incorrect responses. Their difference arises in homogeneous groups, where group-relative advantages vanish while OPDVR can still provide verifier-aligned updates.
Following the reviewer's suggestion, we conduct additional experiments with GRPD on DeepMath and OPDVR on DAPO-Math-17K, enabling direct comparisons under the same training conditions on both datasets.
DeepMath Results.
DAPO-Math-17K Results.
GRPD outperforms OPDVR on both datasets, with a larger improvement on DeepMath (+2.4 points) than on DAPO-Math-17K (+0.3 points). Interestingly, GRPD trained on DeepMath achieves an average score of 51.5, exceeding the GRPO-trained teacher's 50.4. These results show that GRPD's advantage does not depend on switching the training dataset.
Q2: Training Variance and Choice of Student Checkpoints.
(1) Training Variance.
We thank the reviewer for this concern. Due to limited computational resources during the rebuttal period, we were unable to conduct multiple independent training runs with different random seeds. However, all results in Tables 1 and 2 already use avg@16 evaluation, where each problem is evaluated with 16 independently sampled responses and their accuracies are averaged. This reduces stochastic evaluation variance, although it does not directly measure training variance across independent runs.
(2) Choice of Student Checkpoints.
Regarding the choice of Base models in Table 2, successful cross-architecture OPD generally requires carefully selected teacher–student pairs. Therefore, our original experiments followed configurations previously demonstrated to work in the literature, rather than deliberately selecting Base models.
Following the reviewer's suggestion, we conduct additional cross-architecture distillation experiments using Qwen3-4B-RL as the teacher and Qwen3-1.7B as a non-Base student, comparing Top-16 OPD and OPDVR under the same training setting.
OPDVR outperforms Top-16 OPD across all six benchmarks, improving the average score from 33.0 to 33.9. These results demonstrate that OPDVR remains effective in cross-architecture distillation with a non-Base student.