- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates a fundamental tension in sampled-token on-policy distillation (OPD): token-level teacher–student log-probability gaps (Eq. 3) reflect local teacher preference, which can conflict with the student's own continuation value (Eq. 8). In Section 3 (Proposition 1, Eqs. 9–12), the authors show that locally interpolating the student toward the teacher in probability space always reduces local reverse KL to first order (), but simultaneously decreases conditional expected task reward whenever .
To address this without discarding tokens or training a separate value network, Section 4 introduces (Reward-Aligned Reweighting for On-Policy Distillation, Eqs. 13–16, Algorithm 1). Using the signed binary trajectory outcome (Eq. 7) and the per-response median absolute gap (Eq. 13), computes a smooth gate (Eq. 14, default ) and normalizes it via () so that preserves both token signs and total per-response absolute coefficient mass (Propositions 2–3, Eqs. 17–19). Proposition 4 (Eq. 21) characterizes when this reallocation improves first-order projection onto the true task gradient . Empirically, is evaluated on DAPO-Math-17k across 7 math benchmarks (Qwen3-4B-GRPO Qwen3-1.7B-Base and Qwen3-4B-Base, Table 1) and on TACO across 2 code benchmarks (1.7B, Table 2), improving over vanilla sampled-token OPD by +3.5, +2.4, and +1.6 average points, respectively. While the theoretical exposition is admirably careful and the 1.7B math gains are solid, I lean toward weak rejection due to (1) an unexamined scale paradox at the default where median tokens receive virtually zero reweighting () yet 25% of conflict mass is shifted via extreme heavy-tailed outliers, (2) a mismatch between statewise-centered preference in Proposition 1 and raw gating, (3) confounded component ablations and truncated benchmark/baseline matrices in Tables 2 and 3, and (4) incomplete model scaling (omitting Qwen3-8B and teacher-free GRPO while all 4B distilled students beat the frozen teacher) and missing comparisons with the closest outcome- and magnitude-aware OPD methods.
Critical strengths
1. Careful, well-scoped first-order analysis of preference–value misalignment in on-policy distillation.
Proposition 1 (Eqs. 10–12) and the analytic two-route example in Appendix G.5 cleanly formalize why minimizing local reverse KL can hurt student task return when the teacher prefers reasoning routes the student cannot complete. The authors are commendably transparent about the exact assumptions and limitations of their propositions across Appendices F–I.
2. Simple, zero-overhead reweighting rule with consistent improvements over vanilla sampled-token OPD.
requires no auxiliary value head, extra trainable parameters, or additional teacher forward passes, preserves coefficient signs and per-response absolute mass (Propositions 2–3), and improves all 7 math benchmarks over vanilla sampled-token OPD (46.2 \pm 0.4 vs. 42.7 \pm 0.6 on 1.7B and 50.8 vs. 48.4 on 4B in Table 1), with clean directional controls (Reversed alignment and Outcome-permuted in Table 3).
Critical weaknesses
1. Truncated benchmark/baseline matrices in Tables 2–3, omission of Qwen3-8B and pure GRPO, and the unexplained 4B same-size teacher paradox.
- Truncated Table 3 and Table 2 matrices: Main ablation Table 3 (p. 8) reports results on only 3 of the 7 math benchmarks (
AIME24,MATH500,Minerva), omittingAIME25,AMC23,AMC24, andOlympiadBench—even though the exact sameR²-OPD (full)run (30.1, 77.8, 29.4) was already evaluated on all 7 benchmarks in AppendixTable 5(whereSuccess-onlyoutperforms full onAIME24:31.3vs.30.1). Reporting only a 3-benchmark subset in the main ablation table while holding back the other 4 benchmarks weakens the ablation evidence. Similarly, Table 2 (coding) omits the4Bstudent and drops 4 of the 6 baselines from Table 1 (Top-64 OPD,OPD + GRPO,SG-OPD,ExOPD). - Teacher–student checkpoint asymmetry and the 4B same-size paradox (Table 1): While the students start from raw pretrained
-Basecheckpoints (3.3and5.3Avg.), the teacherQwen3-4B-GRPO(46.4Avg.) is trained fromQwen3-4B-Non-Thinking(Section 5.1, line 308). Puzzlingly, in the4Bsame-size setting of Table 1, every single distilled student baseline (48.1to50.8, including vanillaSampled-Token OPDat48.4andTop-64 OPDat48.1) outperforms the frozenQwen3-4B-GRPOteacher (46.4) on the sameDAPO-Math-17kdataset, yet Table 1 includes no pureGRPO(teacher-free RLVR) baseline onQwen3-1.7B-BaseorQwen3-4B-Baseto isolate how much of the gain comes from on-policy exploration vs. teacher supervision. - Omission of
Qwen3-8Band mixed per-benchmark rankings againstOPDVRandExOPD: All experiments are restricted toQwen3-1.7B-BaseandQwen3-4B-Base, skippingQwen3-8B-Base(as well as larger scales and non-Qwen families). Moreover, on several individual benchmarks in Table 1, trails existing gating baselines (OPDVRleads on1.7B AIME24[29.8vs.29.4],1.7B AMC23[63.3vs.63.0], and4B MATH500[84.6vs.83.6];ExOPDleads on4B AMC23[68.9vs.68.3] and4B Minerva[33.9vs.33.5]; andOPD + GRPOleads on1.7B Minerva[30.6vs.29.1]).
2. Missing comparisons with the closest outcome-aware / magnitude-aware OPD methods discussed in Section 6.
Section 6 (p. 9, lines 455–461) explicitly discusses recent OPD methods that also combine trajectory outcomes, teacher–student gap magnitudes, or token importance—most notably EGRSD (Ke et al., 2026), Filter, then reweight (Li et al., 2026d), TIP (Xu et al., 2026b), Asymmetric OPD (Jia et al., 2026), and Uni-OPD (Hou et al., 2026). None of these closest competitors are evaluated in Tables 1 or 2, making it difficult to judge whether 's median-scaled sigmoid gate offers a meaningful empirical advantage over existing outcome- and magnitude-aware reweighting rules.
Questions for authors
-
Assessment-critical (Explaining the scale paradox and testing statewise-centered preference ):
- Why does reduce the batch conflict-mass share from
0.53to0.28in Figure 3(a) when for median tokens ()? Please report the empirical percentiles (P50, P90, P99, Max) of and , as well as the fraction of responses where is near zero or hits the floor . - Since Proposition 1 / Eq. (12) shows that local probability updates are governed by the statewise-centered preference , how does perform if you gate on top--approximated centered preference instead of raw ?
- Why does reduce the batch conflict-mass share from
-
Assessment-critical (Full 7-benchmark Table 3 ablations, matched-strength gates, and
Qwen3-8B/ pureGRPObaselines):- Can you report all 7 mathematical benchmarks for every ablation row in Table 3 (just as reported for
Success-onlyandFailure-onlyin Appendix Table 5), include aSign-only gatetuned to match the conflict-mass share (and a step-size-compensatedNo mass-norm), and includeTop-64 OPD,OPD + GRPO,SG-OPD, andExOPDin Table 2? - Why was
Qwen3-8B-Baseomitted, why do all distilled4Bstudents in Table 1 (48.1–50.8) outperform theQwen3-4B-GRPOteacher (46.4), and what does pureGRPOachieve onQwen3-1.7B-BaseandQwen3-4B-Baseunder the same budget?
- Can you report all 7 mathematical benchmarks for every ablation row in Table 3 (just as reported for
-
Assessment-critical (Comparisons with closest related methods and training-step clarification):
- How does compare against
EGRSD,Filter, then reweight, orTIPunder the matched Table 1 setup, and how were the hyperparameters ofOPD+GRPO,SG-OPD, andExOPDtuned? - Finally, what was the exact number of training steps and checkpoint selection rule for Table 1 (given that Figures 1/6 end at step 140, Figures 2/4 at step 120, and Figure 5 at step 150)?
- How does compare against
5 comments
Response to Reviewer hW4c (2/4)
Q2a (cont.). Full Table 3. All ablations on all seven benchmarks, including the two controls the reviewer requested: a sign-only gate with group multipliers matched to the full gate's inconsistent share (), and a no-mass-norm variant rescaled by a single batch-level constant that restores the batch's total coefficient mass.
| Variant | AIME24 | AIME25 | AMC23 | AMC24 | MATH500 | Minerva | Olympiad | avg |
|---|---|---|---|---|---|---|---|---|
| R2-OPD (full) | 30.1 | 24.5 | 65.0 | 54.4 | 77.8 | 29.4 | 45.9 | – |
| Reversed alignment | 25.4 | 18.1 | 57.2 | 40.6 | 73.5 | 26.9 | 39.2 | −6.6 |
| Outcome-permuted | 27.9 | 20.2 | 59.8 | 50.1 | 75.9 | 28.5 | 43.2 | −3.1 |
| Sign-only gate | 27.3 | 20.0 | 62.9 | 52.6 | 76.6 | 28.7 | 43.1 | −2.3 |
| Strength-matched sign-only | 26.8 | 20.7 | 61.9 | 52.0 | 75.8 | 28.6 | 43.2 | −2.6 |
| Magnitude-only gate | 26.1 | 20.8 | 61.5 | 50.7 | 75.8 | 28.2 | 42.5 | −3.1 |
| No mass-norm | 28.1 | 23.9 | 62.7 | 53.1 | 76.7 | 29.8 | 44.2 | −1.2 |
| Step-size-compensated no mass-norm | 28.5 | 24.1 | 62.5 | 53.5 | 76.4 | 29.8 | 44.4 | −1.1 |
| R2-OPD + GRPO | 31.5 | 25.8 | 65.4 | 54.9 | 78.0 | 30.2 | 46.1 | +0.7 |
The seven-benchmark averages confirm the conclusions drawn from three benchmarks in the submission:
- Every control falls below the full gate, and reversing the alignment costs the most (−6.6).
- Matching the inconsistent share does not rescue the sign-only gate (−2.6): the gain requires magnitude-aware weighting, not just the same amount of reallocation.
- Compensating the global step size recovers almost nothing (−1.1 vs. −1.2), as AdamW already absorbs a global scale; the benefit of mass normalization comes from the per-response coefficient budget.
The Success-only control in App. Table 5 scores higher on AIME24, but the full gate exceeds it by 1.2 points on the seven-benchmark average. This table will replace Table 3 in the revision.
Q2b. Students above the teacher, and pure GRPO. We thank the reviewer for catching this; it exposed an error in our evaluation, for which we apologize. The teacher and the distilled models had been evaluated with scripts that were not fully aligned. In math, the distillation script contained answer-matching fixes (e.g., accepting 40.0 for the answer 40) that the GRPO script lacked, so the teacher was underestimated. In code, the distilled models had been evaluated with a different script from the one used for the student and teacher. We have now unified the evaluation: the teacher was re-evaluated with the corrected math script, and all distilled code models with the student/teacher script (Table 2 above). The corrected teacher scores 53.1, above every distilled 4B student (48.1-50.8), which resolves the inconsistency; we will correct the teacher row of Table 1.
The teacher is itself a pure-GRPO model: Qwen3-4B trained with GRPO on DAPO-Math-17k under the non-thinking chat template ("Qwen3-4B-Non-Thinking" in Sec. 5.1; we will state this explicitly). For the two Base models, we trained pure GRPO from the same checkpoints as the corresponding students for 500 steps (the last 100 after convergence), using the same evaluation and checkpoint-selection rule:
| Pure GRPO | AIME24 | AIME25 | AMC23 | AMC24 | MATH500 | Minerva | Olympiad | Avg. |
|---|---|---|---|---|---|---|---|---|
| Qwen3-1.7B-Base | 28.1 | 23.4 | 63.7 | 53.3 | 78.7 | 30.3 | 44.3 | 46.0 |
| Qwen3-4B-Base | 33.6 | 27.4 | 66.9 | 56.2 | 84.4 | 32.5 | 47.9 | 49.8 |
| Qwen3-4B, non-thinking (= teacher) | 36.5 | 30.1 | 70.7 | 61.2 | 86.3 | 35.2 | 51.5 | 53.1 |
Within 150 training steps, R2-OPD matches or exceeds pure GRPO from the same initialization (1.7B: 46.2 vs. 46.0; 4B: 50.8 vs. 49.8), whereas the GRPO runs train for 500 steps; standard OPD stays below GRPO in both settings (42.7 and 48.4). The two signals are also complementary: adding the GRPO term to R2-OPD yields a further +0.7 (Table 3).
Response to Reviewer hW4c (3/4)
Qwen3-8B and other model families. Compute constraints prevented these runs before the deadline, so the submission covers cross-size (4B → 1.7B) and same-size (4B → 4B) distillation within Qwen3. During the rebuttal, we added a larger teacher and a second model family, with the same baselines and benchmarks.
(a) Qwen3-8B-GRPO → Qwen3-1.7B-Base
| Method | AIME24 | AIME25 | AMC23 | AMC24 | MATH500 | Minerva | Olympiad | Avg. |
|---|---|---|---|---|---|---|---|---|
| Student | 1.9 | 1.7 | 4.4 | 2.2 | 5.8 | 3.1 | 3.7 | 3.3 |
| Teacher | 56.2 | 42.3 | 85.6 | 69.1 | 89.3 | 33.3 | 59.3 | 62.2 |
| Sampled-Token OPD | 39.2 | 27.8 | 77.6 | 58.1 | 78.2 | 29.6 | 50.1 | 51.5 |
| Top-64 OPD | 38.1 | 27.1 | 77.9 | 57.8 | 78.4 | 29.7 | 49.8 | 51.3 |
| OPD + GRPO | 39.9 | 28.2 | 78.4 | 60.0 | 79.0 | 30.1 | 52.7 | 52.6 |
| OPDVR | 41.8 | 29.4 | 81.0 | 62.9 | 80.1 | 31.0 | 51.8 | 54.0 |
| SG-OPD | 40.6 | 28.0 | 78.1 | 60.6 | 79.2 | 30.4 | 50.9 | 52.5 |
| ExOPD | 41.1 | 28.4 | 79.4 | 61.2 | 79.7 | 30.5 | 51.3 | 53.1 |
| R2-OPD | 44.2 | 31.9 | 81.9 | 64.4 | 82.8 | 31.5 | 53.5 | 55.7 |
(b) Llama-3.2-3B-Instruct-GRPO → Llama-3.2-1B
| Method | AIME24 | AIME25 | AMC23 | AMC24 | MATH500 | Minerva | Olympiad | Avg. |
|---|---|---|---|---|---|---|---|---|
| Student | 1.2 | 0.1 | 5.4 | 4.2 | 30.6 | 6.8 | 6.6 | 7.8 |
| Teacher | 18.6 | 4.2 | 35.9 | 19.2 | 56.7 | 22.5 | 20.8 | 25.4 |
| Sampled-Token OPD | 2.3 | 1.2 | 20.6 | 9.2 | 42.1 | 8.1 | 9.3 | 13.3 |
| Top-64 OPD | 2.5 | 1.0 | 20.9 | 8.9 | 41.0 | 7.7 | 9.1 | 13.0 |
| OPD + GRPO | 2.9 | 1.4 | 22.9 | 11.2 | 42.8 | 9.2 | 10.0 | 14.3 |
| OPDVR | 3.1 | 2.2 | 24.0 | 12.4 | 44.5 | 9.9 | 12.4 | 15.5 |
| SG-OPD | 3.3 | 1.9 | 23.8 | 11.8 | 43.2 | 9.6 | 12.1 | 15.1 |
| ExOPD | 3.6 | 2.5 | 24.2 | 12.6 | 43.9 | 10.1 | 12.9 | 15.7 |
| R2-OPD | 4.8 | 3.2 | 26.4 | 13.5 | 45.1 | 11.4 | 13.5 | 16.8 |
R2-OPD is best on every benchmark in both settings. On average, it improves over standard OPD by +4.2 and over the strongest baseline (OPDVR) by +1.7 with the Qwen3-8B teacher, and by +3.5 and +1.1 (ExOPD) in the Llama setting. Both settings will be added to the revision.
Per-benchmark rankings. Each run is reported at a single checkpoint, the one with the best average, so all of its numbers come from the same model. Individual benchmarks peak at different checkpoints; selecting a checkpoint per benchmark would raise every method's per-benchmark scores, ours included, and we deliberately avoid it. Under this stricter protocol, a baseline can lead on an individual benchmark, by at most 1.5 points, with such leads spread over three different baselines. Single small benchmarks are noisy (in Table 1, the seed standard deviation reaches 3.6 points on AMC24), so we rank methods by the average, on which R2-OPD is best in every setting. Fig. 1 also shows R2-OPD ahead of the baselines for most of training.
Response to Reviewer hW4c (1/4)
We thank the reviewer for the detailed and constructive review, and for recognizing our first-order analysis of preference-value misalignment and the simplicity of the zero-overhead reweighting rule. The questions prompted several new experiments and diagnostics, reported below; all will be added to the revision.
Q1a. Why β=10−3 changes the inconsistent mass. The reviewer is right that gt≈0.50025 at a median-magnitude token, where ∣At∣/νo=1 by construction. Such tokens, however, carry almost no coefficient mass. In sampled-token OPD, most sampled tokens are near-deterministic agreements between teacher and student, which makes νo small, while the mass concentrates on a few fork tokens with ∣At∣/νo≫1. To measure these distributions directly, we re-ran R2-OPD (β=10−3, 1.7B setting) with per-token logging and analyzed every rollout of steps 60-90 (31 batches; 3,968 responses, 2,631 of them complete). The run reproduces the one behind Fig. 3: its logged inconsistent share over steps 60-120 is 28.0% (vs. 28.1%).
We will add these statistics to App. C and redraw Fig. 3(a) from the logged rollouts.
Q1b. Statewise-centered preference. Prop. 1 describes the expected update at a state, averaged over all actions. A sampled-token update acts only through the sampled action, and its effect on logπ(ot∣st) has the sign of At. Since Kt depends only on the state, adding it leaves the expected gradient unchanged and cannot be observed from a single sample. R2-OPD's gate asks whether the push actually applied at t agrees with the outcome, i.e. it uses sign(zAt). Centering would also require the teacher's top-k distribution, whereas R2-OPD needs only sampled-token log-probabilities.
We quantified the alternative on the same logged rollouts, approximating Kt with the teacher's top-64 candidates (the student's mass outside them has a mass-weighted P90 of 1.2×10−3, so the approximation is tight):
Gating on z(At+Kt) therefore undoes part of the reallocation that R2-OPD is designed to perform. We will add this analysis to the appendix.
Q2a. Complete Table 2 (code; Qwen3-4B-GRPO → Qwen3-1.7B-Base; avg@4), now with all six baselines:
All rows now use the same evaluation script. The student and teacher were already evaluated with it; we re-evaluated all distilled models with it (see Q2b), which updates the three method rows of the submitted Table 2. R2-OPD is best on both benchmarks: +4.5 over standard OPD, +1.2 over OPDVR, and +1.9 over ExOPD, the strongest of the newly added baselines.