- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Direct On-Policy Distillation (Direct-OPD), a weak-to-strong post-training method designed to transfer the policy change induced by reinforcement learning on a small model to a stronger student without running expensive RL directly on the student. Instead of matching the weak teacher's final post-RL distribution , Direct-OPD defines the teacher policy shift as the log-probability ratio between the post-RL teacher and its pre-RL reference checkpoint , (Section 1 Eq. 1; Section 2.2 Eq. 5), interprets it as a dense implicit reward under KL-regularized RL (Eqs. 6–7), and applies a zero-discount, immediate-token local surrogate on the student's top- () candidate set along student-sampled rollouts while regularizing the student toward its own initialization (Section 2.3, Eqs. 8–15) with a sign-based adaptive KL controller (Section 2.4, Eq. 16).
The method is evaluated using two 1.5B teacher checkpoint pairs (R1-Distill-1.5B JustRL-1.5B and Nemotron-1.5B QuestA-Nemotron-1.5B) and three students (Qwen3-1.7B, Qwen3-4B, and R1-Distill-7B) on AIME 2024 and AIME 2025 (ave@32, Section 3, Table 1, and Appendix B Table 5). In the strict weak-to-strong setting (JustRL-1.5B, which scores 51.3% / 37.5%), Direct-OPD improves stronger students (Qwen3-4B from 72.5% / 65.6% to 77.6% / 68.8% peak, or +3.87 / +1.38 points at step 300 in Table 5; R1-Distill-7B from 56.7% / 40.5% to 63.1% / 48.8% peak, or +4.37 / +6.24 points at step 300). Section 3.2 (Figure 2) further shows that 1,500 steps of 1.5B RL plus 300 steps of Direct-OPD (5,152 A100 GPU-hours) outperforms 1,500 steps of direct 7B RL (10,240 A100 GPU-hours) on R1-Distill-7B, and Section 3.3 (Appendix Figure 7) demonstrates sequential composition of the JustRL and QuestA shifts on Qwen3-1.7B. While the core transfer idea is clean and practically appealing, significant gaps between the idealized sequence objective and the implemented local top- surrogate, the absence of closely related baselines (WSPO, G-OPD, Proxy-Tuning), and single-seed evaluation restricted solely to AIME prevent me from recommending acceptance in the current version.
Critical strengths
1. Clean conceptual decoupling of the RL policy shift from the weak teacher's absolute capability ceiling.
Standard on-policy distillation forces the student toward , which blends the desirable improvements learned during RL with the weak base capacity of . By extracting the log-ratio shift on student-visited states while anchoring the KL penalty to the student's own initialization (Eq. 8), Direct-OPD avoids pulling a stronger student down toward a weaker teacher—as vividly demonstrated in Figure 1(a) on R1-Distill-7B (+6.4 points with Direct-OPD vs. roughly -6.7 points with vanilla OPD).
2. Encouraging weak-to-strong empirical gains and concrete compute savings over direct 7B RL.
In the strict weak-to-strong JustRL-1.5B setting (51.3% / 37.5% on AIME 24/25), Direct-OPD improves both Qwen3-4B (72.5% / 65.6% 77.6% / 68.8% peak in Table 1; 76.37% / 66.98% at step 300 in Table 5) and R1-Distill-7B (56.7% / 40.5% 63.1% / 48.8% peak; 61.04% / 46.77% at step 300). Furthermore, Section 3.2 (Figure 2) shows that training the 1.5B teacher for 1,500 RL steps plus 300 Direct-OPD steps on R1-Distill-7B takes roughly half the compute (5,152 vs. 10,240 A100 GPU-hours) while achieving higher AIME 2025 accuracy than direct 7B GRPO.
Critical weaknesses
1. Missing closest baselines (WSPO, G-OPD, Proxy-Tuning, and exact local target distillation) and single-seed evaluation confined solely to AIME.
Although Section 5 discusses closely related methods that also transfer pre/post-RL log-probability ratios or reference-anchored shifts—most notably WSPO (offline weak-to-strong ratio matching), G-OPD (generalized OPD with a pre-RL reference), and inference-time Proxy-Tuning—none of them are evaluated in the experiments (even vanilla OPD appears only in a single curve in Figure 1a). In addition, every teacher–student configuration is run for only a single training trajectory with no training-seed error bars or per-problem bootstrap confidence intervals (Appendix B, p. 18); Table 1 reports benchmark-specific peak checkpoints across steps 0–300; and AIME 2024/2025 (only 30 problems each) are simultaneously used for hyperparameter sweeps (horizon in Section 4.2, KL in Section 4.3), checkpoint selection, and final reporting, with no independent held-out evaluation benchmark (e.g., MATH-500, OlympiadBench, or HMMT) and no decontamination audit against Skywork-OR1-RL-Data. Note also that at the common step-300 endpoint (Table 5), the gain on Qwen3-4B AIME 2025 is only +1.38 percentage points (65.60% 66.98%), while baseline measurement variations across sweeps in Appendix B already reach 0.8–1.6 percentage points (48.3% vs. 49.10% on QuestA–Qwen3-1.7B; 42.55% vs. 44.18% on JustRL–Qwen3-1.7B). This directly drives my weak-reject rating because the empirical evidence is neither statistically isolated from noise on the strongest student nor compared against the closest existing ratio-transfer methods.
2. Unresolved theoretical and empirical gap between the idealized sequence objective (Eqs. 8–9) and the uncentered, zero-discount top- local update (Eqs. 13–15, 18–20).
The paper motivates Direct-OPD via the sequence-level objective in Eq. (8), yet implements a myopic per-token update (Eq. 15) that discards future return-to-go and includes a non-invariant top- support-mass gradient (Appendix C, Eq. 18). The paper provides no ablation testing whether the empirical gains come from (i) immediate token shift vs. sequence return-to-go, (ii) the uncentered mass-shifting term vs. a centered top- update , or (iii) direct token-level KL minimization toward the local multiplicative target implied by Eq. (9). Similarly, the sequential composition experiment in Section 3.3 (JustRL QuestA for steps, Figure 7) lacks a 600-step single-shift control (JustRL JustRL) or a reversed-order control (QuestA JustRL) to separate true policy-shift composability from simply training for 300 additional steps.
Questions for authors
-
Assessment-critical (Comparisons with closest ratio-transfer baselines): Under matched student rollout budgets on
Qwen3-4BandR1-Distill-7Bwith theJustRL-1.5Bteacher pair, how does Direct-OPD compare against (a)WSPO, (b)G-OPD, (c) inference-timeProxy-Tuning, and (d) directly minimizing the token-level KL divergence toward the normalized local target on ? -
Assessment-critical (Ablating the local top- surrogate in Eq. 15 & Appendix C Eq. 18): Why was the uncentered update chosen despite its sensitivity to additive reward constants? How does performance change if you (a) center rewards within the top- support (, which removes the term), (b) use discounted return-to-go instead of immediate , or (c) vary ? Also, how are tokenizers aligned across
R1-Distill/Nemotronteachers andQwen3students, and is clipped when or is near zero? -
Assessment-critical (Multi-seed reliability, held-out benchmarks, and sequential controls):
- Can you report at least 3 independent training seeds (with fixed step-300 evaluation and paired bootstrap confidence intervals) for the strict weak-to-strong
JustRLQwen3-4BandR1-Distill-7Bruns, as well as results on an untouched held-out math benchmark (e.g.,MATH-500orOlympiadBench) that was not used for KL or checkpoint selection? - Which specific KL configuration (fixed value vs. adaptive KL with initial ) was used for each row of Table 1 and Table 5?
- In Section 3.3 (Figure 7), what happens if
Qwen3-1.7Bis trained for 600 steps usingJustRLalone, or in the reversed orderQuestAJustRL?
- Can you report at least 3 independent training seeds (with fixed step-300 evaluation and paired bootstrap confidence intervals) for the strict weak-to-strong