- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Iterative Reset On-Policy Distillation and Reinforcement Learning (IR-OPD-RL). Each round initializes a student from a fixed SFT checkpoint, distills the latest RL teacher using reward-extrapolated G-OPD, and then applies GRPO. The resulting policy becomes the next teacher. The contribution is this repeated training schedule, rather than a new distillation objective or RL algorithm. Experiments on mathematical reasoning and code generation report advantages over continued RL, while ablations examine stage ordering, repeated alternation, initialization, optimizer restart, and prompt filtering. These results suggest that the schedule can improve post-training endpoints, although the evidence specifically attributing those improvements to reset-induced optimization headroom remains incomplete.
Critical strengths
-
The empirical gains extend across model sizes and task domains. All six model-dataset configurations improve over their initial RL teachers, with substantial gains in several settings. This provides meaningful evidence that the proposed schedule can be more productive than allocating the same measured compute to continued GRPO.
-
The paper includes controls that address plausible alternative explanations. The one-shot versus iterative comparison, initialization ablation, and optimizer-restart and re-filtering controls go beyond simply comparing the final model with its starting teacher. These experiments strengthen the case that the handoff procedure matters, even though they do not fully isolate its essential component.
Critical weaknesses
-
Important inconsistencies in the experimental protocol prevent a confident assessment of comparability and efficiency. Figure 3 reports approximately 6,540 cumulative node-hours for the three-round Qwen3-1.7B Math-57K endpoint, whereas Table 6(b) associates the same 37.00 accuracy with 3,571.90 node-hours. Section 5.5 also describes the comparator as Full-25K, while Table 6(b) labels it Full-57K. Consequently, the claimed cumulative savings do not clearly isolate filtering from changing dataset size. Separately, Appendix A.1 introduces first-round multi-teacher supervision without specifying the teachers or aggregation rule, whereas the main method describes a single-domain teacher recurrence. These ambiguities concern the compute budget and supervision underlying the central results, so they materially affect my assessment.
-
The evidence for the distinctive benefit of resetting is too limited and lacks uncertainty estimates. Table 4 fixes teacher but compares the SFT anchor against the preceding G-OPD checkpoint and initial teacher ; it does not include student initialization from the incoming teacher itself. Table 5 compares against RL-only routes, so it cannot establish whether reset is necessary when the same G-OPD stage is retained. The reset-specific experiment also covers only one model-dataset setting. Moreover, the paper reports no variation across independent training runs or confidence intervals. Sampling 64 responses per problem improves evaluation precision but does not measure training variability. This is important for the relatively small gains supporting repeated alternation and the claimed benefits of later rounds.
Questions for authors
-
Can you reconcile the Math-57K costs in Figure 3 and Table 6(b), specifying which stages each total includes? Is the unfiltered comparator in Section 5.5 Full-25K or Full-57K, and is filtering applied only in round 1? Does the reported cumulative saving compare otherwise identical schedules?
-
Which teachers are used in the first-round multi-teacher configuration, and how are their token-level signals combined? Do the continued-RL, one-shot, OPD, and ExOPD comparisons use equivalent teacher information? Why do first-round G-OPD in Table 1 and ExOPD in Table 2 yield different results despite both being described as reward-extrapolated distillation?
-
Do you have results for initializing the student from the current teacher , resetting its optimizer, and applying the same G-OPD objective and subsequent GRPO budget as the SFT-reset route? This comparison would directly distinguish the benefit of returning to the SFT anchor from the benefit of the distillation stage.
-
How many independent training runs underlie the reported results, and what uncertainty estimates are available for the main compute-matched and reset comparisons? Were endpoints fixed in advance, or selected using benchmark performance?