acceptodds
Withdrawn from ICLR 2027

Reopening Reinforcement Learning Headroom in Language Models with Iterative Reset On-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards can substantially improve language-model reasoning, yet continued optimization often enters a low-gain regime. We introduce **Iterative Reset On-Policy Distillation and Reinforcement Learning** (**IR-OPD-RL**), which repeatedly resets a student to a fixed supervised fine-tuning checkpoint, transfers the current teacher's behavior through reward-extrapolated G-OPD, and launches a new GRPO stage from the resulting student. This design transfers behavioral knowledge without inheriting the teacher's parameters or optimizer history. Across Qwen3-1.7B and Qwen3-4B on six mathematical-reasoning and code-generation configurations, IR-OPD-RL improves domain-average accuracy over the initial RL teachers by 3.30-8.23 points. On Qwen3-4B, IR-OPD-RL further exceeds ExOPD, the strongest tested single-stage baseline, by 5.42 points on Math-57K and 2.45 points on Code-25K. Under matched compute on Qwen3-1.7B, IR-OPD-RL outperforms continued RL by up to 2.35 points on Math-57K and 3.70 points on Code-25K. Order and initialization ablations further favor distillation before RL and resetting the student from the fixed SFT anchor. These results show that immediate checkpoint accuracy need not predict subsequent RL improvement and that reset on-policy distillation can renew optimization headroom.

Primary area:
reinforcement learning
Keywords:
OPD, RL

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_85M3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Iterative Reset On-Policy Distillation and Reinforcement Learning (IR-OPD-RL). Each round initializes a student from a fixed SFT checkpoint, distills the latest RL teacher using reward-extrapolated G-OPD, and then applies GRPO. The resulting policy becomes the next teacher. The contribution is this repeated training schedule, rather than a new distillation objective or RL algorithm. Experiments on mathematical reasoning and code generation report advantages over continued RL, while ablations examine stage ordering, repeated alternation, initialization, optimizer restart, and prompt filtering. These results suggest that the schedule can improve post-training endpoints, although the evidence specifically attributing those improvements to reset-induced optimization headroom remains incomplete.

Critical strengths
  1. The empirical gains extend across model sizes and task domains. All six model-dataset configurations improve over their initial RL teachers, with substantial gains in several settings. This provides meaningful evidence that the proposed schedule can be more productive than allocating the same measured compute to continued GRPO.

  2. The paper includes controls that address plausible alternative explanations. The one-shot versus iterative comparison, initialization ablation, and optimizer-restart and re-filtering controls go beyond simply comparing the final model with its starting teacher. These experiments strengthen the case that the handoff procedure matters, even though they do not fully isolate its essential component.

Critical weaknesses
  1. Important inconsistencies in the experimental protocol prevent a confident assessment of comparability and efficiency. Figure 3 reports approximately 6,540 cumulative node-hours for the three-round Qwen3-1.7B Math-57K endpoint, whereas Table 6(b) associates the same 37.00 accuracy with 3,571.90 node-hours. Section 5.5 also describes the comparator as Full-25K, while Table 6(b) labels it Full-57K. Consequently, the claimed cumulative savings do not clearly isolate filtering from changing dataset size. Separately, Appendix A.1 introduces first-round multi-teacher supervision without specifying the teachers or aggregation rule, whereas the main method describes a single-domain teacher recurrence. These ambiguities concern the compute budget and supervision underlying the central results, so they materially affect my assessment.

  2. The evidence for the distinctive benefit of resetting is too limited and lacks uncertainty estimates. Table 4 fixes teacher T1T_1 but compares the SFT anchor against the preceding G-OPD checkpoint and initial teacher T0T_0; it does not include student initialization from the incoming teacher T1T_1 itself. Table 5 compares against RL-only routes, so it cannot establish whether reset is necessary when the same G-OPD stage is retained. The reset-specific experiment also covers only one model-dataset setting. Moreover, the paper reports no variation across independent training runs or confidence intervals. Sampling 64 responses per problem improves evaluation precision but does not measure training variability. This is important for the relatively small gains supporting repeated alternation and the claimed benefits of later rounds.

Questions for authors
  1. Can you reconcile the Math-57K costs in Figure 3 and Table 6(b), specifying which stages each total includes? Is the unfiltered comparator in Section 5.5 Full-25K or Full-57K, and is filtering applied only in round 1? Does the reported cumulative saving compare otherwise identical schedules?

  2. Which teachers are used in the first-round multi-teacher configuration, and how are their token-level signals combined? Do the continued-RL, one-shot, OPD, and ExOPD comparisons use equivalent teacher information? Why do first-round G-OPD in Table 1 and ExOPD in Table 2 yield different results despite both being described as reward-extrapolated distillation?

  3. Do you have results for initializing the student from the current teacher T1T_1, resetting its optimizer, and applying the same G-OPD objective and subsequent GRPO budget as the SFT-reset route? This comparison would directly distinguish the benefit of returning to the SFT anchor from the benefit of the distillation stage.

  4. How many independent training runs underlie the reported results, and what uncertainty estimates are available for the main compute-matched and reset comparisons? Were endpoints fixed in advance, or selected using benchmark performance?

Reviewer_AZpCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Iterative Reset On-Policy Distillation and Reinforcement Learning (IR-OPD-RL), a multi-round training framework designed to overcome performance plateaus in reinforcement learning with verifiable rewards (RLVR). In each round, a student is reinitialized from a fixed SFT checkpoint, learns the current RL teacher's behavior through reward-extrapolated on-policy distillation (G-OPD), and then undergoes further GRPO training. The resulting policy becomes the teacher for the next round, allowing behavioral knowledge to be transferred without inheriting previous model parameters or optimizer states. Experiments on Qwen3-1.7B and Qwen3-4B across mathematical reasoning and code generation tasks report accuracy improvements of 3.30–8.23 percentage points over the initial RL teachers. The authors also compare the method against existing distillation approaches and continued RL under matched compute budgets. Additional ablation studies examine the effects of student initialization, training-stage ordering, iterative versus single-stage training, and data filtering.

Critical strengths

Practical motivation and encouraging empirical results. The paper addresses the relevant problem of diminishing returns during continued RLVR training. The proposed iterative reset and distillation strategy achieves consistent performance improvements across the evaluated mathematical reasoning and code generation tasks. In particular, the compute-matched comparisons against continued RL provide useful evidence that the gains are not solely attributable to additional training computation.

Critical weaknesses

1. Limited methodological novelty. The proposed framework mainly combines existing techniques, including iterative policy reset and relearning (ITER), on-policy distillation (G-OPD), and GRPO. The core idea of transferring an improved policy to a freshly initialized student before resuming RL has already been explored in prior work. The paper does not sufficiently establish what fundamentally new methodological contributions arise from applying this strategy to LLM post-training beyond combining existing components.

2. Insufficient experimental evidence and mechanistic understanding. The evaluation is limited to two relatively small models from the same Qwen3 family, with compute-matched comparisons primarily conducted on the 1.7B model. Moreover, the main comparisons against existing distillation methods use different training budgets, making it difficult to assess the practical efficiency of the proposed approach. While the authors demonstrate that resetting the student can improve subsequent RL performance, they provide limited analysis of why this reset restores optimization headroom. These limitations weaken the generalizability and scientific significance of the findings.

Questions for authors

1. Methodological novelty. What are the fundamental differences between IR-OPD-RL and prior iterative reset-and-relearning methods such as ITER and MOPD?

2. Baseline comparison. How does IR-OPD-RL compare with multi-round G-OPD or ExOPD under identical training compute budgets?

3. Mechanistic explanation. Why does resetting to the fixed SFT checkpoint improve subsequent RL optimization compared with continuing from the teacher's parameters?

4. Generalizability. Do the observed improvements persist on larger models or different model families beyond Qwen3-1.7B and Qwen3-4B?

Reviewer_D2rHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multi-round post-training framework aimed at addressing the issue that language models often get stuck in the low-yield optimization stage during reinforcement learning training. The core mechanism of this method is to reinitialize the student model to a fixed supervised fine-tuning checkpoint in each training round, then learn the current behavior of the teacher model through the generalized policy distillation technique based on reward extrapolation, and finally start a new round of group relative policy optimization based on this student model. The authors claim that this design can transfer behavioral knowledge while avoiding inheriting the parameters and optimizer history states of the teacher model, thereby achieving significant accuracy improvements in mathematical reasoning and code generation tasks.

Critical strengths
  1. The research motivation addresses the existing performance saturation phenomenon in the current reinforcement learning training of large models, and exploring the influence of the initial state of the model on the subsequent optimization potential has certain practical enlightenment significance.

  2. The paper conducted extensive experimental verification under multiple model scales and different task configurations, and provided relatively abundant ablation experiment data to support the necessity of each component in its method.

Critical weaknesses
  1. The framework proposed in the paper essentially combines the existing iterative re-learning paradigm in deep reinforcement learning with the existing strategy distillation technology directly. This engineering process combination does not bring about a fundamental breakthrough at the algorithmic level, and the theoretical contribution is very weak.

  2. The paper claims to be superior to continuous reinforcement learning under the condition of matching computing resources, but the actual computational cost and convergence time of the process of repeatedly resetting the model and re-performing distillation and training have not been transparently and rigorously calculated. The so-called matching of computing resources may ignore the implicit costs brought by the reset and additional distillation stages.

  3. The author simply attributes the performance gain to avoiding the inheritance of the optimizer's historical state, but does not provide any visual evidence of optimization trajectories or loss surfaces to prove this assumption. The improvement in performance could be simply due to the introduction of regularization effects similar to learning rate restarts caused by multiple resets, rather than the superiority of the design of the method itself.

Questions for authors

In this field, the vast majority of similar papers perform worse than the vanilla algorithm in both real-world scenarios and large-scale scenarios. It's an article with no novelty or persuasiveness at all.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.