- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes EvoLPO, a checkpoint selection method for continuous reinforcement learning with verifiable rewards. The authors first point out that, due to the adaptation and recovery problems in continuous reinforcement learning, continuing training from the latest checkpoint is not necessarily optimal. They propose EvoLPO, which maintains a bounded archive of historical checkpoints and then selects the starting point through a two-stage decision process. First, a checkpoint is selected for training on the new task (adaptation), and then an action is chosen: continued training, on-policy prompt replay, or verified donor repair. The authors demonstrate the effectiveness of the method using extensive experiments.
Critical strengths
- This paper discusses a timely and interesting problem. Checkpoint selection for continuous learning in RLVR is a specific but less-explored decision problem, and the authors' timely investigation fills a gap in this field.
- The authors conduct very detailed, thorough, and rigorous experiments and ablation studies. These include the parent target isolation, confidence interval reporting, and controlled experiments on the archive strategies and the credit-assignment points, covering various tasks. The authors also truthfully report negative results. This makes the entire paper quite convincing.
- The authors' discussion of when recovery changes parent choice is insightful, such as the prevalence of ranking reversals, and demonstrates that the method is nontrivial.
Critical weaknesses
- The proposed method yields small and statistically insignificant gains, and the main gains are not primarily attributable to the proposed idea. The paper's claimed contribution is the recovery-aware parent target. On its own, this shows a rather low improvement (0.7) under the LOFO algorithm, with most of the gains achieved using only a supervised regression variant (EvoLPO-S). Even ignoring the ablation on these components, the method's performance is still low. such as unseen family improvements.
- High cost. The controller costs 2280 H100 hours to prepare and requires approximately 14 deployments to amortize. The high cost coupled with minimal gains significantly diminishes the contribution.
- The method is limited in its originality and overclaimed. The authors acknowledge that the targets are standard two-stage backward induction. The archive size is M=4, and calling this policy evolution and evolutionary lineage inflates a small lookup table.
- The figure formats are inconsistent and somewhat confusing. For example, Figure 3, especially Figure 3(c), has missing y-labels, overlapping text, and a poorly formatted legend.
Questions for authors
See Weaknesses.