acceptodds
Under review as a conference paper at ICLR 2027

EvoLPO: Continual Reinforcement Learning through Policy Evolution

15 views

Abstract

Continual reinforcement learning (RL) with verifiable rewards typically resumes each new task from the latest checkpoint. Although recent work reduces forgetting with replay and regularization and ranks starting checkpoints with short adaptation trials, the recovery step that follows adaptation is left out of this choice. This matters because the latest checkpoint may have the highest current score and still be a poor starting point: adaptation changes its capabilities and recovery changes them again. We therefore treat checkpoint selection as a two-stage decision, in which a controller chooses a parent from an archive of past checkpoints, observes the child produced by adaptation and then chooses how to recover earlier capabilities. Both decisions share the utility of the completed child, so each parent is valued by its outcome after recovery rather than before it. Our approach, , keeps the inner RL optimizer fixed and deploys a single policy at every stage. Across mathematics, code-generation and program-optimization streams, EvoLPO improves new-task learning, retention and final score over sequential training and a sequential two-head controller and exceeds tuned replay in final score at equal total compute; the gain over the two-head controller holds across model scales and unseen model families.

Primary area:
reinforcement learning
Keywords:
continual reinforcement learning, reinforcement learning with verifiable rewards, RLVR, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_geSDOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes EvoLPO, a checkpoint selection method for continuous reinforcement learning with verifiable rewards. The authors first point out that, due to the adaptation and recovery problems in continuous reinforcement learning, continuing training from the latest checkpoint is not necessarily optimal. They propose EvoLPO, which maintains a bounded archive of historical checkpoints and then selects the starting point through a two-stage decision process. First, a checkpoint is selected for training on the new task (adaptation), and then an action is chosen: continued training, on-policy prompt replay, or verified donor repair. The authors demonstrate the effectiveness of the method using extensive experiments.

Critical strengths
  1. This paper discusses a timely and interesting problem. Checkpoint selection for continuous learning in RLVR is a specific but less-explored decision problem, and the authors' timely investigation fills a gap in this field.
  2. The authors conduct very detailed, thorough, and rigorous experiments and ablation studies. These include the parent target isolation, confidence interval reporting, and controlled experiments on the archive strategies and the credit-assignment points, covering various tasks. The authors also truthfully report negative results. This makes the entire paper quite convincing.
  3. The authors' discussion of when recovery changes parent choice is insightful, such as the prevalence of ranking reversals, and demonstrates that the method is nontrivial.
Critical weaknesses
  1. The proposed method yields small and statistically insignificant gains, and the main gains are not primarily attributable to the proposed idea. The paper's claimed contribution is the recovery-aware parent target. On its own, this shows a rather low improvement (0.7) under the LOFO algorithm, with most of the gains achieved using only a supervised regression variant (EvoLPO-S). Even ignoring the ablation on these components, the method's performance is still low. such as unseen family improvements.
  2. High cost. The controller costs 2280 H100 hours to prepare and requires approximately 14 deployments to amortize. The high cost coupled with minimal gains significantly diminishes the contribution.
  3. The method is limited in its originality and overclaimed. The authors acknowledge that the targets are standard two-stage backward induction. The archive size is M=4, and calling this policy evolution and evolutionary lineage inflates a small lookup table.
  4. The figure formats are inconsistent and somewhat confusing. For example, Figure 3, especially Figure 3(c), has missing y-labels, overlapping text, and a poorly formatted legend.
Questions for authors

See Weaknesses.

Reviewer_TRDhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EvoLPO is a continual reinforcement learning with verifiable rewards (RLVR) framework that maintains an archive of checkpoints, selects a parent checkpoint to adapt to an arriving task, and subsequently chooses a recovery operation (continued training, on-policy replay, or verified donor distillation) before running an admission and selection procedure to determine the deployed policy. The primary methodological contribution is valuing candidate parent checkpoints under a learned recovery policy rather than under the behavior recovery mixture used during development data collection. The authors validate the approach across multi-stage task streams and model families, reporting a 0.70-point gain (CI [0.10, 1.30]) for the supervised target change alone (EvoLPO-S vs. Seq-2H on held-out stream composition), a 1.79-point gain (CI [1.17, 2.42]) for full EvoLPO in-family (combining target and controller-fitting changes), and a 0.74-point gain (CI [0.18, 1.30]) over tuned replay under an equal total accelerator budget amortized over 96 deployments, alongside extensive ablations covering matched repair, greedy backups, and archive capacity.

Critical strengths

Concrete demonstration of the motivating interaction and comprehensive empirical controls: The opportunity matrix clearly demonstrates that the preferred parent checkpoint can reverse depending on the subsequent recovery mechanism, appropriately distinguishing nonadditivity from decision-changing ranking reversals. Furthermore, the paper includes unusually thorough experimental controls—most notably prospective greedy-backup deployment (Table 39), matched repair, optimizer resets, archive merging, and frozen cross-model-family transfer—that preempt many standard baseline objections.

Rigorous statistical reporting and transparent compute accounting: The statistical evaluation appropriately avoids treating repeated prompts or deployments as independent, and independent recomputation of the eight root effects (Table 26) and interaction intervals (Table 30) confirms the arithmetic accuracy of the headline confidence intervals. Additionally, the compute comparison in Table 33 explicitly accounts for preparation cost (amortizing 2,280 H100-hours over 96 deployments), providing materially stronger and more transparent evidence than deployment-only comparisons.

Critical weaknesses

Confounded target-only ablation and internal specification inconsistencies in the archive algorithm: The primary methodological contrast (the 0.70-point gain of EvoLPO-S over Seq-2H) is not cleanly isolated by the stated losses: Equation 22 applies Huber loss to individual branch returns (sum_r mu_r * L(s - G_r)), whereas Equations 24–25 apply Huber loss to the pre-averaged return (L(s - sum_r mu_r * G_r)), which can yield different minimizers under Huber regression even with identical weights. Separately, the literal archive update rule in Algorithm 1 and Appendix B.4 protects the current incumbent during capacity eviction before selecting the next incumbent; at capacity one, this permanently evicts any newly admitted child and freezes deployment, contradicting the claim that capacity one reduces to "latest + repair" (Table 19, Appendix I.12).

Unresolved cross-stream data partitioning ambiguity and missing equal-budget matched-repair baseline: Appendix C.2 and Table 7 indicate that math and mixed streams share task families while partitioning development and sealed evaluation identifiers independently per stream, leaving open the possibility that a problem used for controller development in one stream appears in another stream's sealed evaluation panel during LOFO fitting. Additionally, because the underlying backward-induction principle is standard, the empirical case relies heavily on compute efficiency; however, the equal-total-budget comparison (Table 33) only grants extra compute to plain replay rather than testing equal-total-budget latest-parent training with matched verified repair (which Table 1 shows is already substantially stronger than replay).

Questions for authors

Supervised Loss Formulation (Eq. 22 vs. Eqs. 24–25): In the implementation of the EvoLPO-S vs. Seq-2H comparison, are both targets constructed as aggregated returns from complete fork records before applying the exact same Huber loss, or does Seq-2H apply Huber loss per branch as written in Equation 22? If the latter, do all relevant residuals lie strictly within the quadratic region of the Huber loss?

Archive Transitions and Capacity-One Behavior: How does the implementation actually handle the initial seed checkpoint slot and capacity-one eviction so that it reduces to "latest + repair" rather than evicting the newly admitted child before incumbent selection (Algorithm 1 / Appendix B.4)? Please provide exact specification details and archive event records verifying when overflow occurs for capacities 1, 4, and 8.

Global Identifier Overlap Audit: Were development and sealed evaluation subject identifiers partitioned globally across the entire benchmark pool or independently per stream? Please provide a global identifier-overlap audit confirming whether any problem identifier used in controller fitting, probing, or incumbent selection in one stream ever appears in the sealed evaluation panel of another stream during LOFO cross-validation.

Equal-Total-Budget Matched Repair Baseline: Can you report the performance of latest-parent training with matched verified repair under an equal total accelerator budget (accounting for any associated development or recovery-controller overhead), analogous to the tuned replay comparison in Table 33?

Trace 1 Retention Tolerance Reconciliation: In Trace 1, the entering old-task mean is 0.884, whereas the three completed old-task components average approximately 0.767. How is this reconciled with admission under a two-point per-task retention tolerance—are these evaluated on different panels or measuring different quantities?

Executable Configurations and Manifests: Given that key hyperparameters (adaptation/recovery allocation, probe budgets, Huber thresholds) are delegated to run manifests and Appendix H notes that deployment records and audit scripts are omitted from the supplement, can you provide anonymized fork/archive transition records, split manifests, and executable configurations sufficient to reproduce the main tables?

Reviewer_8fTkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents EvoLPO (Evolutionary Lineage Policy Optimization), an evolutionary approach for continual reinforcement learning with verifiable rewards, which adaptively selects which checkpoint to continue to update from, rather than selecting the latest checkpoint. At task boundaries, EvoLPO chooses a parent from a bounded archive of past checkpoints, updates it with an inner RL optimizer, and then selects one of three recovery actions from continued training, on-policy replay, or “verified donor repair”. Experiments on mathematics, code generation, and program optimization on primarily the Qwen3 model family (1.7B, 4B, 8B, 14B, 30B) and other model families show that EvoLPO improves on learning new tasks and old task retention.

Critical strengths

The results of EvoLPO over baseline sequential training and replay-based approach are promising (Figure 3 and Figure 4). Experiments span a range of evaluation domains (math, code generation, program optimization), as well as different model families and parameter counts. The central question posed by the work is definitely true of stochastic optimization, that the weights of the latest checkpoint may be a bad initialization when training subsequent tasks (although this relies on knowing where task boundaries exist).

Critical weaknesses
  • The writing is particularly jargony throughout, such as using terms like “provisional child”, “completed child”, “verified donor repair”, “parent-recovery pair”, “recovery action”, “two-head controller”, “hypervolume”. Particularly, “verified donor repair”, one of the three recovery actions used in EvoLPR, is not formally defined until Appendix B.3 (with nearly 40 pages of appendix). I would recommend the authors do a thorough manual pass through the manuscript to improve the readability, as it is difficult to judge the current presentation of the proposed approach.

  • The method assumes task boundaries and memory capacity to store an archive of past checkpoints. The maximum archive size used seems to be 8. Is there a disk storage limitation for memory sizes beyond this? Figure 8 seems to suggest that archive capacity saturates between 4 and 8, but Table 19 seems to suggest that the archive behaves differently for different sizes (4, hypervolume and 8, no eviction).

Questions for authors
  • L227: What is a “hypervolume”? What is a “disjoint development panel”?

  • Section F.2: Is this definition of retention ratio effectively an inverse measure of catastrophic forgetting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.