acceptodds
Withdrawn from ICLR 2027

Beyond Forgetting: Diagnosing and Harnessing Shared Reasoning in Continual RLVR

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards (RLVR) commonly post-trains reasoning models on multiple tasks, while rerunning multitask RLVR (MTRL) as new tasks are added makes capability expansion costly. We therefore study continual RLVR, which updates the existing model as each task arrives. The central question is whether a model updated this way can perform as well as a jointly trained model. To answer this question, we introduce Continual Reasoning Gym, a continual-RLVR environment that organizes text and visual reasoning tasks into five task sequences. In this setting, we identify two key observations: Continual RLVR exhibits modest forgetting, yet its final performance remains below that of MTRL. To understand the latter, we decompose final performance and show that forgetting accounts for only part of the gap. To explain the former, we identify shared reasoning: transferable reasoning structure allows training on one task to support others on average. We therefore introduce Continual Prompt Replay (CPR), which harnesses shared reasoning to improve learning on the arriving and future tasks by replaying previous-task prompts and regenerating their responses with the current policy. On average, only CPR reaches MTRL-level performance.

Primary area:
reinforcement learning
Keywords:
continual reinforcement learning, reinforcement learning with verifiable rewards (RLVR), large language model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RaPTOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual RLVR for reasoning models and introduces Continual Reasoning Gym (CRG) to evaluate learning and forgetting across tasks. The authors find that sequential RLVR suffers from limited forgetting but still falls behind multitask training. They propose Continual Prompt Replay (CPR), which reuses previous-task prompts with responses generated by the current policy. CPR achieves performance comparable to multitask RLVR on average.

Critical strengths
  1. The paper tackles a practical problem: how to continuously improve reasoning models without retraining on all previous tasks.
  2. The proposed benchmark covers both text and visual reasoning tasks. The performance decomposition also provides a useful perspective on learning and transfer.
  3. The experiments cover several continual learning baselines and include useful ablations to examine the effectiveness of CPR.
Critical weaknesses

Critical Weaknesses

  1. The theoretical analysis does not fully explain why CPR works better than sequential RLVR. Corollary 2 in Section 5.3 only shows that CPR can have a non-negative effect on task performance, which does not necessarily mean it is better than sequential RLVR. Equation (14) in Appendix C.2 also shows that the advantage depends on how the gradients from previous tasks compare with those from the current task. It would help to clarify when CPR is expected to outperform sequential RLVR.
  2. The evidence for shared reasoning is not entirely convincing. Figure 4 shows positive gradient alignment across tasks, but this does not necessarily mean the tasks share reasoning abilities. In Figure 5, the base model already finds the correct solution to the Jugs problem but fails to give a valid answer. The improvement could therefore come from better answer generation rather than better reasoning. More direct evidence would help support the shared reasoning claim.
  3. The benefit of CPR over simple on-policy prompt replay is unclear. The main idea is to replay previous-task prompts and generate new responses with the current policy. However, Figure 7 only compares CPR with replaying old responses, while Figure 12 shows almost no performance difference after removing success-rate filtering (63.2% vs. 63.3%). A comparison with uniform random prompt replay would help show whether the additional design choices in CPR are necessary.
  4. The results may be sensitive to the order of tasks. Table 3 uses fixed task sequences, but the paper does not explain why these orders were chosen. Since task order can affect both transfer and forgetting, it is unclear whether CPR would perform similarly with different sequences. Experiments with other task orders would make the results more convincing.

Other minor Issues

  1. The replay selection rule in Appendix B.2 could be clearer. With five rollouts per prompt, the eligibility range [0.24, 0.70] only includes pass rates of 0.4 and 0.6. Both are equally close to 0.5, so it is unclear how the method prioritizes eligible prompts.
  2. Some wording could be improved. For example, the phrase "Reasoning Gym's largest category algorithmic and its dedicated algebra category" in Section 4.1 reads awkwardly and could be rephrased for clarity.
  3. Some figures take up more space than necessary. The authors could consider reducing the size of Figures 2, 4, 6, and 7 to make room for more important details in the main text.
Questions for authors

See weakness.

Reviewer_LwWVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies continual reinforcement learning with verifiable rewards, focusing on why sequential training underperforms multitask RLVR despite relatively limited catastrophic forgetting. The authors introduce Continual Reasoning Gym, covering five text and visual reasoning task sequences. Motivated by positive gradient alignment across reasoning tasks, they propose Continual Prompt Replay, which replays previous-task prompts while regenerating responses using the current policy.

Critical strengths
  1. The paper highlights that the performance gap between continual and multitask RLVR cannot be explained by forgetting alone. The decomposition into FWT, TLG, and BWT provides a useful diagnostic framework for understanding continual reasoning performance.

  2. CPR substantially improves over sequential RLVR and achieves competitive performance with multitask training on average. The experiments include both LLM and VLM reasoning tasks, and the ablations demonstrate the importance of regenerating responses with the current policy.

Critical weaknesses

1.Limited methodological novelty.CPR primarily combines experience replay with on-policy response regeneration, both of which are established techniques in reinforcement learning. While the proposed motivation is interesting, the method itself is relatively straightforward. The paper would benefit from a clearer distinction between CPR and conventional replay-based RL approaches, particularly on-policy replay with verifiable rewards.

2.The gradient-alignment analysis is primarily conducted on the algorithmic reasoning setting, making it unclear whether the same explanation holds across different reasoning domains and modalities. Moreover, the experiments use fixed task sequences and two model backbones, without clearly reported independent training seeds or uncertainty estimates. Since CPR underperforms sequential RLVR on the VLM positional task, additional analysis of task heterogeneity and negative transfer would strengthen the conclusions.

Questions for authors
  1. How does CPR compare against a simpler baseline that uniformly replays previous-task prompts with fresh on-policy rollouts? This would help isolate the contribution of the proposed replay strategy.

  2. Does the positive gradient-alignment observation hold across the other four task sequences? How does alignment relate to CPR's effectiveness, especially in settings where CPR underperforms sequential RLVR?

Reviewer_yNjsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

the core idea of this paper is to replay previous prompts in RLVR training in continual learning setting where the new tasks gradually arrives. the authors provide some theoretical bounds on their core claims and also demonstrate improvements across several text and multimodal training settings.

Critical strengths

the paper is easy to follow and the setup is clear.

the method is simple but seems effective compared with other baselines.

Critical weaknesses
  1. this previous paper (https://arxiv.org/abs/2603.21177, "Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts") seems very similar to this paper but is not mentioned or compared with.

  2. overall, the method is only proved at a very small scale and it's unclear how it scales

  3. I think the setup is a bit fake and does not reflect real-world model training

Questions for authors

Kind words: if the authors are at school, I strongly suggest in the future to choose a more realistic setup, this paper's toy setting is not how industry actually trains models.

How useful do you think those proofs and bounds are in this paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.