- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies continual RLVR for reasoning models and introduces Continual Reasoning Gym (CRG) to evaluate learning and forgetting across tasks. The authors find that sequential RLVR suffers from limited forgetting but still falls behind multitask training. They propose Continual Prompt Replay (CPR), which reuses previous-task prompts with responses generated by the current policy. CPR achieves performance comparable to multitask RLVR on average.
Critical strengths
- The paper tackles a practical problem: how to continuously improve reasoning models without retraining on all previous tasks.
- The proposed benchmark covers both text and visual reasoning tasks. The performance decomposition also provides a useful perspective on learning and transfer.
- The experiments cover several continual learning baselines and include useful ablations to examine the effectiveness of CPR.
Critical weaknesses
Critical Weaknesses
- The theoretical analysis does not fully explain why CPR works better than sequential RLVR. Corollary 2 in Section 5.3 only shows that CPR can have a non-negative effect on task performance, which does not necessarily mean it is better than sequential RLVR. Equation (14) in Appendix C.2 also shows that the advantage depends on how the gradients from previous tasks compare with those from the current task. It would help to clarify when CPR is expected to outperform sequential RLVR.
- The evidence for shared reasoning is not entirely convincing. Figure 4 shows positive gradient alignment across tasks, but this does not necessarily mean the tasks share reasoning abilities. In Figure 5, the base model already finds the correct solution to the Jugs problem but fails to give a valid answer. The improvement could therefore come from better answer generation rather than better reasoning. More direct evidence would help support the shared reasoning claim.
- The benefit of CPR over simple on-policy prompt replay is unclear. The main idea is to replay previous-task prompts and generate new responses with the current policy. However, Figure 7 only compares CPR with replaying old responses, while Figure 12 shows almost no performance difference after removing success-rate filtering (63.2% vs. 63.3%). A comparison with uniform random prompt replay would help show whether the additional design choices in CPR are necessary.
- The results may be sensitive to the order of tasks. Table 3 uses fixed task sequences, but the paper does not explain why these orders were chosen. Since task order can affect both transfer and forgetting, it is unclear whether CPR would perform similarly with different sequences. Experiments with other task orders would make the results more convincing.
Other minor Issues
- The replay selection rule in Appendix B.2 could be clearer. With five rollouts per prompt, the eligibility range [0.24, 0.70] only includes pass rates of 0.4 and 0.6. Both are equally close to 0.5, so it is unclear how the method prioritizes eligible prompts.
- Some wording could be improved. For example, the phrase "Reasoning Gym's largest category algorithmic and its dedicated algebra category" in Section 4.1 reads awkwardly and could be rephrased for clarity.
- Some figures take up more space than necessary. The authors could consider reducing the size of Figures 2, 4, 6, and 7 to make room for more important details in the main text.
Questions for authors
See weakness.