acceptodds
Under review as a conference paper at ICLR 2027

Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models, yet aggregate accuracy gains can conceal a hidden cost: previously solved problems may regress as training proceeds. We characterize this phenomenon as correct-set turnover, capturing the coupled dynamics of solution acquisition and regression, and make retention an explicit training objective alongside acquisition. In group-relative RLVR, mastered prompts provide no reward gradient when all rollouts are correct, yet can still drift under updates driven by other prompts. This observation motivates a timely-review principle: previously mastered prompts should be revisited before such regression accumulates. We propose ReMind, a retention-aware review mechanism that tracks mastered prompts, periodically revisits them with fresh rollouts, and requeues those that regress. Through pre-rollout batch replacement, ReMind preserves the per-step rollout budget. Across 20 image-text, video, and text-only reasoning benchmarks, ReMind improves average accuracy by 3.81 and 3.44 percentage points over GRPO and DAPO, respectively, while consistently improving retention across modalities and algorithms.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning with Verifiable Rewards, Reasoning Models, GRPO, Multimodal Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_E49fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The idea is intuitive, simple to implement, and adds little complexity to the standard RLVR pipeline. The method also shows consistent improvements when combined with several RLVR algorithms, suggesting that it has practical value beyond a single training setup.

Critical strengths

The idea is intuitive, simple to implement, and adds little complexity to the standard RLVR pipeline. The method also shows consistent improvements when combined with several RLVR algorithms, suggesting that it has practical value beyond a single training setup.

Critical weaknesses

My main concern is that the paper does not clearly separate an anti-forgetting effect from a curriculum or sample-selection effect. The forgetting studied here is different from the conventional continual-learning setting, where forgetting usually results from sequentially learning different tasks or distributions. Here, regression happens within the same RLVR training distribution. More importantly, the ReMind queue naturally focuses training on prompts that were once solved but are no longer consistently solved. These prompts are likely to be neither too easy nor too difficult for the current model, and therefore may simply provide more useful training signals. In this sense, ReMind may also behave like a history-aware curriculum-learning or hard-example-sampling method. The current experiments show that this review strategy works, but do not fully establish that preventing forgetting is the main reason for the improvement. I also find the definition of regression somewhat noisy. A prompt changing from 8/8 correct rollouts to 7/8 does not necessarily mean that the model truly forgot the solution; this may partly result from stochastic generation. Since mastery and regression are defined using a small number of rollouts, a frozen-policy repeated-evaluation control would make this claim more convincing.

Questions for authors
  1. How much of the improvement from ReMind comes from preventing forgetting, and how much may come from selecting samples that are simply better matched to the current model difficulty? Could the authors provide a baseline that matches the current difficulty of ReMind-selected prompts but does not use their mastery history?
  2. Can the authors better separate true regression from rollout variance? For example, if the model parameters are frozen and mastered prompts are repeatedly re-evaluated, how often would they become non-mastered purely due to sampling noise? More generally, since SGD naturally causes per-example performance fluctuations, how much of the observed turnover should really be interpreted as forgetting?
  3. How sensitive is ReMind to the difficulty and reward distribution of the training data? For example, if most prompts are very difficult and rarely become mastered, or if most prompts are very easy and remain consistently correct, would the review queue still provide meaningful benefit?
5 comments

We thank the reviewer for the constructive comments. We ran all three suggested experiments; the results are below.

W1 / Q1: Anti-forgetting vs. curriculum / sample selection

New baseline. As suggested, we added a difficulty-matched review baseline (DM-Review). It uses the same review budget and schedule as ReMind (r=0.10r=0.10, f=5f=5, ≈2%\approx 2\% of training prompts), but its review prompts are previously seen prompts that were never mastered, sampled to match the current success rate q^\hat q of ReMind's review samples. It matches ReMind in budget and difficulty, and differs only in not using mastery history.

MethodVistaVisionWeMathVerseMath AvgMMMU-PMMBMM-SZeroBGen. AvgOverall
GRPO76.2048.8256.5759.5260.2855.7890.2972.2022.6060.2260.25
GRPO + DM-Review (new)75.9049.7557.1959.6960.6356.0490.1971.9023.6560.4560.54
GRPO + ReMind79.1056.8166.3861.5265.9558.7890.4173.4026.0562.1664.06

ReMind is not a data-selection method. DM-Review is on par with GRPO (+0.29+0.29), while ReMind is +3.52+3.52 above it and higher on all eight benchmarks. Selecting prompts of suitable difficulty therefore does not explain the gain; the mastery history does. The main results (Table 1) support the same conclusion: ReMind on plain GRPO (64.06) already outperforms the selection-based methods VCRL (variance-based curriculum, 63.42) and DAPO (online filtering, 61.02), and it further improves both when combined with them (+1.54+1.54 and +3.44+3.44). ReMind is thus complementary to curriculum and filtering, not a variant of them.

W2 / Q2: True regression vs. rollout variance

Frozen-policy control. We froze each model and evaluated the 700 validation prompts twice with independent rollout groups (A and B). KK is the number of rollouts per prompt in each group, and a prompt is all-correct if all KK rollouts are correct. A→B is the fraction of prompts that are all-correct in group A but not in group B, and B→A is the reverse. This fall-out rate measures the apparent regression produced by sampling alone.

Frozen modelKKA→BB→APooled
Base815/84 = 17.86%14/83 = 16.87%17.37%
GRPO823/248 = 9.27%20/245 = 8.16%8.72%
GRPO + ReMind824/260 = 9.23%26/262 = 9.92%9.58%
Base1615/69 = 21.74%9/63 = 14.29%18.18%
GRPO1620/218 = 9.17%20/218 = 9.17%9.17%
GRPO + ReMind1620/240 = 8.33%16/236 = 6.78%7.56%

Sampling noise alone does not explain the observed regression. The reviewer is right that sampling alone produces apparent regression. This noise floor is about 9% for both GRPO and GRPO + ReMind (7–10% across both directions and both values of KK). Two observations go beyond it. First, during training, 21.1% of review records (269/1,275, K=8K=8) are mixed, which is 2.2–2.4$\times$ the floor. Second, since the floor is the same for both methods, it cannot explain the gap between them: on the 700 validation prompts, correct-set loss is 195 for GRPO versus 161 for ReMind (Fig. 1a). The delay sweep (Fig. 2b) points the same way: the retention gap grows monotonically with delay (2.3→4.6→5.4→6.92.3 \to 4.6 \to 5.4 \to 6.9 pp), while sampling noise and zero-mean SGD fluctuation do not depend on delay. We will report the noise floor in the revision.

W3 / Q3: Sensitivity to data difficulty

New experiment. Starting from the MMFineReason-123K training dataset, we froze the initial 8B model and generated 8 responses per prompt. Prompts with 0–3 correct responses form the Hard subset, and prompts with 4–8 correct responses form the Easy subset. We trained GRPO and GRPO + ReMind on each subset, with all other settings unchanged.

MethodMathVistaMathVisionWeMathMathVerseMath AvgMMMU-ProMMBenchMM-StarZeroBenchGeneral AvgOverall
Easy GRPO76.8552.0358.1058.3661.33555.9490.4770.6719.4659.1460.24
Easy ReMind79.4556.6066.2960.5565.7259.2790.9975.0023.8062.2763.99
Hard GRPO78.0054.4260.0058.1862.6557.6690.1971.7019.6159.7961.22
Hard ReMind78.2556.3267.7160.0865.5959.3090.5275.2725.7562.7164.15

ReMind is robust to the difficulty distribution. ReMind improves over GRPO on both subsets (+3.75+3.75 on Easy, +2.93+2.93 on Hard) and on all eight benchmarks in each case, comparable to the gain on the original training set (+3.81+3.81). Thus, ReMind remains effective across the tested difficulty regimes, although the size of the gain depends on the data distribution.

Overall, these experiments show that ReMind's gain is not explained by current prompt difficulty alone, and that the observed online regression signal is larger than the fall-out caused by rollout sampling under a frozen policy.

We hope these results address the reviewer's concerns, and we will include all three experiments in the revision. We would be grateful if the reviewer would consider updating the rating.

Dear Reviewer,

We hope this message finds you well. We are writing to gently follow up on our rebuttal, in which we addressed each of your comments and included the additional experiments you suggested. Since the discussion period will close in less than 24 hours, we would be grateful if you could take a look at our responses when you have a moment and let us know whether they resolve your concerns. If anything remains unclear, we will do our best to clarify it right away.

We understand how demanding the reviewing schedule is, and we sincerely thank you for the time and effort you have devoted to our paper.

Best regards, The Authors

Reviewer_Xfj5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose a method to mitigate the forgetting problem in RLVR. Their approach involves periodically re-solving mastered (1 reward) prompts to ensure that RLVR training does not forget old prompts while acquiring knowledge from new prompts.

Critical strengths
  • The paper tackles an important problem of forgetting that happens in current RLVR training.
  • The work experiments across multiple domains and considers multiple baselines. Experimental protocol is rigorous.
Critical weaknesses
  • I would like to understand the setting in Table 3 better. Specifically, as you increase the review budget from  1%~1\% to 10%10\% could the authors show a plot similar to the one in FIg. 1? Does the number of forgotten responses go down as you increase the review budget?

  • It is interesting to see that a significant portion of the forgetting still remains even after ReMind is applied. It would help to provide some intuition why this is the case. For e.g., if one did not care about sample efficiency of RL training, do the authors think that if the review budget was increased then the model would not forget? (I know learning would be slow as less budget would be spent on acquisition) Or is there some tension between acquisition and forgetting due to which there will always be some irreducible forgetting?

Questions for authors
  • Could the authors provide more intuition (perhaps through a discussion in the appendix) for why their method results in such a drastic change in entropy compared to GRPO, despite differing from GRPO's loss for only about 33% of the wall-clock time?
4 comments

We sincerely thank the reviewer for the thoughtful and constructive comments, and for the positive assessment of our problem setting and experiments. We address each point below with new results.

W1. Does forgetting decrease as the review budget increases?

Yes. At a matched training step, forgetting decreases monotonically with the review budget, and the benefit saturates. The first table compares all settings at step 200, the end of the window plotted in Fig. 1(a). We use a matched step because the ever-solved count is cumulative, while the checkpoints evaluated in Table 3 are [selected per setting by validation accuracy] and fall at different steps.

Setting (step 200)Ever-solvedCurrently correctForgotten ↓\downarrow
GRPO489294195
ReMind $\sim$1%504325179
ReMind $\sim$2%501340161
ReMind $\sim$4%499345154
ReMind $\sim$10%501349152

For direct correspondence with Table 3, the second table reports the same statistics at the checkpoints evaluated there.

Setting (Table 3 checkpoint)Ever-solvedCurrently correctForgotten ↓\downarrowTable 3 Avg.
GRPO51533817760.25
ReMind $\sim$1%53134918262.98
ReMind $\sim$2%52135916264.06
ReMind $\sim$4%52335816563.19
ReMind $\sim$10%52135316862.83

At these checkpoints, the default 2% budget has the fewest forgotten samples (162 vs. 177 for GRPO) and the highest benchmark accuracy, and larger budgets give no further reduction (165 and 168). The 1% setting does not reduce the forgotten count at this point (182 vs. 177), although it has solved more samples (531 vs. 515) and has more currently correct (349 vs. 338). The effect of budget on forgetting is therefore clearest at step 200 and narrows by the end of training. The ever-solved count is not lower than GRPO's in either table, so review does not slow acquisition.

W2. Why does a significant portion of forgetting remain?

We believe it is the latter. Timely review catches regression early and repairs it, but it cannot remove the cause of regression. The cause is learning new problems: all problems share the same parameters, so an update that helps new problems also changes the model's answers on old ones, sometimes for the worse. For instance, if many new problems can be solved by estimating a value from a chart, the model may start doing the same on an old problem that needs exact calculation. Review can act only after a regression has begun, because a sample that is still fully correct gives no training signal. As long as training keeps learning new problems, new regressions keep starting, and at any moment some of them are still in progress. This is the forgetting that remains.

A larger review budget does not change this. Review affects the model only through samples that have already regressed. At the default budget, about 79% of reviewed samples are still fully correct and need no repair, so a larger budget mostly adds more such checks. And because interference continues, regression keeps recurring even in repaired samples: among samples that passed review by step 200, 11.5% are no longer all-correct at a later checkpoint (15.3% under GRPO). Forgetting therefore remains at about 150–170 samples in both tables above instead of approaching zero.

Q1. Why does entropy differ so much with only about 3% of wall-clock time?

Review takes little time, but it is the only point in training where the model is told that a strategy it relies on has started to fail. Under GRPO this signal does not exist. The model trains on its own outputs, so a strategy that often succeeds on new problems is generated more and reinforced more, while other strategies fade. Old problems on which that strategy fails never return, so nothing corrects the over-use, and the outputs concentrate on a few strategies. This is the drop in entropy, and it occurs in the same period in which GRPO's forgetting rises.

Review supplies the missing signal, and it differs in kind from the signal of fresh problems:

Groups with a training signalFreshReview
Mean pass rate46.04%75.66%
1–2 of 8 rollouts correct37.32%2.10%
6–7 of 8 rollouts correct25.67%72.73%

Fresh groups often contain only one or two correct rollouts, so the update reinforces one particular response. Review groups mostly contain only one or two failures, so the update penalizes those failures and keeps the existing solutions. Because the model trains on its own outputs, each correction also changes what is generated afterwards, so its effect accumulates over the whole run. This is why a small amount of review has a large effect: in the step-200 table of W1, a 2% budget removes 34 forgotten samples, and a further fivefold increase removes only 9 more; at the Table 3 checkpoints, budgets above 2% give no further reduction. The higher entropy reflects more retained solutions rather than noisier outputs: among 292 fragile validation samples, ReMind leaves 20 with no correct rollout in 64 attempts, compared with 32 for GRPO.

Revision. We will add Fig. 1(a)-style forgetting curves for all review budgets and the two tables above to the revision. As the reviewer suggests, we will also add the discussion for W2 and Q1 as a new appendix section.

Reviewer_Xfj5OpenReview ↗

Reviewer Response

Thanks for your responses, I will keep my score

Dear Reviewer Xfj5,

Thank you for taking the time to read our responses and for your feedback throughout the review process. We appreciate your constructive comments, which have helped us improve the paper. If any further questions come up before the discussion period ends, we would be happy to address them.

Best regards, The Authors

Reviewer_q85dOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies correct-set turnover in reinforcement learning with verifiable rewards, where acquiring new solutions can coincide with regression on previously solved problems. It proposes ReMind, a lightweight review queue that revisits mastered prompts with fresh rollouts and requeues those showing regression. Experiments across image-text, video, and text-only reasoning report improved accuracy and retention.

Critical strengths
  • The paper proposes a simple and practical method that shows performance gains upon GRPO and DAPO baselines from a small intervention using the review queue.
  • The paper conduct substantial experiments with a diverse set of baselines, multiple evaluation benchmarks and models.
Critical weaknesses
  • The novelty of the proposed method is limited given prior works. For example, MCPO (https://arxiv.org/abs/2604.16972) already identifies the degradation of mastered prompts under GRPO-style training and connects it to vanishing group-relative advantages. Meanwhile, Prompt Replay (https://arxiv.org/abs/2603.21177) already maintains a prompt buffer, mixes reused prompts with fresh examples, and generates new on-policy responses.
  • The gains of the proposed method is relatively modest and simple strategies like random-seen can already recover most of gains.
Questions for authors

N/A

3 comments

We thank the reviewer for the constructive comments. We address both weaknesses below with new results.

W1. Novelty relative to MCPO and Prompt Replay

Thank you for pointing out these two recent preprints. We summarize the comparison as follows:

  • MCPO observes one-step drift on mastered prompts and limits it with a KL regularizer while the prompt is in the current batch. In contrast, ReMind follows mastered prompts over time: it revisits them at later steps and turns any regression it finds into a training signal. MCPO is already a baseline in Table 1.
  • Prompt Replay is a data-selection method for efficiency. It re-samples medium-difficulty prompts (pass rate near 0.50.5), and mastered prompts never enter its buffer. In contrast, ReMind reviews exactly these mastered prompts, and its goal is retention.

ReMind's novelty lies in treating retention as a training objective over time: it tracks which prompts were mastered, reviews them before regression accumulates, and requeues those that regress. Neither work does this.

We also ran Prompt Replay in our setting (same backbone, data and rollout budget) and will add it to Related Work and Table 1:

MethodVistaVisionWeMathVerseMath AvgMMMU-PMMBMM-SZeroBGen. AvgAvg
GRPO76.2048.8256.5759.5260.2855.7890.2972.2022.6060.2260.25
MCPO76.5053.7562.1059.1262.8754.3590.0771.1323.3559.7361.30
Prompt Replay77.6053.1864.1959.4063.5954.4890.1772.4723.9560.2761.93
GRPO + ReMind79.1056.8166.3861.5265.9558.7890.4173.4026.0562.1664.06

ReMind exceeds Prompt Replay by +2.13+2.13 and MCPO by +2.76+2.76 on average.

Compatibility with data selection. Prompt Replay belongs to the family of data-selection methods. Table 1 of the paper already includes two methods from this family, DAPO (dynamic sampling with online filtering) and VCRL (variance-based curriculum), and ReMind improves both:

MethodMath AvgGen. AvgAvg
DAPO61.8460.2061.02
DAPO + ReMind66.4662.4764.46 (+3.44+3.44)
VCRL65.1661.6863.42
VCRL + ReMind67.3662.5764.96 (+1.54+1.54)

This suggests that retention review is complementary to selecting informative prompts, rather than a variant of it.

W2. Size of the gain and the random-seen baseline

Size of the gain. On the same backbone, gains over GRPO are: MCPO +1.05+1.05, RL-ZVP +1.64+1.64, Prompt Replay +1.68+1.68, RePO +1.77+1.77, and ReMind +3.81+3.81 (+5.67+5.67 on math reasoning). Three additional seeds give 64.18±0.0364.18 \pm 0.03. The gain also holds on video (+3.21+3.21) and costs about 3% wall-clock time with no extra rollouts.

Random-seen is a reduced variant of ReMind. It keeps our review mechanism (pre-rollout replacement, fresh rollouts, same $\sim$2% budget) and removes the mastery-keyed queue. Its gain over GRPO (+2.67+2.67) supports our first claim: timely review itself is valuable. The remaining +1.14+1.14 comes from the regression-aware queue (enqueue on mastery, graduate if retained, requeue if regressed). The gap is larger on harder benchmarks, e.g. ZeroBench (26.0526.05 vs. 22.4622.46) and MathVerse (61.5261.52 vs. 58.5058.50).

Retention during training. We also take a fixed set of prompts that were all-correct early in training and repeatedly test retention on them for both strategies:

Training stepsRandom-seenReMindGap
50–9572.11%75.65%+3.54+3.54
100–19577.37%79.35%+1.98+1.98
200–29582.37%85.87%+3.50+3.50
300–39583.68%87.17%+3.49+3.49

ReMind retains more in every window (+1.98+1.98 to +3.54+3.54 pp). So the extra accuracy over random review comes together with better retention, which is consistent with the regression-aware queue being the source of the gain.

Revision. We will add the Prompt Replay comparison, the discussion above, and the retention table to the revision.

Dear Reviewer,

Thank you again for your thoughtful feedback on our submission. As the discussion phase ends in under 24 hours, we wanted to check in briefly regarding our responses and the new results we posted. If time permits, could you kindly let us know whether our replies have addressed the points you raised? We would be glad to answer any further questions or provide more details before the deadline.

We know this is a busy period, and we truly appreciate your consideration and your valuable input on our work.

Best regards, The Authors

Dear Reviewer,

We apologize for reaching out again. With only about 3 hours left in the discussion period, we would like to make one last request: could you kindly take a brief look at our rebuttal and the additional experiments? Even a short note on whether your concerns have been addressed, or an updated score if you find our responses satisfactory, would mean a great deal to us.

If any point is still unclear, we are standing by and will respond immediately.

Thank you very much for your time and for your efforts in reviewing our paper.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.