- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The idea is intuitive, simple to implement, and adds little complexity to the standard RLVR pipeline. The method also shows consistent improvements when combined with several RLVR algorithms, suggesting that it has practical value beyond a single training setup.
Critical strengths
The idea is intuitive, simple to implement, and adds little complexity to the standard RLVR pipeline. The method also shows consistent improvements when combined with several RLVR algorithms, suggesting that it has practical value beyond a single training setup.
Critical weaknesses
My main concern is that the paper does not clearly separate an anti-forgetting effect from a curriculum or sample-selection effect. The forgetting studied here is different from the conventional continual-learning setting, where forgetting usually results from sequentially learning different tasks or distributions. Here, regression happens within the same RLVR training distribution. More importantly, the ReMind queue naturally focuses training on prompts that were once solved but are no longer consistently solved. These prompts are likely to be neither too easy nor too difficult for the current model, and therefore may simply provide more useful training signals. In this sense, ReMind may also behave like a history-aware curriculum-learning or hard-example-sampling method. The current experiments show that this review strategy works, but do not fully establish that preventing forgetting is the main reason for the improvement. I also find the definition of regression somewhat noisy. A prompt changing from 8/8 correct rollouts to 7/8 does not necessarily mean that the model truly forgot the solution; this may partly result from stochastic generation. Since mastery and regression are defined using a small number of rollouts, a frozen-policy repeated-evaluation control would make this claim more convincing.
Questions for authors
- How much of the improvement from ReMind comes from preventing forgetting, and how much may come from selecting samples that are simply better matched to the current model difficulty? Could the authors provide a baseline that matches the current difficulty of ReMind-selected prompts but does not use their mastery history?
- Can the authors better separate true regression from rollout variance? For example, if the model parameters are frozen and mastered prompts are repeatedly re-evaluated, how often would they become non-mastered purely due to sampling noise? More generally, since SGD naturally causes per-example performance fluctuations, how much of the observed turnover should really be interpreted as forgetting?
- How sensitive is ReMind to the difficulty and reward distribution of the training data? For example, if most prompts are very difficult and rarely become mastered, or if most prompts are very easy and remain consistently correct, would the review queue still provide meaningful benefit?
5 comments
W3 / Q3: Sensitivity to data difficulty
New experiment. Starting from the MMFineReason-123K training dataset, we froze the initial 8B model and generated 8 responses per prompt. Prompts with 0–3 correct responses form the Hard subset, and prompts with 4–8 correct responses form the Easy subset. We trained GRPO and GRPO + ReMind on each subset, with all other settings unchanged.
| Method | MathVista | MathVision | WeMath | MathVerse | Math Avg | MMMU-Pro | MMBench | MM-Star | ZeroBench | General Avg | Overall |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Easy GRPO | 76.85 | 52.03 | 58.10 | 58.36 | 61.335 | 55.94 | 90.47 | 70.67 | 19.46 | 59.14 | 60.24 |
| Easy ReMind | 79.45 | 56.60 | 66.29 | 60.55 | 65.72 | 59.27 | 90.99 | 75.00 | 23.80 | 62.27 | 63.99 |
| Hard GRPO | 78.00 | 54.42 | 60.00 | 58.18 | 62.65 | 57.66 | 90.19 | 71.70 | 19.61 | 59.79 | 61.22 |
| Hard ReMind | 78.25 | 56.32 | 67.71 | 60.08 | 65.59 | 59.30 | 90.52 | 75.27 | 25.75 | 62.71 | 64.15 |
ReMind is robust to the difficulty distribution. ReMind improves over GRPO on both subsets ( on Easy, on Hard) and on all eight benchmarks in each case, comparable to the gain on the original training set (). Thus, ReMind remains effective across the tested difficulty regimes, although the size of the gain depends on the data distribution.
Overall, these experiments show that ReMind's gain is not explained by current prompt difficulty alone, and that the observed online regression signal is larger than the fall-out caused by rollout sampling under a frozen policy.
We hope these results address the reviewer's concerns, and we will include all three experiments in the revision. We would be grateful if the reviewer would consider updating the rating.
Dear Reviewer,
We hope this message finds you well. We are writing to gently follow up on our rebuttal, in which we addressed each of your comments and included the additional experiments you suggested. Since the discussion period will close in less than 24 hours, we would be grateful if you could take a look at our responses when you have a moment and let us know whether they resolve your concerns. If anything remains unclear, we will do our best to clarify it right away.
We understand how demanding the reviewing schedule is, and we sincerely thank you for the time and effort you have devoted to our paper.
Best regards, The Authors
We thank the reviewer for the constructive comments. We ran all three suggested experiments; the results are below.
W1 / Q1: Anti-forgetting vs. curriculum / sample selection
New baseline. As suggested, we added a difficulty-matched review baseline (DM-Review). It uses the same review budget and schedule as ReMind (r=0.10, f=5, ≈2% of training prompts), but its review prompts are previously seen prompts that were never mastered, sampled to match the current success rate q^ of ReMind's review samples. It matches ReMind in budget and difficulty, and differs only in not using mastery history.
ReMind is not a data-selection method. DM-Review is on par with GRPO (+0.29), while ReMind is +3.52 above it and higher on all eight benchmarks. Selecting prompts of suitable difficulty therefore does not explain the gain; the mastery history does. The main results (Table 1) support the same conclusion: ReMind on plain GRPO (64.06) already outperforms the selection-based methods VCRL (variance-based curriculum, 63.42) and DAPO (online filtering, 61.02), and it further improves both when combined with them (+1.54 and +3.44). ReMind is thus complementary to curriculum and filtering, not a variant of them.
W2 / Q2: True regression vs. rollout variance
Frozen-policy control. We froze each model and evaluated the 700 validation prompts twice with independent rollout groups (A and B). K is the number of rollouts per prompt in each group, and a prompt is all-correct if all K rollouts are correct. A→B is the fraction of prompts that are all-correct in group A but not in group B, and B→A is the reverse. This fall-out rate measures the apparent regression produced by sampling alone.
Sampling noise alone does not explain the observed regression. The reviewer is right that sampling alone produces apparent regression. This noise floor is about 9% for both GRPO and GRPO + ReMind (7–10% across both directions and both values of K). Two observations go beyond it. First, during training, 21.1% of review records (269/1,275, K=8) are mixed, which is 2.2–2.4$\times$ the floor. Second, since the floor is the same for both methods, it cannot explain the gap between them: on the 700 validation prompts, correct-set loss is 195 for GRPO versus 161 for ReMind (Fig. 1a). The delay sweep (Fig. 2b) points the same way: the retention gap grows monotonically with delay (2.3→4.6→5.4→6.9 pp), while sampling noise and zero-mean SGD fluctuation do not depend on delay. We will report the noise floor in the revision.