- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work focuses on endogenous forgetting in LLM post-training, where models forget previously learned behaviors because the learning algorithm itself makes the effective training distribution non-stationary, rather than because of shifts in the dataset distribution. To investigate this issue, the work proposes two new metrics: attained gain (learning) and unrecovered loss (forgetting). Endogenous forgetting is first shown to occur in SFT experiments where only correct data is used to train new models using two methodologies: ordered (preserves the endogenous curriculum) and shuffled (removes it). Here, the shuffled data shows improved unrecovered metrics (though lower attained gain), supporting the idea that the RL policy itself can induce endogenous forgetting. Following experiments show that adding a replay buffer and an additional naive behavioral cloning loss to the ordered SFT process can mitigate the increase in forgetting. This provides evidence for implementing a replay buffer and additional loss (slightly modified) for RL in post-training. Doing so further decreases forgetting and improves solution coverage.
Critical strengths
- Both the problem formulation and the proposed solution are simple and straight forward. Moreover, the proposed solution parallels solutions in the continual RL space, where integrating separate replay buffers can help reduce forgetting.
- The paper is relatively clear and well structured, with ideas building logically from one section to the next.
- The experiments and metrics are well designed, and the results are promising.
- The paper addresses the issue of forgetting, a well-recognized challenge throughout machine learning, but brings attention to the less explicitly recognized idea of endogenous forgetting while connecting it with prior work.
Critical weaknesses
- Lack of baseline comparisons
- The authors cite related works that are closely tied to the proposed problem (e.g., Nguyen et al. (2025), Qin et al. (2026), Li et al. (2026)) and provide some conceptual reasoning for why they might be worse or not work. However, no empirical evidence is provided. It would be helpful to see how these works compare not only to the proposed method but also to the GRPO baseline. It would also be useful to see how these methods perform on the proposed metrics.
- Replay buffer sampling choice
- The authors provide experiments for various sampling methods for SFT, but the methods do not seem to differ much (Figure 11 and Table 5). This raises the question of how significant the proposed prioritized replay buffer is for RL post-training compared to simple uniform sampling, beyond its conceptual justification.
- Lack of hyperparameter exploration
- Likewise, it is unclear why the values of used for SFT () and RL post-training () were selected and what effect the value of has on performance.
- Model and data concerns
- Model diversity is a concern, as two of the three models are Qwen models, which could limit the generalizability of the results.
- Not all models are applied to all datasets. Clarification on why would be helpful.
- Data selection seems relatively limited. Is there any justification for why this subset of datasets was chosen over others?
Questions for authors
My questions are listed below. I welcome the authors' clarification on these and on the weaknesses above, in case I have misunderstood any aspects of the work. I am open to revising my score in light of the rebuttal.
- From my understanding, Dr. GRPO removes the normalization term , which is still present in Equation 1. Are you only adopting certain aspects of Dr. GRPO (removing the KL penalty and standard-deviation normalization)?
- All models analyzed are relatively small. How might these results generalize to larger models (same, worse, better)?
- Could there be any advantage to discouraging incorrect trajectories as well? I believe only correct responses are stored in the replay buffer.
- The linked code repository seems to be empty. Are there plans to push the code?
7 comments
[2/3] Response to the reviewer sJpX (Weakness 3 & Weakness 4)
W3: Lack of hyperparameter exploration
Re: We use for SFT and for RL post-training, selected through preliminary sweeps over 0.001–1.0 on KK. We then keep the coefficient fixed across all reported model-task settings within each training regime. The consistent results suggest that replay can mitigate endogenous forgetting without extensive task- or model-specific tuning, which support our claim that replay can mitigate endogenous forgetting. We agree that a systematic sensitivity analysis would provide further insight into the tradeoff, but given its substantial computational cost, we leave it to future work. We will include the discussions in the revision. Below, we show a preliminary result to demonstrate the intuition of ’s effect in RL.
Table: Analysis on replay weight (Qwen2.5-3B, 800 steps on Polaris-53K)
| λ | Pass@1↑ | Pass@64↑ | G↑ | F↓ |
|---|---|---|---|---|
| Standard | 22.22 | 59.91 | 5.91 | 2.87 |
| Replay =0.01 | 22.11 | 61.42 | 4.24 | 0.90 |
| Replay =0.1 | 22.00 | 65.06 | 4.17 | 0.21 |
W4.1: Clarifications on model selection
Re: Given our limited compute budget, we selected different model subsets across tasks to cover multiple families and scales. For coding, we focused on smaller models because the task is relatively easy for the 8B model. For math, we included the 8B model to examine a larger scale. Finally, we used one model for Knights and Knaves because it serves as a controlled synthetic setting for studying the phenomenon.
However, we appreciate the suggestion to evaluate the approach on additional model families. We have added experiments on the math task using OLMo-2-1B-SFT, extending our evaluation to a third model family beyond Qwen and Llama. The preliminary standard RL results show the same forgetting pattern: pass@1 improves while pass@64 declines, with an attained gain of 7.93 pp and an unrecovered loss of 2.23 pp at step 1600. We also plan to extend this evaluation to SFT settings and coding in the revision.
Table: Results on OLMo-1B-SFT Standard RL
| Step | 0 | 200 | 400 | 800 | 1000 | 1400 | 1600 |
|---|---|---|---|---|---|---|---|
| Pass@1 | 6.92 | 9.57 | 10.76 | 11.43 | 11.64 | 11.86 | 12.62 |
| Pass@64 | 43.63 | 42.50 | 38.26 | 37.09 | 36.38 | 36.41 | 34.86 |
W4.2: Clarifications on dataset selection
Re: We adopt established and widely used datasets from prior work [1,2,3,4] on reasoning and RL post-training. These tasks provide complementary settings: KK has relatively low initial solution coverage, whereas coding starts with high coverage, allowing us to study forgetting under different initial conditions. Math uses the largest and most diverse training set in our study, allowing us to examine forgetting across a broader range of problems. Our study focuses on prolonged RL training regime. The KK and Math experiments span 4,000–6,000 RL steps per run, with repeated checkpoint evaluations to track acquisition and forgetting. Each additional model–dataset combination therefore incurs substantial computational cost. We prioritize detailed analysis across these three domains over exhaustive dataset coverage. Additional datasets would broaden the evidence, but we do not consider them necessary to establish the phenomenon and mitigation benefits demonstrated in the studied settings.
Still, we appreciate the suggestion to evaluate the approach on additional datasets. We evaluate cryptarithm, another logic game from Reasoning Gym with low initial coverage, and observe results consistent with our main findings, as shown below.
Table: Results on Cryptarithm
| Method | Pass@1 | Pass@64 |
|---|---|---|
| Base | 0.44 | 22.40 |
| Standard step 6000 | 31.77 | 31.80 |
| Replay (buffer 2048, λ=0.1), step 6000 | 58.57 | 59.40 |
[1] Qi, Zhenting, et al. "On the Generalization Gap in Self-Evolving Language Model Reasoning." arXiv preprint arXiv:2606.01075 (2026).
[2] Xie, Tian, et al. "Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning." arXiv preprint arXiv:2502.14768 (2025).
[3] Khatri, Devvrit, et al. "The art of scaling reinforcement learning compute for llms." International Conference on Learning Representations. Vol. 2026. 2026.
[4] Cheng, Jorge Zhoujun, et al. "Revisiting reinforcement learning for llm reasoning from a cross-domain perspective." Advances in Neural Information Processing Systems 38 (2026).
[3/3] Response to the reviewer sJpX (Q1-4)
Q1: Clarification of the length normalization in the algorithm
Re: Yes, we keep the length normalization in the implementation as the current equation (1).
Q2: How do the results generalize to larger models?
Re: Thank you for raising this question. Prior continual learning research suggests that increasing model and pretraining data scale can reduce catastrophic forgetting, with greater separation between representations providing a possible explanation [1]. In our setting, increasing model size does not eliminate the policy-dependent changes in the effective training distribution that give rise to endogenous forgetting. We therefore expect the mechanism to remain relevant to larger models, although greater capacity may reduce interference and alter its severity. Studying how endogenous forgetting varies with model scale is an important direction, further motivated by reports of instability in large-scale RL training. However, prolonged RL training of models larger than 8B exceeds our available computational budget. We hope through the release of our codes that the community can replicate the results as scale. We will discuss the potential effects of model scale in the revision.
[1] Dyer, Ethan, Aitor Lewkowycz, and Vinay Ramasesh. "Effect of scale on catastrophic forgetting in neural networks." (2022).
Q3: Could there be any advantage to discouraging incorrect trajectories?
Re: Yes, our replay buffer stores only successful trajectories. Since our current intervention focuses on mitigating forgetting, reinforcing previously successful solutions is a straightforward way to preserve acquired capabilities. We agree that discouraging incorrect trajectories is an interesting complementary direction, for example, to suppress recurring errors. We will add this possibility to the discussion as a direction for future study.
Q4: Reproduction
Re: Thank you for being interested in the code. Our code has been visible in the anonymous repo and we will make sure all code and datasets be open-sourced.
Please let us know if you have any further questions or concerns. We would be happy to discuss them. If our responses have addressed your concerns, we would be grateful if you would consider revisiting your score.
[1/3] Response to the reviewer sJpX (Weakness 1 & Weakness 2)
Thank you for your time and thoughtful feedback. We appreciate that the reviewer found our formulation and solution simple and straight forward, the paper clear and well structured, and the experiments and metrics well designed. We are pleased to see that the reviewer learned something interesting or useful, as a central aim of our work is to deepen understanding of endogenous forgetting. We respond below to your questions and concerns, which we will incorporate into the revised version.
W1: Lack of baseline comparisons
Re: We would like to clarify that our work focuses on identifying and understanding endogenous forgetting in RL post-training, connecting it to the RL-induced curriculum through controlled experiments, and demonstrating the benefits of explicitly mitigating it. We study replay as a simple intervention to investigate the mitigation, without claiming that it outperforms all alternatives. Some interventions in the cited papers are not directly comparable to ours: they either address additional objectives beyond forgetting or operate at test time [1] rather than during training. Rather than implementing their exact methods, we discuss how the forms of forgetting they study are particular cases of endogenous forgetting and provide empirical evidence that endogenous forgetting also occurs beyond the cases in prior studies.
First, Qin et al. [2] study forgetting of previously solved prompts, identified by all-correct rollout groups. They capture particular cases of forgetting but will miss declines in success probability on partially solved prompts. In Fig.16 (p.21), we have shown that the unrecovered loss distributed more in the lower pass rate region. Also, we test this by including successful trajectories from all-correct groups in our ordered SFT experiment (i.e., these samples are receiving zero-advantage in GRPO but we can keep their traces, which preserves these samples). The results below show that forgetting still occurs, as shown below.
Table 1: Unrecovered loss results on SFT runs with all correct trajectories
Second, Nguyen et al. [3] consider preserving base-model capabilities to prevent coverage collapse, which may miss declines in capability acquired during RL training. We evaluate a standard reference-policy reverse-KL penalty as a related comparison (Table 2). Intuitively, a fixed base-model reference does not explicitly preserve successes acquired later during RL, and regularizing the full response distribution may constrain useful learning. In our experiments, KL regularization with coefficient 0.01 achieves lower attained gain and incurs higher unrecovered loss than replay. If we decompose unrecovered loss by its source, both KL settings incur slightly less loss from the base model than replay, but more loss from intermediate RL checkpoints. Replay also achieves higher final pass@64 in this setting.
Table 2: Results of Qwen2.5-3B on Math for 2200 training steps
In summary, our formulation captures a broader range of endogenous forgetting, and we agree that a more comprehensive understanding of different interventions will be interesting but we think this could leave for future work.
W2: Replay buffer sampling choice
Re: We compare the standard RL, RL with prioritized replay, and RL with uniform sampling replay below. Both replay settings improve over standard RL, with prioritized replay yielding larger improvements in the tested setting.
Table 2: Ablation on the sampling strategy for replay buffer on KK at step 800
More generally, by introducing prioritized replay, our main point is that the replay sampling distribution can be adapted to prioritize prompts vulnerable to forgetting. Our success-rate-based priority is a simple choice reflecting problem difficulty relative to the model, motivated by the relationship between initial success rate and unrecovered loss in Fig. 16 (p. 21) and supported by its empirical benefits during training. Future strategies could investigate more direct signals of forgetting, as discussed in Appx. A. We will include these results and discussions in the revision.