acceptodds
Under review as a conference paper at ICLR 2027

Endogenous Forgetting in RL Post-Training

Abstract

A central challenge in scaling reinforcement learning (RL) for LLM post-training is how models continue to learn new behaviors while preserving previously acquired ones. We study *endogenous forgetting*–where a model loses previously accessible behaviors due to non-stationarity that arises because the evolving model determines the effective data distribution–as a potential barrier to scaling on-policy RL. We first isolate endogenous forgetting via a controlled experiment: we collect successful traces from an RL run and show that supervised learning on these traces in temporal order leads to substantially more forgetting and lower solution coverage than training with a shuffled order. We then investigate successful-trajectory replay as a way to mitigate endogenous forgetting. Across multiple reasoning tasks and model families, replay reduces forgetting and prompt-level performance oscillations, achieves comparable or higher final pass@1, and substantially improves solution coverage relative to standard RL.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Endogenous Forgetting, RL Post-training, Experience Replay

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sJpXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work focuses on endogenous forgetting in LLM post-training, where models forget previously learned behaviors because the learning algorithm itself makes the effective training distribution non-stationary, rather than because of shifts in the dataset distribution. To investigate this issue, the work proposes two new metrics: attained gain (learning) and unrecovered loss (forgetting). Endogenous forgetting is first shown to occur in SFT experiments where only correct data is used to train new models using two methodologies: ordered (preserves the endogenous curriculum) and shuffled (removes it). Here, the shuffled data shows improved unrecovered metrics (though lower attained gain), supporting the idea that the RL policy itself can induce endogenous forgetting. Following experiments show that adding a replay buffer and an additional naive behavioral cloning loss to the ordered SFT process can mitigate the increase in forgetting. This provides evidence for implementing a replay buffer and additional loss (slightly modified) for RL in post-training. Doing so further decreases forgetting and improves solution coverage.

Critical strengths
  • Both the problem formulation and the proposed solution are simple and straight forward. Moreover, the proposed solution parallels solutions in the continual RL space, where integrating separate replay buffers can help reduce forgetting.
  • The paper is relatively clear and well structured, with ideas building logically from one section to the next.
  • The experiments and metrics are well designed, and the results are promising.
  • The paper addresses the issue of forgetting, a well-recognized challenge throughout machine learning, but brings attention to the less explicitly recognized idea of endogenous forgetting while connecting it with prior work.
Critical weaknesses
  • Lack of baseline comparisons
    • The authors cite related works that are closely tied to the proposed problem (e.g., Nguyen et al. (2025), Qin et al. (2026), Li et al. (2026)) and provide some conceptual reasoning for why they might be worse or not work. However, no empirical evidence is provided. It would be helpful to see how these works compare not only to the proposed method but also to the GRPO baseline. It would also be useful to see how these methods perform on the proposed metrics.
  • Replay buffer sampling choice
    • The authors provide experiments for various sampling methods for SFT, but the methods do not seem to differ much (Figure 11 and Table 5). This raises the question of how significant the proposed prioritized replay buffer is for RL post-training compared to simple uniform sampling, beyond its conceptual justification.
  • Lack of λ\lambda hyperparameter exploration
    • Likewise, it is unclear why the values of λ\lambda used for SFT (λ=1\lambda = 1) and RL post-training (λ=0.1\lambda = 0.1) were selected and what effect the value of λ\lambda has on performance.
  • Model and data concerns
    • Model diversity is a concern, as two of the three models are Qwen models, which could limit the generalizability of the results.
    • Not all models are applied to all datasets. Clarification on why would be helpful.
    • Data selection seems relatively limited. Is there any justification for why this subset of datasets was chosen over others?
Questions for authors

My questions are listed below. I welcome the authors' clarification on these and on the weaknesses above, in case I have misunderstood any aspects of the work. I am open to revising my score in light of the rebuttal.

  • From my understanding, Dr. GRPO removes the normalization term 1∣oi∣\frac{1}{|o_i|}, which is still present in Equation 1. Are you only adopting certain aspects of Dr. GRPO (removing the KL penalty and standard-deviation normalization)?
  • All models analyzed are relatively small. How might these results generalize to larger models (same, worse, better)?
  • Could there be any advantage to discouraging incorrect trajectories as well? I believe only correct responses are stored in the replay buffer.
  • The linked code repository seems to be empty. Are there plans to push the code?
7 comments
AuthorsOpenReview ↗

[1/3] Response to the reviewer sJpX (Weakness 1 & Weakness 2)

Thank you for your time and thoughtful feedback. We appreciate that the reviewer found our formulation and solution simple and straight forward, the paper clear and well structured, and the experiments and metrics well designed. We are pleased to see that the reviewer learned something interesting or useful, as a central aim of our work is to deepen understanding of endogenous forgetting. We respond below to your questions and concerns, which we will incorporate into the revised version.

W1: Lack of baseline comparisons

Re: We would like to clarify that our work focuses on identifying and understanding endogenous forgetting in RL post-training, connecting it to the RL-induced curriculum through controlled experiments, and demonstrating the benefits of explicitly mitigating it. We study replay as a simple intervention to investigate the mitigation, without claiming that it outperforms all alternatives. Some interventions in the cited papers are not directly comparable to ours: they either address additional objectives beyond forgetting or operate at test time [1] rather than during training. Rather than implementing their exact methods, we discuss how the forms of forgetting they study are particular cases of endogenous forgetting and provide empirical evidence that endogenous forgetting also occurs beyond the cases in prior studies.

First, Qin et al. [2] study forgetting of previously solved prompts, identified by all-correct rollout groups. They capture particular cases of forgetting but will miss declines in success probability on partially solved prompts. In Fig.16 (p.21), we have shown that the unrecovered loss distributed more in the lower pass rate region. Also, we test this by including successful trajectories from all-correct groups in our ordered SFT experiment (i.e., these samples are receiving zero-advantage in GRPO but we can keep their traces, which preserves these samples). The results below show that forgetting still occurs, as shown below.

Table 1: Unrecovered loss results on SFT runs with all correct trajectories

DomainModelOrderedShuffledΔ
KKQwen2.5-3B9.57.2-2.3
MathQwen2.5-3B7.93.6-4.3
-Llama3.2-3B-Instruct7.34.4-2.9
CodingQwen2.5-3B4.42.6-1.8
-Llama3.2-3B-Instruct12.64.2-8.4
-Qwen3-1.7B-Base7.42.9-4.5

Second, Nguyen et al. [3] consider preserving base-model capabilities to prevent coverage collapse, which may miss declines in capability acquired during RL training. We evaluate a standard reference-policy reverse-KL penalty as a related comparison (Table 2). Intuitively, a fixed base-model reference does not explicitly preserve successes acquired later during RL, and regularizing the full response distribution may constrain useful learning. In our experiments, KL regularization with coefficient 0.01 achieves lower attained gain and incurs higher unrecovered loss than replay. If we decompose unrecovered loss by its source, both KL settings incur slightly less loss from the base model than replay, but more loss from intermediate RL checkpoints. Replay also achieves higher final pass@64 in this setting.

Table 2: Results of Qwen2.5-3B on Math for 2200 training steps

Method.Pass@1(%)↑Pass@64(%)↑G↑F↓
Base model15.5163.61––
Standard22.9954.3614.376.89
Standard + KL 0.00123.0759.9411.373.81
Standard + KL 0.0120.8561.337.692.35
Replay23.8764.609.821.46

In summary, our formulation captures a broader range of endogenous forgetting, and we agree that a more comprehensive understanding of different interventions will be interesting but we think this could leave for future work.

W2: Replay buffer sampling choice

Re: We compare the standard RL, RL with prioritized replay, and RL with uniform sampling replay below. Both replay settings improve over standard RL, with prioritized replay yielding larger improvements in the tested setting.

Table 2: Ablation on the sampling strategy for replay buffer on KK at step 800

MethodPass@1↑Pass@64↑G↑F↓
Standard44.4865.0049.9612.00
Replay (uniform)56.1177.2058.128.53
Replay (prioritized)63.2887.6066.399.63

More generally, by introducing prioritized replay, our main point is that the replay sampling distribution can be adapted to prioritize prompts vulnerable to forgetting. Our success-rate-based priority is a simple choice reflecting problem difficulty relative to the model, motivated by the relationship between initial success rate and unrecovered loss in Fig. 16 (p. 21) and supported by its empirical benefits during training. Future strategies could investigate more direct signals of forgetting, as discussed in Appx. A. We will include these results and discussions in the revision.

AuthorsOpenReview ↗

[2/3] Response to the reviewer sJpX (Weakness 3 & Weakness 4)

W3: Lack of hyperparameter exploration

Re: We use λ=1\lambda=1 for SFT and λ=0.1\lambda=0.1 for RL post-training, selected through preliminary sweeps over 0.001–1.0 on KK. We then keep the coefficient fixed across all reported model-task settings within each training regime. The consistent results suggest that replay can mitigate endogenous forgetting without extensive task- or model-specific tuning, which support our claim that replay can mitigate endogenous forgetting. We agree that a systematic sensitivity analysis would provide further insight into the tradeoff, but given its substantial computational cost, we leave it to future work. We will include the discussions in the revision. Below, we show a preliminary result to demonstrate the intuition of ’s effect in RL.

Table: Analysis on replay weight (Qwen2.5-3B, 800 steps on Polaris-53K)

λPass@1↑Pass@64↑G↑F↓
Standard22.2259.915.912.87
Replay =0.0122.1161.424.240.90
Replay =0.122.0065.064.170.21

W4.1: Clarifications on model selection

Re: Given our limited compute budget, we selected different model subsets across tasks to cover multiple families and scales. For coding, we focused on smaller models because the task is relatively easy for the 8B model. For math, we included the 8B model to examine a larger scale. Finally, we used one model for Knights and Knaves because it serves as a controlled synthetic setting for studying the phenomenon.

However, we appreciate the suggestion to evaluate the approach on additional model families. We have added experiments on the math task using OLMo-2-1B-SFT, extending our evaluation to a third model family beyond Qwen and Llama. The preliminary standard RL results show the same forgetting pattern: pass@1 improves while pass@64 declines, with an attained gain of 7.93 pp and an unrecovered loss of 2.23 pp at step 1600. We also plan to extend this evaluation to SFT settings and coding in the revision.

Table: Results on OLMo-1B-SFT Standard RL

Step0200400800100014001600
Pass@16.929.5710.7611.4311.6411.8612.62
Pass@6443.6342.5038.2637.0936.3836.4134.86

W4.2: Clarifications on dataset selection

Re: We adopt established and widely used datasets from prior work [1,2,3,4] on reasoning and RL post-training. These tasks provide complementary settings: KK has relatively low initial solution coverage, whereas coding starts with high coverage, allowing us to study forgetting under different initial conditions. Math uses the largest and most diverse training set in our study, allowing us to examine forgetting across a broader range of problems. Our study focuses on prolonged RL training regime. The KK and Math experiments span 4,000–6,000 RL steps per run, with repeated checkpoint evaluations to track acquisition and forgetting. Each additional model–dataset combination therefore incurs substantial computational cost. We prioritize detailed analysis across these three domains over exhaustive dataset coverage. Additional datasets would broaden the evidence, but we do not consider them necessary to establish the phenomenon and mitigation benefits demonstrated in the studied settings.

Still, we appreciate the suggestion to evaluate the approach on additional datasets. We evaluate cryptarithm, another logic game from Reasoning Gym with low initial coverage, and observe results consistent with our main findings, as shown below.

Table: Results on Cryptarithm

MethodPass@1Pass@64
Base0.4422.40
Standard step 600031.7731.80
Replay (buffer 2048, λ=0.1), step 600058.5759.40

[1] Qi, Zhenting, et al. "On the Generalization Gap in Self-Evolving Language Model Reasoning." arXiv preprint arXiv:2606.01075 (2026).

[2] Xie, Tian, et al. "Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning." arXiv preprint arXiv:2502.14768 (2025).

[3] Khatri, Devvrit, et al. "The art of scaling reinforcement learning compute for llms." International Conference on Learning Representations. Vol. 2026. 2026.

[4] Cheng, Jorge Zhoujun, et al. "Revisiting reinforcement learning for llm reasoning from a cross-domain perspective." Advances in Neural Information Processing Systems 38 (2026).

AuthorsOpenReview ↗

[3/3] Response to the reviewer sJpX (Q1-4)

Q1: Clarification of the length normalization in the algorithm

Re: Yes, we keep the length normalization in the implementation as the current equation (1).

Q2: How do the results generalize to larger models?

Re: Thank you for raising this question. Prior continual learning research suggests that increasing model and pretraining data scale can reduce catastrophic forgetting, with greater separation between representations providing a possible explanation [1]. In our setting, increasing model size does not eliminate the policy-dependent changes in the effective training distribution that give rise to endogenous forgetting. We therefore expect the mechanism to remain relevant to larger models, although greater capacity may reduce interference and alter its severity. Studying how endogenous forgetting varies with model scale is an important direction, further motivated by reports of instability in large-scale RL training. However, prolonged RL training of models larger than 8B exceeds our available computational budget. We hope through the release of our codes that the community can replicate the results as scale. We will discuss the potential effects of model scale in the revision.

[1] Dyer, Ethan, Aitor Lewkowycz, and Vinay Ramasesh. "Effect of scale on catastrophic forgetting in neural networks." (2022).

Q3: Could there be any advantage to discouraging incorrect trajectories?

Re: Yes, our replay buffer stores only successful trajectories. Since our current intervention focuses on mitigating forgetting, reinforcing previously successful solutions is a straightforward way to preserve acquired capabilities. We agree that discouraging incorrect trajectories is an interesting complementary direction, for example, to suppress recurring errors. We will add this possibility to the discussion as a direction for future study.

Q4: Reproduction

Re: Thank you for being interested in the code. Our code has been visible in the anonymous repo and we will make sure all code and datasets be open-sourced.

Please let us know if you have any further questions or concerns. We would be happy to discuss them. If our responses have addressed your concerns, we would be grateful if you would consider revisiting your score.

Reviewer_pQ6YOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies forgetting during on-policy RL when the prompts still come from the same fixed dataset. In group-relative RL, a prompt stops contributing to the gradient when all sampled responses receive the same reward. As the policy changes, the subset of prompts that actually drives training changes with it. The paper calls the resulting loss of earlier behavior endogenous forgetting.

The authors first collect successful trajectories from an RL run and train on them with SFT. They compare the original collection order with a shuffled version of the same data. The ordered runs have higher unrecovered loss and worse pass@64 in all six settings. The authors try to address this by replaying successful trajectories from earlier in training with an auxiliary SFT loss. Their sampler leans toward prompts with low stored success rates, but uses a cooldown so that a few prompts do not dominate replay. They test the approach on Knights and Knaves, math, and coding tasks with several model families. Replay improves large-k pass@k in all seven RL settings and improves pass@1 in six.

Critical strengths
  1. The ordered and shuffled SFT experiment is the part I found most convincing.

    Both runs use the same successful trajectories and the same hyperparameters. Only the order changes. The gap in pass@64 is therefore hard to dismiss as a data-quality difference. The higher loss on old rollouts gives a second, more direct view of the same retention problem.

  2. The replay result is consistent across the tested settings.

    Pass@64 improves in every RL setting, while pass@1 improves in six of seven. The SFT study also obtains similar results with uniform, hard, and cross-entropy sampling. Replay is simple enough that these results could be useful even if the exact mechanism needs more work.

Critical weaknesses
  1. I don't think unrecovered loss can yet be read as the amount of behavior the model forgot.

    The metric keeps the highest estimated pass@1 for each prompt and compares it with the final estimate. Since both numbers come from sampled responses, a frozen policy would still look unusually good at one checkpoint sooner or later. That high estimate is then counted as forgetting when the final estimate returns to a more typical value. The effect grows as more checkpoints are included.

    Appendix E.1.3 handles sampling noise when labeling changes between adjacent checkpoints, but that correction isn't used for attained gain or unrecovered loss. I'd like to see the authors evaluate a frozen checkpoint several times under the same protocol. The apparent unrecovered loss in that experiment would show how much of the reported value can arise without any policy change. The number of responses per prompt and the checkpoint spacing also need to be stated.

  2. The replay comparison changes more than whether old trajectories are retained.

    Replay adds an SFT objective with batches of 256 to 2048 verified trajectories, while standard RL has no comparable auxiliary batch. The runs may have the same number of gradient steps, but they don't process the same amount of training data. I can't separate the value of historical replay from the value of simply doing more supervised training on correct answers.

    A useful control would keep the auxiliary-loss weight and sample budget fixed but draw successful trajectories from the current or recent policy. Without such a comparison, I read the result as evidence that extra verified SFT helps, not yet as clean evidence that historical replay works by preventing endogenous forgetting.

Questions for authors
  1. How many responses are sampled per prompt, and how often is the model evaluated? If a frozen checkpoint is evaluated repeatedly, how much unrecovered loss appears from sampling alone?

  2. Does historical replay still outperform an equally sized auxiliary SFT batch drawn from current or recent successes?

3 comments
AuthorsOpenReview ↗

Response to the reviewer pQ6Y

Thank you for your time and thoughtful feedback. We appreciate that the reviewer found the paper clear and easy to follow, the SFT experiment most convincing, and the replay result consistent. We are glad the reviewer gained a substantially deeper understanding of the problem, which was a central aim of our work. We respond below to your questions and concerns, which we will incorporate into the revised version.

W1 & Q1: Clarification of Metric Empirical Estimation & Baseline for frozen checkpoint

Re: Thank you for raising this concern. Our metrics use two-fold cross-fitting to address peak-selection bias [1,2]. At each checkpoint, we split the 160 responses per prompt into two folds of 80. We select the best historical checkpoint on one fold and estimate its performance difference from the current checkpoint on the independent fold, then swap folds and average. For nondecreasing true pass@1, the estimator satisfies E[F^unrec(q,t)]≤max⁡s≤tm(q,s)−m(q,t)=0.\mathbb{E}\left[\widehat F^{\mathrm{unrec}}(q,t)\right]\leq \max_{s\leq t}m(q,s)-m(q,t)=0.

For a frozen policy, this expectation is exactly zero, regardless of the number of checkpoints. A significantly positive average estimate therefore provides statistical evidence of forgetting. We have clarified this procedure and its guarantee in the revised PDF (Appx. E1.3 p.13).

For standard RL, we additionally test whether the unrecovered loss is positive. All one-sided p-values are below 0.001:

SettingF [95% CI], pp
KK / Qwen2.5-3B5.65 [5.46, 5.83]
Math / Qwen2.5-3B9.53 [9.28, 9.78]
Math / Llama-3.2-3B-Instruct7.83 [7.61, 8.05]
Code / Qwen3-1.7B8.59 [8.38, 8.79]
Code / Qwen2.5-3B6.03 [5.79, 6.26]
Code / Llama-3.2-3B-Instruct8.69 [8.46, 8.93]

Regarding Q1, we evaluate every 30 training steps on Coding, 200 on Math, and 100 on KK, using 160 responses per prompt.

[1] Stone, Mervyn. "Cross‐validatory choice and assessment of statistical predictions." Journal of the royal statistical society: Series B (Methodological) 36.2 (1974): 111-133.

[2] Hasselt, Hado. "Double Q-learning." Advances in neural information processing systems 23 (2010).

W2 & Q2: Historical replay versus additional verified supervision

Re: Thank you for the suggestion. Our replay method reuses previously generated successful trajectories and therefore requires no additional rollout generation. By contrast, if we use only distinct successful trajectories from the current on-policy data for the SFT loss, we would need additional rollouts whenever the batch contains fewer successes than the replay batch size.

Still to probe the importance of updating on older sequences, we compare against a FIFO buffer with capacity equal to the replay batch size, adding current-batch successful trajectories and discarding the oldest entries when full. Both variants use the same generation budget, auxiliary batch size, and SFT loss. The main difference is prompt coverage: FIFO retains only the most recent successes across prompts, whereas our original buffer preserves successes for every previously solved prompt.

The FIFO baseline achieves slightly higher attained gain but retains greater unrecovered loss in both tasks, indicating that more updates on recent successes alone are insufficient to prevent forgetting.

Table: Comparison with FIFO baseline (Qwen2.5-3B,Polaris-53K,1200 steps)

TaskMethodPass@1↑Pass@64↑G↑F↓
MathStandard22.9555.9911.994.55
FIFO21.9253.2712.035.61
Replay22.1865.187.410.74
-
CodeStandard71.6081.5541.727.54
FIFO70.4483.2443.1110.09
Replay76.4693.9740.971.93

Please let us know if you have any further questions or concerns. We would be happy to discuss them. If our responses have addressed your concerns, we would be grateful if you would consider revisiting your score.

Thanks to the authors for the reply. After reading all the reviews and rebuttals, I still lean toward a weak accept of this work. I will maintain my scores.

AuthorsOpenReview ↗

Thank you for the reply!

Thank you for your reply. We appreciate the time and effort you have devoted to reviewing our submission, as well as the constructive feedback. Should any questions or concerns remain, we would be happy to address them during the remaining discussion period.

Reviewer_Ck46OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies forgetting that arises during on-policy reinforcement learning post-training of large language models even when the prompt distribution is fixed. The authors argue that the evolving policy itself makes the effective training distribution non-stationary: prompts that the policy has mastered yield zero group-relative advantage and therefore receive no learning signal, while updates driven by still-active prompts can damage them. They call this endogenous forgetting, in contrast to classical catastrophic forgetting caused by an externally changing task distribution. The paper makes three moves. First, a controlled supervised experiment: successful trajectories from a reinforcement learning run are replayed as supervised data either in their original temporal order or in shuffled order, and the ordered variant shows more forgetting and lower solution coverage across six model-and-domain settings. Second, a measurement framework that decomposes the change in average pass@1 into attained gain and unrecovered loss, with a hypergeometric calibration for detecting per-prompt changes. Third, a mitigation: a replay buffer of verified-correct historical trajectories, optimized with an auxiliary token-normalized supervised loss alongside the reinforcement learning objective, with prompt prioritization by one minus the estimated success rate plus a cooldown for stale priorities.

Critical strengths

The definition of endogenous forgetting on a fixed prompt distribution, its mechanistic link to the zero-advantage property of group-relative policy optimization, and the attained-gain/unrecovered-loss decomposition (which subsumes recently proposed binary temporal-forgetting metrics as special cases) give the community a reusable language and toolkit.

Critical weaknesses
  1. Every table cell appears to come from a single run, with no seeds, error bars, or significance tests, while several headline margins are at the one-to-two-point level. More importantly, the replay variant consumes exactly 50 percent more sequences per gradient step in every setting (for example, 2048 replay sequences on top of 4096 on-policy sequences), and no compute-matched or token-matched control is provided; a plausible alternative explanation for much of the improvement is simply more supervised signal per step rather than reduced forgetting. The claimed acceleration on logic puzzles ("1.7 times faster") is therefore not interpretable without wall-clock or floating-point-operation accounting. There is also no comparison against any published anti-forgetting baseline (forward-divergence regularization toward the base model, dynamic sampling, entropy regularization, or correct-set revisiting), and no ablation of the replay weight or of the prioritization scheme in the reinforcement learning setting — so the claim that the win-rate-prioritized variant "works especially well" is untested where it matters.
  2. Ordered supervised training differs from shuffled training not only in the induced curriculum but also in recency (the strongest policy's data is seen last) and in difficulty and trace-length composition (later reinforcement learning data contains harder problems and longer solutions). A reverse-order control, a difficulty-sorted control, or a second-epoch control would separate these, and none is provided. Relatedly, ordered training attains higher attained gain in six of six settings, which cuts against a purely harmful reading of the endogenous curriculum and is never reconciled. Finally, all evaluation is in-distribution, and the coding setting trains on only 320 deduplicated problems for what amounts to roughly 60 epochs, which is close to a memorization regime; the suggestion that replay should be "a default component of extended reinforcement learning deployments" substantially outruns the evidence.
Questions for authors
  1. How many samples per prompt per checkpoint were used to estimate per-prompt pass rates, what was the checkpoint stride, and how many random seeds were run per cell? Can you report confidence intervals for the main table?
5 comments
AuthorsOpenReview ↗

[1/4] Response to the reviewer Ck46 (Weakness 1.1, 1.2 & Q1)

Thank you for your time and thoughtful feedback. We appreciate the reviewer’s recognition of a reusable language and toolkit and are glad the reviewer gained a substantially deeper understanding of the problem, which was a central aim of our work. We respond below to your questions and concerns, which we will incorporate into the revised version.

W1.1 & Q1: “Every table cell appears to come from a single run, with no seeds.”

Re: All of our evaluation uses 160 responses per prompt, with checkpoints evaluated every 30 training steps in RL on Coding, 200 on Math, and 100 on KK, same for standard and replay RL. Each model–task setting uses one random seed. Prolonged RL training is computationally expensive in practice. Our experiments on KK and Math require 4,000–6,000 RL steps, equivalent to 500-700 H100 GPU-hours per job for a 3B model. Repeating every model–task setting across multiple seeds would multiply this substantial cost and is not standard practice in comparable studies [1,2]. We therefore prioritize multiple model-task training over multi-seed replication, given the scope and budget of this study. We hope through the release of our codes that the community can replicate the results as scale.

Our claims do not rest on isolated one- or two-point improvements. Our findings are consistent across multiple models and three tasks with replay reducing unrecovered loss in all seven RL settings. These reductions are substantial: unrecovered loss decreases from 17.5 to 4.5 points on Math with Qwen3-8B-Base, from 9.6 to 3.4 points on Math with Qwen2.5-3B, and from 8.6 to 3.4 points on Coding with Qwen3-1.7B. Replay also improves pass@64 across all seven settings, by 2.0–15.1 percentage points. We also report prompt-level total variation and the fraction of prompts exhibiting monotonic improvement, providing complementary evidence of reduced learning–forgetting oscillations and more sustained progress.

[1] Liu, Mingjie, et al. "Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models." Advances in Neural Information Processing Systems 38 (2026): 17998-18031.

[2] Khatri, Devvrit, et al. "The art of scaling reinforcement learning compute for llms." International Conference on Learning Representations. Vol. 2026. 2026.

W 1.2: Replay consumes 50% more sequences per gradient steps and the acceleration claim

Re: We clarify that replay requires no additional rollout generation, the main bottleneck in RL training. Replay only accounts for 1%-2% of average training step time in Coding and 8.5–12.5% of average training step time in Math and KK, arising from forward and backward passes on cached trajectories. After accounting for this overhead, the reported 1.7× reduction in required training steps translates to approximately 1.5× acceleration in wall clock time to reach the same performance threshold. We will report replay’s time overhead and include the wall clock time comparison on KK in the revised manuscript.

AuthorsOpenReview ↗

[2/4] Response to the reviewer Ck46 (Weakness 1.3)

W 1.3: Comparisons with anti-forgetting baselines

Re: Thank you for your suggestion. We would like to clarify that our work focuses on understanding whether the RL-induced curriculum causes forgetting and demonstrating the benefits of explicitly mitigating it. We study replay as a simple intervention, without claiming that it outperforms all alternatives.

We agree that a comprehensive comparison of mitigation methods would be valuable. Below, we discuss four approaches mentioned by the reviewer.

First, could the reviewer specify the reference for “forward-divergence regularization”? We evaluate the standard reference-policy reverse-KL penalty as a related baseline. Conceptually, a base-model reference does not explicitly preserve successes acquired during RL, and regularizing the full response distribution may also constrain useful learning since it preserves failures. Table 1 below shows that for different values of KL coefficients, KL regularization induces higher unrecovered loss (F) than replay, which suggests more forgetting. For 0.01, the attained gain is lower than replay. For 0.001, the attained gain is higher but final pass@1 is comparable. Also, replay achieves the highest final pass@64 among the tested settings.

Table 1: Results of Qwen2.5-3B on Math for 2200 training steps

Method.Pass@1(%)↑Pass@64(%)↑G↑F↓
Base model15.5163.61––
Standard22.9954.3614.376.89
Standard + KL 0.00123.0759.9411.373.81
Standard + KL 0.0120.8561.337.692.35
Replay23.8764.609.821.46

Second, we understand correct-set revising as replaying prompts for which all sampled responses were correct in RL [2]. We do not directly implement this in RL, but use our SFT experiment to examine whether forgetting persists when trajectories from all-correct groups receive a learning signal. Our SFT setting retains all successful rollouts, including those from all-correct prompts. Unlike GRPO, where an all-correct response group yields zero advantages, SFT continues to reinforce these responses. Nevertheless, ordered SFT still exhibits forgetting (p.4, Table 1 in the manucript), suggesting that reinforcing successful examples alone does not necessarily resolve forgetting because their temporal position also matters.

Finally, we consider dynamic sampling and entropy regularization. DAPO-style dynamic sampling [1] filters zero-advantage groups and replenishes the batch with gradient-contributing prompts, preserving the policy-dependent curriculum. Entropy regularization encourages diversity, but does not explicitly preserve previously successful responses. These two methods were introduced to improve training efficiency and exploration, rather than explicitly mitigate forgetting.

We additionally evaluate entropy regularization on Coding with Qwen3-1.7B-Base. Coefficients of 0.1 and 0.01 resulted in training reward failing to improve or rapidly collapsing, so we do not report evaluation results for those runs. The tables below report the comparison using a coefficient of 0.001. At RL step 300, entropy regularization maintains entropy at 0.109 while standard RL collapses to 0.026. However, as shown in Table 2, the unrecovered loss for Standard w. Ent is numerically higher than under Standard RL, while pass@1 is similar and pass@64 is better modestly. In contrast, RL with replay maintains substantially lower unrecovered loss,, improves pass@64, and increases the fraction of prompts with monotonic improvement from 59.26% to 75.93%. These results show that higher policy entropy alone does not reduce forgetting in this setting.

Table 2: Performance comparison for three settings

Settingpass@1pass@64G↑F↓Mono. improv.(%)↑
Standard71.6081.5541.727.5459.26
Standard w. Ent71.4283.2142.628.6357.14
Replay76.4693.9740.971.9375.93

We agree that further evaluating how these RLVR interventions affect forgetting is an interesting direction, which we leave to future work.

[1] Yu, Qiying, et al. "Dapo: An open-source llm reinforcement learning system at scale." Advances in Neural Information Processing Systems 38 (2026): 113222-113244.

[2] Qin, Chuanyu, et al. "Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR." arXiv preprint arXiv:2606.03087 (2026).

AuthorsOpenReview ↗

[3/4] Response to the reviewer Ck46 (Weakness 1.4 & Weakness 2.1, 2.2)

W 1.4: Replay weight and prioritization ablations

Re: First, for the replay weight, we use λ=1\lambda=1 for SFT and λ=0.1\lambda=0.1 for RL post-training, selected through preliminary sweeps over 0.001–1.0 on KK. We then keep the coefficient fixed across all reported models–task settings within each training regime. The consistent results suggest that replay can mitigate endogenous forgetting without extensive task- or model-specific tuning, which support our claim that replay can mitigate endogenous forgetting. We agree that a systematic sensitivity analysis would provide further insight into the tradeoff, but given its substantial computational cost, we leave it to future work.

Below, we show a preliminary result to demonstrate the intuition of ’s effect in RL.

Table 1: Analysis on replay weight (Qwen2.5-3B, 800 steps on Polaris-53K)

λPass@1Pass@64GainLoss
Standard22.2259.915.912.87
Replay =0.0122.1161.424.240.90
Replay =0.122.0065.064.170.21

Second, for the prioritized replay, we compare the standard RL, RL with prioritized replay, and RL with uniform sampling replay below. Both replay settings improve over standard RL, with prioritized replay yielding larger improvements in the tested setting.

Table 2: Ablation on the sampling strategy for replay buffer on KK at step 800

MethodPass@1↑Pass@64↑G↑F↓
Standard44.4865.0049.9612.00
Replay (uniform)56.1177.2058.128.53
Replay (prioritized)63.2887.6066.399.63

More generally, by introducing prioritized replay, our main point is that the replay sampling distribution can be adapted to prioritize prompts vulnerable to forgetting. Our success-rate-based priority is a simple choice reflecting problem difficulty relative to the model, motivated by the relationship between initial success rate and unrecovered loss in Fig. 16 (p. 21) and supported by its empirical benefits during training. Future strategies could investigate more direct signals of forgetting, as discussed in Appx. A. We will include these results and discussions in the revision.

W 2.1: Confounds in the ordered-versus-shuffled comparison

Re: We agree that ordering changes recency and the temporal composition of problem difficulty and response length. These are exactly part of the RL-induced curriculum we've discussed, rather than separate confounders. As formulated in Section 3 (line 119-128), we characterize RL training distribution as μt(q,o)∝Dtr(q)\mu_t(q,o)\propto\mathcal{D}_{tr}(q) ∏i=1Gπt(oi∣q)1[Vari(R(q,oi))>0]\prod_{i=1}^{G} \pi_t(o_i \mid q) \mathbb{1}[Var_i(R(q, o_i)) > 0]. As the policy evolves, both the effective prompt distribution and the response distribution can change, inducing a temporal curriculum.

Also, we would like to clarify that our SFT experiment aims to test whether the temporal ordering inherited from RL can induce forgetting even when RL optimization itself is removed. We do not propose Shuffled SFT as an online method or attribute the effect to any single feature of the ordering. Shuffling provides the required control by removing the original ordering while preserving the training examples and budget. We think the additional controls could help distinguish and explain what individual factors in the curriculum affect more, but are not necessary to establish our claim and beyond the current scope of the work.

W 2.2: Reconciling higher attained gain under ordered training

Re: First, in our manuscript, we have explicitly discussed the tradeoff of learning and forgetting (line 212-214) and higher attained gain need not translate into better final performance, which is the concern our decomposition highlights (line 140-142). Ordered SFT achieves higher attained gain but also incurs greater unrecovered loss. In four of six settings, shuffled SFT achieves higher final pass@1 despite lower attained gain because its reduction in unrecovered loss more than compensates for the difference. These complementary measures reveal how greater gain can coexist with more forgetting, which final performance alone cannot distinguish.

Second, in general, we would also like to clarify that our paper claims that the RL-induced curriculum can induce forgetting and learning–forgetting oscillations, not that it is uniformly harmful. Our work provides a perspective for understanding training dynamics by examining learning and forgetting jointly. Their relative importance depends on the use case, and we agree that greater attained gain can be valuable even when accompanied by greater forgetting. We will explicitly discuss this distinction in the revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We thank the Area Chair and all reviewers for their time and effort. We appreciate Ck46’s recognition of a reusable language and toolkit, pQ6Y finding the SFT experiment convincing, and sJpX finding our formulation and solution simple and straightforward, our paper relatively clear and well structured, and our experiments and metrics well designed. We appreciate the constructive feedback from all reviewers.

Below, we summarize the main clarifications and additions addressing shared concerns raised during the rebuttal period, with references to our detailed responses and the revised PDF:

  • Clarifying the paper’s scope and contribution (Ck46 W1.3, sJpX W1). Our main contribution is to identify and characterize endogenous forgetting, distinguish learning from forgetting through our measurements, and investigate the role of the RL-induced curriculum. We study replay as a simple intervention, without claiming that it outperforms all alternatives. We have expanded the conclusion and limitations in the revised PDF to clarify this scope and discuss potential alternative or complementary interventions.

  • Additional baseline comparisons (Ck46 W1.3, pQ6Y W2/Q2, sJpX W1). We compare replay with reference-policy reverse-KL regularization, entropy regularization, and a FIFO buffer that uses the same rollout budget, auxiliary batch size, and SFT objective. In the tested settings, replay yields lower unrecovered loss than these alternatives, supporting the value of preserving historical successes. See Ck46 W1.3 for KL and entropy comparisons, pQ6Y W2/Q2 for FIFO, and Appx. E.5, Fig. 19 for the KL learning curves.

  • Replay strategy and weight ablations (Ck46 W1.4, sJpX W2–W3). We compare uniform and prioritized replay and provide additional discussions on replay loss weights. We’ve included results in Appx. E.4, Figs. 17–18.

  • Broader model and task coverage (sJpX W4). We add Cryptarithm, a new logic task, and extend evaluation to the OLMo family on Math and Coding, where standard RL also exhibits forgetting. The revised PDF includes Cryptarithm results in Appx. E.6, Figs. 20–21 and OLMo learning curves in Figs. 12–13. We also clarify model and dataset selection in Appx. D.1–D.2 and acknowledge the limitations of the evaluated model scales in the conclusion section and Appx. A.

We have responded to all reviewers’ concerns point by point and will carefully integrate the additional results and clarifications into our revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.