acceptodds
Under review as a conference paper at ICLR 2027

RL Does Not Always Forget Less: Understanding Retention through Target Learning in LLM Post-Training

20 views

Abstract

Reinforcement learning (RL) often preserves prior capabilities better than supervised fine-tuning (SFT) during post-training, but we show that this retention advantage is not universal and can reverse. We study this variation by connecting target learning to non-target retention through gradient analysis and controlled experiments across diverse models, target tasks, and learning rate settings. We first decompose the SFT gradient into a capability component that increases correct-response probability and a supervision component that fits the distribution over provided correct responses. Under binary rewards, the expected on-policy RL gradient is proportional only to the capability component. This distinction reveals how the initial model affects target learning differently under RL and SFT. For RL, low initial correct-response probabilities can limit reward variation and restrict target learning at small learning rates. For SFT, compatible supervision can enable comparable target gains at smaller learning rates and, in some settings, better retention than RL. We further show that the learning rate regime is closely tied to non-target retention: at small learning rates, degradation is typically localized and cross-entropy changes are largely explained by first-order gradient update interactions, whereas at larger learning rates, nonlinear contributions can become substantial and degradation can extend across more non-target tasks. Together, these results explain when RL's retention advantage holds or breaks down and identify the roles of the initial model, target learning efficiency, and learning rate regime in the relative retention of RL and SFT.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Catastrophic Forgetting, Reinforcement Learning, Supervised Fine-Tuning, Large Language Model Post-Training, Learning Dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Sk9wOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors claim that while RL is known for preserving prior model capabilities better than SFT during post-training, this retention advantage is not universal and can reverse.

The authors investigate this phenomenon by connecting target learning to non-target retention through gradient analysis and controlled experiments across base models, target tasks, and learning rate regimes.

They show that RL's retention benefits depend critically on the initial model's competency on the target task, how efficiently each method learns, and the chosen learning rate regime.

Critical strengths
  1. Rather than treating RL and SFT as black boxes, the paper provides a mathematical bridge by decomposing the SFT gradient into a capability component and a supervision component, they prove that expected on-policy RL gradients with binary rewards follow only the capability component.

  2. The authors introduce a Taylor-expansion-based CE decomposition of non-target retention into first-order gradient-update interactions and nonlinear curvature contributions, this allow the authors to isolate when non-target degradation is driven by first-order gradient alignment versus nonlinear curvature effects.

  3. Empirically, the authors Systematically compares SFT and RL at matched target accuracy gains across 12 model-target-task combinations, 4 base models, 3 target tasks, and extensive learning-rate sweeps.

Some of the insights are interesting:

  1. their results suggest that post-training retention is a direct consequence of how hard the optimizer has to push to achieve target mastery. If an initialization is misaligned with the target task (Llama on Countdown), on-policy RL starves for reward variance.

  2. When RL lacks informative reward signals, practitioners are forced to increase learning rates or scale up updates. This causes high-norm gradients and nonlinear curvature effects that scramble unrelated pretraining knowledge.

  3. This is also interesting implication: SFT bypasses the reward-scarcity trap of RL if the supervision data is pre-aligned (compatible) with the model's likelihood landscape. Under such conditions, SFT can gently go down a smooth loss basin at small learning rates, preserving non-target capabilities better than RL.

Critical weaknesses
  1. The analytical gradient decomposition (Proposition 3.1) primarily assumes binary rewards, which may not fully capture the nuances of dense, continuous, or partial-credit reward functions used in some advanced RL post-training pipelines.

  2. The investigation focuses on text-based autoregressive language models and specific reasoning/instruction tasks like Countdown, MMLU, and IFEval, direct transferability of these specific gradient dynamics to multimodal vision-language models or other domains like coding or long context agentic tasks requires separate verification.

Questions for authors
  1. Proposition 3.1 and your core gradient decomposition rely heavily on a binary reward assumption. How do these dynamics change when applying dense, continuous, or partial-credit reward functions (e.g., format rewards, step-by-step verification, or soft correctness scores)? Does the supervision component of SFT still map cleanly relative to a continuous reward signal?

  2. You use the initial per-token negative log-likelihood to measure supervision compatibility. However, SFT supervision data in your experiments was generated by a much larger model (Llama-3.3-70B-Instruct). To what extent is SFT retention driven by the inherent compatibility of the target task's distribution versus the style, verbosity, or format of the 70B teacher model's outputs? Would using on-policy self-generated trajectories for SFT alter these findings?

  3. For models with near-zero initial success on a target task (Llama on Countdown), RL is trapped between "zero learning at small learning rates" and "severe non-target forgetting at large learning rates". Could curriculum learning and warm-up phases successfully bridge this reward-variance gap without incurring heavy nonlinear retention costs?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Sk9w [1/2]

Weakness 1 & Question 1 Proposition 3.1 assumes binary rewards. How do the dynamics change under dense, continuous, or partial-credit rewards, and does the SFT supervision component still map cleanly onto a continuous reward signal?

We thank the reviewer for these thoughtful questions. We address these questions by extending the gradient decomposition to general rewards and clarifying how the additional terms relate to the SFT supervision component.

We agree that the capability-direction result depends on the reward structure, rather than being an intrinsic property of RL. However, our paper investigates why RL often forgets less than SFT and under what conditions this ordering reverses. Binary correctness rewards are one of the conditions under which we study the RL–SFT retention comparison. We do not systematically investigate the effect of reward design in this paper and leave that extension to future work.

1. Discussion about general rewards.

Proposition 3.1 indeed assumes binary rewards, and the decomposition extends naturally to general rewards in a way that also answers how the SFT supervision component relates to them. For a reward R(x,y)R(x,y), let sθ(x)s_\theta(x) be the probability of generating a correct response, qθ±q_\theta^{\pm} the conditional distributions over correct and incorrect responses, and Rˉ±=Eqθ±[R]\bar R_\pm=\mathbb E_{q_\theta^\pm}[R]. Then

−∇θEπθ[R]=(Rˉ+−Rˉ−)sθ(x)gcap⏟capability−sθ(x)∇θEqθ+[R]⏟preferences among correct responses−(1−sθ(x))∇θEqθ−[R]⏟preferences among incorrect responses.-\nabla_\theta \mathbb E_{\pi_\theta}[R] = \underset{\text{capability}}{\underbrace{(\bar R_+-\bar R_-) s_\theta(x) g_{\mathrm{cap}}}} - \underset{\text{preferences among correct responses}}{\underbrace{s_\theta(x) \nabla_\theta \mathbb E_{q_\theta^+}[R]}} - \underset{\text{preferences among incorrect responses}}{\underbrace{(1-s_\theta(x)) \nabla_\theta \mathbb E_{q_\theta^-}[R]}}.

With binary rewards, RR is constant within each class, the last two terms vanish, and we recover Proposition 3.1 (αReinforce=sθ(x)\alpha_{\mathrm{Reinforce}}=s_\theta(x)). General rewards can change the mean reward gap between correctness classes and introduce preferences within each class. These within-class preferences are captured by the two additional terms above, so the expected reward gradient need not remain proportional to gcapg_{\mathrm{cap}}.

2. Mapping to the SFT supervision component.

The correspondence remains clear. The term among correct responses is the RL counterpart of the supervision component gsupg_{\mathrm{sup}}: both reshape the distribution over correct responses qθ+q_\theta^+. The difference lies in the target of this reshaping. SFT pulls qθ+q_\theta^+ toward a fixed external distribution Q+Q^+, whereas a continuous reward pulls it toward a reweighting of the model's own correct responses, since ∇θEqθ+[R]=Eqθ+[(R−Rˉ+)∇θlog⁡qθ+]\nabla_\theta \mathbb E_{q_\theta^+}[R]=\mathbb E_{q_\theta^+}[(R-\bar R_+)\nabla_\theta\log q_\theta^+]. This suggests an analogous role for reward–policy compatibility, but extending our SFT diagnostic and establishing its relationship require separate analysis.

Weakness 2 Discuss whether the observed dynamics generalize beyond the tested text tasks to multimodal, coding, or long-context agentic settings.

We thank the reviewer for this helpful comment, and we agree that the empirical findings for other modalities and domains require separate verification.

At the same time, the analytical parts of our framework do not depend on the modality or the task. Proposition 3.1 only requires an autoregressive policy and a binary correctness reward, so it applies directly to vision-language models, where the image is part of the input xx, and to coding with pass/fail unit tests. For multi-turn agentic tasks with a binary success signal, it applies at the level of whole trajectories. The CE decomposition is an identity on the realized parameter updates and holds for any differentiable model.

Our analysis also suggests what to expect in the domains the reviewer mentions. Long-horizon agentic and hard coding tasks often start with a low success rate, and Finding 2 predicts that RL then receives few informative groups and needs larger learning rates, which makes retention reversals more likely. Testing this prediction in these domains is a natural direction for future work.

AuthorsOpenReview ↗

Response to Reviewer Sk9w [2/2]

Question 2. Does the compatibility measure capture the target task distribution or the teacher’s style, verbosity, and format? Would on-policy self-generated supervision change the findings?

We thank the reviewer for this insightful question. We address it by clarifying what the compatibility measure captures and adding controlled comparisons with alternative teachers and self-generated supervision. These comparisons strengthen the evidence that more compatible supervision generally yields a better SFT gain–retention trade-off, including beyond the original Llama teacher.

Specifically:

1. What the compatibility measure captures. Both task content and response characteristics contribute to the measure. It captures how well the correct supervision responses match the student, including their style, verbosity, and format.

2. Controlled supervision-source comparisons. We added Qwen2.5-72B-Instruct and self-generated correct responses alongside the original Llama-3.3-70B-Instruct supervision. Within each setting, the student, target task, prompts, training protocol, and LR grid are fixed; only the supervision responses change.

3. Whether self-generated supervision changes the findings. Changing supervision can change retention and reverse the relative ordering of Llama and Qwen, while the association between compatibility and retention generally persists. Across all supervision-source comparisons for the same student and task, the more compatible source yields the better gain–retention frontier. On Countdown, Qwen-generated responses are more compatible with both Llama students and yield better frontiers than Llama-generated responses, further showing that the relationship extends across teacher families.

Figure 7 summarizes these results. Appendix C.4.3, Figures 17–18 report the compatibility measurements and gain–retention frontiers for each student–task setting.

Question 3. Can curriculum learning or warm-up make RL updates informative for low-success starts such as Llama on Countdown while preserving retention?

We thank the reviewer for this insightful question. We address it by adding an SFT warm-start experiment and discussing the implications for curriculum learning. Yes, in the tested setting, a short SFT warm start increases informative RL groups and enables GRPO to reach a comparable target gain at a smaller learning rate with better retention.

Specifically:

1. SFT warm start and informative groups. We ran this experiment on Llama-3.2-1B / Countdown, where informative groups are rarest. We first apply SFT for 0.2 epoch, then run GRPO for two epochs from the resulting model. The warm start raises the fraction of informative groups at every evaluated learning rate, addressing the scarcity of reward feedback identified in Finding 2.

2. Learning rate and retention at matched target gain. We report final checkpoints (two epochs), following the protocol of Table 1. With an SFT warm start, GRPO reaches a comparable target gain at a smaller learning rate and with better retention.

MethodLearning rateTarget gain (pp)Avg. non-target change (pp)
GRPO (Table 1)8e-6+22.6−1.48
SFT warm start + GRPO1e-6+21.3+0.88

These results support the relevance of initialization and informative reward feedback in Finding 2. The warm start increases informative groups and improves the observed learning–retention trade-off.

3. Implications for curriculum learning. We appreciate this suggestion and leave its experimental evaluation to future work. Our analysis suggests that selecting prompts with intermediate success probabilities could increase informative groups early in training, since these prompts are more likely to produce groups containing both correct and incorrect responses.


We thank the reviewer again and are happy to answer further questions.

Dear Reviewer,

Thank you for your careful review and feedback. Completing the full set of additional experiments and incorporating the results took some time, so we posted our rebuttal relatively late in the discussion period. With the discussion deadline approaching, we’d be grateful if you had a chance to read our rebuttal and share any thoughts. We’d be happy to clarify any remaining points. Thank you again for your time.

The Authors

Reviewer_cK6AOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates when reinforcement learning preserves existing capabilities better than supervised fine-tuning during LLM post-training. It compares SFT and GRPO post training on four instruction-tuned Llama and Qwen checkpoints across three target tasks, while evaluating changes in non-target performance. The authors decompose the SFT gradient into a component that increases the probability of correct responses and a component that fits the distribution of supervised responses. Under their binary-reward assumptions, the expected RL gradient appears to follow the former component. Their experiments show that SFT can sometimes achieve comparable target-task improvements with better retention, and they interpret these reversals through initial reward informativeness, supervision compatibility, and learning-rate-dependent training dynamics.

Critical strengths
  • Target-task improvement seems valid. Retention is difficult to interpret without considering how much the model learned during adaptation. The experiments allude to plausible counterexamples to the claim that RL always preserves existing capabilities better than SFT.
  • The study examines different qwen and llama model sizes, several target tasks, and learning-rate sweeps. The distinction between increasing total probability on correct responses and matching particular supervised responses provides a useful framework for understanding why initialization and supervision choice could affect retention.
  • The discussion of cases where the likelihood of previously correct responses decreases while aggregate accuracy improves is valuable. It helps clarify what the proposed retention decomposition measures and where its interpretation requires care.
Critical weaknesses
  • The experiments use two epochs with constant learning rates, but the paper does not report a validation based early-stopping analysis. Figure 5 shows training reward trajectories and endpoint performance across learning rates; it does not show whether the retention reversals persist when checkpoints are selected jointly over learning rate and training duration. Table 1 also combines approximately matched-gain comparisons with cases where one method dominates on both gain and retention.
  • All starting models are instruction-tuned, so their prior post-training histories may influence their response distributions and subsequent adaptation. However, the paper does not establish that Qwen has undergone mode collapse or has received too much RL to benefit from SFT. A more concrete concern is that all SFT supervision comes from Llama-3.3-70B-Instruct: "we use correct responses generated by Llama-3.3-70B-Instruct". Since supervision compatibility is central to the explanation, the observed family differences could depend on the teacher’s response style. Testing another teacher family or self-generated correct responses would directly probe this mechanism.
  • The main result concerns expected, per-prompt, sequence-level gradients under specific reward assumptions. The experiments use AdamW, and selected RL runs additionally use KL regularization. These choices affect the relationship between raw gradient magnitude, learning rate, and the realized parameter update. A Dr.GRPO comparison would be a useful robustness test, given the normalization effects identified in its motivating analysis.
  • The general observation that larger updates can damage non-target performance is unsurprising and elementary. The more substantive contribution would be showing that the proposed initialization and supervision diagnostics predict when a retention reversal will occur, or identify useful operating regimes before training. Comparisons against actual update magnitude, total parameter drift, or other simple diagnostics would help establish the additional explanatory value of the framework.
  • Binary correctness rewards are reasonable for the selected tasks, but they are essential to the stated capability-direction result. More general rewards or advantages can distinguish between responses within the same correctness class and therefore introduce additional preferences. The policy gradient is zero in expectation so anything can be added to the advantage.
  • The experiments isolate an additional SFT or GRPO stage from an already instruction-tuned checkpoint. This is a valid controlled comparison, but it provides limited evidence about staged or mixed training procedures. In particular, a short SFT warm start could change the initial correct-response probability and the availability of informative RL updates. Replication on a more recent qwen model would also strengthen the empirical scope.
Questions for authors
  1. Do the retention reversals persist with validation-selected checkpoints? Could the authors report held-out target accuracy and non-target performance throughout training, and construct target-gain–retention frontiers across both learning rates and training steps?
  2. How sensitive are the results to the teacher and the starting checkpoint? Would Qwen-generated or self-generated correct supervision change the relative SFT retention of Llama and Qwen?
  3. Do the main results hold with Dr.GRPO?
  4. Would graded rewards or an explicit retention objective change the observed trade-off? A brief discussion connecting this distinction to the reward hypothesis, including ((https://proceedings.mlr.press/v202/bowling23a.html , http://incompleteideas.net/rlai.cs.ualberta.ca/RLAI/rewardhypothesis.html), would help separate questions about the expressiveness of reward objectives from the optimization behavior of the particular objective studied here.
  5. Could a short SFT warm start increase the frequency of informative RL groups enough to reach the same target gain at a smaller learning rate and with better retention?
8 comments
AuthorsOpenReview ↗

Response to Reviewer cK6A [1/5]

Weakness 1 and Question 1 Assess whether retention reversals persist with validation-selected checkpoints across learning rates and training steps, and clarify matched-gain versus dominance comparisons.

We appreciate the reviewer’s thoughtful questions. We have addressed them in three ways.

1. The scope of the comparison in the paper.

We would first like to clarify the scope of the comparison in the paper. As stated in Section 4.1, SFT and RL use the same training prompts and the same number of optimizer steps for each model and target task. Table 1 therefore compares the two objectives under a fixed optimization budget, sweeping the learning rate for each method. Keeping the number of optimizer steps constant allows us to study how the two objectives and their learning dynamics shape the gain–retention trade-off, without confounding this comparison with differences in the number of optimizer steps. The frontier analysis below complements this comparison by additionally varying optimizer steps.

2. Gain–retention comparison across both learning rates and checkpoints.

For Countdown and IFEval with Llama-3.2-1B, the table below summarizes the best average non-target accuracy change at each target gain threshold and the maximum target gain achieved across evaluated learning rates and training checkpoints. The full frontiers are reported in Appendix C.2.2, Figure 8.

SettingTarget gain ≥SFT Best average non-target accuracy changeGRPO Best average non-target accuracy change
Countdown / Llama-3.2-1B+20+1.80−0.11
+30+1.40—
max. gain+40.8+24.3
IFEval / Llama-3.2-1B+20−4.90−0.97
+25−7.88−5.88
max. gain+28.3+35.2

Considering learning rates and training checkpoints jointly, the observed frontiers show both outcomes:

  • Countdown: the reversal persists. SFT retains better than GRPO over the shared observed gain range and reaches gains that GRPO does not reach (+40.8 vs. +24.3).
  • IFEval: no reversal. GRPO retains better than SFT at comparable gains.

This is consistent with our central claim: RL does not always forget less, and whether it does depends on the setting. Countdown is the setting with the lowest initial correct-response probability, where GRPO receives almost no learning signal (Finding 2). The frontier figure and setup are now included in Appendix C.2.2, Figure 8.

3. Table 1 includes both approximately matched-gain comparisons and dominance comparisons.

We thank the reviewer for highlighting this distinction. Both inform our analysis of retention through target learning: matched gains allow direct comparison, while dominance shows that better retention is not simply due to learning less. Both therefore provide meaningful comparisons of retention in relation to target learning.

AuthorsOpenReview ↗

Response to Reviewer cK6A [2/5]

Weakness 2 and Question 2 Differences in SFT gain–retention between Llama and Qwen may reflect prior post-training histories or the single teacher’s response style. The compatibility explanation requires stronger evidence across alternative teachers and self-generated supervision.

We sincerely thank the reviewer for this constructive comment and helpful suggestion. To address these concerns, we ran additional SFT experiments comparing the original Llama-generated supervision with Qwen-generated and self-generated supervision. These additional experiments provide strong empirical support for our finding that more compatible supervision generally yields a better SFT gain–retention trade-off across the evaluated teachers and starting checkpoints. We added these results to the revised paper: the paragraph "Sensitivity to the supervision source" in Section 4.4 and Figure 7 summarize the comparison, and Appendix C.4.3 reports the initial NLL of each source (Figure 17) and the gain–retention frontiers for every student–task setting (Figure 18).

Specifically, we address the reviewer’s concerns as follows:

  1. Qwen can still benefit from SFT. Keeping the Qwen starting checkpoint fixed, replacing Llama-generated supervision with more compatible Qwen-generated responses improves its gain–retention trade-off. This shows that Qwen’s prior training history does not prevent it from benefiting from SFT with compatible supervision.

  2. Changing supervision can alter the relative SFT retention of Llama and Qwen students. It shows that Llama students do not necessarily achieve a better SFT gain–retention trade-off than Qwen; the comparison depends on the compatibility between the student model and the supervision.

  3. The compatibility explanation extends beyond the Llama teacher style. The relationship between measured compatibility and retention persists across alternative supervision sources. This addresses the concern that the explanation relies on a single teacher.

Together, these controlled comparisons provide strong empirical support for our explanation: with the student and task held fixed, the more compatible source yields the better gain–retention frontier.

Weakness 3 and Question 3 The experimental settings differ from the proposition (AdamW vs. SGD), and the gradient analysis does not include the KL term. Test robustness with Dr.GRPO.

We thank the reviewer for highlighting this distinction and suggesting the Dr.GRPO comparison. We address these concerns by clarifying the scope of the analysis and adding experimental comparisons. The additional experiments support the empirical relevance of our analysis, and the retention reversals also occur under Dr.GRPO in both tested settings. Specifically, we address the reviewer’s concerns as follows:

1. Scope of Proposition 3.1.

We agree that Proposition 3.1 characterizes expected objective gradients, rather than the full dynamics of AdamW with KL regularization.

Nevertheless, the analysis provides a useful basis for understanding and testing the observed retention differences. It distinguishes increasing the total probability of correct responses from fitting their conditional distribution to supervision, and identifies when GRPO receives sparse reward feedback. These observations motivate concrete experimental tests: changing the supervision source while keeping the student fixed, and introducing an SFT warm start before RL. Their results support the roles of compatibility and initialization under the actual training procedure. Our contribution combines this local theoretical analysis with controlled experiments to understand retention through target learning.

2. Robustness to Dr.GRPO.

Yes, the retention reversals also occur under Dr.GRPO in the two tested settings. This suggests that the observed reversals are not specific to GRPO’s normalization scheme. We ran Dr.GRPO on Countdown and IFEval with Llama-3.2-1B under the same protocol as Table 1. Proposition 3.1 extends directly to it: the expected gradient is (1−1M)sθ(x)gcap(θ;x)(1-\frac{1}{M})s_\theta(x)g_{\mathrm{cap}}(\theta;x). The main results hold.

TargetLR (Dr.GRPO / SFT)Target gain (Dr.GRPO / SFT)Avg. non-target change (Dr.GRPO / SFT)
Countdown†8e-6∗ / 5e-6+7.5 / +23.4−0.13 / +1.64
IFEval1e-5 / 3e-5+27.6 / +27.8−15.27 / −8.82

∗ KL-regularized run (β = 0.05), used where training without KL collapses, as in Table 1. † No Dr.GRPO run comes within 2 points of SFT's gain; the run with the highest gain is shown, and SFT dominates it on both gain and retention.

The reversals hold under Dr.GRPO. On Countdown, no Dr.GRPO run exceeds +7.5 points, and SFT attains both a higher gain and better retention, consistent with Finding 2 for the setting with the lowest initial correct-response probability. On IFEval, at a matched gain, Dr.GRPO forgets more than SFT (−15.27 vs. −8.82), as GRPO does in Table 1.

AuthorsOpenReview ↗

Response to Reviewer cK6A [3/5]

Weakness 4 Larger updates causing forgetting is elementary; show that the initialization and supervision diagnostics predict retention reversals or identify operating regimes before training, and compare them against update magnitude and parameter drift.

We thank the reviewer for this constructive suggestion. We agree that larger updates harming non-target performance is, by itself, unsurprising. Our framework aims to explain what update magnitude alone does not: which non-target capabilities are affected, and why one objective needs a larger update than the other to reach the same target gain. Following the reviewer's suggestion, we measured the parameter drift ∥θ−θ0∥2/∥θ0∥2\Vert\theta-\theta_0\Vert_2/\Vert\theta_0\Vert_2 at every evaluated checkpoint and compared it with our diagnostics.

These comparisons support the value of our framework beyond update magnitude, both in explaining retention reversals and in guiding the choice of training regime. Specifically, we address the reviewer’s concerns in three parts:

1. Comparison with parameter drift.

Drift correlates with average forgetting. However, drift is a single scalar and cannot explain three observations:

  • Which tasks are forgotten. At small learning rates, forgetting is selective, concentrated on a few non-target tasks, and is largely captured by the first-order interaction between each task's gradient and the training update (Finding 3; Section 4.3).
  • The retention reversals. In 3 of the 4 settings where SFT retains better than RL, SFT has the larger drift.
  • The large-learning-rate regime. Forgetting there grows faster than drift, as the nonlinear contribution in our CE decomposition becomes dominant (Section 4.3).

2. Initialization and supervision diagnostics.

The initial correct-response probability sθ0s_{\theta_0} and the supervision NLL ℓsup\ell_{\mathrm{sup}}, both measured at θ0\theta_0, determine the learning rate each objective needs to reach a given target gain. This learning rate predicts retention better than drift:

  • For SFT, ℓsup\ell_{\mathrm{sup}} correlates with the required learning rate. The required learning rate correlates with SFT's forgetting more strongly (Spearman 0.80) than SFT's drift does (Spearman 0.55).
  • The largest reversals occur where both conditions hold: a low sθ0s_{\theta_0}, which limits RL, and compatible supervision (ℓsup≤ℓself+\ell_{\mathrm{sup}}\le\ell^+_{\mathrm{self}}), which lets SFT learn at small learning rates.
  • Raising sθ0s_{\theta_0} with a short SFT warm start lets GRPO reach the target gain at a smaller learning rate with better retention (response to W6/Q5).

3. Identifying operating regimes before training.

These diagnostics can be computed before training. When sθ0s_{\theta_0} is low and the supervision is compatible, SFT, or a short SFT warm start before RL, is likely to retain better. When the supervision is incompatible, SFT needs large learning rates and is likely to forget.

Reviewer_3vcZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates when RL preserves non-target capabilities better than SFT and when this advantage reverses. It decomposes the SFT gradient into a capability component, which increases the probability of producing a correct response, and a supervision component, which fits the distribution of demonstrated correct responses. Under binary rewards, the expected on-policy RL gradient is proportional to the capability component. Experiments across four models, three target tasks, and multiple learning rates compare retention at comparable target gains and investigate how initialization, supervision compatibility, and optimization dynamics relate to the observed differences.

Critical strengths
  1. The paper offers an original and insightful analytical framing of the differences between RL and SFT. Although the mathematical ingredients are standard or build on prior work, their combination provides a useful distinction between acquiring target capability and fitting particular correct response distribution. This offers a plausible interpretation of why the relative retention of RL and SFT may depend on initialization and supervision.

  2. The experimental scope is extensive, covering multiple model families, scales, target tasks, non-target evaluations, and learning rates. Comparing retention at comparable target gains is valuable, and the results provide concrete evidence that RL does not always retain better than SFT.

Critical weaknesses
  1. The main concern is that the proposed explanations are not systematically reported across the broad experimental setup. Several analyses rely on selected model–task settings, vary multiple factors simultaneously, or report associations without isolating the proposed effects. This applies particularly to the learning-rate and retention patterns in Figures 3–5 and the supervision-compatibility interpretation in Figure 6. Although the overall picture is plausible, it remains unclear how consistently these explanations hold across the settings studied. A more systematic analysis of the existing experiments, including exceptions to the proposed patterns, would substantially strengthen the conclusions.

  2. Some interpretations extend beyond what the theoretical analysis establishes. The analysis characterizes per-prompt expected gradients, local first-order improvements, and changes in the likelihood of fixed correct responses. These quantities provide useful intuition, but do not directly establish the behavior of complete training runs or their final retention of task accuracy. Although the authors acknowledge some of these distinctions, the subsequent discussion sometimes presents the proposed connections more strongly than the analysis and empirical evidence support.

Questions for authors

Across the full learning-rate sweeps, how consistently do the proposed patterns hold within fixed model–task settings? Could you summarize selective versus widespread forgetting across the remaining settings?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3vcZ [1/1]

Weakness 1 and Question 1 Insufficient systematic evidence for the proposed patterns. Report their consistency and exceptions across full learning-rate sweeps within fixed model–task settings, including selective versus widespread forgetting.

We thank the reviewer for the insightful questions and constructive suggestion. Overall, our expanded analyses provide empirical support for the proposed patterns across the evaluated settings. We have addressed the questions in three ways.

1. Additional experimental results for systematic analyses.

We revised the paper to include more systematic analyses across model–task settings. The cross-setting summaries (Figures 3(d) and 4(c)) summarize the forgetting patterns and test the first-order explanation across settings. The full results (Appendices C.3 and C.4.1) report these patterns for each setting, while the exceptions in Appendix C.3.1 clarify the limits of the first-order explanation. The controlled supervision-source comparisons (Figures 6–7 and Appendix C.4.3) provide further evidence for the compatibility explanation.

2. Consistency and exceptions.

Across the evaluated learning-rate sweeps and controlled supervision-source comparisons, the proposed patterns show empirical consistency, as quantified below:

  • CE decomposition: For each training run, we identify which CE term dominates on each of the six non-target tasks. The first-order term dominates in 97.2% of results at small learning rates, while the nonlinear term dominates in 72.2% at large learning rates. Figure 3(c) in Section 4.3 shows the systematic comparison across model–task settings.

  • Forgetting patterns. As the learning rate increases, more non-target tasks are forgotten (accuracy drop > 2 pp) in 95.8% of sweeps. At small learning rates, 88.9% of forgotten tasks have a positive cumulative first-order term. Among forgotten tasks in the same run, the task with the larger first-order term loses more accuracy in 75% of comparisons. Figures 3(d) and 4(c) in Section 4.3 show the systematic comparisons, and Appendix C.3, Figures 10–11 show the accuracy changes for each task.

  • Learning–retention trade-off. The RL learning-rate sweeps cover 12 model–task pairs, including four models and three target tasks. In 11 of the 12 pairs, increasing the learning rate yielded a larger target accuracy gain at the cost of more forgetting. Figure 5 shows four examples, and Figure 12 in Appendix C.4.1 shows the full sweeps.

  • Supervision compatibility. We compare three sources of SFT supervision: correct responses generated by Llama-3.3-70B, Qwen2.5-72B, and the student model itself. For the same student model and target task, the more compatible source gives better retention at similar target gains in 90.6% of comparisons. Figure 7 summarizes the results, and Figure 18 in Appendix C.4.3 shows the gain–retention frontiers for each setting.

We also discuss exceptions in Appendices C.3.1 and C.4.3, including cases where correct-response CE and task accuracy differ.

3. Selective versus widespread forgetting (Question 1, second part).

Higher learning rates are associated with forgetting on more non-target tasks in 95.8% of the RL and SFT sweeps. Figure 3(d) summarizes this trend, while Appendix C.3, Figures 10–11 provide RL and SFT heatmaps covering all 12 model–task combinations.

Weakness 2 Scope of the theoretical analysis and its interpretation.

We thank the reviewer for pointing this out. We agree that our wording sometimes made it unclear where the theoretical results ended and our interpretation of the empirical findings began.

We revised Sections 4.2 and 4.4 and Appendix C.4.2 to make that distinction explicit. The theoretical results characterize per-prompt expected gradients, local first-order changes in target capability, and changes in the likelihood of fixed correct responses. As clarified in Section 3.2, changes in response likelihood and changes in task accuracy are distinct quantities. Our conclusions about retention at the end of training come from the empirical accuracy measurements. Where we relate those observations to the theory, we now state which patterns are consistent with the analysis and describe the relationships with initialization and supervision compatibility as empirical associations.

Our main contribution is to characterize the RL–SFT retention gap by examining it alongside target learning. The decompositions explain how the training objectives shape local learning signals and how individual parameter updates affect non-target losses. The experiments characterize the resulting gain–retention trade-offs across the settings we study.


We thank the reviewer again and are happy to answer further questions.

Dear Reviewer,

Thank you for your careful review and feedback. Completing the full set of additional experiments and incorporating the results took some time, so we posted our rebuttal relatively late in the discussion period. With the discussion deadline approaching, we’d be grateful if you had a chance to read our rebuttal and share any thoughts. We’d be happy to clarify any remaining points. Thank you again for your time.

The Authors

Thank you for the detailed response and for adding a broader analysis and evidence for some of the claims. The discussion of the exceptions to the first-order patterns, as well as the additional experiments with different supervision sources, address some important parts of my concerns.

However, some of the claims are still not fully supported by the evidence presented, or are reported in a confusing way. For example, the claim connecting limited correct-behavior information, larger learning rates, and increased forgetting is framed as a general claim, while the experiments support it only in a particular case (Llama/Countdown). A concern also remains regarding Figure 3. Panels (a) and (b) are presented in a way that is supposed to demonstrate the difference between the effects of large and small learning rates, while in fact they represent two settings that differ in model, training task, and evaluation task, and therefore do not isolate the effect of the learning rate. The scatter plot in panel (c) provides broader support for the claim, but it can also be misleading, since the points are not annotated by setting. The global association may therefore reflect differences between settings rather than the association within each setting.

Overall, the revision considerably improves the paper. However, as explained above, I still have concerns about the experimental rigor, and therefore I maintain my weak-accept score.

AuthorsOpenReview ↗

Follow-up Response: Revised Figure 3

Weakness 1

Thank you again for your helpful comments. As described in our earlier response, we added cross-setting summaries (Figures 3(d) and 4(c)), per-setting results and exceptions (Appendices C.3, C.3.1, and C.4.1), controlled supervision-source comparisons (Figure 7 and Appendix C.4.3), and revised the wording in Sections 4.2 and 4.4 and Appendix C.4.2 to distinguish the theoretical results from the empirical associations.

To further isolate the effect of the learning rate, we have revised Figure 3. Panels (a) and (b) now use the same training setting (Llama 3.2 1B trained with RL on IFEval and evaluated on MMLU) and differ only in the learning rate. The comparison shows that increasing the learning rate alone shifts the CE change from being dominated by the first-order term to being dominated by the nonlinear term, consistent with the cross-setting results in panels (c) and (d).

We would be happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.