- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors claim that while RL is known for preserving prior model capabilities better than SFT during post-training, this retention advantage is not universal and can reverse.
The authors investigate this phenomenon by connecting target learning to non-target retention through gradient analysis and controlled experiments across base models, target tasks, and learning rate regimes.
They show that RL's retention benefits depend critically on the initial model's competency on the target task, how efficiently each method learns, and the chosen learning rate regime.
Critical strengths
-
Rather than treating RL and SFT as black boxes, the paper provides a mathematical bridge by decomposing the SFT gradient into a capability component and a supervision component, they prove that expected on-policy RL gradients with binary rewards follow only the capability component.
-
The authors introduce a Taylor-expansion-based CE decomposition of non-target retention into first-order gradient-update interactions and nonlinear curvature contributions, this allow the authors to isolate when non-target degradation is driven by first-order gradient alignment versus nonlinear curvature effects.
-
Empirically, the authors Systematically compares SFT and RL at matched target accuracy gains across 12 model-target-task combinations, 4 base models, 3 target tasks, and extensive learning-rate sweeps.
Some of the insights are interesting:
-
their results suggest that post-training retention is a direct consequence of how hard the optimizer has to push to achieve target mastery. If an initialization is misaligned with the target task (Llama on Countdown), on-policy RL starves for reward variance.
-
When RL lacks informative reward signals, practitioners are forced to increase learning rates or scale up updates. This causes high-norm gradients and nonlinear curvature effects that scramble unrelated pretraining knowledge.
-
This is also interesting implication: SFT bypasses the reward-scarcity trap of RL if the supervision data is pre-aligned (compatible) with the model's likelihood landscape. Under such conditions, SFT can gently go down a smooth loss basin at small learning rates, preserving non-target capabilities better than RL.
Critical weaknesses
-
The analytical gradient decomposition (Proposition 3.1) primarily assumes binary rewards, which may not fully capture the nuances of dense, continuous, or partial-credit reward functions used in some advanced RL post-training pipelines.
-
The investigation focuses on text-based autoregressive language models and specific reasoning/instruction tasks like Countdown, MMLU, and IFEval, direct transferability of these specific gradient dynamics to multimodal vision-language models or other domains like coding or long context agentic tasks requires separate verification.
Questions for authors
-
Proposition 3.1 and your core gradient decomposition rely heavily on a binary reward assumption. How do these dynamics change when applying dense, continuous, or partial-credit reward functions (e.g., format rewards, step-by-step verification, or soft correctness scores)? Does the supervision component of SFT still map cleanly relative to a continuous reward signal?
-
You use the initial per-token negative log-likelihood to measure supervision compatibility. However, SFT supervision data in your experiments was generated by a much larger model (Llama-3.3-70B-Instruct). To what extent is SFT retention driven by the inherent compatibility of the target task's distribution versus the style, verbosity, or format of the 70B teacher model's outputs? Would using on-policy self-generated trajectories for SFT alter these findings?
-
For models with near-zero initial success on a target task (Llama on Countdown), RL is trapped between "zero learning at small learning rates" and "severe non-target forgetting at large learning rates". Could curriculum learning and warm-up phases successfully bridge this reward-variance gap without incurring heavy nonlinear retention costs?
3 comments
Response to Reviewer Sk9w [2/2]
Question 2. Does the compatibility measure capture the target task distribution or the teacher’s style, verbosity, and format? Would on-policy self-generated supervision change the findings?
We thank the reviewer for this insightful question. We address it by clarifying what the compatibility measure captures and adding controlled comparisons with alternative teachers and self-generated supervision. These comparisons strengthen the evidence that more compatible supervision generally yields a better SFT gain–retention trade-off, including beyond the original Llama teacher.
Specifically:
1. What the compatibility measure captures. Both task content and response characteristics contribute to the measure. It captures how well the correct supervision responses match the student, including their style, verbosity, and format.
2. Controlled supervision-source comparisons. We added Qwen2.5-72B-Instruct and self-generated correct responses alongside the original Llama-3.3-70B-Instruct supervision. Within each setting, the student, target task, prompts, training protocol, and LR grid are fixed; only the supervision responses change.
3. Whether self-generated supervision changes the findings. Changing supervision can change retention and reverse the relative ordering of Llama and Qwen, while the association between compatibility and retention generally persists. Across all supervision-source comparisons for the same student and task, the more compatible source yields the better gain–retention frontier. On Countdown, Qwen-generated responses are more compatible with both Llama students and yield better frontiers than Llama-generated responses, further showing that the relationship extends across teacher families.
Figure 7 summarizes these results. Appendix C.4.3, Figures 17–18 report the compatibility measurements and gain–retention frontiers for each student–task setting.
Question 3. Can curriculum learning or warm-up make RL updates informative for low-success starts such as Llama on Countdown while preserving retention?
We thank the reviewer for this insightful question. We address it by adding an SFT warm-start experiment and discussing the implications for curriculum learning. Yes, in the tested setting, a short SFT warm start increases informative RL groups and enables GRPO to reach a comparable target gain at a smaller learning rate with better retention.
Specifically:
1. SFT warm start and informative groups. We ran this experiment on Llama-3.2-1B / Countdown, where informative groups are rarest. We first apply SFT for 0.2 epoch, then run GRPO for two epochs from the resulting model. The warm start raises the fraction of informative groups at every evaluated learning rate, addressing the scarcity of reward feedback identified in Finding 2.
2. Learning rate and retention at matched target gain. We report final checkpoints (two epochs), following the protocol of Table 1. With an SFT warm start, GRPO reaches a comparable target gain at a smaller learning rate and with better retention.
| Method | Learning rate | Target gain (pp) | Avg. non-target change (pp) |
|---|---|---|---|
| GRPO (Table 1) | 8e-6 | +22.6 | −1.48 |
| SFT warm start + GRPO | 1e-6 | +21.3 | +0.88 |
These results support the relevance of initialization and informative reward feedback in Finding 2. The warm start increases informative groups and improves the observed learning–retention trade-off.
3. Implications for curriculum learning. We appreciate this suggestion and leave its experimental evaluation to future work. Our analysis suggests that selecting prompts with intermediate success probabilities could increase informative groups early in training, since these prompts are more likely to produce groups containing both correct and incorrect responses.
We thank the reviewer again and are happy to answer further questions.
Dear Reviewer,
Thank you for your careful review and feedback. Completing the full set of additional experiments and incorporating the results took some time, so we posted our rebuttal relatively late in the discussion period. With the discussion deadline approaching, we’d be grateful if you had a chance to read our rebuttal and share any thoughts. We’d be happy to clarify any remaining points. Thank you again for your time.
The Authors
Response to Reviewer Sk9w [1/2]
We thank the reviewer for these thoughtful questions. We address these questions by extending the gradient decomposition to general rewards and clarifying how the additional terms relate to the SFT supervision component.
We agree that the capability-direction result depends on the reward structure, rather than being an intrinsic property of RL. However, our paper investigates why RL often forgets less than SFT and under what conditions this ordering reverses. Binary correctness rewards are one of the conditions under which we study the RL–SFT retention comparison. We do not systematically investigate the effect of reward design in this paper and leave that extension to future work.
1. Discussion about general rewards.
Proposition 3.1 indeed assumes binary rewards, and the decomposition extends naturally to general rewards in a way that also answers how the SFT supervision component relates to them. For a reward R(x,y), let sθ(x) be the probability of generating a correct response, qθ± the conditional distributions over correct and incorrect responses, and Rˉ±=Eqθ±[R]. Then
−∇θEπθ[R]=capability(Rˉ+−Rˉ−)sθ(x)gcap−preferences among correct responsessθ(x)∇θEqθ+[R]−preferences among incorrect responses(1−sθ(x))∇θEqθ−[R].
With binary rewards, R is constant within each class, the last two terms vanish, and we recover Proposition 3.1 (αReinforce=sθ(x)). General rewards can change the mean reward gap between correctness classes and introduce preferences within each class. These within-class preferences are captured by the two additional terms above, so the expected reward gradient need not remain proportional to gcap.
2. Mapping to the SFT supervision component.
The correspondence remains clear. The term among correct responses is the RL counterpart of the supervision component gsup: both reshape the distribution over correct responses qθ+. The difference lies in the target of this reshaping. SFT pulls qθ+ toward a fixed external distribution Q+, whereas a continuous reward pulls it toward a reweighting of the model's own correct responses, since ∇θEqθ+[R]=Eqθ+[(R−Rˉ+)∇θlogqθ+]. This suggests an analogous role for reward–policy compatibility, but extending our SFT diagnostic and establishing its relationship require separate analysis.
We thank the reviewer for this helpful comment, and we agree that the empirical findings for other modalities and domains require separate verification.
At the same time, the analytical parts of our framework do not depend on the modality or the task. Proposition 3.1 only requires an autoregressive policy and a binary correctness reward, so it applies directly to vision-language models, where the image is part of the input x, and to coding with pass/fail unit tests. For multi-turn agentic tasks with a binary success signal, it applies at the level of whole trajectories. The CE decomposition is an identity on the realized parameter updates and holds for any differentiable model.
Our analysis also suggests what to expect in the domains the reviewer mentions. Long-horizon agentic and hard coding tasks often start with a low success rate, and Finding 2 predicts that RL then receives few informative groups and needs larger learning rates, which makes retention reversals more likely. Testing this prediction in these domains is a natural direction for future work.