acceptodds
Under review as a conference paper at ICLR 2027

RL Fine-Tuning Heals OOD Forgetting in SFT

20 views

Abstract

Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) is a standard post-training recipe for improving Large Language Models (LLM) reasoning, but why it works remains unclear. We revisit the common claim that “SFT memorizes, RL generalizes” through checkpoint-wise analyses of in-distribution (ID) and out-of-distribution (OOD) reasoning. We find that OOD performance often peaks early during SFT and then declines despite continued improvement in ID reasoning. RL typically does not surpass this early SFT peak; rather, it restores OOD capability lost during later SFT, and only from a bounded range of SFT checkpoints. Further spectral analysis and direction interventions show that forgetting and recovery are correlated with singular-vector directions, whereas singular-value substitutions have much smaller effects. These findings suggest a more precise view of post-training dynamics: SFT can forget, RL can recover, and controlling singular-vector rotation may improve OOD robustness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning Fine-tuning, Supervised Fine-tuning, OOD Forgetting, Two-stage Fine-tuning, Post-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PxUvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates impact of SFT duration on out-of-domain (OOD) performance (the authors show it degrades with prolonged SFT), and on recovery of that performance via RL (the authors claim the RL stage applied on intermediate SFT ckpts restores OOD performance).

The claimed contributions are:

  • Showing that OOD performance declines with prolonged SFT, while in-domain (ID) performance improves.
  • Demonstrating a pattern of recovery via RL, where early SFT checkpoints degrade OOD perf. with RL, then RL improves OOD, and for later checkpoints RL degrades again.
  • Spectral analysis and interventions
Critical strengths

None

Critical weaknesses

There are two main issues with the paper. One of them is novelty of the paper -- the effect of increasing ID and declining OOD performance with SFT duration, and the fact that both perf. improve with RL, are established in literature. It was investigated e.g. in the work "SFT Memorizes, RL Generalizes" by Chu et al. That limits the first mentioned contribution by the authors.

Another issue is the visibly incorrect RL setup, that invalidates the findings and the authors' second contribution. We can see in Figure 2d that RL degrades in-domain performance for every SFT checkpoint, even though the RL is applied on ID setting. Moreover, the claim about restoring OOD performance via RL is not supported by Figure 3, where for some middle checkpoints RL improves performance only by ~1pp, and never comes close to the original accuracy in the initial SFT checkpoint. In this setting, the RL doesn't improve ID performance as well -- the results are noisy and there's no clear pattern of RL improving the ID accuracy, while RL on math settings has been demonstrated in existing literature to work remarkably well.

Questions for authors

none

3 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for the comments on the contribution and RL evaluation. We address both through the checkpoint-wise results, appendix controls, and additional paired evaluations.

W1 — Contribution beyond the SFT/RL comparison

We map when RL recovers previously observed OOD performance and test which parameter directions support recovery. Building on Chu et al., we measure the paired effect of a fixed RL procedure at successive SFT checkpoints:

ΔOOD(k)=Acc⁡OOD(RL(θk))−Acc⁡OOD(θk).\Delta_{\mathrm{OOD}}(k)=\operatorname{Acc}_{\mathrm{OOD}}(\mathrm{RL}(\theta_k))-\operatorname{Acc}_{\mathrm{OOD}}(\theta_k).

The dense sweeps locate both the entry and exit of the recovery window (Figures 2–3). Mean OOD gains are +5.12 pp across seven LLaMA/PPO handoffs and +2.89 pp across eight Qwen/GRPO handoffs. The early SFT peak gives recovery a concrete reference.

Early stopping and late-SFT+RL (Table 2). Values are mean ± one SD over three training runs on GeneralPoints.

Model / RLStageID accuracyOOD accuracy
LLaMA / PPOEarly SFT, checkpoint 14024.790 ± 2.82017.520 ± 2.290
LLaMA / PPOLate SFT, checkpoint 110068.380 ± 3.2426.920 ± 1.529
LLaMA / PPOLate SFT + RL62.340 ± 3.17016.240 ± 2.410
Qwen / GRPOEarly SFT, checkpoint 12026.070 ± 2.87019.670 ± 2.600
Qwen / GRPOLate SFT, checkpoint 80043.590 ± 3.24014.100 ± 2.460
Qwen / GRPOLate SFT + RL46.600 ± 3.21019.660 ± 2.600

Late-SFT+RL nearly reaches the early SFT OOD peak while retaining 37.55 more ID points for LLaMA and 20.53 more for Qwen than the stopped early model.

The spectral interventions add a second contribution: they test which parameter directions support recovery.

Singular values and directions (Table 19). The natural-scale factorial over 168 Qwen matrices gives the following effects on the equal-six OOD macro:

Factorial quantityEffect (pp)Paired 95% CI
Singular-value attribution+0.27[−0.63, 1.19]
Singular-vector direction attribution+2.81[1.56, 4.09]
Direction-minus-value contrast+2.54[1.07, 4.04]

Geometry-matched controls (Table 20). Pretrained-direction substitutions and random controls are matched in perturbation magnitude and principal-angle geometry. IFEval changes are measured from the same 55.268% SFT baseline:

DosePretrained-direction gain (pp)Random-control gain (pp)Difference (pp)Paired 95% CI
0.125+2.773+0.185 ± 0.370+2.588[−0.185, 5.360]
0.250+4.067+0.370 ± 0.847+3.697[0.678, 6.778]
0.500+5.915+0.678 ± 0.282+5.237[2.341, 8.195]
0.750+8.688+0.801 ± 1.337+7.887[4.868, 10.906]
1.000+10.536+0.062 ± 0.949+10.474[7.209, 13.740]

Random-control values are mean ± one SD across three matched controls. The full-dose advantage is 10.47 pp [7.21, 13.74], supporting the importance of pretrained direction identity.

Protected SFT (Table 13). At the matched one-epoch checkpoint, protecting top-256 subspaces improves mean ID and OOD accuracy on both training tasks:

SFT datasetMethodID accuracyOOD macro
CommonsenseQAStandard SFT84.30 ± 0.9026.67 ± 0.78
CommonsenseQAProtected SFT84.58 ± 0.2128.33 ± 0.61
ProofWriterStandard SFT89.42 ± 1.324.71 ± 0.35
ProofWriterProtected SFT91.16 ± 1.547.05 ± 2.72

Values are mean ± one SD over three training seeds. The OOD macro gives equal weight to IFEval, MMLU-Pro, and SuperGPQA.

AuthorsOpenReview ↗

Response (2/2)

W2 — RL evaluation and ID performance

Our PPO implementation follows the protocol and budget of Chu et al.; the code accompanies the submission. Appendix C.3 checks the result across four budgets, from 0.5× to 2×:

PPO budget sweep (Table 6). OOD accuracy is mean ± one SD over three training seeds for each setting.

SFT checkpointSFTPPO 0.5×PPO 1×PPO 1.5×PPO 2×
30016.24 ± 2.587.26 ± 1.208.12 ± 1.549.12 ± 2.589.54 ± 1.62
8005.98 ± 1.0811.11 ± 0.129.40 ± 1.6710.26 ± 0.4311.11 ± 1.28
13009.40 ± 2.216.41 ± 1.446.84 ± 1.516.84 ± 1.866.70 ± 1.93

The before/inside/after-window pattern persists across all four budgets, with +3.42–5.13 pp OOD gains at checkpoint 800.

The submitted experiments also show joint ID/OOD gains. Figures 13–14 show that LLaMA/GRPO improves both metrics on GeneralPoints, providing a comparison on the same model and task as Figure 2.

Qwen/GRPO recovery-window rows from Table 8:

SFT checkpointID: SFT → GRPOOOD: SFT → GRPO
40032.48 → 33.2915.38 → 16.67
50032.05 → 32.8515.38 → 18.38
60035.04 → 35.9016.24 → 18.80
70032.91 → 34.1916.67 → 18.80
80043.59 → 47.7414.10 → 19.67
90047.86 → 49.0611.97 → 14.96
100049.57 → 50.8113.68 → 16.67
110057.26 → 63.2513.68 → 16.24

All eight handoffs improve both metrics, averaging +2.04 pp ID and +2.89 pp OOD.

These sweep values are reported separately from the three-run endpoint comparison in Table 2.

Public math ID results (Table 7). Each cell shows SFT-end → RL.

Benchmark / metricQwen-7B / GRPOQwen3B / DAPO
AIME24 avg@1615.21 → 18.753.96 → 4.79
AIME25 avg@1619.38 → 20.835.00 → 5.42
AMC avg@1650.47 → 58.1335.00 → 37.50
MATH-500 avg@472.85 → 79.3560.95 → 65.90
Minerva avg@831.52 → 38.9722.75 → 25.32
Olympiad avg@437.41 → 43.4426.25 → 27.44
Average37.81 → 43.24 (+5.43 pp)25.65 → 27.73 (+2.08 pp)

Additional paired ID/OOD evidence

We added exact Qwen3B SFT-to-DAPO40 comparisons on MATH-500 and IFEval:

Raw SFT handoffMATH-500, SFT → DAPOID gain [95% CI]IFEval, SFT → DAPOOOD gain [95% CI]
240040.70 → 54.05+13.35 [10.00, 16.75]38.45 → 44.73+6.28 [2.96, 9.61]
640046.70 → 57.15+10.45 [7.25, 13.70]33.64 → 37.71+4.07 [0.55, 7.58]

Both handoffs improve both metrics, with all four pointwise 95% intervals above zero. Each SFT/RL pair uses identical evaluation settings. MATH-500 reports avg@4; IFEval reports prompt-level loose all-instructions accuracy on 541 prompts. Paired intervals resample problems or prompts for these fixed checkpoints. These additional evaluations are reported separately from Figure 3; settings are recorded in the accompanying evaluation note.

MATH-500 adds broader coverage alongside the 30-problem AIME-25 evaluation. Figure 3 measures OOD improvement relative to each initializer; Table 2 measures near-restoration to the early SFT peak in the controlled tasks. The additional pairs establish joint gains at two specified handoffs and are reported separately from the original trajectory. Together, these results support the paper's central finding: the SFT handoff helps determine the OOD recovery achieved by subsequent RL.

My concerns remain. The main issue is novelty, as the fact that SFT might hurt OOD performance and RL can restore it is already established. The paper adds a sweep over SFT checkpoints and reports where RL helps and where doesn't, but it does not explain why the recovery window exists or how to predict its boundaries. The additionally proposed protected SFT gives only minor gains. I will maintain my score.

Reviewer_EWTEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the effect of RL on the OOD performance on checkpoints at different SFT stages. It firstly identifies that the OOD performance of the SFT model during SFT training will firstly increase, then decrease, whereas the ID performance constantly increases. And RL training via PPO can help to restore the OOD performance for the SFT checkpoints at certain intermediate stages, that can approach the best performance of the SFT checkpoints. The authors further investigate the weight matrix components and find that instead of the singular values, the singular vectors can have a larger influence on this decrease on OOD tasks. They also show that a suppression on the rotation of the OOD-related subspaces can help the retention of the OOD performance during SFT.

Critical strengths
  • This paper investigates the effect of SFT on the follow-up RL training, which provides insights on the strength of SFT needed for RL training. In particular, little SFT cannot benefit neither ID nor OOD performance, while too much SFT can hurt the restoration of OOD performance via RL training.
  • It also identifies the restoration of OOD performance via RL training, and provides preliminary approaches via retaining the related subspace to keep the OOD performance of the models. This also challenges the previous “SFT memorizes, RL generalizes”, in the sense that SFT can also generalizes in the early stage then declines and RL restores such generalization ability.
Critical weaknesses
  • Given that there are many works considering the elements that influence the generalization ability of SFT (e.g., [1,2]), it remains unclear whether the findings remain valid with different SFT training data sources, e.g., using the data generated by the RL trained model as the SFT training data. Nevertheless, I acknowledge that the manuscript does challenges the previous statement “SFT memorizes, RL generalizes” under the given training setups. The scope of the the new statement “SFT forgets, RL recovers” should be restricted to the examined setups.

[1] Ren, et al. Rethinking Generalization in Reasoning Sft: A Conditional Analysis on Optimization, Data, and Model Capability. 2026.

[2] Hou, et al. Rethinking “RL Generalizes, SFT Memorizes”: The Role of SFT Data. 2026

Questions for authors
  • For Figure 2(d), it is confusing that why SFT+PPO has worse ID test accuracy compared to the corresponding initial SFT checkpoint before PPO training. Does this indicate that PPO training worsens the ID performance? In addition, in Figure 3, the ID test accuracy does not improve. Is it better to augment the evaluation by using another ID test benchmarks for Figure 3, as AIME25 might be too difficult for a 3B model?
  • The results in Figure 2 and Figure 3 show different trend of ID and OOD performance. Can the authors explain this difference here?

Minors:

  • I suggest that the authors should move the important results for Direction-sensitive recovery paragraph and Algorithmic implication paragraph to the main paper in the revision to improve reading experience.
  • I also suggest the authors to be precise of the statement “SFT forgets, RL recovers”. It is meant for the OOD tasks.
2 comments

Thank you for the suggestions on data sources, ID evaluation, and presentation. We have added MATH-500 results and clarified the scope of the findings.

W1 — SFT data source and scope

Our public math setting already uses demonstrations from an RL-trained model. OpenR1-Math-220k contains DeepSeek-R1-generated reasoning traces. We now state this provenance explicitly and discuss the cited work by Ren et al. and Hou et al.

Our claim is: In the examined settings, continued SFT can improve ID performance while reducing OOD performance, and subsequent RL can recover OOD performance at suitable handoffs.

We reproduce both forgetting and recovery with RL-generated SFT data. The additional three-stage evaluation is:

StageCheckpointID: MATH500 avg@4OOD: IFEval
MAX_OOD reference*Base model55.95%39.56%
Further SFT: forgettingSFT640059.35%33.09%
RL recoverySFT6400 → DAPO4060.65%38.26%

*Here, MAX_OOD denotes the highest IFEval score among the three reported stages; the reference is the Base model.

MATH500 improves through both stages: 55.95% → 59.35% → 60.65%. IFEval follows the forgetting-and-recovery pattern: 39.56% → 33.09% → 38.26%. DAPO recovers 79.9% of the observed OOD drop while adding 1.30 pp ID accuracy.

Two separately identified SFT trajectories provide further evidence of ID/OOD divergence on MATH-500 and IFEval:

SFT trajectoryMATH-500, first → finalIFEval, first → final
First trajectory, raw steps 400 → 800047.10 → 60.00; +12.90 pp39.37 → 34.57; −4.81 pp
Separate SFT training seed, raw steps 400 → 800048.05 → 59.25; +11.20 pp42.70 → 36.60; −6.10 pp

Q1/Q2 — ID performance, OOD recovery, and the different trends

Further paired evaluations show joint ID/OOD improvement. The following Qwen3B SFT-to-DAPO40 results are presented separately from the three-stage comparison above:

Raw SFT handoffMATH-500, SFT → DAPOID gain [95% CI]IFEval, SFT → DAPOOOD gain [95% CI]
240040.70 → 54.05+13.35 [10.00, 16.75]38.45 → 44.73+6.28 [2.96, 9.61]
640046.70 → 57.15+10.45 [7.25, 13.70]33.64 → 37.71+4.07 [0.55, 7.58]

Both handoffs improve both metrics, with all four pointwise 95% intervals above zero. Each SFT/RL pair uses identical evaluation settings. MATH-500 reports avg@4; IFEval reports prompt-level loose all-instructions accuracy on 541 prompts. Paired intervals resample problems or prompts for these fixed checkpoints. These additional evaluations are reported separately from Figure 3; settings are recorded in the accompanying evaluation note.

The figures describe different model/RL configurations. Figure 2 shows an ID/OOD tradeoff for LLaMA/PPO. On the same model and task, GRPO improves both ID and OOD (Appendix F.2, Figures 13–14). The submitted Qwen results also show joint gains:

Qwen/GRPO recovery-window rows from Table 8:

SFT checkpointID: SFT → GRPOOOD: SFT → GRPO
40032.48 → 33.2915.38 → 16.67
50032.05 → 32.8515.38 → 18.38
60035.04 → 35.9016.24 → 18.80
70032.91 → 34.1916.67 → 18.80
80043.59 → 47.7414.10 → 19.67
90047.86 → 49.0611.97 → 14.96
100049.57 → 50.8113.68 → 16.67
110057.26 → 63.2513.68 → 16.24

All eight handoffs improve both metrics, averaging +2.04 pp ID and +2.89 pp OOD.

Public math ID results (Table 7). Each cell shows SFT-end → RL.

Benchmark / metricQwen-7B / GRPOQwen3B / DAPO
AIME24 avg@1615.21 → 18.753.96 → 4.79
AIME25 avg@1619.38 → 20.835.00 → 5.42
AMC avg@1650.47 → 58.1335.00 → 37.50
MATH-500 avg@472.85 → 79.3560.95 → 65.90
Minerva avg@831.52 → 38.9722.75 → 25.32
Olympiad avg@437.41 → 43.4426.25 → 27.44
Average37.81 → 43.24 (+5.43 pp)25.65 → 27.73 (+2.08 pp)

AIME-25 has 30 problems, so one problem changes single-output accuracy by about 3.33 pp. MATH-500 adds broader problem coverage. Figure 3 maps checkpoint-wise OOD recovery; the new paired evaluations measure joint improvement at specified handoffs under separately reported settings.

Presentation updates

We have moved representative direction-sensitive recovery and protected-SFT results into the main text and clarified checkpoint definitions, data sources, and metrics. The phrase “SFT forgets, RL recovers” now explicitly refers to OOD performance in the examined settings.

Thank the authors for their response and most of my concerns have been resolved.

The response to W1 clarifies and provides further evidence that the RL can restore the performance decreased by SFT forgetting.

The response to Q1 and Q2 provide further evidence of the correctness on ID and OOD tasks trained via GRPO and DAPO, the authors also provided results on more ID benchmarks to support their results. Apart from these results, it still concerns me why Figure 2 shows worse results for models trained via PPO, which have not been directly answered.

My concerns about the presentation issues have been solved.

Given that the authors have solved most of my concerns raised in the initial review, I increase my evaluation to 3. Nevertheless, the results related to PPO still concern me, and the reported results seem to only support GRPO-style (e.g. GRPO and DAPO) RL algorithms, instead of general RL algorithms (e.g. PPO). I noticed that other reviewers share the same concerns.

Reviewer_nLaPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the OOD effect of downstream RL changes as the RL initializer moves along one SFT trajectory. On controlled rule-shift tasks and a public Qwen2.5-3B OpenR1-to-DAPO sweep, it reports that OOD accuracy peaks early in SFT and then declines, and that RL recovers OOD accuracy only when started from an intermediate range of checkpoints. A spectral analysis finds that singular-vector directions matter much more than singular values, and a projected-gradient protected SFT is proposed as a design implication.

Critical strengths
  • Useful object of study. Measuring the paired RL effect from many handoff points along one SFT trajectory makes the handoff checkpoint an explicit design variable.
  • Careful controls. The Base-directed dose response is compared with geometry-matched random directions, and results that go against the paper's own account are reported openly.
Critical weaknesses

1. Headline magnitudes rest on favorable handoffs. In each sweep the reported SFT end lies inside the recovery region rather than at the last checkpoint, where RL lowers OOD accuracy:

SweepReported SFT endΔOOD there (pp)Last checkpointΔOOD there (pp)
LLaMA / PPO / GeneralPoints1100+9.321600negative
Qwen / GRPO / GeneralPoints800+5.571600−2.99
Qwen2.5-3B / DAPO / IFEval600 (by Table 9)+1.112400−1.29

The Qwen gain at 800 is the largest of the 16 in Table 8, and SFTMaxOOD\mathrm{SFT}_{\mathrm{MaxOOD}} is a test-set maximum, so the magnitudes in Table 3 are likely optimistic.

2. Recovery not attributed to RL. PPO lowers ID accuracy at all 17 initializers (Fig. 2d), while a Base-directed weight substitution alone raises IFEval by about 10.5 pp (Table 20). The OOD gain may therefore partly reflect an undoing of SFT specialization, consistent with reports that forgetting often reflects lost task alignment [1] and that on-policy data drive the lower forgetting of RL [2]; no non-RL control separates these explanations.

Typos. L310 "A SVD Analysis" → "An SVD Analysis"; L1268 stray "vspace6pt" → remove; L1292 "e.g.,," → "e.g.,"; L1403 "performance … are shown" → "performance … is shown".

Questions for authors
  1. Please report the RL effect averaged over each recovery window and at the last checkpoint, and evaluate the Appendix C.2 selector on a held-out OOD split.
  2. Non-RL controls at matched ID accuracy, such as weight interpolation between SFTEnd\mathrm{SFT}_{\mathrm{End}} and SFTMaxOOD\mathrm{SFT}_{\mathrm{MaxOOD}} or SFT on on-policy data [2], should be reported, and the spectral analysis should be related to the finding that RLVR rotates principal subspaces less than SFT [3].

References

[1] Spurious Forgetting in Continual Learning of Language Models. ICLR 2025.
[2] Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting. ICML 2026.
[3] The Path Not Taken: RLVR Learns Off the Principals. NeurIPS 2026.

2 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for the helpful suggestions. We have added trajectory summaries, held-out checkpoint selection, and the proposed interpolation control.

W1/Q1 — Recovery magnitudes and held-out checkpoint selection

Recovery covers a range of handoffs. We distinguish the selected SFT endpoints from the final checkpoints of the extended sweeps:

ExperimentObserved recovery windowMean OOD gain in windowOOD change at final checkpoint
LLaMA-11B/PPO/GeneralPoints500–1100; 7 checkpoints+5.121600: 6.84 → 4.70; −2.14
Qwen-7B/GRPO/GeneralPoints400–1100; 8 checkpoints+2.891600: 12.82 → 9.83; −2.99
Qwen3B/DAPO/IFEval, sparse Table 9 experiment600 and 1400+0.752400: 39.37 → 38.08; −1.29

These summaries quantify the recovery window and its boundary. Separately, the dense public trajectory in Figure 3 shows positive mean IFEval changes at all 11 checkpoints from 600–1600, across three training runs. The early SFT maximum serves as a retrospective reference.

Held-out selection improves test OOD. We use 128 validation and 256 test examples per distribution. A fixed-2-pp version of Appendix C.2 retains checkpoints within 2 pp of the best OOD validation score, then selects the candidate with the highest ID validation score.

Selection ruleSelected checkpointTest IDTest OOD
Highest ID validation accuracy160073.054.69
OOD tolerance, then highest eligible ID30034.7710.55

The test OOD gain is 5.86 pp [1.56, 10.16]. This experiment evaluates standalone SFT selection, with both ID and OOD outcomes shown above.

Early stopping and late-SFT+RL (Table 2). Values are mean ± one SD over three training runs on GeneralPoints.

Model / RLStageID accuracyOOD accuracy
LLaMA / PPOEarly SFT, checkpoint 14024.790 ± 2.82017.520 ± 2.290
LLaMA / PPOLate SFT, checkpoint 110068.380 ± 3.2426.920 ± 1.529
LLaMA / PPOLate SFT + RL62.340 ± 3.17016.240 ± 2.410
Qwen / GRPOEarly SFT, checkpoint 12026.070 ± 2.87019.670 ± 2.600
Qwen / GRPOLate SFT, checkpoint 80043.590 ± 3.24014.100 ± 2.460
Qwen / GRPOLate SFT + RL46.600 ± 3.21019.660 ± 2.600

Late-SFT+RL nearly reaches the early SFT OOD peak while retaining 37.55 more ID points for LLaMA and 20.53 more for Qwen than the stopped early model.

W2/Q2 — Joint ID/OOD recovery and interpolation control

Forgetting and recovery with RL-generated SFT data. The additional three-stage evaluation shows the full sequence:

StageCheckpointID: MATH500 avg@4OOD: IFEval
MAX_OOD reference*Base model55.95%39.56%
Further SFT: forgettingSFT640059.35%33.09%
RL recoverySFT6400 → DAPO4060.65%38.26%

*Here, MAX_OOD denotes the highest IFEval score among the three reported stages; the reference is the Base model.

SFT raises ID by 3.40 pp while OOD falls by 6.47 pp. DAPO then improves both: +1.30 pp ID and +5.17 pp OOD, recovering 79.9% of the observed OOD drop. The final model retains 4.70 pp more ID accuracy than Base. This shows OOD recovery alongside further ID improvement.

Interpolation control.

We added a separate LLaMA control using θα=(1−α)θSFT+αθearly\theta_\alpha=(1-\alpha)\theta_{\mathrm{SFT}}+\alpha\theta_{\mathrm{early}}. At checkpoint 800, validation selected α=0.4\alpha=0.4: interpolation and PPO achieved 56.84% and 57.27% validation ID, within the fixed 2 pp matching tolerance.

ModelTest IDTest OOD
SFT80061.119.40
Interpolation, α=0.4\alpha=0.450.859.83
SFT800 followed by PPO2057.699.83

PPO retains 6.84 pp more test ID accuracy [1.71, 11.97] at the same observed OOD accuracy as the validation-selected interpolation control. The PPO-minus-interpolation OOD contrasts are 0.00 pp [−2.99, 2.99] at checkpoint 800 and −1.17 pp [−3.91, 1.56] at checkpoint 1100.

Qwen/GRPO recovery-window rows from Table 8:

SFT checkpointID: SFT → GRPOOOD: SFT → GRPO
40032.48 → 33.2915.38 → 16.67
50032.05 → 32.8515.38 → 18.38
60035.04 → 35.9016.24 → 18.80
70032.91 → 34.1916.67 → 18.80
80043.59 → 47.7414.10 → 19.67
90047.86 → 49.0611.97 → 14.96
100049.57 → 50.8113.68 → 16.67
110057.26 → 63.2513.68 → 16.24

All eight handoffs improve both metrics, averaging +2.04 pp ID and +2.89 pp OOD.

These sweep values are reported separately from the three-run endpoint comparison in Table 2. The joint gains show that RL recovery can accompany stronger task performance. Our contribution concerns when recovery occurs and the joint ID/OOD performance it achieves; parameter interventions provide another route to recovery.

AuthorsOpenReview ↗

Response (2/2)

Related work and spectral evidence

We connect Spurious Forgetting to task alignment and access to existing capability, and Retaining by Doing to the role of on-policy data. The Path Not Taken studies how much principal subspaces rotate; our interventions test which directions improve OOD performance.

Singular values and directions (Table 19). The natural-scale factorial over 168 Qwen matrices gives the following effects on the equal-six OOD macro:

Factorial quantityEffect (pp)Paired 95% CI
Singular-value attribution+0.27[−0.63, 1.19]
Singular-vector direction attribution+2.81[1.56, 4.09]
Direction-minus-value contrast+2.54[1.07, 4.04]

Geometry-matched controls (Table 20). Pretrained-direction substitutions and random controls are matched in perturbation magnitude and principal-angle geometry. IFEval changes are measured from the same 55.268% SFT baseline:

DosePretrained-direction gain (pp)Random-control gain (pp)Difference (pp)Paired 95% CI
0.125+2.773+0.185 ± 0.370+2.588[−0.185, 5.360]
0.250+4.067+0.370 ± 0.847+3.697[0.678, 6.778]
0.500+5.915+0.678 ± 0.282+5.237[2.341, 8.195]
0.750+8.688+0.801 ± 1.337+7.887[4.868, 10.906]
1.000+10.536+0.062 ± 0.949+10.474[7.209, 13.740]

Random-control values are mean ± one SD across three matched controls. The full-dose advantage is 10.47 pp [7.21, 13.74], supporting the importance of pretrained direction identity.

We have incorporated all four typographical corrections.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.