- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates impact of SFT duration on out-of-domain (OOD) performance (the authors show it degrades with prolonged SFT), and on recovery of that performance via RL (the authors claim the RL stage applied on intermediate SFT ckpts restores OOD performance).
The claimed contributions are:
- Showing that OOD performance declines with prolonged SFT, while in-domain (ID) performance improves.
- Demonstrating a pattern of recovery via RL, where early SFT checkpoints degrade OOD perf. with RL, then RL improves OOD, and for later checkpoints RL degrades again.
- Spectral analysis and interventions
Critical strengths
None
Critical weaknesses
There are two main issues with the paper. One of them is novelty of the paper -- the effect of increasing ID and declining OOD performance with SFT duration, and the fact that both perf. improve with RL, are established in literature. It was investigated e.g. in the work "SFT Memorizes, RL Generalizes" by Chu et al. That limits the first mentioned contribution by the authors.
Another issue is the visibly incorrect RL setup, that invalidates the findings and the authors' second contribution. We can see in Figure 2d that RL degrades in-domain performance for every SFT checkpoint, even though the RL is applied on ID setting. Moreover, the claim about restoring OOD performance via RL is not supported by Figure 3, where for some middle checkpoints RL improves performance only by ~1pp, and never comes close to the original accuracy in the initial SFT checkpoint. In this setting, the RL doesn't improve ID performance as well -- the results are noisy and there's no clear pattern of RL improving the ID accuracy, while RL on math settings has been demonstrated in existing literature to work remarkably well.
Questions for authors
none
3 comments
Response (2/2)
W2 — RL evaluation and ID performance
Our PPO implementation follows the protocol and budget of Chu et al.; the code accompanies the submission. Appendix C.3 checks the result across four budgets, from 0.5× to 2×:
PPO budget sweep (Table 6). OOD accuracy is mean ± one SD over three training seeds for each setting.
| SFT checkpoint | SFT | PPO 0.5× | PPO 1× | PPO 1.5× | PPO 2× |
|---|---|---|---|---|---|
| 300 | 16.24 ± 2.58 | 7.26 ± 1.20 | 8.12 ± 1.54 | 9.12 ± 2.58 | 9.54 ± 1.62 |
| 800 | 5.98 ± 1.08 | 11.11 ± 0.12 | 9.40 ± 1.67 | 10.26 ± 0.43 | 11.11 ± 1.28 |
| 1300 | 9.40 ± 2.21 | 6.41 ± 1.44 | 6.84 ± 1.51 | 6.84 ± 1.86 | 6.70 ± 1.93 |
The before/inside/after-window pattern persists across all four budgets, with +3.42–5.13 pp OOD gains at checkpoint 800.
The submitted experiments also show joint ID/OOD gains. Figures 13–14 show that LLaMA/GRPO improves both metrics on GeneralPoints, providing a comparison on the same model and task as Figure 2.
Qwen/GRPO recovery-window rows from Table 8:
| SFT checkpoint | ID: SFT → GRPO | OOD: SFT → GRPO |
|---|---|---|
| 400 | 32.48 → 33.29 | 15.38 → 16.67 |
| 500 | 32.05 → 32.85 | 15.38 → 18.38 |
| 600 | 35.04 → 35.90 | 16.24 → 18.80 |
| 700 | 32.91 → 34.19 | 16.67 → 18.80 |
| 800 | 43.59 → 47.74 | 14.10 → 19.67 |
| 900 | 47.86 → 49.06 | 11.97 → 14.96 |
| 1000 | 49.57 → 50.81 | 13.68 → 16.67 |
| 1100 | 57.26 → 63.25 | 13.68 → 16.24 |
All eight handoffs improve both metrics, averaging +2.04 pp ID and +2.89 pp OOD.
These sweep values are reported separately from the three-run endpoint comparison in Table 2.
Public math ID results (Table 7). Each cell shows SFT-end → RL.
| Benchmark / metric | Qwen-7B / GRPO | Qwen3B / DAPO |
|---|---|---|
| AIME24 avg@16 | 15.21 → 18.75 | 3.96 → 4.79 |
| AIME25 avg@16 | 19.38 → 20.83 | 5.00 → 5.42 |
| AMC avg@16 | 50.47 → 58.13 | 35.00 → 37.50 |
| MATH-500 avg@4 | 72.85 → 79.35 | 60.95 → 65.90 |
| Minerva avg@8 | 31.52 → 38.97 | 22.75 → 25.32 |
| Olympiad avg@4 | 37.41 → 43.44 | 26.25 → 27.44 |
| Average | 37.81 → 43.24 (+5.43 pp) | 25.65 → 27.73 (+2.08 pp) |
Additional paired ID/OOD evidence
We added exact Qwen3B SFT-to-DAPO40 comparisons on MATH-500 and IFEval:
| Raw SFT handoff | MATH-500, SFT → DAPO | ID gain [95% CI] | IFEval, SFT → DAPO | OOD gain [95% CI] |
|---|---|---|---|---|
| 2400 | 40.70 → 54.05 | +13.35 [10.00, 16.75] | 38.45 → 44.73 | +6.28 [2.96, 9.61] |
| 6400 | 46.70 → 57.15 | +10.45 [7.25, 13.70] | 33.64 → 37.71 | +4.07 [0.55, 7.58] |
Both handoffs improve both metrics, with all four pointwise 95% intervals above zero. Each SFT/RL pair uses identical evaluation settings. MATH-500 reports avg@4; IFEval reports prompt-level loose all-instructions accuracy on 541 prompts. Paired intervals resample problems or prompts for these fixed checkpoints. These additional evaluations are reported separately from Figure 3; settings are recorded in the accompanying evaluation note.
MATH-500 adds broader coverage alongside the 30-problem AIME-25 evaluation. Figure 3 measures OOD improvement relative to each initializer; Table 2 measures near-restoration to the early SFT peak in the controlled tasks. The additional pairs establish joint gains at two specified handoffs and are reported separately from the original trajectory. Together, these results support the paper's central finding: the SFT handoff helps determine the OOD recovery achieved by subsequent RL.
My concerns remain. The main issue is novelty, as the fact that SFT might hurt OOD performance and RL can restore it is already established. The paper adds a sweep over SFT checkpoints and reports where RL helps and where doesn't, but it does not explain why the recovery window exists or how to predict its boundaries. The additionally proposed protected SFT gives only minor gains. I will maintain my score.
Response (1/2)
Thank you for the comments on the contribution and RL evaluation. We address both through the checkpoint-wise results, appendix controls, and additional paired evaluations.
W1 — Contribution beyond the SFT/RL comparison
We map when RL recovers previously observed OOD performance and test which parameter directions support recovery. Building on Chu et al., we measure the paired effect of a fixed RL procedure at successive SFT checkpoints:
ΔOOD(k)=AccOOD(RL(θk))−AccOOD(θk).
The dense sweeps locate both the entry and exit of the recovery window (Figures 2–3). Mean OOD gains are +5.12 pp across seven LLaMA/PPO handoffs and +2.89 pp across eight Qwen/GRPO handoffs. The early SFT peak gives recovery a concrete reference.
Early stopping and late-SFT+RL (Table 2). Values are mean ± one SD over three training runs on GeneralPoints.
Late-SFT+RL nearly reaches the early SFT OOD peak while retaining 37.55 more ID points for LLaMA and 20.53 more for Qwen than the stopped early model.
The spectral interventions add a second contribution: they test which parameter directions support recovery.
Singular values and directions (Table 19). The natural-scale factorial over 168 Qwen matrices gives the following effects on the equal-six OOD macro:
Geometry-matched controls (Table 20). Pretrained-direction substitutions and random controls are matched in perturbation magnitude and principal-angle geometry. IFEval changes are measured from the same 55.268% SFT baseline:
Random-control values are mean ± one SD across three matched controls. The full-dose advantage is 10.47 pp [7.21, 13.74], supporting the importance of pretrained direction identity.
Protected SFT (Table 13). At the matched one-epoch checkpoint, protecting top-256 subspaces improves mean ID and OOD accuracy on both training tasks:
Values are mean ± one SD over three training seeds. The OOD macro gives equal weight to IFEval, MMLU-Pro, and SuperGPQA.