acceptodds
Under review as a conference paper at ICLR 2027

When Does Reasoning SFT Generalize Better Than GRPO?

Abstract

Supervised fine-tuning (SFT) is often associated with memorization, whereas reinforcement learning (RL) is credited with stronger generalization in large language models (LLMs). We show that sufficiently trained SFT can achieve stronger out-of-distribution (OOD) generalization than Group Relative Policy Optimization (GRPO). Tracking SFT with long chain-of-thought (Long-CoT) supervision and GRPO over multiple epochs under matched training-problem sequences and checkpoints, we find that SFT surpasses GRPO across consecutive checkpoints after sufficient training, whereas GRPO is generally stronger or comparable at early checkpoints. Response analysis shows that strict verification language, identified by markers that directly check intermediate or final results, remains common in SFT OOD generations. The SFT–GRPO difference in verification density is already visible near GRPO’s early high-performing checkpoint, before its later weakening in our matched run. Crucially, SFT’s OOD advantage is not explained by simply generating longer reasoning: successful SFT responses are typically shorter and use verification more densely than failed responses. With further training, SFT’s OOD performance declines from the stronger levels it reaches, although GRPO does not overtake SFT again within the observed training horizon. These results challenge the view that SFT memorizes while RL generalizes. SFT’s generalization gains can be missed by stopping too early and eroded by training too long.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-CoT Reasoning SFT, GRPO, Out-of-Distribution Generalization, Multi-Epoch Training, Training Dynamics, Reasoning Trajectories

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qbzBOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares long-CoT SFT and GRPO throughout multi-epoch training on the same 7,500 mathematical problems. SFT learns from Qwen3-32B-generated demonstrations, whereas GRPO learns from outcome rewards on its own rollouts. Across Qwen3-4B/8B and Olmo3-7B, sufficiently trained SFT achieves stronger OOD performance under the evaluated configurations, although further training eventually reduces its gains. The paper also analyzes verification language and presents a synthetic task illustrating shortcut dependence under GRPO.

Critical strengths

Following both methods across training provides useful evidence that early checkpoints can underestimate long-CoT SFT. Table 2 additionally compares a fixed SFT checkpoint against GRPO’s best observed checkpoints, addressing the concern that the reported advantage simply results from selecting a deteriorated late GRPO checkpoint.

The controlled shortcut task provides an interpretable demonstration of outcome-reward optimization reinforcing a spurious strategy. The warm-start experiments are particularly informative: sufficiently trained SFT initialization allows subsequent GRPO to preserve and improve OOD performance, highlighting the importance of initialization.

Critical weaknesses

Teacher-provided capabilities are not separated from the effects of the training method. Appendix A.1 states that SFT targets are generated by Qwen3-32B, whose post-training includes reinforcement learning (Yang et al., 2025, Section 4). SFT therefore receives complete demonstrations from a substantially larger, post-trained teacher, while GRPO must discover successful trajectories from the smaller base model using outcome feedback. Matching questions and optimizer updates does not control this difference in available supervision. The results demonstrate that teacher-distilled SFT can generalize, but do not establish how much of its advantage arises from training duration versus the teacher’s capabilities, potentially including behaviors developed through RL. This substantially limits the explanatory answer to “when” SFT generalizes better. Controls varying demonstration provenance and a main-task comparison from a shared SFT initialization would strengthen the contribution.

The robustness of the GRPO comparison remains insufficiently established. The main experiments use one GRPO learning rate and one training run per configuration, as acknowledged in Section 7. Appendix B.3 varies the SFT learning rate, which does not establish that the GRPO configuration is competitive. Reporting GRPO’s best observed checkpoint is helpful, but does not address sensitivity to its optimization settings. Moreover, Appendix D.3 shows that GRPO after sixteen SFT updates achieves nearly perfect ID and OOD accuracy in the synthetic task. This makes the absence of an analogous SFT-initialized GRPO comparison on the main reasoning benchmarks consequential. The evidence supports a useful observation under the selected recipe, but leaves its practical scope uncertain.

Questions for authors

Supervision-source control: Does the SFT advantage persist with human-written solutions, verified self-generated solutions, or demonstrations from a teacher before reasoning RL? A controlled comparison using the same teacher before and after RL, with problem coverage and solution quality characterized, would help distinguish teacher capability transfer from training-method effects.

Shared-initialization comparison: On the main reasoning benchmarks, what happens when the same SFT checkpoint is followed by either continued SFT or GRPO under comparable additional budgets? Given the successful sixteen-update warm-start result in Appendix D.3, this comparison could materially change the interpretation.

GRPO robustness: How was the main GRPO learning rate selected? Does the ordering persist across a reasonable learning-rate sweep and independent training seeds, with stopping points selected using held-out validation data?

Reviewer_vexdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper compares SFT and GRPO over matched training-problem sequences and optimizer checkpoints. It finds that GRPO is often stronger early, while SFT becomes stronger after additional training and eventually declines with further training. However, the current experiments establish this result for a limited training configuration, while leaving important questions about baseline optimization, inference budgets, and the proposed explanation unresolved. it provides neither a theoretical account of when this ordering should emerge nor sufficiently comprehensive experiments to establish its empirical scope.

Critical strengths
  1. The paper addresses an important methodological issue. And the advantage appears across mathematical and cross-domain benchmarks.
  2. The paired shortcut intervention provides clear evidence of suffix dependence, and the warm-up experiments reveal useful dependence on initialization.
Critical weaknesses
  1. Limited theoretical or conceptual explanation: the paper documents an early-to-late performance crossover, but does not develop a framework explaining the conditions that produce it.
  2. Given the limited theoretical development, the paper’s contribution depends heavily on the breadth and robustness of its experiments. The GRPO baseline is insufficiently explored to establish when the observed ordering should hold, such as one GRPO learning rate and one run per configuration, GRPO from a trained SFT checkpoint, and so on.
  3. The synthetic task demonstrates shortcut dependence, but contributes limited evidence about the broader SFT–GRPO comparison.
  4. The paper argues that SFT’s advantage is not explained by simply generating longer reasoning because successful SFT responses are shorter than failed SFT responses. This observation is informative, but it does not establish whether SFT benefits from generating substantially more tokens than GRPO.
Questions for authors
  1. Evaluation inconsistencies and missing details reduce confidence in the quantitative analysis. Table 2 and Table B.1 report Qwen3-8B SFT accuracy of 32.7% on AIME 2024 and 48.8% on GPQA-D, whereas Table C.1 reports 33.13% and 50.06%. Table B.1 reports step-708 GRPO accuracy of 15.3% on AIME 2024, whereas Table C.1 reports 14.33%.
Reviewer_H7LXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper conducts multi-epoch tracking experiments comparing SFT on long-cot data with GRPO under a consistent training problem sequence and checkpoint setup. The results show that, with sufficient training, SFT surpasses GRPO across multiple consecutive checkpoints, whereas GRPO performs better than or on par with SFT at early checkpoints.

Furthermore, an analysis of model-generated responses reveals that explicit verification statements occur frequently in SFT’s OOD generations. The difference in verification density between SFT and GRPO emerges as early as the early checkpoints, when GRPO achieves its peak performance.

Critical strengths

The paper challenges an established assumption by examining the learning behaviors of SFT and GRPO from a new perspective, making its investigation valuable.

The results provide strong support for the paper’s main conclusions.

The analysis of explicit verification statements offers a more fine-grained characterization of the underlying behaviors of SFT and GRPO.

Critical weaknesses
  1. The experiments match the number of optimization steps rather than computational budgets. Moreover, the experimental setup uses only a single random seed and a single learning rate.

  2. The paper uses base models, making it difficult to determine whether the transfer of SFT to OOD tasks stems from improved instruction-following capabilities induced by SFT or from other factors.

  3. The gains attributable to the SFT training data itself remain difficult to disentangle. The quality of the SFT data may independently contribute to the observed improvements. Since the SFT data are not generated on-policy, the current experiments do not fully establish a fair and direct comparison between SFT and GRPO.

Questions for authors
  1. can the authors report results across multiple random seeds to assess the robustness of their findings?

  2. can the authors provide experimental results using Instruct models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.