- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper compares long-CoT SFT and GRPO throughout multi-epoch training on the same 7,500 mathematical problems. SFT learns from Qwen3-32B-generated demonstrations, whereas GRPO learns from outcome rewards on its own rollouts. Across Qwen3-4B/8B and Olmo3-7B, sufficiently trained SFT achieves stronger OOD performance under the evaluated configurations, although further training eventually reduces its gains. The paper also analyzes verification language and presents a synthetic task illustrating shortcut dependence under GRPO.
Critical strengths
Following both methods across training provides useful evidence that early checkpoints can underestimate long-CoT SFT. Table 2 additionally compares a fixed SFT checkpoint against GRPO’s best observed checkpoints, addressing the concern that the reported advantage simply results from selecting a deteriorated late GRPO checkpoint.
The controlled shortcut task provides an interpretable demonstration of outcome-reward optimization reinforcing a spurious strategy. The warm-start experiments are particularly informative: sufficiently trained SFT initialization allows subsequent GRPO to preserve and improve OOD performance, highlighting the importance of initialization.
Critical weaknesses
Teacher-provided capabilities are not separated from the effects of the training method. Appendix A.1 states that SFT targets are generated by Qwen3-32B, whose post-training includes reinforcement learning (Yang et al., 2025, Section 4). SFT therefore receives complete demonstrations from a substantially larger, post-trained teacher, while GRPO must discover successful trajectories from the smaller base model using outcome feedback. Matching questions and optimizer updates does not control this difference in available supervision. The results demonstrate that teacher-distilled SFT can generalize, but do not establish how much of its advantage arises from training duration versus the teacher’s capabilities, potentially including behaviors developed through RL. This substantially limits the explanatory answer to “when” SFT generalizes better. Controls varying demonstration provenance and a main-task comparison from a shared SFT initialization would strengthen the contribution.
The robustness of the GRPO comparison remains insufficiently established. The main experiments use one GRPO learning rate and one training run per configuration, as acknowledged in Section 7. Appendix B.3 varies the SFT learning rate, which does not establish that the GRPO configuration is competitive. Reporting GRPO’s best observed checkpoint is helpful, but does not address sensitivity to its optimization settings. Moreover, Appendix D.3 shows that GRPO after sixteen SFT updates achieves nearly perfect ID and OOD accuracy in the synthetic task. This makes the absence of an analogous SFT-initialized GRPO comparison on the main reasoning benchmarks consequential. The evidence supports a useful observation under the selected recipe, but leaves its practical scope uncertain.
Questions for authors
Supervision-source control: Does the SFT advantage persist with human-written solutions, verified self-generated solutions, or demonstrations from a teacher before reasoning RL? A controlled comparison using the same teacher before and after RL, with problem coverage and solution quality characterized, would help distinguish teacher capability transfer from training-method effects.
Shared-initialization comparison: On the main reasoning benchmarks, what happens when the same SFT checkpoint is followed by either continued SFT or GRPO under comparable additional budgets? Given the successful sixteen-update warm-start result in Appendix D.3, this comparison could materially change the interpretation.
GRPO robustness: How was the main GRPO learning rate selected? Does the ordering persist across a reasonable learning-rate sweep and independent training seeds, with stopping points selected using held-out validation data?