- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes on-policy supervised fine-tuning (SFT) to address the problem of excessive thinking in large reasoning models. Specifically, the model first generates multiple rollouts for each question using its current policy, retains only responses that are both correct and shorter than a predefined threshold, and then performs SFT on the filtered positive samples. The authors argue that the KL term and group-wise reward normalization in GRPO are unnecessary for verifiable mathematical reasoning tasks. They therefore remove these components and show, from a gradient perspective, that the resulting simplified objective is closely related to SFT on on-policy data. Experiments conducted on multiple datasets demonstrate promising performance.
Critical strengths
-
The paper proposes an insightful on-policy SFT method.
-
The experimental design is relatively systematic, providing a comprehensive evaluation of the method in terms of accuracy, reasoning length, training stability, and computational cost.
-
The proposed method is relatively simple and clearly presented, providing a solid basis for reproducibility.
Critical weaknesses
- The paper does not sufficiently establish the necessity of on-policy training or its advantages over GRPO. The paper uses on-policy data as a central motivation, but it does not clearly explain why this problem requires an on-policy approach. Moreover, on-policy SFT still requires rollouts, correctness verification, length-threshold filtering, and continual resampling. This leaves a central question unanswered: if all of these steps are still required, why not directly apply GRPO with the same truncation-based correctness-and-length reward? The paper claims that on-policy SFT achieves lower training costs and more stable optimization by removing the reference-model KL term and group-wise normalization, but these advantages are not clearly disentangled from the effects of output length, the number of accepted samples, and the training dynamics. Only a comparison with GRPO under matched rollout, verifier, and computational budgets can determine whether the gains come from a superior optimization method or merely from a cheaper implementation of a similar positive-sample update. This is my primary concern and the main reason for my low score, because it directly affects the paper’s central motivation and contribution.
- The distinction between the proposed method and related self-training and distillation approaches is unclear. If the goal is simply to learn from correct and short trajectories, the proposed method appears conceptually close to rejection-sampling SFT, on-policy distillation, and best-of- SFT. Although the paper discusses on-policy distillation and includes CRST as a related baseline, it does not provide a strictly matched positive-only rejection-sampling SFT comparison.
- The training-efficiency comparison is not sufficiently normalized by computational cost. Figure 3 compares GPU memory and wall-clock time per training step, but different methods may generate responses of substantially different lengths at each step. FLOPs could be reported as an additional metric, but FLOPs alone may not fully capture the practical costs of autoregressive generation, verifier calls, memory access, and GPU utilization.
Questions for authors
Please refer to the Weaknesses section. My primary concern is Weakness 1. If the authors can adequately address this concern, I would be willing to raise my score to 3.
7 comments
Part 2/6
Under the same binary correctness and length criterion, on-policy SFT achieves higher accuracy and generates shorter responses than GRPO, DAPO, and Dr.GRPO on all three benchmarks (Table 1). On AIME24, on-policy SFT achieves 30.7% accuracy compared with 25.3% for each baseline, a gain of 5.4 percentage points. Its responses are 22.1%, 35.3%, and 28.4% shorter on average than those of GRPO, DAPO, and Dr.GRPO, respectively.
Table 1. On-policy SFT and GRPO variants with the same truncation-based correctness reward on the 1.5B model (main-paper Table 7).
| Method | GSM8K Acc / Tokens | MATH-500 Acc / Tokens | AIME24 Acc / Tokens |
|---|---|---|---|
| Truncation-based GRPO | 84.7 / 710 | 84.2 / 1,546 | 25.3 / 4,888 |
| Truncation-based DAPO | 84.5 / 840 | 84.3 / 1,824 | 25.3 / 5,888 |
| Truncation-based Dr.GRPO | 85.3 / 816 | 85.2 / 1,668 | 25.3 / 5,321 |
| On-policy SFT | 86.1 / 696 | 85.7 / 1,354 | 30.7 / 3,808 |
2b. On-policy SFT trains more efficiently than both the five length-penalized RL baselines and GRPO with the same truncation-based reward. For the same number of training steps, on-policy SFT uses 6.40 GPU-hours compared with 10.37 to 16.28 GPU-hours for the five length-penalized RL baselines (Table 1 in Part 4). At matched GPU-hours, it achieves higher overall accuracy and shorter mean responses than each baseline (Table 1 in Part 3). On the same batch of verified responses, a policy update takes 7.42 s for on-policy SFT and 14.83 s for GRPO under the same truncation-based reward (Table 2 in Part 4).
2c. Systematic analysis of length-penalized GRPO yields a simple yet effective training procedure for efficient reasoning.
-
We systematically analyze and simplify GRPO for efficient reasoning. Our goal is a simple yet effective training procedure: strong accuracy–length performance with fewer optimization components and lower training overhead. Starting from the directly verifiable correctness and length criteria, we examine which inherited GRPO components remain necessary, including KL regularization, group-wise reward normalization, and length-reward design. Section 3 derives on-policy SFT from this analysis, while the component ablations in main-paper Table 8 and the comparisons in Table 1 in Part 1 and Table 1 validate the resulting simplification. The contribution comprises the task-specific analysis, the minimalist training procedure it yields, and the experimental validation of its accuracy–length and training-efficiency benefits.
-
The resulting optimization is simpler and sampling is more flexible. Training uses the familiar supervised-likelihood form on correct responses satisfying the length criterion, without reference-policy KL, group-relative advantages, or shaped length rewards. Standard outcome-supervised GRPO constructs advantages by subtracting the within-question reward mean and dividing by the group standard deviation [1]; with one rollout per question, no informative group-relative advantage is available. Our objective permits learning from a single rollout per question whenever it passes the correctness-and-length filter.
References
[1] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.
Part 3/6
3. Controlled comparisons quantify the efficiency gains and training variability.
3a. On-policy SFT retains a strong accuracy–length trade-off with little variation across seeds at fixed training steps or fixed GPU-hours. We compare all five main-paper RL baselines and on-policy SFT from the same R1-Distill-Qwen-1.5B checkpoint on the same underlying DeepScaleR prompt stream, using eight rollouts per prompt, rollout temperature 1.0, top-p 0.95, a 3,500-token generation limit, the same correctness verifier, global prompt batch size 64, optimizer settings, and a single 94GB H100. Each baseline retains its method-specific reward and prompt conditioning as described in Appendix H. The first condition fixes 630 optimizer steps and 322,560 verifier calls for every method; the second matches each method's end-to-end GPU-hour budget to the on-policy SFT reference run for the same seed, averaging 6.40 GPU-hours. Evaluation follows Appendix F, with the evaluation seed fixed across three paired training seeds. In Table 1, all accuracy and response length entries report mean ± sample SD; Overall statistics are computed from each seed's five-benchmark macro-average. The on-policy SFT reference endpoint is shared by both budget conditions.
Table 1. Five-benchmark results at fixed training steps or fixed GPU-hours on the 1.5B model. All metrics report mean ± sample SD across three training seeds.
| Budget condition | Method | GSM8K Acc / Tokens | MATH-500 Acc / Tokens | AMC23 Acc / Tokens | AIME24 Acc / Tokens | AIME25 Acc / Tokens | Overall Acc (%) | Overall Tokens |
|---|---|---|---|---|---|---|---|---|
| Fixed training steps | ThinkPrune | 84.7 ± 0.37 / 710 ± 55.7 | 84.2 ± 0.44 / 1,546 ± 118.6 | 72.0 ± 0.74 / 2,672 ± 155.9 | 25.3 ± 1.50 / 4,888 ± 322.0 | 18.0 ± 1.31 / 4,213 ± 296.1 | 56.84 ± 0.78 | 2,805.8 ± 168.4 |
| Fixed training steps | O1-Pruner | 85.2 ± 0.27 / 916 ± 39.8 | 83.0 ± 0.37 / 2,383 ± 107.9 | 72.5 ± 0.91 / 3,942 ± 206.4 | 27.3 ± 1.48 / 8,164 ± 361.1 | 18.0 ± 1.26 / 8,528 ± 378.5 | 57.20 ± 0.81 | 4,786.6 ± 216.7 |
| Fixed training steps | L1 | 86.0 ± 0.32 / 1,607 ± 111.5 | 85.3 ± 0.46 / 1,922 ± 137.3 | 74.5 ± 0.80 / 2,170 ± 122.0 | 23.3 ± 1.61 / 2,881 ± 203.6 | 20.0 ± 1.30 / 2,762 ± 178.7 | 57.82 ± 0.82 | 2,268.4 ± 133.6 |
| Fixed training steps | ER-RL | 83.1 ± 0.45 / 721 ± 44.6 | 85.3 ± 0.42 / 2,466 ± 130.1 | 72.4 ± 0.81 / 4,913 ± 201.3 | 27.3 ± 1.80 / 11,584 ± 574.3 | 23.3 ± 1.20 / 9,373 ± 355.8 | 58.28 ± 0.86 | 5,811.4 ± 244.9 |
| Fixed training steps | LASER | 86.0 ± 0.35 / 1,178 ± 33.6 | 85.3 ± 0.26 / 2,714 ± 91.5 | 72.0 ± 0.78 / 4,262 ± 108.1 | 33.3 ± 1.50 / 7,821 ± 271.8 | 22.6 ± 1.10 / 8,569 ± 241.8 | 59.84 ± 0.75 | 4,908.8 ± 149.0 |
| Fixed training steps | On-policy SFT | 86.1 ± 0.26 / 696 ± 22.9 | 85.7 ± 0.36 / 1,354 ± 48.0 | 73.5 ± 0.89 / 1,977 ± 85.4 | 30.7 ± 1.30 / 3,808 ± 136.9 | 23.3 ± 1.20 / 3,096 ± 128.7 | 59.86 ± 0.67 | 2,186.2 ± 83.9 |
| Fixed GPU-hours | ThinkPrune | 84.5 ± 0.42 / 932 ± 82.3 | 83.8 ± 0.51 / 1,937 ± 181.1 | 71.6 ± 1.21 / 3,194 ± 282.7 | 24.6 ± 1.64 / 6,027 ± 448.8 | 18.2 ± 1.36 / 5,396 ± 352.6 | 56.54 ± 0.94 | 3,497.2 ± 247.6 |
| Fixed GPU-hours | O1-Pruner | 84.8 ± 0.34 / 1,183 ± 59.1 | 82.5 ± 0.44 / 2,947 ± 143.1 | 71.6 ± 1.16 / 4,816 ± 295.0 | 25.6 ± 2.23 / 10,473 ± 717.3 | 17.9 ± 1.51 / 10,928 ± 607.6 | 56.48 ± 0.97 | 6,069.4 ± 312.6 |
| Fixed GPU-hours | L1 | 85.8 ± 0.31 / 1,819 ± 131.4 | 84.9 ± 0.51 / 2,187 ± 192.7 | 73.6 ± 0.93 / 2,534 ± 178.4 | 22.9 ± 1.88 / 3,491 ± 265.5 | 19.2 ± 1.29 / 3,378 ± 212.1 | 57.28 ± 0.89 | 2,681.8 ± 179.3 |
| Fixed GPU-hours | ER-RL | 82.8 ± 0.46 / 933 ± 58.6 | 84.6 ± 0.41 / 3,017 ± 132.6 | 71.8 ± 1.35 / 6,028 ± 375.9 | 25.9 ± 1.75 / 13,947 ± 679.1 | 22.4 ± 1.71 / 11,586 ± 630.5 | 57.50 ± 1.02 | 7,102.2 ± 337.4 |
| Fixed GPU-hours | LASER | 85.7 ± 0.32 / 1,437 ± 62.8 | 84.9 ± 0.46 / 3,289 ± 149.4 | 71.8 ± 0.89 / 5,073 ± 207.4 | 31.4 ± 1.40 / 9,364 ± 336.1 | 22.1 ± 1.40 / 10,219 ± 413.6 | 59.18 ± 0.88 | 5,876.4 ± 231.7 |
| Fixed GPU-hours | On-policy SFT | 86.1 ± 0.26 / 696 ± 22.9 | 85.7 ± 0.36 / 1,354 ± 48.0 | 73.5 ± 0.89 / 1,977 ± 85.4 | 30.7 ± 1.30 / 3,808 ± 136.9 | 23.3 ± 1.20 / 3,096 ± 128.7 | 59.86 ± 0.67 | 2,186.2 ± 83.9 |
At matched GPU-hours, on-policy SFT attains 59.86% accuracy with 2,186.2 tokens, compared with accuracy ranging from 56.48% to 59.18% and response lengths from 2,681.8 to 7,102.2 tokens across the five RL baselines. At fixed training steps, it maintains accuracy comparable to LASER, 59.86% versus 59.84%, while using 55.5% shorter responses and 54.9% fewer GPU-hours. Across training seeds, the SD is 0.67 percentage points for overall accuracy and 3.8% of the mean for response length.
Part 1/6
W1 / Q1. On-policy sampling, advantages over RL methods, and the sources of efficiency and stability.
(1) Why is continual on-policy sampling needed?
(2) What performance and methodological advantages does on-policy SFT offer over RL baselines with length penalties and GRPO with the same truncation-based reward?
(3) How do the methods compare when training steps or GPU-hours are matched, and where do the training-time savings come from?
Response
Thank you for highlighting these central questions. Our core contribution is a systematic, experimentally validated simplification of length-penalized GRPO, from which on-policy SFT emerges naturally. We seek a simple yet effective training procedure for efficient reasoning that achieves a strong accuracy–length trade-off with low training overhead.
1. Continual on-policy sampling enables substantial compression while preserving accuracy.
From a theoretical perspective, on-policy training lets us sample responses from the current policy and train it on those that are concise and correct. This update makes the policy more likely to generate concise, correct responses in the next round. By sampling again from the updated policy, we may find even shorter solutions that are still correct and use them for the next update. From an empirical perspective, Figure 6 in the main paper compares resampling after every policy update with resampling only after one epoch of SFT on a fixed batch of generated responses. Both variants use the same 7B model, sample eight responses per prompt, and retain correct responses within 3,500 tokens. On MATH-500, resampling after every update substantially shortens responses while preserving accuracy. Refreshing the data once per epoch yields less compression and a drop in accuracy.
2. For efficient reasoning, on-policy SFT offers a better accuracy–length trade-off and simpler training than both length-penalized RL baselines and GRPO trained with the same truncation-based reward.
2a. On-policy SFT achieves a strong accuracy–length trade-off relative to five length-penalized RL baselines and outperforms GRPO under the same truncation-based reward. Table 1 reports the complete results for ThinkPrune, O1-Pruner, L1, ER-RL, LASER, and on-policy SFT. At 1.5B, our method achieves 59.9% overall accuracy with 2,186 tokens: compared with ThinkPrune, accuracy increases by 3.1 percentage points and response length decreases by 22.1%; compared with ER-RL, accuracy increases by 1.6 points while length decreases by 62.4%. Relative to LASER, accuracy is 59.9% versus 59.8%, with 55.5% shorter responses. At 7B, our 72.8% overall accuracy matches O1-Pruner and LASER while reducing response length by 65.7% and 33.0%, respectively. Relative to L1, accuracy increases from 67.3% to 72.8%, a 5.5-percentage-point gain, at comparable average lengths of 2,356 and 2,454 tokens. Figure 2 additionally compares accuracy at matched generation budgets on MATH-500 and AIME24 at both scales.
Table 1. Five-benchmark results for all main-paper RL baselines at both model scales (main-paper Table 1).