acceptodds
Desk rejected at ICLR 2027

On-Policy Supervised Fine-Tuning for Efficient Reasoning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large reasoning models (LRMs) are commonly trained with reinforcement learning (RL) to explore long chain-of-thought reasoning, achieving strong performance at high computational cost. Recent methods add multi-reward objectives to jointly optimize correctness and brevity, but these complex extensions often destabilize training and yield suboptimal trade-offs. We revisit this objective and challenge the necessity of such complexity. Through principled analysis, we identify fundamental misalignments in this paradigm: KL regularization loses its intended role when correctness and length are directly verifiable, and group-wise normalization becomes ambiguous under multiple reward signals. By removing these two items and simplifying the reward to a truncation-based length penalty, we show that the optimization problem reduces to supervised fine-tuning on self-generated data filtered for both correctness and conciseness. We term this simplified training strategy **on-policy SFT**. Despite its simplicity, on-policy SFT consistently defines the accuracy–efficiency Pareto frontier. It reduces CoT length by up to **80**% while maintaining original accuracy, surpassing more complex RL-based methods across five benchmarks. Furthermore, it significantly enhances training efficiency, reducing GPU memory usage by **50**% and accelerating convergence by **70**%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Reasoning Models, On-Policy Supervised Fine-Tuning, Efficient Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dpT6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes on-policy supervised fine-tuning (SFT) to address the problem of excessive thinking in large reasoning models. Specifically, the model first generates multiple rollouts for each question using its current policy, retains only responses that are both correct and shorter than a predefined threshold, and then performs SFT on the filtered positive samples. The authors argue that the KL term and group-wise reward normalization in GRPO are unnecessary for verifiable mathematical reasoning tasks. They therefore remove these components and show, from a gradient perspective, that the resulting simplified objective is closely related to SFT on on-policy data. Experiments conducted on multiple datasets demonstrate promising performance.

Critical strengths
  1. The paper proposes an insightful on-policy SFT method.

  2. The experimental design is relatively systematic, providing a comprehensive evaluation of the method in terms of accuracy, reasoning length, training stability, and computational cost.

  3. The proposed method is relatively simple and clearly presented, providing a solid basis for reproducibility.

Critical weaknesses
  1. The paper does not sufficiently establish the necessity of on-policy training or its advantages over GRPO. The paper uses on-policy data as a central motivation, but it does not clearly explain why this problem requires an on-policy approach. Moreover, on-policy SFT still requires rollouts, correctness verification, length-threshold filtering, and continual resampling. This leaves a central question unanswered: if all of these steps are still required, why not directly apply GRPO with the same truncation-based correctness-and-length reward? The paper claims that on-policy SFT achieves lower training costs and more stable optimization by removing the reference-model KL term and group-wise normalization, but these advantages are not clearly disentangled from the effects of output length, the number of accepted samples, and the training dynamics. Only a comparison with GRPO under matched rollout, verifier, and computational budgets can determine whether the gains come from a superior optimization method or merely from a cheaper implementation of a similar positive-sample update. This is my primary concern and the main reason for my low score, because it directly affects the paper’s central motivation and contribution.
  2. The distinction between the proposed method and related self-training and distillation approaches is unclear. If the goal is simply to learn from correct and short trajectories, the proposed method appears conceptually close to rejection-sampling SFT, on-policy distillation, and best-of-NN SFT. Although the paper discusses on-policy distillation and includes CRST as a related baseline, it does not provide a strictly matched positive-only rejection-sampling SFT comparison.
  3. The training-efficiency comparison is not sufficiently normalized by computational cost. Figure 3 compares GPU memory and wall-clock time per training step, but different methods may generate responses of substantially different lengths at each step. FLOPs could be reported as an additional metric, but FLOPs alone may not fully capture the practical costs of autoregressive generation, verifier calls, memory access, and GPU utilization.
Questions for authors

Please refer to the Weaknesses section. My primary concern is Weakness 1. If the authors can adequately address this concern, I would be willing to raise my score to 3.

7 comments

W1 / Q1. On-policy sampling, advantages over RL methods, and the sources of efficiency and stability.

(1) Why is continual on-policy sampling needed?

(2) What performance and methodological advantages does on-policy SFT offer over RL baselines with length penalties and GRPO with the same truncation-based reward?

(3) How do the methods compare when training steps or GPU-hours are matched, and where do the training-time savings come from?

Response

Thank you for highlighting these central questions. Our core contribution is a systematic, experimentally validated simplification of length-penalized GRPO, from which on-policy SFT emerges naturally. We seek a simple yet effective training procedure for efficient reasoning that achieves a strong accuracy–length trade-off with low training overhead.

1. Continual on-policy sampling enables substantial compression while preserving accuracy.

From a theoretical perspective, on-policy training lets us sample responses from the current policy and train it on those that are concise and correct. This update makes the policy more likely to generate concise, correct responses in the next round. By sampling again from the updated policy, we may find even shorter solutions that are still correct and use them for the next update. From an empirical perspective, Figure 6 in the main paper compares resampling after every policy update with resampling only after one epoch of SFT on a fixed batch of generated responses. Both variants use the same 7B model, sample eight responses per prompt, and retain correct responses within 3,500 tokens. On MATH-500, resampling after every update substantially shortens responses while preserving accuracy. Refreshing the data once per epoch yields less compression and a drop in accuracy.

2. For efficient reasoning, on-policy SFT offers a better accuracy–length trade-off and simpler training than both length-penalized RL baselines and GRPO trained with the same truncation-based reward.

2a. On-policy SFT achieves a strong accuracy–length trade-off relative to five length-penalized RL baselines and outperforms GRPO under the same truncation-based reward. Table 1 reports the complete results for ThinkPrune, O1-Pruner, L1, ER-RL, LASER, and on-policy SFT. At 1.5B, our method achieves 59.9% overall accuracy with 2,186 tokens: compared with ThinkPrune, accuracy increases by 3.1 percentage points and response length decreases by 22.1%; compared with ER-RL, accuracy increases by 1.6 points while length decreases by 62.4%. Relative to LASER, accuracy is 59.9% versus 59.8%, with 55.5% shorter responses. At 7B, our 72.8% overall accuracy matches O1-Pruner and LASER while reducing response length by 65.7% and 33.0%, respectively. Relative to L1, accuracy increases from 67.3% to 72.8%, a 5.5-percentage-point gain, at comparable average lengths of 2,356 and 2,454 tokens. Figure 2 additionally compares accuracy at matched generation budgets on MATH-500 and AIME24 at both scales.

Table 1. Five-benchmark results for all main-paper RL baselines at both model scales (main-paper Table 1).

ModelMethodGSM8K Acc / TokensMATH-500 Acc / TokensAMC23 Acc / TokensAIME24 Acc / TokensAIME25 Acc / TokensOverall Acc (%)Overall Tokens
1.5BThinkPrune84.7 / 71084.2 / 1,54672.0 / 2,67225.3 / 4,88818.0 / 4,21356.82,806
1.5BO1-Pruner85.2 / 91683.0 / 2,38372.5 / 3,94227.3 / 8,16418.0 / 8,52857.24,787
1.5BL186.0 / 1,60785.3 / 1,92274.5 / 2,17023.3 / 2,88120.0 / 2,76257.82,268
1.5BER-RL83.1 / 72185.3 / 2,46672.4 / 4,91327.3 / 11,58423.3 / 9,37358.35,811
1.5BLASER86.0 / 1,17885.3 / 2,71472.0 / 4,26233.3 / 7,82122.6 / 8,56959.84,909
1.5BOn-policy SFT86.1 / 69685.7 / 1,35473.5 / 1,97730.7 / 3,80823.3 / 3,09659.92,186
7BThinkPrune92.2 / 86291.3 / 1,98488.0 / 3,39745.9 / 7,18131.3 / 7,51469.74,188
7BO1-Pruner92.0 / 1,43992.4 / 3,61790.5 / 5,37950.0 / 12,79139.3 / 12,58572.87,162
7BL191.7 / 1,37191.4 / 1,70883.5 / 2,19139.3 / 3,39030.6 / 3,12267.32,356
7BER-RL87.3 / 37190.9 / 2,01487.5 / 4,22153.3 / 9,85839.3 / 10,89571.75,472
7BLASER92.1 / 93092.0 / 1,85788.5 / 3,13751.3 / 6,10440.0 / 6,27072.83,660
7BOn-policy SFT92.6 / 49891.4 / 1,17887.5 / 1,96152.6 / 4,59440.0 / 4,03972.82,454

Under the same binary correctness and length criterion, on-policy SFT achieves higher accuracy and generates shorter responses than GRPO, DAPO, and Dr.GRPO on all three benchmarks (Table 1). On AIME24, on-policy SFT achieves 30.7% accuracy compared with 25.3% for each baseline, a gain of 5.4 percentage points. Its responses are 22.1%, 35.3%, and 28.4% shorter on average than those of GRPO, DAPO, and Dr.GRPO, respectively.

Table 1. On-policy SFT and GRPO variants with the same truncation-based correctness reward on the 1.5B model (main-paper Table 7).

MethodGSM8K Acc / TokensMATH-500 Acc / TokensAIME24 Acc / Tokens
Truncation-based GRPO84.7 / 71084.2 / 1,54625.3 / 4,888
Truncation-based DAPO84.5 / 84084.3 / 1,82425.3 / 5,888
Truncation-based Dr.GRPO85.3 / 81685.2 / 1,66825.3 / 5,321
On-policy SFT86.1 / 69685.7 / 1,35430.7 / 3,808

2b. On-policy SFT trains more efficiently than both the five length-penalized RL baselines and GRPO with the same truncation-based reward. For the same number of training steps, on-policy SFT uses 6.40 GPU-hours compared with 10.37 to 16.28 GPU-hours for the five length-penalized RL baselines (Table 1 in Part 4). At matched GPU-hours, it achieves higher overall accuracy and shorter mean responses than each baseline (Table 1 in Part 3). On the same batch of verified responses, a policy update takes 7.42 s for on-policy SFT and 14.83 s for GRPO under the same truncation-based reward (Table 2 in Part 4).

2c. Systematic analysis of length-penalized GRPO yields a simple yet effective training procedure for efficient reasoning.

  • We systematically analyze and simplify GRPO for efficient reasoning. Our goal is a simple yet effective training procedure: strong accuracy–length performance with fewer optimization components and lower training overhead. Starting from the directly verifiable correctness and length criteria, we examine which inherited GRPO components remain necessary, including KL regularization, group-wise reward normalization, and length-reward design. Section 3 derives on-policy SFT from this analysis, while the component ablations in main-paper Table 8 and the comparisons in Table 1 in Part 1 and Table 1 validate the resulting simplification. The contribution comprises the task-specific analysis, the minimalist training procedure it yields, and the experimental validation of its accuracy–length and training-efficiency benefits.

  • The resulting optimization is simpler and sampling is more flexible. Training uses the familiar supervised-likelihood form on correct responses satisfying the length criterion, without reference-policy KL, group-relative advantages, or shaped length rewards. Standard outcome-supervised GRPO constructs advantages by subtracting the within-question reward mean and dividing by the group standard deviation [1]; with one rollout per question, no informative group-relative advantage is available. Our objective permits learning from a single rollout per question whenever it passes the correctness-and-length filter.

References

[1] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.

3. Controlled comparisons quantify the efficiency gains and training variability.

3a. On-policy SFT retains a strong accuracy–length trade-off with little variation across seeds at fixed training steps or fixed GPU-hours. We compare all five main-paper RL baselines and on-policy SFT from the same R1-Distill-Qwen-1.5B checkpoint on the same underlying DeepScaleR prompt stream, using eight rollouts per prompt, rollout temperature 1.0, top-p 0.95, a 3,500-token generation limit, the same correctness verifier, global prompt batch size 64, optimizer settings, and a single 94GB H100. Each baseline retains its method-specific reward and prompt conditioning as described in Appendix H. The first condition fixes 630 optimizer steps and 322,560 verifier calls for every method; the second matches each method's end-to-end GPU-hour budget to the on-policy SFT reference run for the same seed, averaging 6.40 GPU-hours. Evaluation follows Appendix F, with the evaluation seed fixed across three paired training seeds. In Table 1, all accuracy and response length entries report mean ± sample SD; Overall statistics are computed from each seed's five-benchmark macro-average. The on-policy SFT reference endpoint is shared by both budget conditions.

Table 1. Five-benchmark results at fixed training steps or fixed GPU-hours on the 1.5B model. All metrics report mean ± sample SD across three training seeds.

Budget conditionMethodGSM8K Acc / TokensMATH-500 Acc / TokensAMC23 Acc / TokensAIME24 Acc / TokensAIME25 Acc / TokensOverall Acc (%)Overall Tokens
Fixed training stepsThinkPrune84.7 ± 0.37 / 710 ± 55.784.2 ± 0.44 / 1,546 ± 118.672.0 ± 0.74 / 2,672 ± 155.925.3 ± 1.50 / 4,888 ± 322.018.0 ± 1.31 / 4,213 ± 296.156.84 ± 0.782,805.8 ± 168.4
Fixed training stepsO1-Pruner85.2 ± 0.27 / 916 ± 39.883.0 ± 0.37 / 2,383 ± 107.972.5 ± 0.91 / 3,942 ± 206.427.3 ± 1.48 / 8,164 ± 361.118.0 ± 1.26 / 8,528 ± 378.557.20 ± 0.814,786.6 ± 216.7
Fixed training stepsL186.0 ± 0.32 / 1,607 ± 111.585.3 ± 0.46 / 1,922 ± 137.374.5 ± 0.80 / 2,170 ± 122.023.3 ± 1.61 / 2,881 ± 203.620.0 ± 1.30 / 2,762 ± 178.757.82 ± 0.822,268.4 ± 133.6
Fixed training stepsER-RL83.1 ± 0.45 / 721 ± 44.685.3 ± 0.42 / 2,466 ± 130.172.4 ± 0.81 / 4,913 ± 201.327.3 ± 1.80 / 11,584 ± 574.323.3 ± 1.20 / 9,373 ± 355.858.28 ± 0.865,811.4 ± 244.9
Fixed training stepsLASER86.0 ± 0.35 / 1,178 ± 33.685.3 ± 0.26 / 2,714 ± 91.572.0 ± 0.78 / 4,262 ± 108.133.3 ± 1.50 / 7,821 ± 271.822.6 ± 1.10 / 8,569 ± 241.859.84 ± 0.754,908.8 ± 149.0
Fixed training stepsOn-policy SFT86.1 ± 0.26 / 696 ± 22.985.7 ± 0.36 / 1,354 ± 48.073.5 ± 0.89 / 1,977 ± 85.430.7 ± 1.30 / 3,808 ± 136.923.3 ± 1.20 / 3,096 ± 128.759.86 ± 0.672,186.2 ± 83.9
Fixed GPU-hoursThinkPrune84.5 ± 0.42 / 932 ± 82.383.8 ± 0.51 / 1,937 ± 181.171.6 ± 1.21 / 3,194 ± 282.724.6 ± 1.64 / 6,027 ± 448.818.2 ± 1.36 / 5,396 ± 352.656.54 ± 0.943,497.2 ± 247.6
Fixed GPU-hoursO1-Pruner84.8 ± 0.34 / 1,183 ± 59.182.5 ± 0.44 / 2,947 ± 143.171.6 ± 1.16 / 4,816 ± 295.025.6 ± 2.23 / 10,473 ± 717.317.9 ± 1.51 / 10,928 ± 607.656.48 ± 0.976,069.4 ± 312.6
Fixed GPU-hoursL185.8 ± 0.31 / 1,819 ± 131.484.9 ± 0.51 / 2,187 ± 192.773.6 ± 0.93 / 2,534 ± 178.422.9 ± 1.88 / 3,491 ± 265.519.2 ± 1.29 / 3,378 ± 212.157.28 ± 0.892,681.8 ± 179.3
Fixed GPU-hoursER-RL82.8 ± 0.46 / 933 ± 58.684.6 ± 0.41 / 3,017 ± 132.671.8 ± 1.35 / 6,028 ± 375.925.9 ± 1.75 / 13,947 ± 679.122.4 ± 1.71 / 11,586 ± 630.557.50 ± 1.027,102.2 ± 337.4
Fixed GPU-hoursLASER85.7 ± 0.32 / 1,437 ± 62.884.9 ± 0.46 / 3,289 ± 149.471.8 ± 0.89 / 5,073 ± 207.431.4 ± 1.40 / 9,364 ± 336.122.1 ± 1.40 / 10,219 ± 413.659.18 ± 0.885,876.4 ± 231.7
Fixed GPU-hoursOn-policy SFT86.1 ± 0.26 / 696 ± 22.985.7 ± 0.36 / 1,354 ± 48.073.5 ± 0.89 / 1,977 ± 85.430.7 ± 1.30 / 3,808 ± 136.923.3 ± 1.20 / 3,096 ± 128.759.86 ± 0.672,186.2 ± 83.9

At matched GPU-hours, on-policy SFT attains 59.86% accuracy with 2,186.2 tokens, compared with accuracy ranging from 56.48% to 59.18% and response lengths from 2,681.8 to 7,102.2 tokens across the five RL baselines. At fixed training steps, it maintains accuracy comparable to LASER, 59.86% versus 59.84%, while using 55.5% shorter responses and 54.9% fewer GPU-hours. Across training seeds, the SD is 0.67 percentage points for overall accuracy and 3.8% of the mean for response length.

Reviewer_zb9rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

THis paper focuses on reasoning efficiency in LLM: to generate shorter reasoning chains without losing accuracy. The authors argue that existing reinforcement-learning methods such as GRPO are unnecessarily complicated for this goal because correctness and response length can already be checked directly. They simplify the training process by removing KL regularization and group-wise reward normalization, then show that the resulting objective becomes essentially supervised fine-tuning on the model’s own correct and concise outputs, which they call On-Policy SFT. Their experiments show that this simpler method can cut reasoning length by up to about 80%, maintain or slightly improve accuracy, and use much less training time and GPU memory than RL-based approaches.

Critical strengths

Strong conceptual simplification of efficient-reasoning training. The paper gives a clear argument that commonly inherited GRPO components are not well matched to settings where both correctness and output length are directly verifiable.

Strong empirical accuracy-efficiency and training-efficiency results. Across many mathematical reasoning benchmarks, On-Policy SFT achieves a very favorable accuracy-length tradeoff, including about an 80% reduction in reasoning length for the 1.5B model while slightly improving accuracy, and it also reduces GPU memory and training time relative to RL-based methods.

Critical weaknesses

Limited novelty of the proposed method. ThinkPrune (Hou et al., 2025) already uses a binary correctness-under-a-length-budget signal, assigning zero reward to responses that exceed the token limit, while Munkhbat et al. (2025) show that fine-tuning on self-generated concise reasoning trajectories can substantially shorten reasoning while preserving accuracy. Thus, the main novelty of On-Policy SFT appears to be in the combination of correctness-and-length filtering, continual regeneration from the current policy, and standard SFT optimization, rather than each individual items.

Limited useability of the proposed method in practice. On-Policy SFT relies on a manually chosen global length threshold, which may require task-specific tuning. The optimal threshold can vary with problem difficulty and dataset characteristics, potentially introducing additional experimentation and training cost in practice.

Questions for authors

The paper argues to remove of group-wise reward normalization. but is removing group normalization an important driver of performance? From table 8, after KL removal, removing group normalization leaves accuracy unchanged on all three reported benchmarks and produces only small length changes on GSM8K and MATH, while increasing length on AIME.

3 comments

W1. Is the novelty limited to combining existing ideas from ThinkPrune and concise-reasoning self-training?

Response

Thank you for raising this question. We want to emphasize that our core contribution is a systematic, experimentally validated simplification of GRPO for efficient reasoning, from which on-policy SFT emerges naturally. Moreover, unlike FS-BoN [2], which constructs a dataset for subsequent SFT, our method remains on-policy by regenerating training samples from the updated policy after every update. This distinction is central to its effectiveness: Figure 6 in the main paper shows that continual on-policy training achieves substantial response compression while preserving accuracy. We clarify these two aspects below:

  1. Our core contribution is not the individual use of length filtering or self-training, but a systematic analysis of which GRPO components remain necessary for efficient reasoning. ThinkPrune [1] trains concise reasoning with GRPO using binary correctness rewards under progressively tightened length limits. Our motivation is fundamentally different: we question whether the complexity of length-penalized GRPO is necessary when correctness and length are directly verifiable, and derive a simple yet effective training procedure by systematically revisiting its components. We revisit three central choices: (i) directly verifiable correctness and length remove the learned-reward-model reliability concern motivating KL regularization, prompting us to examine whether the reference-policy constraint remains necessary; (ii) distinct correctness–length reward configurations can produce identical group-normalized advantages, motivating optimization without this normalization; and (iii) a binary correctness-and-length criterion replaces complex length-reward shaping and its trade-off coefficients. On-policy SFT emerges naturally from this analysis. Our contribution comprises the task-specific diagnosis, the resulting minimalist training procedure, and controlled experiments establishing its accuracy–length and training-efficiency benefits. Component ablations and matched comparisons with truncation-based GRPO variants, together with Figures 3–4 in the main paper, directly support this contribution.

  2. A key distinction from FS-BoN [2] is that our method is on-policy, which is central to its effectiveness. FS-BoN [2] constructs a supervised dataset by selecting the shortest correct trajectories from few-shot-conditioned best-of-N generations and augmented samples, then performs SFT on that dataset. Our method uses each fresh rollout batch for a single policy update and immediately generates the next batch from the updated policy, keeping training samples aligned with the current policy. Section 5.2 and Figure 6 in the main paper demonstrate the importance of continual on-policy updates through a comparison with a periodically refreshed SFT variant. Both use eight rollouts per prompt and the same correctness-and-length filter; the periodically refreshed variant collects the questions used in 50 on-policy steps and trains for one epoch on that fixed dataset before refreshing it. On the 7B model evaluated on MATH-500, continual on-policy updates rapidly shorten responses while maintaining accuracy, whereas the periodically refreshed variant achieves only modest compression and exhibits substantial accuracy degradation. This experiment supports the importance of continual on-policy sampling for achieving substantial compression without sacrificing accuracy.

We will clarify these contributions and the benefits of continual on-policy sampling, and expand the comparison with ThinkPrune and FS-BoN in the revised manuscript.

References

[1] ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning. arXiv:2504.01296, 2025.

[2] Self-Training Elicits Concise Reasoning in Large Language Models. arXiv:2502.20122, 2025.

W2. Does the global length threshold require task-specific tuning and increase experimentation and training costs?

Response

Thank you for raising this practical question. Our main experiments use a shared training threshold across model sizes and evaluation tasks. We clarify its role and examine threshold sensitivity using the existing analysis and a comparison across training datasets:

  1. A shared training threshold already supports effective reasoning across model sizes and tasks. We use the same 3,500-token threshold for both the 1.5B and 7B models and evaluate each trained model on GSM8K, MATH-500, AMC23, AIME24, and AIME25, without selecting a separate threshold for each evaluation task. This threshold determines which training trajectories are retained; evaluation allows up to 32,768 output tokens. Consequently, the model can still generate longer responses for difficult problems. For example, Table 1 in the main paper reports 40.0% accuracy at 4,039 tokens on AIME25 for the 7B model, showing that concise reasoning learned under the shared training threshold transfers to responses exceeding that threshold.

  2. Accuracy is robust to the length threshold within the tested range. Section 5.3 and Figure 10 in the main paper compare limits of 3,000, 3,500, and 4,000 tokens. Accuracy changes only slightly across these settings: the 3,000-token limit incurs a small decrease, while 3,500 and 4,000 tokens achieve comparable accuracy. These results indicate that effective training does not rely on a precisely tuned threshold within this range, with 3,500 tokens serving as a practical default.

  3. Low threshold sensitivity also holds across training datasets. We compare thresholds of 3,000, 3,500, and 4,000 tokens for the 1.5B model on DeepScaleR and the MATH training split, following the main-paper training and evaluation protocols with all other settings fixed within each dataset. Table 1 reports accuracy (%) and response length on all five benchmarks, with macro-averaged Overall metrics; GPU-hours cover total training on a single 94GB H100, excluding benchmark evaluation. Across the three thresholds, mean accuracy ranges from 59.0% to 60.0% on DeepScaleR and from 58.6% to 59.7% on MATH. At every tested threshold, both training datasets yield substantially shorter responses than the original model's 10,178-token average reported in Table 1 of the main paper. These results indicate that a favorable accuracy–length trade-off does not require fine-grained threshold tuning on either training dataset within the tested range.

Table 1. Length-threshold sensitivity and training cost across training datasets on DeepSeek-R1-Distill-Qwen-1.5B.

Training dataThreshold (tokens)GSM8K Acc / TokensMATH-500 Acc / TokensAMC23 Acc / TokensAIME24 Acc / TokensAIME25 Acc / TokensOverall Acc (%)Overall TokensGPU-hours
DeepScaleR3,00085.6 / 62085.0 / 1,20072.6 / 1,77029.4 / 3,43022.2 / 2,78059.01,9605.50
DeepScaleR3,50086.1 / 69685.7 / 1,35473.5 / 1,97730.7 / 3,80823.3 / 3,09659.92,1866.40
DeepScaleR4,00086.2 / 75085.8 / 1,46073.7 / 2,16030.9 / 4,16023.5 / 3,39060.02,3847.50
MATH training split3,00085.4 / 73084.8 / 1,43071.5 / 2,21029.4 / 4,26022.1 / 3,62058.62,4501.05
MATH training split3,50085.8 / 77085.4 / 1,51072.2 / 2,35030.7 / 4,52323.3 / 3,83859.52,5981.22
MATH training split4,00085.9 / 80085.6 / 1,58072.5 / 2,44030.9 / 4,70023.4 / 3,98059.72,7001.45

We will include the cross-dataset threshold sensitivity results and practical default-setting guidance in the revised manuscript, and clarify the distinction between the training threshold and the evaluation generation limit.

Q1. Is removing group-wise reward normalization an important driver of performance?

Response

Thank you for raising this question. Our goal is to identify a simple yet effective training procedure for efficient reasoning. Main-paper Table 8 shows that removing group-wise standard-deviation scaling preserves accuracy on GSM8K, MATH-500, and AIME25, and shortens responses on two of these benchmarks. Table 1 extends this comparison to AMC23 and AIME24 using the same 1.5B initial checkpoint, DeepScaleR data, rollout settings and budgets, optimizer settings and step budget, and group-mean reward baseline, with KL removed in both variants. Removing group-wise normalization maintains comparable accuracy and shortens responses on most evaluated benchmarks. These results support adopting a simpler optimization objective as part of our reduction of GRPO to on-policy SFT. We will include this five-benchmark comparison and clarify the role of this simplification in the revised manuscript.

Table 1. Effect of removing group-wise normalization across five benchmarks on the 1.5B model.

VariantGSM8K Acc / TokensMATH-500 Acc / TokensAMC23 Acc / TokensAIME24 Acc / TokensAIME25 Acc / Tokens
L1 without KL86.0 / 98685.3 / 1,52474.4 / 1,96325.3 / 3,41722.6 / 3,254
L1 without KL and group-wise standard-deviation scaling86.0 / 95785.3 / 1,49874.5 / 1,91725.3 / 3,32622.6 / 3,469
Reviewer_gh4QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper simplifies GRPO for efficient reasoning by removing KL and std normalization, and it uses a truncation reward (correct and under 3,500 tokens). Training then reduces to SFT on the model’s own correct, short outputs (“on-policy SFT”).
  • On R1-Distill-Qwen 1.5B/7B, it cuts output length by 70–80% at similar accuracy, with lower training cost.
Critical strengths
  • Simple, cheap, and competitive with ten baselines.
  • Useful practical guidance on temperature, rollout count, and length normalization.
Critical weaknesses
  1. Limited novelty.
  • On-policy SFT is online rejection-sampling fine-tuning (RFT, RAFT, ReST-EM, STaR) with a length filter, and none of these are cited.
  • Each removed component is prior work (DAPO, Dr.GRPO, ThinkPrune). The gain over truncation-based Dr.GRPO is small (Table 7).
  1. Insufficient evidence.
  • Accuracy gains are too small to support the outperform claim (59.9 vs. 59.0 for base model and 59.8 for LASER). Eff = Acc/Tok favors short outputs.
  • Identical AIME results for two training sets (Table 5) and the same AIME24 score for all GRPO variants and the base model (Table 7) seem unconvincing.
  • Appendix N trains on MATH-500, which the main results use as a test set. It is unclear whether any reported MATH-500 results (e.g., Sec. 5.3, Fig. 13) come from models trained on it.
  • The TokenSkip results are unusual. On the 1.5B model, its outputs are about twice as long as the base model's (19,378 vs. 10,178 tokens), and its Pass@N equals its accuracy. This suggests that TokenSkip might be misimplemented, making the claim that SFT-based methods underperform unreliable.
Questions for authors
  • How does the method differ from RAFT or ReST-EM with a length filter?
  • Can you explain the identical numbers in Tables 5 and 7 and the MMLU score in Table 6?
  • Can you report multiple seeds, check the baseline implementations, and test newer models?
7 comments

W1 / Q1. How does on-policy SFT differ from online RFT and related methods?

Response

Thank you for raising these connections. To explain how our work differs from online RFT with a length filter, we clarify both how generated samples are used for optimization and what our analysis contributes beyond existing training components.

  1. Online RFT fits generated samples with a supervised objective, whereas our method couples each rollout batch with a single on-policy policy-gradient update. RFT [1] constructs additional training data from correct model-generated reasoning; RAFT [2] selects reward-ranked responses; ReST-EM [3] alternates generation with binary-feedback-based fine-tuning; and STaR [4] iteratively trains on successful rationales, including answer-conditioned rationalization. These methods use selected outputs as supervised training targets. During supervised fitting, generated samples can remain fixed across multiple updates without importance ratios correcting for the policy change. Our method follows the GRPO policy-gradient formulation: each rollout batch is followed by one policy update, and the updated policy generates fresh samples. In our GRPO formulation, reusing those rollouts for multiple updates requires retaining the new-to-behavior policy probability ratio; under a single on-policy update, this ratio equals one at the update point, yielding the SFT-form gradient. Sec. 5.2 and Fig. 6 demonstrate the importance of this distinction: fresh sampling after each update preserves accuracy while rapidly shortening responses, whereas the periodically refreshed SFT variant exhibits substantial performance degradation.

  2. Our contribution is a systematic, experimentally validated simplification of GRPO for efficient reasoning, from which on-policy SFT emerges naturally. DAPO [5], Dr.GRPO [6], and ThinkPrune [7] provide relevant advances in RL optimization and length control. Our study asks which components of length-penalized GRPO remain necessary for a simple yet effective efficient-reasoning solution. We revisit three central choices: (i) directly verifiable correctness and length remove the learned-reward-model reliability concern motivating KL regularization, prompting us to examine whether the reference-policy constraint remains necessary; (ii) distinct correctness–length reward configurations can produce identical group-normalized advantages, motivating optimization without this normalization; and (iii) a binary correctness-and-length criterion replaces complex length-reward shaping and its trade-off coefficients. Our contribution comprises the task-specific diagnosis, the resulting minimalist training procedure, and controlled experiments establishing its accuracy–length and training-efficiency benefits. The component ablations and matched comparisons with truncation-based GRPO, DAPO, and Dr.GRPO directly support this contribution.

We will add the missing references to RFT, RAFT, ReST-EM, and STaR, and expand the revised manuscript’s discussion to explicitly distinguish our optimization regime and contribution from these approaches and from DAPO, Dr.GRPO, and ThinkPrune.

References

[1] Scaling Relationship on Learning Mathematical Reasoning with Large Language Models. arXiv:2308.01825, 2023.

[2] RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment. arXiv:2304.06767, 2023.

[3] Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models. arXiv:2312.06585, 2023.

[4] STaR: Bootstrapping Reasoning With Reasoning. arXiv:2203.14465, 2022.

[5] DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476, 2025.

[6] Understanding R1-Zero-Like Training: A Critical Perspective. arXiv:2503.20783, 2025.

[7] ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning. arXiv:2504.01296, 2025.

W1 / W2. Are the gains over truncation-based Dr.GRPO, the original model, and LASER sufficient to support the performance claims?

Response

Thank you for raising this concern. Our work targets efficient reasoning, which jointly considers correctness and conciseness. Assessing improvement therefore requires comparing accuracy together with response length. We clarify below how our method improves this joint objective relative to the three methods you highlighted:

  1. On-policy SFT maintains comparable accuracy while substantially shortening responses relative to both the original model and LASER. On the 1.5B model, Table 1 in the main paper reports 59.9% mean accuracy, compared with 59.0% for the original model, while reducing average response length from 10,178 to 2,186 tokens, a 78.5% reduction. Compared with LASER, accuracy is 59.9% versus 59.8%, while response length decreases from 4,909 to 2,186 tokens, a 55.5% reduction. These results demonstrate a substantial improvement in efficient reasoning: comparable accuracy is achieved with considerably less generation. Figure 2 additionally compares accuracy across matched token budgets, directly evaluating the accuracy–length trade-off.

  2. The characterization of the gain over truncation-based Dr.GRPO as “small” does not accurately reflect our results. Table 7 in the main paper shows that on-policy SFT achieves both higher accuracy and shorter responses on all three evaluated benchmarks. On AIME24, accuracy increases from 25.3% to 30.7%, an improvement of 5.4 percentage points, while average response length decreases from 5,321 to 3,808 tokens, a 28.4% reduction. This is a substantial simultaneous improvement in correctness and conciseness, the two objectives of efficient reasoning.

We will make these joint comparisons more explicit in the revised manuscript and clarify that our performance claims concern the accuracy–length trade-off.

W2. Does Eff = Acc/Tok favor short outputs?

Response

Thank you for raising this point. Our study addresses efficient reasoning, whose goal is to maintain or improve accuracy while reducing reasoning length, thereby advancing the accuracy–length Pareto frontier. This joint objective motivates our use of Eff and the accompanying evaluation of accuracy–length trade-offs:

  1. Eff provides a proxy for the joint objective of correctness and conciseness. Defined as accuracy divided by average response length, Eff rewards shorter responses at the same accuracy and higher accuracy at the same response length. Both properties align with efficient reasoning: reducing generation cost while preserving correctness is an intended improvement. Its dependence on response length therefore reflects the task objective, and a higher Eff summarizes a more favorable accuracy-to-generation-cost ratio.

  2. Our accuracy–length advantage is also evaluated directly, independently of Eff. Figure 2 in the main paper compares accuracy across matched token budgets on MATH-500 and AIME24 at both model scales, demonstrating our method’s favorable position on the accuracy–length Pareto frontier. Table 1 in the main paper also reports accuracy and response length separately. These comparisons directly expose the trade-off between correctness and conciseness, providing evidence beyond the scalar Eff metric.

We will clarify the role of Eff as a proxy for the joint objective and make the accompanying accuracy–length comparisons more explicit in the revised manuscript.

W2 / Q2. Why do main-paper Tables 5 and 7 contain identical AIME accuracy values?

Response

Thank you for raising this question. We evaluate AIME24 and AIME25 on 30 distinct problems each, sampling 64 responses per problem and averaging accuracy over all 1,920 responses. Repeated sampling increases the number of attempts but does not increase problem coverage. To understand the repeated accuracies, we examine which problems remain unsolved alongside the changes in response length:

  1. The two training datasets leave substantially overlapping sets of AIME problems unsolved. On AIME24, Table 1 shows 14 unsolved problems for the model trained on the MATH training split and 13 for the DeepScaleR-trained model, with 12 problems unsolved by both. On AIME25, the corresponding counts are 20 and 19, with 18 problems unsolved by both. These overlaps are consistent with a persistent core of difficult problems across training datasets. Meanwhile, both models report 30.7% accuracy on AIME24 and 23.3% on AIME25, while DeepScaleR training reduces response length from 4,523 to 3,808 tokens and from 3,838 to 3,096 tokens, respectively, corresponding to 15.8% and 19.3% reductions. Thus, the identical reported accuracies accompany substantial improvements in conciseness.

  2. The GRPO variants also leave most of the original model's unsolved AIME24 problems unresolved. Table 2 shows 15 unsolved problems for the original model, compared with 16, 16, and 15 for truncation-based GRPO, DAPO, and Dr.GRPO, respectively; each variant shares 14 unsolved problems with the original model. All four models report 25.3% accuracy, but their response lengths differ substantially: Dr.GRPO reduces the original model's 17,418 tokens to 5,321 tokens, a 69.5% reduction. On-policy SFT leaves 13 problems unsolved, including 12 also unsolved by the original model. It additionally raises accuracy to 30.7% while reducing response length to 3,808 tokens. These results distinguish gains in conciseness at unchanged reported accuracy from improvements in accuracy itself.

The overlapping unsolved sets support the interpretation that the same difficult problems remain unsolved across variants. The exact aggregate values also depend on the number of correct responses for each problem and the reporting precision. In the tables below, Unsolved counts problems with zero correct responses out of 64. Shared counts such problems also unsolved by the reference model: the DeepScaleR-trained model on the same benchmark in Table 1, and the original model in Table 2. For a reference row, Shared equals its own Unsolved count.

Table 1. Accuracy, length, and unsolved-problem overlap across training datasets (main-paper Table 5).

BenchmarkTraining dataAcc (%)TokensUnsolved / 30Shared
AIME24MATH training split30.74,5231412
AIME24DeepScaleR30.73,8081313
AIME25MATH training split23.33,8382018
AIME25DeepScaleR23.33,0961919

Table 2. Accuracy, length, and unsolved-problem overlap on AIME24 (main-paper Table 7).

MethodAcc (%)TokensUnsolved / 30Shared
Original model25.317,4181515
Truncation-based GRPO25.34,8881614
Truncation-based DAPO25.35,8881614
Truncation-based Dr.GRPO25.35,3211514
On-policy SFT30.73,8081312

We will clarify the evaluation protocol and include these per-problem comparisons in the revised manuscript alongside the accuracy and length results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.