- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RazorReward-RL aims to improve the accuracy–efficiency trade-off of reasoning models by training them to stop near the earliest reasoning prefix that supports a correct answer. It constructs serial samples by truncating a reasoning trajectory at semantic boundaries, then applies a reward that favors the first correct prefix, penalizes unsuccessful prefixes, and discourages continued reasoning after a sufficient point. Experiments on DeepSeek-R1-Distill-Qwen models report accuracy improvements together with shorter responses across several mathematical and non-mathematical benchmarks. The empirical results are promising, but the incremental contribution over S-GRPO is narrower than presented, and the baseline reproduction is not documented well enough to support the strength of the comparative claims.
Critical strengths
-
The paper addresses an important practical problem. Reducing reasoning tokens without sacrificing accuracy is directly relevant to latency- and cost-sensitive deployment.
-
The experiments cover three model sizes and a reasonably broad set of benchmarks. The non-math and Chinese evaluations provide useful evidence that the learned behavior is not entirely specific to the English mathematical training distribution.
-
The ablations separate semantic segmentation from the full reward to some extent. The continuation experiments, stopping-prompt analysis, and inspection of verification behavior are also useful attempts to understand what the method changes.
Critical weaknesses
-
The novelty over S-GRPO is limited and somewhat overstated. S-GRPO already constructs serial prefixes from one reasoning trajectory and exponentially decreases rewards according to the order of correct early exits. In fact, the correct-prefix branch of RazorReward with closely matches this existing reward. The main additions appear to be semantic rather than random truncation and stronger penalties for failed prefixes. These may be useful, but the paper currently frames the difference as a broader capability-aware reformulation without isolating which new component is responsible for the gain.
-
The baseline reproduction and evaluation protocol are insufficiently documented. The paper appears to re-evaluate released checkpoints under a common 16K-token protocol, which is more defensible than copying results from prior papers. However, several results differ substantially from the original reports. The reproduced S-GRPO drops from 93.8 to 88.7 on GSM8K and from 56.0 to 51.0 on AIME24. DAST drops from 70.0 to 51.5 on AIME24. These differences may arise from legitimate protocol changes: DAST originally uses a 32K limit and greedy decoding, ShorterBetter uses temperature 0.9, and S-GRPO averages 4–16 trials depending on the benchmark. The paper does not clearly report the temperature, number of trials, prompt templates, stopping rules, or exact checkpoint selected for each baseline.
-
The S-GRPO reproduction needs a fidelity check. The original implementation oversamples and filters queries so that serial groups contain correct early-exit samples. The paper does not state whether this procedure is retained, or report the reproduced baseline’s training-set size, update budget, rollout limit, and checkpoint-selection rule. This matters because RazorReward explicitly learns from groups with no correct prefix, whereas the original S-GRPO data construction attempts to avoid such groups. The authors should first show that their implementation approximately recovers the published S-GRPO results under its original protocol, then evaluate that checkpoint under the common protocol.
-
The advantage over the strongest baselines is modest. On the 7B model, RazorReward improves over the reproduced S-GRPO by 1.9 accuracy points while reducing tokens by 4.4%. Relative to AdaptThink, it gains 1.9 points but uses slightly more tokens. In the more controlled Table 2 comparison, RazorReward improves structured S-GRPO by 1.5 points while increasing token use by 2.5%. Thus, the proposed reward changes the trade-off but does not clearly dominate the closest baseline. Since no uncertainty estimates are reported, the paper does not establish that these relatively small gains are statistically reliable. The large AES improvement also depends on its manually chosen asymmetric weights.
-
The claimed protection of useful long reasoning does not follow from the objective. The paper argues that a correct trajectory always receives a positive reward regardless of length. However, the optimization uses group-centered advantages, so a correct but later prefix may still receive negative advantage when earlier correct prefixes score higher. Tables 5–6 show that forced continuation rarely rescues selected failures, but this is not equivalent to showing that independent test-time sampling with an equal additional token budget is ineffective.
Questions for authors
-
Can the authors report, for every baseline, the exact checkpoint, decoding temperature, number of generations per question, prompt template, answer extractor, and stopping condition used in Table 1?
-
Does the reproduced S-GRPO recover its published results when evaluated with the original S-GRPO protocol? If not, what accounts for the differences?
-
Does the S-GRPO reproduction retain the original oversampling and positive-group filtering procedure? Was it trained on exactly the same 30,190 examples and with the same update and rollout budget as RazorReward?
-
What is the result of changing only the new failed-prefix penalties while holding segmentation, correctness reward, training data, and compute fixed?
-
Are the 1–2 point gains over S-GRPO and AdaptThink reliable under repeated evaluation or paired confidence intervals?
4 comments
Response to Reviewer pTXK (2)
Q2: Explanation of evaluation discrepancies.
We also evaluate five benchmarks using S-GRPO's sampling counts: four samples per question on GSM8K, 16 on AIME/AMC, and eight on MATH-500/GPQA. S-GRPO(Max AES Checkpoint) achieves 73.72% / 3,641 tokens, and RazorReward-RL achieves 74.90% / approximately 3,710 tokens.
These results still do not reproduce the original paper's reported values. Differences in training data and evaluation details may contribute to the discrepancy between our reproduction and S-GRPO’s reported results. Although S-GRPO specifies the number of evaluation trials for each benchmark, we could not find detailed descriptions of its answer-extraction and correctness-checking procedures in the paper, which limits our ability to exactly match its evaluation setup.
In addition, the difference in DAST's evaluation results arises because DAST uses a maximum response length of 32K tokens, whereas most work on overthinking uses a 16K limit.
The reproduction results under S-GRPO’s reported evaluation settings are shown below:
| Method | GSM8K | AIME24 | AMC2023 | MATH-500 | GPQA-D | Avg. Acc | Avg. Tokens | AES |
|---|---|---|---|---|---|---|---|---|
| Vanilla (original-paper baseline) | 92.40 / 1,833.0 | 55.40 / 13,232.0 | 77.20 / 9,693.0 | 85.80 / 5,590.0 | 50.10 / 15,385.0 | 72.18% | 9,147.0 | 0.0000 |
| S-GRPO (original-paper report) | 93.80 / 906.0 | 56.00 / 7,377.0 | 87.50 / 3,494.0 | 92.40 / 2,252.0 | 50.80 / 3,751.0 | 76.10% | 3,556.0 | 0.7742 |
| S-GRPO (New, Max AES) | 89.16 / 527.2 | 51.67 / 7,340.0 | 85.00 / 3,590.2 | 90.63 / 2,097.5 | 52.15 / 4,651.8 | 73.72% | 3,641.3 | 0.6659 |
| RazorReward-RL (New) | 91.00 / 575.5 | 50.83 / ≈7,856.0 | 87.19 / 3,781.6 | 91.70 / 2,242.3 | 53.79 / 4,096.5 | 74.90% | 3,710.4 | 0.7075 |
W4 / Q5: Statistical reliability and AES.
Using the saved outputs of fixed checkpoints, we perform 10,000 question-level paired bootstrap resamples, resampling within each benchmark and aggregating the six benchmarks with equal weights. Razor's differences relative to the baselines are shown below (90% pointwise confidence intervals):
| Model | Acc (%)[90% CI] | Average Tokens [90% CI] | ΔAcc (pp)[90% CI] | ΔToken [90% CI] |
|---|---|---|---|---|
| RazorReward-RL | 69.2 [66.1, 72.2] | 4,395.9 [4,096.2, 4,690.3] | — | — |
| S-GRPO | 68.1 [65.2, 71.1] | 4,308.3 [3,991.5, 4,621.8] | +1.0 [−0.7, +2.7] | +87.6 [−66.3, +245.9] |
| AdaptThink | 68.0 [64.9, 71.1] | 4,850.5 [4,499.6, 5,193.5] | +1.2 [−0.5, +2.8] | −454.6 [−621.5, −283.3] |
| Vanilla | 67.4 [64.3, 70.5] | 6,473.5 [6,098.8, 6,835.8] | +1.7 [+0.1, +3.3] | −2,077.6 [−2,265.2, −1,886.7] |
Relative to the new S-GRPO and AdaptThink, RazorReward's accuracy point estimates are higher by 1.021 and 1.162 percentage points, respectively. However, both paired intervals include 0, so we do not claim statistically significant accuracy improvements. Relative to AdaptThink and Vanilla, average token usage decreases by 9.37% and 32.09%, respectively, and the paired intervals for these differences lie entirely below 0. The interval for the accuracy difference relative to Vanilla also lies entirely above 0, supporting an accuracy improvement and a reduction in token usage at the 90% confidence level.
The AES weights follow Luo et al. (2025), and we also report the raw Acc/Token values in the main results table.
W5: Preserving effective long reasoning. We accept that this wording is too strong. In the revision, we will change it to “improves the trade-off between accuracy and reasoning cost under the evaluated settings.”
Thank you again for your detailed and constructive feedback. We took your concerns seriously and devoted substantial time and effort to reproducing the data-filtering procedure, retraining both methods on the same filtered data, clarifying the evaluation details, and conducting paired bootstrap analyses. Your comments helped us strengthen the experimental evidence. We hope these additions address your main concerns, and we sincerely ask you to reconsider your assessment and raise your score in light of the additional evidence.
Follow-up on our response before the discussion deadline
Dear Reviewer pTXK,
Thank you again for your thoughtful feedback. With approximately 2 hours remaining in the discussion period, we would be grateful if you could take another look at our responses when time permits.
We would particularly appreciate hearing whether our responses address your main concerns, or whether any questions remain. We are available to provide further clarification before the discussion closes.
Thank you for your time and consideration.
Best regards,
The Authors
Final Follow-up Before the Discussion Closes
Dear Reviewer pTXK,
As the discussion period draws to a close, we would sincerely appreciate your feedback on the additional evidence provided in response to your concerns:
- Contribution and ablations: We clarified our relationship to S-GRPO and provided a controlled comparison holding segmentation fixed, showing an accuracy increase from 68.4% to 69.9% with our reward design.
- Reproduction and evaluation: We implemented S-GRPO’s offline filtering and retrained both methods on the same 27,308 examples, reported both its highest-accuracy and highest-AES checkpoints, and clarified the evaluation procedures.
- Statistical reliability and claim scope: We added 10,000 paired bootstrap resamples, and narrowed the claim concerning preservation of useful long reasoning. Could you please let us know whether these additions address your main concerns, or which issues remain unresolved? We remain available to provide any final clarification.
Thank you again for your time and consideration.
Best regards,
The Authors
Response to Reviewer pTXK (1)
Thank you for your careful review and for recognizing the practical importance of reducing reasoning costs, the breadth of our evaluation across model scales and domains, and the value of our ablations and behavioral analyses. We address your concerns point by point below.
W1 / Q4: Contributions and ablations.
We adopt S-GRPO's serial-group framework and reward design for correct samples. Our main improvements are capability-aware reward design and semantic segmentation.
W2, W3 / Q1, Q3: Reproduction and evaluation protocols.
To keep the training data consistent, our previous reproduction did not perform S-GRPO's offline oversampling and filtering of all-incorrect queries. We have now added this step: using 12 H20 GPUs, we completed preprocessing of 30,190 examples in nearly six hours of inference, retaining 27,308 preprocessed examples, and retrained both methods on the same filtered data.
Our main evaluation follows the AdaptThink protocol: T = 0.6, top-p = 0.95, top-k = 50, a maximum generation length of 16,384 tokens, and termination at EOS or the length limit. We follow AdaptThink’s answer extraction procedure, primarily extracting the final answer enclosed in \boxed{...} from the response after </think>. We use its original prompts without appending any additional prompt to the questions. For GSM8K, AIME24, AMC23, MATH-500, GPQA-D, and AIME25, we generate 1, 16, 4, 1, 1, and 16 responses per question, respectively. DAST, ShorterBetter, and AdaptThink use publicly available weights from Hugging Face or ModelScope. For S-GRPO, which has not released model weights, and our own method, we select checkpoints with a good overall balance between accuracy and token usage.
The results of RL training for the 7B models on the preprocessed data are shown below:
The two S-GRPO checkpoints correspond to the highest accuracy and highest AES among the evaluated checkpoints, respectively. After training on the high-quality RL data filtered as required by S-GRPO, RazorReward-RL uses fewer tokens than the S-GRPO maximum-accuracy checkpoint and achieves higher accuracy with slightly more tokens than the maximum-AES checkpoint. Its overall AES is higher than that of both S-GRPO checkpoints.