acceptodds
Under review as a conference paper at ICLR 2027

RazorReward-RL: Capability-Aware Efficiency Optimization for LLM Reasoning via Razor Reward

Abstract

Chain-of-Thought (CoT) prompting is a de facto standard for unlocking strong reasoning performance in large language models (LLMs), but its step-by-step generation often introduces redundant computation in cost- and latency-sensitive production deployments. Existing reinforcement learning (RL) approaches for CoT efficiency rely on length-based rewards to mitigate redundant steps, but overlook the model's intrinsic query-specific reasoning capacity. This leads to two key limitations: for Beyond-capability queries that exceed the model's reasoning capacity, the reward mechanism encourages longer but futile reasoning, wasting massive token budgets; for Near-capability queries lying at the edge of the model's capacity, smoothed length-based rewards fail to provide clear discriminative signals to identify optimal stopping points, causing the model to either stop early with insufficient reasoning or drift away from the correct answer after redundant reasoning. To address these issues, we propose RazorReward-RL, a capability-aware RL framework for efficient CoT reasoning. As its name suggests, the core RazorReward function acts as a razor for reasoning chains. By segmenting CoT trajectories at natural reasoning step boundaries to construct semantically coherent prefixes, it sharply amplifies the reward gap in favor of the shortest sufficient prefix, i.e., the shortest prefix that yields a correct answer, imposes steep penalties for under- and over-reasoning on all within-capability and near-capability queries, while imposing heavy penalties for generation on beyond-capability queries to block futile reasoning. Experiments across multiple domains and model scales show that RazorReward-RL significantly reduces token consumption while improving accuracy, achieving a better accuracy-efficiency trade-off than prior methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reasoning model, overthinking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pTXKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RazorReward-RL aims to improve the accuracy–efficiency trade-off of reasoning models by training them to stop near the earliest reasoning prefix that supports a correct answer. It constructs serial samples by truncating a reasoning trajectory at semantic boundaries, then applies a reward that favors the first correct prefix, penalizes unsuccessful prefixes, and discourages continued reasoning after a sufficient point. Experiments on DeepSeek-R1-Distill-Qwen models report accuracy improvements together with shorter responses across several mathematical and non-mathematical benchmarks. The empirical results are promising, but the incremental contribution over S-GRPO is narrower than presented, and the baseline reproduction is not documented well enough to support the strength of the comparative claims.

Critical strengths
  1. The paper addresses an important practical problem. Reducing reasoning tokens without sacrificing accuracy is directly relevant to latency- and cost-sensitive deployment.

  2. The experiments cover three model sizes and a reasonably broad set of benchmarks. The non-math and Chinese evaluations provide useful evidence that the learned behavior is not entirely specific to the English mathematical training distribution.

  3. The ablations separate semantic segmentation from the full reward to some extent. The continuation experiments, stopping-prompt analysis, and inspection of verification behavior are also useful attempts to understand what the method changes.

Critical weaknesses
  1. The novelty over S-GRPO is limited and somewhat overstated. S-GRPO already constructs serial prefixes from one reasoning trajectory and exponentially decreases rewards according to the order of correct early exits. In fact, the correct-prefix branch of RazorReward with α=1/2\alpha=1/2 closely matches this existing reward. The main additions appear to be semantic rather than random truncation and stronger penalties for failed prefixes. These may be useful, but the paper currently frames the difference as a broader capability-aware reformulation without isolating which new component is responsible for the gain.

  2. The baseline reproduction and evaluation protocol are insufficiently documented. The paper appears to re-evaluate released checkpoints under a common 16K-token protocol, which is more defensible than copying results from prior papers. However, several results differ substantially from the original reports. The reproduced S-GRPO drops from 93.8 to 88.7 on GSM8K and from 56.0 to 51.0 on AIME24. DAST drops from 70.0 to 51.5 on AIME24. These differences may arise from legitimate protocol changes: DAST originally uses a 32K limit and greedy decoding, ShorterBetter uses temperature 0.9, and S-GRPO averages 4–16 trials depending on the benchmark. The paper does not clearly report the temperature, number of trials, prompt templates, stopping rules, or exact checkpoint selected for each baseline.

  3. The S-GRPO reproduction needs a fidelity check. The original implementation oversamples and filters queries so that serial groups contain correct early-exit samples. The paper does not state whether this procedure is retained, or report the reproduced baseline’s training-set size, update budget, rollout limit, and checkpoint-selection rule. This matters because RazorReward explicitly learns from groups with no correct prefix, whereas the original S-GRPO data construction attempts to avoid such groups. The authors should first show that their implementation approximately recovers the published S-GRPO results under its original protocol, then evaluate that checkpoint under the common protocol.

  4. The advantage over the strongest baselines is modest. On the 7B model, RazorReward improves over the reproduced S-GRPO by 1.9 accuracy points while reducing tokens by 4.4%. Relative to AdaptThink, it gains 1.9 points but uses slightly more tokens. In the more controlled Table 2 comparison, RazorReward improves structured S-GRPO by 1.5 points while increasing token use by 2.5%. Thus, the proposed reward changes the trade-off but does not clearly dominate the closest baseline. Since no uncertainty estimates are reported, the paper does not establish that these relatively small gains are statistically reliable. The large AES improvement also depends on its manually chosen asymmetric weights.

  5. The claimed protection of useful long reasoning does not follow from the objective. The paper argues that a correct trajectory always receives a positive reward regardless of length. However, the optimization uses group-centered advantages, so a correct but later prefix may still receive negative advantage when earlier correct prefixes score higher. Tables 5–6 show that forced continuation rarely rescues selected failures, but this is not equivalent to showing that independent test-time sampling with an equal additional token budget is ineffective.

Questions for authors
  1. Can the authors report, for every baseline, the exact checkpoint, decoding temperature, number of generations per question, prompt template, answer extractor, and stopping condition used in Table 1?

  2. Does the reproduced S-GRPO recover its published results when evaluated with the original S-GRPO protocol? If not, what accounts for the differences?

  3. Does the S-GRPO reproduction retain the original oversampling and positive-group filtering procedure? Was it trained on exactly the same 30,190 examples and with the same update and rollout budget as RazorReward?

  4. What is the result of changing only the new failed-prefix penalties while holding segmentation, correctness reward, training data, and compute fixed?

  5. Are the 1–2 point gains over S-GRPO and AdaptThink reliable under repeated evaluation or paired confidence intervals?

4 comments
AuthorsOpenReview ↗

Response to Reviewer pTXK (1)

Thank you for your careful review and for recognizing the practical importance of reducing reasoning costs, the breadth of our evaluation across model scales and domains, and the value of our ablations and behavioral analyses. We address your concerns point by point below.

W1 / Q4: Contributions and ablations.

We adopt S-GRPO's serial-group framework and reward design for correct samples. Our main improvements are capability-aware reward design and semantic segmentation.

  • We replace random truncation with segmentation at semantic boundaries so that probing positions better align with reasoning-step boundaries.
  • In reward design, for near-capability groups, we aim to reduce near-misses before the first correct answer and drift after it; for beyond-capability cases, we aim to reduce futile exploration on overly difficult problems. This design enables all-incorrect groups to provide differentiated within-group training signals, reducing reliance on additional rollout-based filtering: if filtering is performed separately, checking whether each early-exit branch is correct requires essentially the same generation process as constructing training groups, and its inference cost can approach that of one rollout pass over a training set of the same size.
  • To distinguish the contributions of reward design and segmentation, S-GRPO with Structured Sampling in Table 2 retains the S-GRPO reward, introduces only the same semantic segmentation as ours, and sets alpha = 0.5. On top of this setting, RazorReward further improves the accuracy by 1.5 percentage points, supporting the additional benefit of the failure-reward design beyond semantic segmentation.
  • Furthermore, inspection of training shows that all-incorrect groups still occur after data filtering: their proportions in the early and late stages of RazorReward-RL training are 2.66% and 2.44%, respectively. This indirectly supports the necessity of introducing a reward design for failed samples.

W2, W3 / Q1, Q3: Reproduction and evaluation protocols.

To keep the training data consistent, our previous reproduction did not perform S-GRPO's offline oversampling and filtering of all-incorrect queries. We have now added this step: using 12 H20 GPUs, we completed preprocessing of 30,190 examples in nearly six hours of inference, retaining 27,308 preprocessed examples, and retrained both methods on the same filtered data.

Our main evaluation follows the AdaptThink protocol: T = 0.6, top-p = 0.95, top-k = 50, a maximum generation length of 16,384 tokens, and termination at EOS or the length limit. We follow AdaptThink’s answer extraction procedure, primarily extracting the final answer enclosed in \boxed{...} from the response after </think>. We use its original prompts without appending any additional prompt to the questions. For GSM8K, AIME24, AMC23, MATH-500, GPQA-D, and AIME25, we generate 1, 16, 4, 1, 1, and 16 responses per question, respectively. DAST, ShorterBetter, and AdaptThink use publicly available weights from Hugging Face or ModelScope. For S-GRPO, which has not released model weights, and our own method, we select checkpoints with a good overall balance between accuracy and token usage.

The results of RL training for the 7B models on the preprocessed data are shown below:

ModelGSM8K ↑AIME24 ↑AIME25 ↑AMC23 ↑MATH500 ↑GPQA ↑Summary Acc ↑Summary Tokens ↓Token reduction ↑AES ↑
7B backbone87.8%50.8%37.1%86.3%91.0%51.5%67.40%6,473.00.00%0.000
New Razor91.51%50.83%34.58%88.13%92.80%57.07%69.15%4,395.932.09%0.399
New S-GRPO (Max Acc)88.2%55.0%37.1%88.1%93.0%53.0%69.07%5,299.018.14%0.256
New S-GRPO (Max AES)89.1%51.7%36.7%85.6%90.2%55.6%68.15%4,308.233.44%0.368

The two S-GRPO checkpoints correspond to the highest accuracy and highest AES among the evaluated checkpoints, respectively. After training on the high-quality RL data filtered as required by S-GRPO, RazorReward-RL uses fewer tokens than the S-GRPO maximum-accuracy checkpoint and achieves higher accuracy with slightly more tokens than the maximum-AES checkpoint. Its overall AES is higher than that of both S-GRPO checkpoints.

AuthorsOpenReview ↗

Response to Reviewer pTXK (2)

Q2: Explanation of evaluation discrepancies.

We also evaluate five benchmarks using S-GRPO's sampling counts: four samples per question on GSM8K, 16 on AIME/AMC, and eight on MATH-500/GPQA. S-GRPO(Max AES Checkpoint) achieves 73.72% / 3,641 tokens, and RazorReward-RL achieves 74.90% / approximately 3,710 tokens.

These results still do not reproduce the original paper's reported values. Differences in training data and evaluation details may contribute to the discrepancy between our reproduction and S-GRPO’s reported results. Although S-GRPO specifies the number of evaluation trials for each benchmark, we could not find detailed descriptions of its answer-extraction and correctness-checking procedures in the paper, which limits our ability to exactly match its evaluation setup.

In addition, the difference in DAST's evaluation results arises because DAST uses a maximum response length of 32K tokens, whereas most work on overthinking uses a 16K limit.

The reproduction results under S-GRPO’s reported evaluation settings are shown below:

MethodGSM8KAIME24AMC2023MATH-500GPQA-DAvg. AccAvg. TokensAES
Vanilla (original-paper baseline)92.40 / 1,833.055.40 / 13,232.077.20 / 9,693.085.80 / 5,590.050.10 / 15,385.072.18%9,147.00.0000
S-GRPO (original-paper report)93.80 / 906.056.00 / 7,377.087.50 / 3,494.092.40 / 2,252.050.80 / 3,751.076.10%3,556.00.7742
S-GRPO (New, Max AES)89.16 / 527.251.67 / 7,340.085.00 / 3,590.290.63 / 2,097.552.15 / 4,651.873.72%3,641.30.6659
RazorReward-RL (New)91.00 / 575.550.83 / ≈7,856.087.19 / 3,781.691.70 / 2,242.353.79 / 4,096.574.90%3,710.40.7075

W4 / Q5: Statistical reliability and AES.

Using the saved outputs of fixed checkpoints, we perform 10,000 question-level paired bootstrap resamples, resampling within each benchmark and aggregating the six benchmarks with equal weights. Razor's differences relative to the baselines are shown below (90% pointwise confidence intervals):

ModelAcc (%)[90% CI]Average Tokens [90% CI]ΔAcc (pp)[90% CI]ΔToken [90% CI]
RazorReward-RL69.2 [66.1, 72.2]4,395.9 [4,096.2, 4,690.3]——
S-GRPO68.1 [65.2, 71.1]4,308.3 [3,991.5, 4,621.8]+1.0 [−0.7, +2.7]+87.6 [−66.3, +245.9]
AdaptThink68.0 [64.9, 71.1]4,850.5 [4,499.6, 5,193.5]+1.2 [−0.5, +2.8]−454.6 [−621.5, −283.3]
Vanilla67.4 [64.3, 70.5]6,473.5 [6,098.8, 6,835.8]+1.7 [+0.1, +3.3]−2,077.6 [−2,265.2, −1,886.7]

Relative to the new S-GRPO and AdaptThink, RazorReward's accuracy point estimates are higher by 1.021 and 1.162 percentage points, respectively. However, both paired intervals include 0, so we do not claim statistically significant accuracy improvements. Relative to AdaptThink and Vanilla, average token usage decreases by 9.37% and 32.09%, respectively, and the paired intervals for these differences lie entirely below 0. The interval for the accuracy difference relative to Vanilla also lies entirely above 0, supporting an accuracy improvement and a reduction in token usage at the 90% confidence level.

The AES weights follow Luo et al. (2025), and we also report the raw Acc/Token values in the main results table.

W5: Preserving effective long reasoning. We accept that this wording is too strong. In the revision, we will change it to “improves the trade-off between accuracy and reasoning cost under the evaluated settings.”


Thank you again for your detailed and constructive feedback. We took your concerns seriously and devoted substantial time and effort to reproducing the data-filtering procedure, retraining both methods on the same filtered data, clarifying the evaluation details, and conducting paired bootstrap analyses. Your comments helped us strengthen the experimental evidence. We hope these additions address your main concerns, and we sincerely ask you to reconsider your assessment and raise your score in light of the additional evidence.

AuthorsOpenReview ↗

Follow-up on our response before the discussion deadline

Dear Reviewer pTXK,

Thank you again for your thoughtful feedback. With approximately 2 hours remaining in the discussion period, we would be grateful if you could take another look at our responses when time permits.

We would particularly appreciate hearing whether our responses address your main concerns, or whether any questions remain. We are available to provide further clarification before the discussion closes.

Thank you for your time and consideration.

Best regards,

The Authors

AuthorsOpenReview ↗

Final Follow-up Before the Discussion Closes

Dear Reviewer pTXK,

As the discussion period draws to a close, we would sincerely appreciate your feedback on the additional evidence provided in response to your concerns:

  • Contribution and ablations: We clarified our relationship to S-GRPO and provided a controlled comparison holding segmentation fixed, showing an accuracy increase from 68.4% to 69.9% with our reward design.
  • Reproduction and evaluation: We implemented S-GRPO’s offline filtering and retrained both methods on the same 27,308 examples, reported both its highest-accuracy and highest-AES checkpoints, and clarified the evaluation procedures.
  • Statistical reliability and claim scope: We added 10,000 paired bootstrap resamples, and narrowed the claim concerning preservation of useful long reasoning. Could you please let us know whether these additions address your main concerns, or which issues remain unresolved? We remain available to provide any final clarification.

Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_cyB3OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RazorReward-RL, a capability-aware reinforcement learning framework for making chain-of-thought reasoning in LLMs more efficient. The authors argue that existing length-based RL methods ignore whether a query is within the model's reasoning ability. As a result, they encourage long, futile reasoning on queries beyond the model's capacity, and on queries near its capacity, they give signals too weak to separate barely sufficient reasoning from sufficient reasoning. RazorReward-RL cuts each reasoning trace into prefixes at natural step boundaries and checks whether each prefix, after an early-stop prompt, leads to the correct answer. The reward is highest for the shortest prefix that does. Wrong answers just before that point and after it are penalized sharply, which discourages both under- and over-reasoning. When no prefix is correct, shorter failed attempts are preferred. On six math and science benchmarks with models from 1.5B to 14B parameters, it improves accuracy and cuts output tokens by 17–36%, beating prior methods on the accuracy–efficiency trade-off.

Critical strengths
  1. Clear motivation: it shows that length-based rewards ignore the model's capability and supports this with prefix correctness patterns across three query types.
  2. Simple, interpretable reward: the reward is built around the first sufficient prefix, so one mechanism handles under-reasoning, over-reasoning, and futile reasoning.
  3. Consistent results: accuracy goes up, and tokens go down across three model scales and several benchmarks.
Critical weaknesses
  1. The reward may penalize correct early reasoning steps. All samples in a probing group are truncations of the same rollout, so the earlier blocks are shared. T₁ appears in every sample. Eq. (3) masks only P_es, so each sample's advantage is also applied to its prefix tokens. Under per-sample length normalization, the short, incorrect samples get the largest weights, and their negative advantages dominate the shared early blocks. For example, with the pattern ✗✗✓✓ and α = 1/2, T₁ gets a net negative signal of about −1.35, even though it is part of a trajectory that reaches the correct answer. The error was the decision to stop after T₁, but the penalty falls on generating T₁. The authors should discuss this and run an ablation that applies advantages only to cᵢ and the newly added block.
  2. Prompt sensitivity: results change with the wording of the early-stop prompt.
  3. The experimental results are based on the DeepSeek-R1-Distill-Qwen series models and lack validation on other architectures and newer models such as Qwen3.
Questions for authors
  1. Does the reward encourage early guessing? The first correct prefix gets the highest reward, so the policy could learn to state a candidate answer in the first block and verify it afterward. That would be rewarding guessing, especially on multiple-choice tasks (e.g., GPQA) or tasks whose answers are small integers. How often does the final answer already appear in the first block, before and after training? Does the accuracy gain on GPQA remain if you evaluate in a format that makes guessing less effective, for example, by removing the answer options?
  2. As described in the first point under weaknesses, would this method incorrectly penalize early correct reasoning steps?
  3. When more than 8 matching tokens are found, how are these 8 split points selected? The appendix only mentions the term "selected prefix blocks"; the specific rule is not written. It could be uniform sampling, or it could be taking the first few.
  4. The mathematical benchmarks covered in the paper are a bit old. How does the method perform on the latest AIME26?
  5. Readability issue: the image does not appear to be embedded in the paper in PDF format, and it becomes very blurry when zoomed in. In the formula in Section 3.2.2, the notation G∼πθold(⋅∣q)G\sim\pi_{\theta_\text{old}} (\cdot\mid q) does not conform to the construction rule for G described in the paper.
8 comments
AuthorsOpenReview ↗

Response to Reviewer cyB3 (1)

Thank you for your careful review and for recognizing the motivation, interpretability of the reward, and consistency of the experimental results.

W1 / Q2: Incorrect penalization of shared prefixes.

We considered a scheme that applies the policy loss only to the answer cic_i generated after the reasoning prefix. Under this scheme, the answer outcomes of these truncated branches no longer provide direct policy-gradient supervision for prefix generation. Training would optimize how to answer correctly given a prefix, but applying the policy loss only to cic_i does not optimize how to generate a prefix that better enables the model to answer correctly. This motivates our decision to retain prefix updates: we aim to optimize both the ability to generate a correct answer given a prefix and the ability to generate reasoning prefixes that facilitate subsequent correct answers. An ablation using the same training data and evaluation protocol achieves 68.64% accuracy across the six benchmarks, with an average generation length of 4,933 tokens; the current method performs better, achieving 69.8% accuracy and 4,910 tokens. This result supports the empirical benefit of retaining prefix updates in the current experimental setting.

W2: Sensitivity to the early-stopping prompt.

Prompt wording affects the results to some extent, but both prompts improve accuracy and reduce token usage (these results are reported in Appendix E.8):

SettingSummary AccSummary Token
Vanilla67.4%6,473
Neutral69.9%4,910
Strict68.48%5,420

W3: Validation on a new model architecture.

We provide additional experiments on Qwen3-1.7B using the same evaluation protocol as the main experiments:

Each benchmark cell reports Acc (%) / average Tokens.

BenchmarkVanillaS-GRPO‡RazorReward-RL
GSM8K90.4 / 2,46390.0 / 1,45689.6 / 1,772
AIME2436.5 / 13,49738.8 / 11,39037.7 / 11,810
AMC202373.1 / 8,69376.9 / 6,72476.9 / 7,425
MATH-50087.0 / 5,61485.8 / 3,60187.6 / 4,165
GPQA-D33.3 / 9,15036.9 / 7,83243.4 / 7,908
AIME2532.5 / 13,40729.6 / 11,32529.4 / 11,969
Summary Acc (%)58.859.660.8
Summary Tokens8,8047,0557,508
Summary Aes0.0000.2420.248

On Qwen3-1.7B, RazorReward-RL likewise improves accuracy and reduces token usage on the new architecture, achieving the highest accuracy and AES among the methods using the same backbone.

Q1: Whether the reward encourages guessing.

To address your concern, we remove the answer options and exclude questions unsuitable for open-ended answering, retaining 118 GPQA questions (113 retain their original question stems and 5 are locally reworded). We sample four responses per question and use Qwen3.8-flash to judge whether each response agrees with the ground truth. Razor125 achieves 32.84% accuracy, compared with Vanilla's 30.51%, while reducing token usage from 5,650 to 4,597. This shows that Razor125 retains advantages in accuracy and generation efficiency in an open-ended setting without answer options, alleviating the concern that the gains depend on guessing from the options. In addition, the analysis of MATH-500 reasoning traces in Appendix E.1 provides supplementary evidence: RazorReward obtains an overall logical quality score of 4.888 and a global coherence score of 4.916, both the highest among the five models compared (Table 8). These results support that shorter generations are accompanied by good logical quality and coherence.

Q3: Selection of split points.

Each group contains seven truncated branches and one complete response. When there are more than seven candidate split points, we uniformly sample seven without replacement and then sort them by their positions along the trajectory. We will correct the number in the appendix and add the selection rule.

Q4: AIME26.

In the avg@16 evaluation with 30 questions and 16 samples per question, RazorReward-RL reduces accuracy by 2.08 percentage points and token usage by 32% relative to Vanilla. RazorReward achieves the highest AES (0.095) among the compared methods, outperforming the other baselines on this metric.

ModelAccRelative to BaseTokAES
Base 7B46.25%—10,550—
RazorReward-RL44.17%−2.08pp7,174 (−32%)0.095
S-GRPO43.75%−2.50pp7,1650.051
AdaptThink43.33%−2.92pp9,018−0.170
DAST43.54%−2.71pp10,335−0.273
ShorterBetter35.83%−10.42pp4,387−0.542
AuthorsOpenReview ↗

Response to Reviewer cyB3 (2)

Q5: Figure quality and notation.

We will re-export vector PDFs from the original figure source files, replace the raster images in the main text, and check the readability of the final PDF when zoomed in.

Regarding Section 3.2.2, we agree that G∼πθold(⋅∣q)G\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid q) does not accurately express the construction of the prefix probing group. We will replace it with G∼Pprobeθold(⋅∣q)G\sim\mathcal P_{\mathrm{probe}}^{\theta_{\mathrm{old}}}(\cdot\mid q) and clarify that this distribution is induced by the following procedure: sample one complete rollout from the pre-update policy, construct prefixes at semantic boundaries, generate an early-answer response conditioned on each prefix, and retain the original complete branch. The revision will clarify that the group members share the same original trajectory, rather than being independently sampled complete responses. This change clarifies the existing training procedure; the reward definition and policy loss remain unchanged.


We hope these clarifications and additional results address your concerns, and we respectfully ask you to revise your score upward.

AuthorsOpenReview ↗

Follow-up on our response before the discussion deadline

Dear Reviewer cyB3,

Thank you again for your thoughtful feedback. With approximately one day remaining in the discussion period, we would be grateful if you could take another look at our responses when time permits.

We would particularly appreciate hearing whether our responses address your main concerns, or whether any questions remain. We are available to provide further clarification before the discussion closes.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_fdbdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RazorReward-RL, a capability-aware reinforcement learning framework designed to mitigate the overthinking problem in Chain-of-Thought (CoT) reasoning for Large Language Models. Building upon serial truncation sampling (e.g., S-GRPO), the method segments a full CoT trajectory into semantically coherent prefixes using an adaptively generated list of linguistic transition markers (e.g., "wait", "thus", "first"), appends an early-stopping prompt to each prefix, and probes whether the policy model can produce the correct final answer. Based on the prefix correctness distribution, the authors introduce a piecewise exponential length reward (RazorReward) that sharply rewards the shortest sufficient prefix, heavily penalizes near-miss under-reasoning and post-sufficiency over-reasoning, and suppresses lengthy generation when no prefix succeeds. Experiments on DeepSeek-R1-Distill-Qwen models (1.5B, 7B, and 14B) across math, science, and out-of-distribution benchmarks show that RazorReward-RL improves accuracy while reducing token usage by 17%–36%.

Critical strengths
  1. The problem and the formulation are timely, practical, and well-diagnosed.
  2. The Razor reward concept is intuitive and clean.
  3. The evaluation across model scales, domains, and languages is extensive with strong empirical gains.
  4. The behavioral and mechanistic analysis of reasoning traces is deep.
Critical weaknesses
  1. The novelty of methodology is limited compared with S-GRPO. The general pipeline of the method is: a. generating a CoT; b. truncating it into multiple prefixes; c. appending an early-stopping prompt; d. generating answers from each prefix; e. applying GRPO over the serial prefix group This is very similar to S-GRPO. The innovation is mainly on the two refinement: (1) matching strings from a GPT-5-curated marker list instead of random truncation, and (2) replacing S-GRPO's reward formula with a piecewise exponential decay function. This is quite limited.

  2. The RL baselines on the 14B scale is missing. In Table 1, for the 14B model, the authors only compare against Vanilla and the training-free DEER baseline, omitting all RL baselines (AdaptThink, ShorterBetter, DAST, S-GRPO) with the justification that no baseline releases 14B weights. Similarly, in the OOD generalization experiments, RazorReward-RL is only compared against Vanilla 7B, with no RL baselines included. It would be more convincing if the 14B results can be presented.

  3. The reward can penalize exploration. In RL, solving hard problems requires exploration over long trajectories that are initially unsuccessful. In RazorReward-RL, if all prefixes in a rollout fail, the reward function assigns the least negative penalty to the shortest prefix and the heaviest penalty to the longest prefixes. If a query is actually solvable by the model with deeper thinking, but happens to fail in the sampled rollout, this reward actively trains the model to give up immediately on hard prompts rather than exploring deeper reasoning paths.

Questions for authors

See the comments above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer fdbd (1)

Thank you for your careful review and for recognizing the motivation, reward design, and depth of our analysis. We respond to your comments below.

W1: Contributions relative to S-GRPO.

We adopt S-GRPO's serial-group framework, with two contributions:

  • Structured segmentation: We construct prefixes at reasoning-step boundaries to reduce interference from random truncation. Early-answer accuracy is 2.75 percentage points higher than with random segmentation (Appendix E.5).
  • Capability-aware reward design: For all-correct groups (within-capability), we use the same reward mechanism for correct samples as S-GRPO. For incorrect samples, S-GRPO assigns a reward of 0 to all of them, whereas we differentiate among them: in near-capability groups, we distinguish failures before the first correct answer from degradation after it; in beyond-capability groups, we provide within-group learning signals for all-incorrect groups. This improves training-sample utilization and broadens applicability to training data without requiring all-incorrect queries to be filtered out beforehand. With structured segmentation held fixed, replacing the reward increases accuracy from 68.4% to 69.9% (Table 2). Offline oversampling and filtering in our S-GRPO reproduction required approximately 72 H20 GPU-hours; our method does not incur this overhead.

W2: 14B and OOD baselines.

These RL baselines were previously omitted because none had released 14B weights. To address your concern, we have additionally trained S-GRPO at the 14B scale and provide its results as an RL baseline:

MethodGSM8K ↑ / ↓AIME24 ↑ / ↓AIME25 ↑ / ↓AMC23 ↑ / ↓MATH500 ↑ / ↓GPQA ↑ / ↓Summary Acc ↑Summary Tokens ↓AES ↑
Vanilla 14B90.4 / 61163.1 / 9,78344.2 / 10,99793.1 / 5,47491.2 / 3,51557.6 / 5,84773.3%6,0380.000
S-GRPO-14B90.9 / 53860.4 / 7,59541.3 / 8,40491.3 / 3,72091.6 / 2,31557.1 / 4,35772.1%4,4880.174
RazorReward-RL 14B92.0 / 60761.0 / 8,36644.2 / 9,40995.0 / 4,16192.8 / 2,81861.6 / 4,60974.4%4,9950.221

At the 14B scale, RazorReward achieves 74.4% accuracy, higher than S-GRPO's 72.1%, and also achieves a higher overall AES than S-GRPO.

For OOD evaluation, we follow the AdaptThink evaluation protocol and provide additional evaluations of 7B models, including RL baselines. Compared with Vanilla, RazorReward improves accuracy and reduces token usage on CommonsenseQA, ARC-Challenge, and MMLU-Pro CS; on COPA, accuracy is slightly lower and token usage is slightly higher. Compared with S-GRPO, RazorReward achieves higher accuracy on ARC-Challenge, COPA, and MMLU-Pro CS. Together with the 14B experiments above, these results provide additional RL-baseline evidence across model scales and domains.

Each cell reports Acc (%) / average Tokens.

7B modelCommonsenseQA Acc ↑ / Tok ↓ARC-C Acc ↑ / Tok ↓COPA Acc ↑ / Tok ↓MMLU-Pro CS Acc ↑ / Tok ↓
Vanilla66.01 / 794.280.80 / 844.790.00 / 531.758.78 / 2,878.6
AdaptThink66.58 / 673.180.20 / 697.187.00 / 710.059.27 / 2,198.8
S-GRPO66.75 / 512.478.50 / 558.387.00 / 345.856.59 / 1,576.9
RazorReward66.58 / 579.681.14 / 772.389.00 / 551.560.00 / 2,139.5

These results show that RazorReward-RL achieves OOD performance broadly comparable to Vanilla 7B and slightly outperforms some RL baselines.

AuthorsOpenReview ↗

Response to Reviewer fdbd (2)

W3: Whether the reward suppresses exploration.

Penalizing excessively long reasoning in all-incorrect groups is a design choice for cost-sensitive settings. The available evidence does not show a clear contraction in capability coverage:

  • The reasoning boundary does not narrow: We evaluate pass@16 on AIME24+25, obtaining 45/60 for RazorReward and 44/60 for Vanilla. This indicates that our training does not suppress the model's exploration ability: across 16 samples, it retains stronger exploration ability on difficult problems than the baseline.
  • The model does not give up on difficult problems during training: We keep the average difficulty of each batch approximately consistent during training, while the proportion of all-incorrect groups decreases from 15.3% to 10.6% (Table 12, 1.5B). If the model were increasingly giving up on difficult problems, this proportion should increase.
  • Clear gains on problems requiring more thought: On MATH-500 in Table 4, accuracy on problems that Vanilla sometimes answers correctly and sometimes incorrectly—that is, problems requiring more thought and exploration—increases from 72.4% to 85.2%; on problems for which all eight Vanilla samples are incorrect, accuracy increases from 0.0% to 5.9% (Table 4).
  • Better performance on difficult benchmarks: On the challenging mathematical benchmarks AIME24 and AIME25, RazorReward improves over Vanilla by +5.4pp (1.5B) / +4.8pp (7B) on AIME24 and +5.7 / +0.6pp on AIME25.
  • Additional budget remains beneficial: On near-capability problems in MMLU-Pro CS, continuing after an initial answer improves accuracy by +2.2pp (Vanilla: +2.7pp; Table 6).

In light of these clarifications and additional experimental results, we respectfully ask you to reconsider your assessment and raise your score.

AuthorsOpenReview ↗

Follow-up on our response before the discussion deadline

Dear Reviewer fdbd,

Thank you again for your thoughtful feedback. With approximately 2 hours remaining in the discussion period, we would be grateful if you could take another look at our responses when time permits.

We would particularly appreciate hearing whether our responses address your main concerns, or whether any questions remain. We are available to provide further clarification before the discussion closes.

Thank you for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.