- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper provides the argument that when reasoning models fail under tight token budgets, it's usually because they don't stop, not because they can't solve the problem. To address this, the authors propose R3, a reward that gives full credit only to correct answers finishing under a threshold that tracks the model's own recent correct-solution lengths. Compared with the standard GRPO reward and with methods like LASER-DE and DECS, R3 gets higher accuracy at every budget while using roughly half the tokens.
Critical strengths
-
The definition of budget-constrained failure as termination failure rather than capability failure is interesting. Decomposing budgeted accuracy into completion rate and per-completion accuracy, together with the latent-answer scans of truncated outputs, gives a clean diagnostic that other work could reuse.
-
The R3 reward is simple: it is a single plug-in scalar that requires no auxiliary judge model, token-level labels, or curriculum. This makes it easy to reproduce and straightforward to drop into other GRPO-style RL algorithms, since only the reward function needs to change.
Critical weaknesses
-
Length-aware RL rewards for reducing overthinking seem to be a well-studied problem, and the proposed reward resembles existing methods. LAPO (Wu et al., 2025) also uses the median length of correct rollouts as the target and applies length rewards only to correct responses. The paper should compare R3 reward against LAPO or similar adaptive-target methods (e.g., ALP, GRPO-λ).
-
The paper’s claim is that a threshold-conditional reward outperforms the length-penalty family, but the only matched baseline (R1) is correctness-only, while LASER-DE and DECS are released checkpoints trained with different data. A standard length-penalized reward trained under identical data, steps, and backbone is needed to attribute R3’s gains to the reward design.
-
Training and most evaluation focus on math reasoning. The authors could evaluate on code (e.g., LiveCodeBench), scientific reasoning (e.g., GPQA), and open-ended or agentic tasks, and ideally include non-math data in training to show that this training method generalizes.
-
All backbones are early-2025 DeepSeek-R1 distilled models (1.5B–8B) without any RL stage; it is unclear whether the findings hold for newer reasoning models such as the Qwen3 series.
-
The “termination calibration” claim is not well supported. The reward only checks whether the final answer is correct and whether the output is short enough; it never sees when the model actually reached its answer. The evidence for “learning when to stop” is indirect, and much of it could also be explained by the model simply producing shorter outputs.
Questions for authors
See Weakness.
5 comments
Response to Reviewer nEU3 (Part 2/3)
W2: A matched length-penalty arm
We trained it (new). R2 uses the same backbone, data (GSM8K + MATH), 450 steps, G = 8, learning rate and 200-step correctness-only warm-up as R1 and R3. Only the reward after step 200 differs: a correct answer earns 1 − 0.7·ℓ/4096 and a wrong one 0. This is the standard linear penalty, with the same reward range [0.3, 1] as R3, so only the shape differs (re-implemented on a current TRL release for our hardware). Evaluated under the Table 1 protocol:
| MATH 300 / 500 / 1k / 2k / 4k | GSM8K 300 / 500 / 1k / 2k / 4k | tokens @4k (MATH / GSM8K) | Δtok 2k→4k (MATH) | |
|---|---|---|---|---|
| R2 (linear penalty) | 21.4 / 30.4 / 49.2 / 61.5 / 77.6 | 80.3 / 82.3 / 82.3 / 82.3 / 82.3 | 1,889 / 130 | +45.7% |
| R3 | 18.4 / 39.5 / 71.2 / 85.0 / 90.3 | 48.7 / 85.3 / 92.0 / 94.3 / 94.3 | 738 / 353 | +6.0% |
In paired McNemar tests, R3 is significantly more accurate on MATH at 500–4k (p ≤ 0.005) and on GSM8K at 1k–4k (p ≤ 3×10⁻⁵). R2 is better only at GSM8K@300 (p < 10⁻¹⁷); MATH@300 is not significant. On ARC, R3 is better at 300–500 (p < 10⁻²²) and comparable at 1k–4k.
This is what Proposition M.1 predicts. A strictly decreasing reward has no stopping boundary, so it compresses wherever it can rather than where the answer is ready:
- GSM8K: R2 compresses to 130 tokens and its accuracy plateaus at 82.3 at every budget. That is 12 points below R3 on the same problems (p < 10⁻⁶): brevity bought with correctness.
- MATH: R2 barely compresses (1,889 tokens; 21.1% truncated at 4k vs. 0.7% for R3, Table 13). It keeps reasoning for 552 tokens after first stating its answer, vs. 22 for R3 (the W5 metric; an answer statement is detected in 54% of R2's and 80% of R3's correct traces).
Supporting evidence. Under the identical recipe on the released DECS checkpoint (App. J), R3 lifts GSM8K@300 from 33.7 to 60.0 and MATH@4k from 87.6 to 88.3 at 957 vs. 1,515 tokens, while R1 does not. On the same backbone, R3 is also significantly more accurate than the released Arora & Zanette (α = 0.2) checkpoint on MATH at 1k–4k (p ≤ 0.007) with 0.46× the tokens.
Response to Reviewer nEU3 (Part 3/3)
W3: Generalisation beyond math
Training is math-only, so every row below is zero-shot transfer:
| Never trained on | Base | DECS | R3 |
|---|---|---|---|
| HumanEval pass@1 @2k / @4k (App. K) | 50.6 / 58.5 | 54.3 / 57.9 | 64.6 / 67.1 |
| HumanEval truncation @4k | 29.3% | 13.4% | 7.3% |
| ARC-Challenge @300 / 1k / 4k (Table 1; in no method's corpus, App. D.1) | 26.0 / 49.0 / 63.3 | 26.3 / 57.0 / 70.7 | 58.0 / 79.7 / 87.7 |
| GPQA-Diamond @1k / @4k (new, n = 198) | 1.5 / 24.7 | 8.6 / 41.4 | 17.2 / 36.9 |
On GPQA-Diamond, R3 leads every model at 1k (vs. Base, R1, DECS, LASER-DE: +8.6 to +15.7 pp, paired McNemar p ≤ 0.0023). At 4k it remains well above Base and R1 (+12.1, +8.6 pp; p ≤ 0.016) with the fewest tokens (2,497 mean vs. 2,700–3,595), while DECS and LASER-DE are ahead (+4.5 and +6.6 pp; not significant, p = 0.19 and 0.053). Non-math training data is future work. The reward needs only a verifier, so it transfers unchanged.
W4: Newer backbones (Qwen3)
This is a fair limitation. We use DeepSeek-R1-Distill models because every comparison method (DECS, LASER-DE, DLER, LC-R1, Arora & Zanette, AdaptThink) releases checkpoints on them, which allows like-for-like evaluation. Across three backbones from two families (Qwen-1.5B, Qwen-7B, Llama-8B), the ordering R3 > R1 > Base replicates in 8 of 9 backbone–dataset cells (App. I). Qwen3 already includes RL stages and native thinking-mode controls, so whether it still fails to terminate under hard budgets is open, and we state it as such.
W5: Termination calibration vs. shorter outputs
Correct: the reward never sees when the answer is reached. Our claim is about observed behaviour, and we now measure it directly (new). We keep only correct generations that ended on their own before 4k, so each stopping point is observed rather than censored by the cap. Inside the reasoning block (before </think>), we find where the final answer is first stated and count the reasoning tokens that follow.
| MATH@4k (median) | Base | R1 | DECS | LASER-DE | R2 (linear, W2) | R3 |
|---|---|---|---|---|---|---|
| Reasoning tokens | 1,281 | 1,127 | 741 | 771 | 656 | 230 |
| Reasoning after the answer is first stated | 661 | 418 | 119 | 169 | 552 | 22 |
| Same, as % of reasoning | 58% | 44% | 19% | 26% | 45% | 9% |
GSM8K: 257 / 159 / 56 / 122 / 18 tokens (Base / R1 / DECS / LASER-DE / R3). R2 emits no reasoning block on GSM8K (median generation 112 tokens), so the metric does not apply there.
- "Just shorter". At matched reasoning length, R3 still continues least: 6% of its reasoning vs. 22–48% at 300–600 tokens, and 13% vs. 19–59% at 600–1,200.
- Same data, steps and optimiser. RL alone reduces post-answer reasoning modestly (Base 661 → R1 418 tokens); R3, which differs from R1 only in the termination term, reduces it to 22.
- Length penalty vs. termination. R2 shares R3's data, steps and reward range but prices length linearly. Its reasoning is shorter than R1's (656 vs. 1,127 tokens), yet it still continues for 552 tokens (45%) after stating the answer. A length penalty shortens traces; it does not teach the model to stop at the answer.
- Forced-readout control. With an s1-style forced readout for every model (cue and answer within the budget), R3 still beats R1 by +14.7 / +21.7 / +14.0 pp on MATH at 300 / 500 / 1k and +36.3 pp on GSM8K@300 (all p < 10⁻⁵). Giving R1 the same answer-emission mechanism does not close the gap, so R3's advantage is not only answer emission.
This is observed behaviour, measured on correct, completed traces with a numeric final answer where an answer statement is detected (96–98% of baseline traces, 80% of R3's, 54% of R2's). It does not show that the model internally detects readiness.
Revisions (we will upload the revised PDF with all of these within the next couple of days)
- §4 and App. B: LAPO, ALP and GRPO-λ, with the reward comparison above.
- New appendix: the matched linear-penalty arm (R2), the uncensored readiness analysis and forced readout, referenced from §3.3.
- Table 1: four added same-backbone baselines; App. K: GPQA-Diamond.
- Limitations: math-only training and Qwen3 as future work.
Response to Reviewer nEU3 (Part 1/3)
We thank Reviewer nEU3 for the expert review, and for noting that the completion/accuracy decomposition "gives a clean diagnostic that other work could reuse" and that R3 is "easy to reproduce". We address each weakness below with new evidence from the rebuttal period. All new analyses use the Table 1 checkpoints, prompts, problems and extractor (Qwen-7B, n = 300; MATH 299; GPQA-Diamond 198); where an analysis changes decoding (forced readout), we say so.
W1: Relation to LAPO, ALP and GRPO-λ
We agree these must be discussed and add them to §4 and App. B. All three differ from R3 in reward shape; for ALP and GRPO-λ the difference is formal.
<think>in trainingLAPO: a per-problem target vs. a ceiling. LAPO learns a target length per problem and, in both stages, reduces credit for correct answers below it. R3 sets only a ceiling: every correct answer within τ gets equal credit. Under a hard budget the only failure is running too long, so a ceiling targets exactly that and never pulls a short solution longer. LAPO also needs per-problem memory across two stages and a length statement after
<think>; R3 is one scalar in one stage, with no prompt or trace change. LAPO's reward is not length-monotone, so it lies outside Proposition M.1 (Remark M.2). LAPO trains only 1.5B models from different bases and data (4,096-token cap) and evaluates at 32k tokens, so a grid comparison would confound base, data and reward (the reviewer's W2); we compare designs instead.ALP and GRPO-λ fall under Proposition M.1(i). ALP prices each token by the solve rate, so for a fixed prompt it is the linear R_γ family with a prompt-dependent γ and cannot express a stopping boundary (its β = 10⁻⁷ keeps correctness dominance in practice, so the missing boundary is the difference). GRPO-λ is the Arora & Zanette sigmoid penalty (α = 0.2), strictly decreasing in length, applied only to the most accurate groups; W2 evaluates the released Arora & Zanette α = 0.2 checkpoint on our backbone.