acceptodds
Under review as a conference paper at ICLR 2027

Termination Calibration Recovers Answers That Reasoning Models Compute but Never Express

Abstract

Large reasoning models trained with reinforcement learning from verifiable rewards (RLVR) achieve state-of-the-art performance, but they are evaluated under generous token limits while deployment imposes hard budgets. We show that under such budgets, these models fail primarily through ***termination failure*** rather than capability failure: nearly half of a distilled model's truncated generations already contain the correct answer—computed, but never expressed. Existing efficiency methods do not resolve this. Length-penalized rewards teach models to be shorter, yet their stopping points remain coupled to whatever budget is available; and we find the standard correctness-only RLVR recipe can actively worsen termination, lengthening typical traces and dropping tight-budget accuracy below that of the untrained base model. We introduce ***termination calibration*** through **R3**, a single trajectory-level reward granting full credit only when a correct answer terminates within a threshold that tracks the policy's own recent correct-solution lengths. Our design (i) requires no auxiliary judge, token-level labels, or curriculum, and (ii) includes a matched correctness-only control that isolates the causal mechanism. Across budgets and datasets, R3 matches or exceeds specialized efficiency methods at roughly half their tokens and twice the speed; its learned stopping rule is budget-invariant yet extends adaptively to problems harder than any seen in training, and the effect replicates across three backbones. Efficient reasoning, we argue, is less about learning to be brief than learning when to stop.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning from verifiable rewards, reasoning models, inference efficiency, token budgets, reward design, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nEU3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides the argument that when reasoning models fail under tight token budgets, it's usually because they don't stop, not because they can't solve the problem. To address this, the authors propose R3, a reward that gives full credit only to correct answers finishing under a threshold that tracks the model's own recent correct-solution lengths. Compared with the standard GRPO reward and with methods like LASER-DE and DECS, R3 gets higher accuracy at every budget while using roughly half the tokens.

Critical strengths
  1. The definition of budget-constrained failure as termination failure rather than capability failure is interesting. Decomposing budgeted accuracy into completion rate and per-completion accuracy, together with the latent-answer scans of truncated outputs, gives a clean diagnostic that other work could reuse.

  2. The R3 reward is simple: it is a single plug-in scalar that requires no auxiliary judge model, token-level labels, or curriculum. This makes it easy to reproduce and straightforward to drop into other GRPO-style RL algorithms, since only the reward function needs to change.

Critical weaknesses
  1. Length-aware RL rewards for reducing overthinking seem to be a well-studied problem, and the proposed reward resembles existing methods. LAPO (Wu et al., 2025) also uses the median length of correct rollouts as the target and applies length rewards only to correct responses. The paper should compare R3 reward against LAPO or similar adaptive-target methods (e.g., ALP, GRPO-λ).

  2. The paper’s claim is that a threshold-conditional reward outperforms the length-penalty family, but the only matched baseline (R1) is correctness-only, while LASER-DE and DECS are released checkpoints trained with different data. A standard length-penalized reward trained under identical data, steps, and backbone is needed to attribute R3’s gains to the reward design.

  3. Training and most evaluation focus on math reasoning. The authors could evaluate on code (e.g., LiveCodeBench), scientific reasoning (e.g., GPQA), and open-ended or agentic tasks, and ideally include non-math data in training to show that this training method generalizes.

  4. All backbones are early-2025 DeepSeek-R1 distilled models (1.5B–8B) without any RL stage; it is unclear whether the findings hold for newer reasoning models such as the Qwen3 series.

  5. The “termination calibration” claim is not well supported. The reward only checks whether the final answer is correct and whether the output is short enough; it never sees when the model actually reached its answer. The evidence for “learning when to stop” is indirect, and much of it could also be explained by the model simply producing shorter outputs.

Questions for authors

See Weakness.

5 comments
AuthorsOpenReview ↗

Response to Reviewer nEU3 (Part 1/3)

We thank Reviewer nEU3 for the expert review, and for noting that the completion/accuracy decomposition "gives a clean diagnostic that other work could reuse" and that R3 is "easy to reproduce". We address each weakness below with new evidence from the rebuttal period. All new analyses use the Table 1 checkpoints, prompts, problems and extractor (Qwen-7B, n = 300; MATH 299; GPQA-Diamond 198); where an analysis changes decoding (forced readout), we say so.

W1: Relation to LAPO, ALP and GRPO-λ

We agree these must be discussed and add them to §4 and App. B. All three differ from R3 in reward shape; for ALP and GRPO-λ the difference is formal.

Length signal on correct responsesStatisticCorrect answers below the target / threshold
LAPO stage 11 + 0.7 inside band [P30, P70]; bonus decays linearly to 0 within 100 tokens outside, on both sidesper problem, from the correct rollouts among that step's 8penalised (below the band)
LAPO stage 21 + β·exp(−(ℓ − n)² / 2σ²), with "I will answer the question with n tokens" placed after <think> in trainingper-problem n from stage 1, refined in stage 2penalised (symmetric around n)
ALP1[correct] − β·N·max(p_solved, 1/K), described as a per-token cost; applied to all rolloutsper-prompt solve rateno flat region: every token priced
GRPO-λ1 − α·σ(z-scored length), α = 0.2, top-20% most-accurate groups onlyper groupno flat region: strictly decreasing
R31.0 if ℓ ≤ τ, 0.3 otherwiseone policy-level τ (1.2 × median of last 5,000 correct rollouts)flat: all equal credit

LAPO: a per-problem target vs. a ceiling. LAPO learns a target length per problem and, in both stages, reduces credit for correct answers below it. R3 sets only a ceiling: every correct answer within τ gets equal credit. Under a hard budget the only failure is running too long, so a ceiling targets exactly that and never pulls a short solution longer. LAPO also needs per-problem memory across two stages and a length statement after <think>; R3 is one scalar in one stage, with no prompt or trace change. LAPO's reward is not length-monotone, so it lies outside Proposition M.1 (Remark M.2). LAPO trains only 1.5B models from different bases and data (4,096-token cap) and evaluates at 32k tokens, so a grid comparison would confound base, data and reward (the reviewer's W2); we compare designs instead.

ALP and GRPO-λ fall under Proposition M.1(i). ALP prices each token by the solve rate, so for a fixed prompt it is the linear R_γ family with a prompt-dependent γ and cannot express a stopping boundary (its β = 10⁻⁷ keeps correctness dominance in practice, so the missing boundary is the difference). GRPO-λ is the Arora & Zanette sigmoid penalty (α = 0.2), strictly decreasing in length, applied only to the most accurate groups; W2 evaluates the released Arora & Zanette α = 0.2 checkpoint on our backbone.

AuthorsOpenReview ↗

Response to Reviewer nEU3 (Part 2/3)

W2: A matched length-penalty arm

We trained it (new). R2 uses the same backbone, data (GSM8K + MATH), 450 steps, G = 8, learning rate and 200-step correctness-only warm-up as R1 and R3. Only the reward after step 200 differs: a correct answer earns 1 − 0.7·ℓ/4096 and a wrong one 0. This is the standard linear penalty, with the same reward range [0.3, 1] as R3, so only the shape differs (re-implemented on a current TRL release for our hardware). Evaluated under the Table 1 protocol:

MATH 300 / 500 / 1k / 2k / 4kGSM8K 300 / 500 / 1k / 2k / 4ktokens @4k (MATH / GSM8K)Δtok 2k→4k (MATH)
R2 (linear penalty)21.4 / 30.4 / 49.2 / 61.5 / 77.680.3 / 82.3 / 82.3 / 82.3 / 82.31,889 / 130+45.7%
R318.4 / 39.5 / 71.2 / 85.0 / 90.348.7 / 85.3 / 92.0 / 94.3 / 94.3738 / 353+6.0%

In paired McNemar tests, R3 is significantly more accurate on MATH at 500–4k (p ≤ 0.005) and on GSM8K at 1k–4k (p ≤ 3×10⁻⁵). R2 is better only at GSM8K@300 (p < 10⁻¹⁷); MATH@300 is not significant. On ARC, R3 is better at 300–500 (p < 10⁻²²) and comparable at 1k–4k.

This is what Proposition M.1 predicts. A strictly decreasing reward has no stopping boundary, so it compresses wherever it can rather than where the answer is ready:

  • GSM8K: R2 compresses to 130 tokens and its accuracy plateaus at 82.3 at every budget. That is 12 points below R3 on the same problems (p < 10⁻⁶): brevity bought with correctness.
  • MATH: R2 barely compresses (1,889 tokens; 21.1% truncated at 4k vs. 0.7% for R3, Table 13). It keeps reasoning for 552 tokens after first stating its answer, vs. 22 for R3 (the W5 metric; an answer statement is detected in 54% of R2's and 80% of R3's correct traces).

Supporting evidence. Under the identical recipe on the released DECS checkpoint (App. J), R3 lifts GSM8K@300 from 33.7 to 60.0 and MATH@4k from 87.6 to 88.3 at 957 vs. 1,515 tokens, while R1 does not. On the same backbone, R3 is also significantly more accurate than the released Arora & Zanette (α = 0.2) checkpoint on MATH at 1k–4k (p ≤ 0.007) with 0.46× the tokens.

AuthorsOpenReview ↗

Response to Reviewer nEU3 (Part 3/3)

W3: Generalisation beyond math

Training is math-only, so every row below is zero-shot transfer:

Never trained onBaseDECSR3
HumanEval pass@1 @2k / @4k (App. K)50.6 / 58.554.3 / 57.964.6 / 67.1
HumanEval truncation @4k29.3%13.4%7.3%
ARC-Challenge @300 / 1k / 4k (Table 1; in no method's corpus, App. D.1)26.0 / 49.0 / 63.326.3 / 57.0 / 70.758.0 / 79.7 / 87.7
GPQA-Diamond @1k / @4k (new, n = 198)1.5 / 24.78.6 / 41.417.2 / 36.9

On GPQA-Diamond, R3 leads every model at 1k (vs. Base, R1, DECS, LASER-DE: +8.6 to +15.7 pp, paired McNemar p ≤ 0.0023). At 4k it remains well above Base and R1 (+12.1, +8.6 pp; p ≤ 0.016) with the fewest tokens (2,497 mean vs. 2,700–3,595), while DECS and LASER-DE are ahead (+4.5 and +6.6 pp; not significant, p = 0.19 and 0.053). Non-math training data is future work. The reward needs only a verifier, so it transfers unchanged.

W4: Newer backbones (Qwen3)

This is a fair limitation. We use DeepSeek-R1-Distill models because every comparison method (DECS, LASER-DE, DLER, LC-R1, Arora & Zanette, AdaptThink) releases checkpoints on them, which allows like-for-like evaluation. Across three backbones from two families (Qwen-1.5B, Qwen-7B, Llama-8B), the ordering R3 > R1 > Base replicates in 8 of 9 backbone–dataset cells (App. I). Qwen3 already includes RL stages and native thinking-mode controls, so whether it still fails to terminate under hard budgets is open, and we state it as such.

W5: Termination calibration vs. shorter outputs

Correct: the reward never sees when the answer is reached. Our claim is about observed behaviour, and we now measure it directly (new). We keep only correct generations that ended on their own before 4k, so each stopping point is observed rather than censored by the cap. Inside the reasoning block (before </think>), we find where the final answer is first stated and count the reasoning tokens that follow.

MATH@4k (median)BaseR1DECSLASER-DER2 (linear, W2)R3
Reasoning tokens1,2811,127741771656230
Reasoning after the answer is first stated66141811916955222
Same, as % of reasoning58%44%19%26%45%9%

GSM8K: 257 / 159 / 56 / 122 / 18 tokens (Base / R1 / DECS / LASER-DE / R3). R2 emits no reasoning block on GSM8K (median generation 112 tokens), so the metric does not apply there.

  • "Just shorter". At matched reasoning length, R3 still continues least: 6% of its reasoning vs. 22–48% at 300–600 tokens, and 13% vs. 19–59% at 600–1,200.
  • Same data, steps and optimiser. RL alone reduces post-answer reasoning modestly (Base 661 → R1 418 tokens); R3, which differs from R1 only in the termination term, reduces it to 22.
  • Length penalty vs. termination. R2 shares R3's data, steps and reward range but prices length linearly. Its reasoning is shorter than R1's (656 vs. 1,127 tokens), yet it still continues for 552 tokens (45%) after stating the answer. A length penalty shortens traces; it does not teach the model to stop at the answer.
  • Forced-readout control. With an s1-style forced readout for every model (cue and answer within the budget), R3 still beats R1 by +14.7 / +21.7 / +14.0 pp on MATH at 300 / 500 / 1k and +36.3 pp on GSM8K@300 (all p < 10⁻⁵). Giving R1 the same answer-emission mechanism does not close the gap, so R3's advantage is not only answer emission.

This is observed behaviour, measured on correct, completed traces with a numeric final answer where an answer statement is detected (96–98% of baseline traces, 80% of R3's, 54% of R2's). It does not show that the model internally detects readiness.

Revisions (we will upload the revised PDF with all of these within the next couple of days)

  1. §4 and App. B: LAPO, ALP and GRPO-λ, with the reward comparison above.
  2. New appendix: the matched linear-penalty arm (R2), the uncensored readiness analysis and forced readout, referenced from §3.3.
  3. Table 1: four added same-backbone baselines; App. K: GPQA-Diamond.
  4. Limitations: math-only training and Qwen3 as future work.
Reviewer_hjyvOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

RL increases response length. Published solutions assign a variety of length penalties during training and/or during inference, including linear per token, accuracy-constrained per-token, judge-based credit assignment per token, and step-wise depending on the accuracy at each length. This manuscript proposes a stepwise length penalty, with just one key innovation: the step threshold depends on the median length of recent correct responses, thus the threshold decreases as the model parameters converge.

Critical strengths
  • The method is simple and intuitively reasonable, in that all correct responses get a positive reward, while short correct responses get more reward.
  • The stepwise length penalty neatly sidesteps the problems with linear and gated length penalties identified by the DECS paper (Jiang et al., 2026).
  • The adaptive threshold neatly solves the hyperparameter tuning problem.
  • Ablation studies are very well done. Experimentally, median rollout length and the adaptive threshold are shown to converge almost monotonically, from the point at which the step function is turned on, to a nonzero threshold that can be reasonably interpreted as the smallest length at which most answers are correct. The R1 internal baseline convincingly makes the point that RL increases rollout length, and the step function decreases it.
  • Mean length of rollouts converges to values much smaller than the token budget, confirming the authors' claim that the reward function is training the model to generate a distribution in which most responses are short and only difficult problems merit a long answer
Critical weaknesses
  • Comparison to only two external and two internal baselines feels a little limited, considering the number of papers addressing this problem.
  • Hyperparameters are arbitrary, not supported by any theory or ablations.
Questions for authors

See "weaknesses"

1 comment
AuthorsOpenReview ↗

Response to Reviewer hjyv

We thank Reviewer hjyv for the positive assessment, and for noting that "ablation studies are very well done", that the R1 baseline "convincingly makes the point that RL increases rollout length", and that "the adaptive threshold neatly solves the hyperparameter tuning problem." We address both weaknesses below with new evidence from the rebuttal period.

W1: Only two external baselines

We added four further released checkpoints on the same backbone (DeepSeek-R1-Distill-Qwen-7B), all evaluated under the Table 1 protocol: same prompts, greedy decoding, extractor and problems. Table 1 now compares six external methods.

MethodMATH 300 / 1k / 4kGSM8K 300 / 1k / 4kARC 300 / 1k / 4kMATH tok@4kΔtok 2k→4k (MATH)
DLER9.7 / 65.6 / 91.036.3 / 89.3 / 91.019.7 / 83.3 / 84.01,180+9.9%
LC-R123.1 / 63.2 / 78.957.0 / 83.0 / 83.030.3 / 82.0 / 82.3940+20.3%
Arora & Zanette (α = 0.2)17.1 / 57.2 / 83.659.3 / 89.3 / 91.327.0 / 49.7 / 60.01,621+31.2%
AdaptThink8.0 / 36.8 / 82.928.7 / 76.3 / 81.32.7 / 78.0 / 86.02,132+33.5%
R3 (ours)18.4 / 71.2 / 90.348.7 / 92.0 / 94.358.0 / 79.7 / 87.7738+6.0%
  • Significance. Across 60 exact paired McNemar tests (4 methods × 3 datasets × 5 budgets), R3 is significantly better in 35, worse in 4 and not significantly different in 21.
  • Where baselines win. The four losses fall at one budget per dataset: GSM8K@300 against LC-R1 and Arora & Zanette, and ARC@1k against DLER and LC-R1. On MATH, R3 is never significantly worse than any added baseline.
  • Tokens and coupling. At 4k, R3 generates the fewest tokens of all seven methods on MATH (738 vs. 940–2,132) and ARC (331 vs. 390–1,620). On GSM8K, LC-R1 is shorter (282 vs. 353 tokens) but 11.3 points less accurate (83.0 vs. 94.3). R3 also has the lowest budget coupling on MATH: its output barely grows when the budget doubles.

We also ran two further comparisons:

  • Inference-time forced readout for every model (s1-style, cue and answer within the budget). R3 still beats R1 under the same readout (same data and steps) by +14.7 / +21.7 / +14.0 pp on MATH at 300 / 500 / 1k (p < 10⁻⁵).
  • An uncensored readiness test. On MATH, R3 continues reasoning for a median of 22 tokens after first stating its final answer, vs. 418 for R1, 119 for DECS and 169 for LASER-DE.

W2: Hyperparameters lack theory or ablation

What supports each constant:

ConstantSupport
Factor 1.2App. L: recomputing each logged correct rollout's tier under factors 1.0–2.0 flips only 4.7% of assignments between 1.1 and 1.3, so 1.2 sits in a locally stable region. At 2.0, 93.3% of correct rollouts already receive full credit and the threshold stops discriminating length.
Partial credit 0.3Theory (Prop. M.1(ii), Remark M.1): any value in (0, 1) yields the same optimal set (correct answers within τ) and keeps correctness dominance; only the ordering 1 > 0.3 > 0 matters. Only the endpoints fail: 1 collapses to R1 (no boundary), and 0 lets a verbose correct answer tie a wrong one.
W = 5,000; floor 100; warm-up 200Design: the window smooths τ across steps, the floor prevents collapse on trivially short problems, and the warm-up lets correctness rise before the brake engages (τ binds at 1,631 tokens at activation, §3.2).

As the reviewer notes, the adaptive threshold removes the constant that length penalties tune most, the target length or penalty weight. The remaining constants are either ordinal (partial credit) or sit in a locally stable range (the factor).

Revisions (we will upload the revised PDF with all of these within the next couple of days)

  1. Table 1 and App. D.6: the four added baselines with full five-budget grids. New appendices: forced readout and readiness analyses.
  2. App. L: the hyperparameter rationale above.
Reviewer_cBTEOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies failures of reasoning models to produce usable answers within hard generation budgets. It decomposes budgeted accuracy into completion frequency and accuracy conditional on completion, then introduces R3, which gives partial credit to longer correct outputs and full credit to shorter correct outputs using an adaptive threshold. Experiments compare R3 with the base model and a same-data correctness-only RL control across generation budgets, backbones, and sampling settings. The results show substantial improvements in budgeted accuracy and output efficiency, which the paper interprets as improved termination calibration rather than increased reasoning capability.

Critical strengths
  1. The paper addresses a consequential deployment problem and evaluates accuracy together with generation length, latency, and completion behavior. These measurements expose failures that generous-budget accuracy can obscure. The same-data correctness-only control is particularly useful for assessing the effect of the reward modification rather than additional RL training alone.

  2. The practical gains are large: at a 4,000-token budget on MATH, R3 reaches 90.3% accuracy versus 82.6% for the correctness-only control and 75.0% for the base model. Mean generation length also falls from the base model's 2,358 tokens to 738, and additional backbone and sampling evaluations broaden the evidence. These benefits remain valuable even if the proposed mechanistic interpretation is narrowed.

Critical weaknesses
  1. The completion definition and the truncated-output analysis are not reconciled. The stated classifier treats a complete boxed answer as terminated, yet the Strict scan reports 8.5% complete correct boxed answers within a population described as truncated by that classifier. The disclosed use of legacy flags may explain a different implemented population, but it does not resolve the stated restriction. This matters because the population definition underlies the evidence for answers that were computed but never expressed, although it does not by itself invalidate the headline accuracy gains.

  2. The stronger mechanism and capability-preservation claims exceed the evidence. When greedy decoding differs only in its cap, observed length is censored by that cap, so reduced budget–length coupling can follow from shorter traces without independently establishing answer-readiness calibration. Conditional accuracy also compares different completed subsets, while the AIME accuracy difference of −2.9 percentage points has an interval of [−11.7, 5.8], which does not establish equivalence. These limitations leave me unconvinced by the central explanation despite the practical improvements.

Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer cBTE (Part 1/2)

We thank Reviewer cBTE for a precise review, and for recognising that "the same-data correctness-only control is particularly useful" and that the gains "remain valuable even if the proposed mechanistic interpretation is narrowed." We agree the mechanism deserves direct evidence and provide it below, from the rebuttal period. All new analyses use the Table 1 checkpoints, prompts, problems and extractor (Qwen-7B, n = 300; MATH 299); where an analysis changes decoding (forced readout) or scoring (completed-only), we say so.

W1: Completion definition vs. the Strict scan

The reviewer is right. The error is in how the classifier is described, not in which population the scans used. The truncation flag stored for every generation in the grid is a length rule: a generation is flagged as truncated if ℓ ≥ B − 5, i.e. it ends within five tokens of the cap (e.g., Table 13, R3 on MATH: 90.3 / 62.2 / 17.7 / 3.0 / 0.7% at 300–4k). The latent-answer scans of App. G.3 run on this length-flagged population. Under Algorithm 2's box test the Strict scan would be 0% by construction, so the App. D.2 / G.1 statement that re-scanning with Algorithm 2 reproduces the reported rates is incorrect, and we remove it. The 8.5% are generations that contain a correct boxed answer yet continue to within five tokens of the cap: the failure to stop that the paper studies.

Forced-readout experiment (new; changes decoding). We preserve the original answer only for generations that emit EOS by B − 32 (generation finish reason, not the five-token length flag); a response that would finish naturally after that cutoff still receives the forced readout. For every remaining generation, including those that hit the cap, we keep the first B − 32 tokens, append </think> **Final Answer** \boxed{ and let the model complete the answer; the cue and readout fit within the 32 reserved tokens, so the total stays ≤ B (s1-style). The forced readout recovers additional correct answers from the retained prefix: on the base model, GSM8K@300 rises from 41.0 to 76.7 and MATH@1k from 31.8 to 55.5. Because the readout generates new tokens, this does not show that each answer was already settled at the cutoff; it shows that the prefix supports a correct answer that the model did not emit. R3's advantage is not just answer emission: R3 + forced readout beats R1 + forced readout (same data, steps and optimiser; only the termination term differs) in all five comparisons shown (exact paired McNemar, all p < 10⁻⁵).

R3 vs. R1, both with forced readoutMATH@300MATH@500MATH@1kGSM8K@300GSM8K@500
Δ accuracy (pp)+14.7+21.7+14.0+36.3+10.3

No credit for cut-off text (new; changes scoring). We rescored accuracy so that cut-off responses (ℓ ≥ B) count as incorrect, with all evaluated problems kept in the denominator. R3's margin over base and R1 stays positive in all 20 in-domain comparisons against Base and R1 and grows in 18 (e.g., MATH@1k vs. R1: +30.1 → +47.5 pp).

AuthorsOpenReview ↗

Response to Reviewer cBTE (Part 2/2)

(Part 2 of 2)

W2: Censoring: shorter traces vs. answer-readiness

We agree that budget–length coupling alone cannot separate "stops when the answer is ready" from "writes less". We add a more direct measurement of observed behaviour that is not censored by the cap (new). We keep correct generations that ended on their own before B = 4,000, so each stopping point is observed rather than imposed by the cap, and whose final answer is a single number, because the detector needs an exact numeric match (this keeps 188 of R3's 267 correct completed MATH traces and 152 of Base's 215). Inside the reasoning block (text before </think>), we find where the final answer is first stated (the first occurrence of the final value after "=", "is", "answer", …) and count the reasoning tokens that follow.

MATH@4k (median)BaseR1DECSLASER-DER3
Correct, completed traces with a numeric answer (n)152166181181188
Answer statement detected98.0%97.6%96.7%96.7%80.3%
Reasoning tokens1,2811,127741771230
Reasoning after the answer is first stated66141811916922
Same, as % of reasoning58%44%19%26%9%
Traces with > 300 tokens after the answer79%71%20%29%6%

Medians are over traces where an answer statement is detected. GSM8K shows the same ordering: 257 / 159 / 56 / 122 / 18 tokens. (R3's reasoning block is short. Its full median generation, including the written solution after </think>, is 576 tokens; App. O.)

  • Matched length. Within each reasoning-length bin, R3 continues least: at 300–600 tokens, 6% of its reasoning vs. 22–48% for the other models; at 600–1,200, 13% vs. 19–59%.
  • Matched training. RL training alone reduces post-answer reasoning modestly (Base 661 → R1 418 tokens). R3, which differs from R1 only in the termination term, reduces it to 22.
  • Robustness. Excluding small-integer answers, where an early value could match by coincidence, leaves the ordering unchanged (R3 20, R1 357, Base 633 tokens).

We present this as observed behaviour: R3 continues far less after stating an answer. It does not show that the model internally detects readiness.

W3: Conditional accuracy compares different completed subsets

Agreed. We now compare per-completion accuracy on the same problems: those both models complete before B, with paired McNemar tests (new).

R3 vs. (same problems)MATH@2kMATH@4kGSM8K@2kGSM8K@4k
Base94.4 vs 95.8 (n=142)94.6 vs 96.4 (n=222)96.3 vs 94.2 (n=242)96.4 vs 94.0 (n=250)
R194.3 vs 95.4 (n=175)93.3 vs 92.9 (n=253)95.2 vs 95.5 (n=289)95.2 vs 94.9 (n=293)

We ran 24 in-domain comparisons: MATH and GSM8K × budgets 1k / 2k / 4k × Base, R1, DECS, LASER-DE. Differences range from −3.0 to +3.7 pp. Paired bootstrap 95% intervals include zero in 22 of 24; the other two favour R3 (DECS on GSM8K@2k and @4k). The lowest bound is −7.6 pp (MATH@1k vs. R1, n = 66). At ≤ 500 tokens too few problems are completed by both models to test (n ≤ 15 on MATH). These results concern the problems both models complete.

W4: AIME does not establish equivalence

Agreed: [−11.7, +5.8] spans zero in both directions. Throughout the paper, we replace "parity" and "matches" for AIME with "no significant difference detected (n = 30; underpowered for equivalence)". We withdraw the equivalence claim. What our data support are accuracy comparisons under the tested budgets: the matched-subset analysis above (up to 293 problems per comparison) and MATH-500 sampled pass@1–8, where R3 exceeds Base with paired bootstrap intervals excluding zero (Table 2, App. E). Broader capability preservation remains open, and we say so.

Revisions (we will upload the revised PDF with all of these within the next couple of days)

  1. G.1, G.3 and D.2 now document the historical near-cap flag (ℓ ≥ B − 5), and we remove the statement about rescanning with Algorithm 2. The stricter accuracy analysis separately uses ℓ ≥ B.
  2. New appendix: forced readout for all five models.
  3. New appendix: the uncensored readiness analysis, referenced from §3.3.
  4. §3.4 and App. D.5: matched-subset accuracy with paired intervals replaces the unmatched A_c comparison; AIME equivalence claim withdrawn; capability claims limited to the tested budgets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.