acceptodds
Under review as a conference paper at ICLR 2027

Token Caps, Not Probes: Managing Non-Termination in Distilled Reasoning Models

Abstract

We study how thinking-token budgets relate to accuracy in DeepSeek-R1-Distill-Qwen-7B on GSM8K, MATH-500, and AIME\@. On GSM8K, accuracy peaks at a very small forced thinking budget, while on MATH-500 short budgets remain well below uncapped accuracy. On AIME, generations divide by whether the model ends its reasoning within a 10,000-token ceiling. Generations that terminate are usually correct; nearly half never terminate and consume most of the inference compute. Whether a problem terminates changes for about a quarter of problems under temperature sampling or across hardware. Forcing termination at a fixed cap requires no predictor and traces a compute–accuracy frontier: at 4,000 tokens it saves about 39% of inference compute with accuracy no lower than generating to the 10,000-token ceiling. Linear probes on early hidden states do not predict non-termination at the primary evaluation point; one position in a nine-position sweep (token 125) is at the threshold for significance after correction for multiple testing; replication is needed. We also show that the correction, valid for unsupervised probes, inflates the performance of supervised ones and produced a spurious positive result in our initial analysis. These results suggest that token-count caps, rather than activation probes, are the practical lever for reducing inference cost in this setting.

Primary area:
interpretability and explainable AI
Keywords:
chain-of-thought reasoning, reasoning non-convergence, inference-time compute, budget-forced decoding, linear probing, distilled language models, DeepSeek-R1, compute-accuracy tradeoff

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ygWwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a series of analyses on the termination of DeepSeek-R1-Distill-Qwen-7B and 1.5B on three math benchmarks: GSM8K, MATH-500, and AIME 1983–2024. They showed that short thinking budgets can improve accuracy on GSM8K but reduce accuracy on MATH-500, and that termination on AIME is sensitive to decoding settings and hardware. They also showed that a 4,000-token cap reduces inference compute by approximately 39% without lowering accuracy compared with a 10,000-token ceiling. They trained probes across model layers and token positions to predict thinking termination on AIME, and showed that the probes were largely unable to do so. They concluded that setting token caps is enough to reduce inference costs, while probing for termination is of limited use in this setting.

Critical strengths
  1. The paper describes the experimental configurations in detail, supporting the reproduction of their results. This is a strength, but it does not outweigh my concerns about the paper’s contributions and whether its conclusions are well supported.
Critical weaknesses
  1. The paper’s contribution is not obvious. Sections 4 - 6 largely present trivial results or reinforce well-known findings, while Section 7, which most directly addresses the main claim, is short and not convincing. This leaves the paper without a clear, substantial contribution to justify acceptance.
  2. The probe training setup is problematic. With n << d, the training data may be insufficient to learn a meaningful probe, so the conclusions in Section 7 are not well supported. The reported failure could reflect limitations of the training setup rather than the absence of predictive information, undermining the paper’s main conclusion about probing.
  3. There is no direct comparison between token caps and probes, despite the main claim being that token caps are better than probes. Instead, the authors present a series of analyses that are not directly comparable. Without a comparison under matched compute budgets or accuracy targets, the paper does not establish the claimed advantage of token caps, which is central to my overall assessment.
Questions for authors
  1. With n << d, how do you distinguish insufficient probe training from a lack of predictive information in the hidden states?
  2. What evidence supports the claim that token caps are better than probes, given the lack of a direct comparison?
2 comments

Thank you. Your central point, that we claimed caps beat probes without comparing them, is correct. We added the comparison.

Direct comparison at matched compute (new Sec. 7.5). For each problem we have the outcome and token count under every cap and under an early forced exit at ~150 tokens. A probe flags likely non-terminating generations at token 150 (simulated at a given AUC, since our real probe was trained on a different sample); flagged runs exit, the rest run under cap C. Best accuracy at a compute saving at least that of each cap:

savedcap onlyAUC 0.65AUC 0.85AUC 0.999
59.7%31.034.639.045.6
48.6%40.541.043.745.9
39.0%46.546.546.948.3
30.5%48.548.549.749.9
22.9%52.052.052.052.0

At ≤40% savings, even a near-perfect probe adds ≤2 points, because forced exits at the cap already rescue looping generations (16% at 4k) and an exit at token 150 rescues 3%. Probes matter only at aggressive budgets, and only at AUC ≥0.85. This is the evidence for the title claim, and it does not depend on our probe failing.

n≪d. Agreed: our original null alone cannot separate "insufficient training" from "no information", and we no longer claim it does. New analyses: (1) PCA-reduced probes (k=2–50, fit within folds) and strong L2 (C=10⁻⁴) at all nine positions: best 0.69–0.70 post hoc, 0.61 under nested CV with the configuration selected inside training folds (p=0.016, 500 permutations), i.e. a weak signal; (2) a problem-level ceiling of AUC 0.83–0.87, so information exists, and a better-trained probe may find it. Section 7 is now framed as a bounded result for early-activation linear probes at n=200.

Contribution. We restated it: on hard problems, compute waste in R1-distilled models is dominated by generations that never terminate (65% of tokens), and termination is unstable (23–24% item flips, replicated on a second sample). Forced exits rescue part of those generations, which bounds what any early predictor can add over a cap. We now present Sec. 4 as supporting context rather than as a contribution.

Thank you for the response and revisions.

  1. The matched-compute comparison is a useful addition, but the reliance on simulated probe scores limits what it establishes about how a trained probe would perform against a fixed cap.
  2. The new analyses make the conclusion better supported, though the narrower claim is less significant.
  3. The revision makes the contribution clearer, but I still find it limited given the narrow experimental setting and the reliance on simulation for the central comparison.

The revised manuscript is better, but the reliance on simulation for the central comparison and the limited contribution still fall short of the standard for ICLR, so I retain my score of 1 (clear rejection).

Reviewer_c9o5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how the thinking budget affects accuracy for DeepSeek-R1-Distill-Qwen-7B/1.5B on GSM8K, MATH-500, and 200 AIME problems. The authors use a logits processor to force the end-of-thinking token at a set budget and compare small budgets against a 10k-token ceiling. On AIME, generations that finish thinking within 10k tokens are usually right, while the rest never emit an answer and consume about two-thirds of total tokens. Around a quarter of problems also flip termination status across hardware or temperature runs. Based on this, the paper argues for a fixed cap around 4k tokens to cut cost and reports that linear probes on early hidden states do not reliably predict termination.

Critical strengths

(1) The paper frames test-time compute waste in distilled reasoning models around whether generation terminates at all, rather than how long a finished trace should be. On AIME, traces that close the thinking block within 10k tokens get over 80% accuracy, while the rest hit the ceiling and eat up roughly two-thirds of total tokens. In Sec 5, the authors show that about a quarter of problems switch between terminating and non-terminating across two greedy runs on different machines or under T=0.6 sampling. That means part of the looping comes from the run itself rather than question difficulty, which is a useful empirical point for early exit or routing.

(2) The reporting is transparent and easy to audit. The authors check their AIME findings across multiple seeds, a second machine, T=0.6 sampling, and the post-cutoff AIME 2025 set. They also include a power analysis for their negative probe result and openly retract an earlier positive probe number after spotting an evaluation error in Sec 7.2.

Critical weaknesses

(1) The main claim is that a 4k cap saves ~39% compute on AIME at no accuracy cost, but this rests on an uneven comparison with the uncapped baseline. Uncapped runs that hit the 10k ceiling are scored zero with no answer phase, while the capped policy forces </think> and gives 1k extra tokens to answer. Since forced exits recover a right answer 20-30% of the time and the converged half is unchanged, running C=10k with the same 1k answer window should beat the 44.5% baseline, yet that point is never tested. The baseline itself is also weak. All main runs use greedy decoding in fp16, which the model card advises against and which is known to cause run-to-run swings on this checkpoint [1]. In Table 1, uncapped MATH-500 is only ~61% vs ~93% in the official report, so it's hard to tell whether caps beat the model or just fix a handicapped evaluation setup.

(2) The second half of the title claims that activation probes cannot predict non-termination, yet the probing setup is too small and noisy to support a broad negative claim. The power check in Sec 7.4 shows that n=200 only reaches 80% power around 0.65 AUC, and token 125 in App. D actually hits ~0.65 right at the Bonferroni cutoff. The probe also reads a single fixed token index to predict a greedy label that flips ~24% of the time across machines. Prior work on the same R1-Distill-Qwen models showed that simple probes do work for early exit when trained on ~1000 examples at reasoning chunk boundaries eg after intermediate steps, even if their target was answer correctness rather than termination [2]. Testing one linear classifier at fixed token offsets on 200 noisy labels is not enough to rule out probing.

ref: [1] Yuan et al., Give Me FP32 or Give Me Death? Challenges and Solutions for Reproducible Reasoning, arXiv:2506.09501, 2025. [2] Zhang et al., Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification, arXiv:2504.05419, 2025.

Questions for authors

(1) What happens on AIME if you run C=10k with the same 1k forced-answer window used in Table 6, and how does the cap curve look under T=0.6 majority voting? Also, how does a fixed cap compare against a training-free confidence exit like DEER [3], which needs no probe and cuts 30-40% of tokens on R1-distill models without losing accuracy?

(2) In Table 1 what fraction of uncapped GSM8K and MATH-500 runs hit the 10k ceiling without emitting an answer, and why is uncapped MATH-500 ~61% instead of the ~93% reported by DeepSeek?

(3) What is the probe AUC if you train and test only on the ~150 problems in Table 5 with stable labels between greedy and temperature runs, or if you include the rest of the 933 AIME problems? Also in section 5.1, the cross-node run changes the machine, attention kernel, and logits processor at once, so did you run a seperate check isolating which of those caused the 24% flip rate?

ref: [3] Yang et al., Dynamic Early Exit in Reasoning Models, arXiv:2504.15895, 2025.

1 comment

Thank you for a detailed and constructive review. Each point led to a change.

W1/Q1: uneven baseline. You are right, and this changes our headline. Per problem, forced exits rescue 10/93 (C=2k) to 22/93 (C=6k) of the never-terminating generations. A C=10k run with the same window keeps the 89 correct terminating problems and adds such rescues, so it beats C=4k (46.5%) if more than 4/93 (4.3%) are rescued. Bracketing by the observed rates gives 49.5–55.5%. We removed "no accuracy cost". The revised fixed-cap table (Table 5) reports savings against both baselines: C=4k saves 42.4% at a projected 3–9 points; C=6k (52.0%) is within the projected range at 27.2% savings. We have a run script for C=10k that uses the identical code path as the fixed-cap runs and hash-verified problem alignment, and the measured value will replace the projection. We could not run it within the discussion period.

Cap selected on the evaluation set: we added held-out selection (1,000 random 50/50 splits). Choosing the most accurate cap on one half picks C=6k in 86% of splits and, on the held-out half, gains +7.0 points over uncapped-as-scored (95% interval +2 to +12) at 22% savings.

T=0.6 cap curve and DEER: not run yet. We list both as the primary missing experiments. DEER is now discussed: it targets how long a terminating trace should be, which complements non-termination.

Baseline strength / fp16. We agree, and we now state that all main results use greedy fp16 decoding with a 10k ceiling, unlike the model card's recommended configuration (T=0.6, 32k). We cite Yuan et al. (2025), who document the same kind of cross-hardware nondeterminism under reduced-precision greedy decoding.

Q2: ceiling hits and the MATH-500 gap. Uncapped runs that hit the ceiling without an answer: GSM8K 1/500 (0.2%), MATH-500 50/500 (10.0%), all scored wrong. That explains at most 10 of the ~31 points. Prompted by your question, we re-audited the scorer and found the main cause: our answer extractor stopped at the first closing brace, so all 103 MATH-500 problems whose answer contains braces (e.g. fractions) were scored wrong at every budget. On the 325 problems with numeric answers, uncapped accuracy is 88.6% (96.0% among terminating runs), far closer to the reported 92.8%. Table 1 now reports this subset (budget-forced 58.5/67.4/69.2/74.8/74.5% vs. 88.6% uncapped), so the conclusion that short budgets stay well below uncapped on MATH-500 still holds. GSM8K may also be affected (comma-formatted large answers), so we softened the GSM8K claim and will re-score both with a robust grader. AIME answers are integers below 1000 and are unaffected; all AIME results stand.

W2/Q3: probe scope. Agreed; we narrowed the claim throughout. New evidence: (1) predicting one run's termination from other runs of the same problem gives AUC 0.83–0.87, so the label is learnable in principle and our probe is far below that ceiling; (2) low-dimensional PCA probes reach up to 0.70 at the best post hoc configuration and 0.61 under nested CV (p=0.016, 500 permutations), i.e. a weak signal that we now report as such; (3) the matched-compute analysis shows that even a probe at AUC 0.85 would add 0.4 points over a cap at 39% savings. So the practical conclusion does not rest on our probe failing.

Stable-label subset: on the 149 problems whose label agrees between the probe's greedy run and the T=0.6 majority (111 terminating, 38 not), the pre-specified token-150 probe stays at chance (AUC 0.512, p=0.50). The best position, token 175 (also second-best on the full set), reaches AUC 0.72 (nominal p=0.01; PCA-10 variant 0.71, p≤0.005), which does not survive correction for 9 positions × 2 probe types. This is consistent with a weak signal that label noise partly masks, and token 175 is the position to test in a replication.

Flip isolation: correct, the cross-node comparison changes machine, attention backend and logits processor together. We now say 24.0% is their combined effect; a factorial check is planned.

Reviewer_QR3aOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper checks the effect of imposing various compute budget caps on two DeepSeek instances and three mathematics benchmarks. The main finding is that early termination of token sequence generation during inference can lead to sizable computational savings and sometimes improved benchmark accuracy. This seems an intuitively correct result but difficult to generalize across models and benchmarks.

Critical strengths

None

Critical weaknesses

(Broad criticism):-The authors’ framing consistently contributes the incorrect but dangerously prevalent idea that “reasoning” or “thinking” in any meaningful sense occurs within large transformer models (as opposed to statistical, purely correlative next-probable-token generation, which is what, mathematically and by the construction of these models, is actually happening). I understand that this is common in the field but it is wrong and should be corrected. This substantially reduces my confidence in the authors’ technical understanding of these models and undermines my overall opinion of the paper.

The contributions from the paper seem very limited to me for two reasons: First, no theory is provided to explain the results and direct future optimization efforts. Without this, and especially as only a small subclass of models and benchmark problems were tested, I’m not sure I gained any generalizable insight into the (admittedly very real) problem of token cost tradeoffs vs. model performance. In particular, as only accuracy is reported, I can’t draw conclusions as to how real-world performance (which is a more complex problem than benchmark accuracy) will be affected. Further supporting this assessment is the authors’ note that model termination behavior depends on hardware. Finally, this can be viewed as an empirical study on DeepSeek with math benchmark, how does this generalize across model families and non-math benchmarks.

Second, there is plenty of existing work covering the fact that transformer models may use large portions of their total token cost on irrelevant computations that worsen outcomes (for example, by finding the correct answer quickly and then using additional compute to follow another trajectory to a wrong answer). This is very relevant to the authors’ work but as far as I can tell is only briefly mentioned (one reference, Chen et al, is included).

Questions for authors

N/A (I don’t think clarifying questions would materially affect my review).

1 comment

Thank you for the review.

Terminology. We agree the words can suggest more than we mean. We use "reasoning"/"thinking" only as names for an interface: the tokens between <think> and </think> in the DeepSeek-R1 template. We make no claim about cognition. A footnote now says this at first use.

Generalization and theory. We agree that one model family and math benchmarks do not support general claims, and we narrowed the abstract and conclusion to this setting. We offer no theory of non-termination. Our contribution is empirical: where compute goes (non-terminating generations use 65% of AIME tokens), how unstable termination is (23–24% item-level flips, now replicated on an independent sample), and a matched-compute comparison that bounds what early predictors can add over token caps. On hardware dependence: Yuan et al. (2025) independently document reduced-precision non-reproducibility in reasoning models. We report it as a property of the deployment setting.

Related work. Agreed. We added a paragraph on overthinking and efficient reasoning (Sui et al. 2025; Wang et al. 2025; Han et al. 2024; Aggarwal & Welleck 2025; Ma et al. 2025; Hassid et al. 2025; Yang et al. 2025; Zhang et al. 2025). We distinguish non-termination within a ceiling from shortening traces that do terminate.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for careful and fair reviews. Several criticisms are correct, and we have revised the paper accordingly. All new numbers below are computed from our existing per-problem results, and the analysis scripts will be included in our code release.

1. The cap comparison was uneven (c9o5). We agree. The uncapped run never asks non-terminating generations for an answer, while capped runs get a 1,000-token answer window. Per-problem analysis shows that forced exits rescue 10–22 of the 93 never-terminating problems (10.8–23.7%, depending on the cap). A matched baseline (10k cap with the same window) therefore keeps the 89 correct terminating problems and adds these rescues. It beats the 4k cap (46.5%) whenever more than 4.3% are rescued, below every rate we observe. Bracketing by the observed rates projects it at 49.5–55.5%. We have removed the claim that the 4k cap is "no lower than" the baseline. The revised fixed-cap table (Table 5) reports savings against both baselines: 4k saves 42.4% at a projected cost of 3–9 points; 6k (52.0%, 27.2% saved) lies within the projected range. We will run C=10k directly with the identical code path and replace the projection with the measured value.

2. No direct cap-vs-probe comparison (ygWw). We added one (new Sec. 7.5, Table 6 and Figure 5). Using per-problem outcomes under each cap and under an early forced exit at ~150 tokens, we evaluate "probe flags at token 150 → exit; otherwise cap C" at matched compute, as a function of probe AUC. At the savings of the 4k cap (39%), accuracy is 46.5% (cap only), 46.5% (probe AUC 0.65), 46.9% (AUC 0.85) and 48.3% (AUC 0.999). The reason is that forced exits at the cap already rescue looping generations (16% at 4k), while an exit at token 150 rescues only 3%. A probe pays off only at aggressive budgets: at ~60% savings, 31.0% (cap) vs. 39.0% (AUC 0.85). This makes "caps, not probes" a quantitative statement that holds however well a probe could be trained.

3. Probe scope and n≪d (c9o5, ygWw). We agree that our negative result cannot rule out probing, and we have narrowed our claims to early-activation linear probes at n=200. New analyses: (a) termination is largely problem-determined: predicting one run's label from other runs of the same problem gives AUC 0.83–0.87, so information exists in principle; (b) low-dimensional probes (PCA inside each fold) reach up to 0.70 at their best configuration; under nested CV, which selects the configuration inside training folds, the AUC is 0.61 (mean of five outer seeds; permutation p=0.016, 500 permutations). So early activations carry a weak signal, which we now report as such rather than as a null.

4. Replication and corrections. The item-level termination flip replicates on an independent 200-problem sample (23.0% vs. 23.5%; κ=0.526). Re-auditing our code against the paper, we corrected: the probe layer (16, not 20; conclusions unchanged); the MATH-500 scorer, which truncated answers containing braces (Table 1 now uses the 325 numeric-answer problems; uncapped 88.6%); the answer-window description (forced generations without a boxed answer are re-prompted and can be correct, 12/98 at C=4k); the surface-feature baseline (chance up to token 200, nominally above chance at 250–300, not significant after correction); Fig. 7, which now shows the actual permutation null; and the stated 11,000-token generation limit and GSM8K seed. AIME results are unaffected.

5. Terminology and related work (QR3a). We added a footnote defining "reasoning"/"thinking" as the tokens in the <think> block, with no claim about cognition, plus a related-work paragraph on overthinking and efficient reasoning (including DEER, Zhang et al. 2025, and Yuan et al. 2025).

Still to do (GPU): the measured C=10k matched baseline, a T=0.6 cap curve, DEER as a baseline, probes trained on more AIME problems, and isolating the cause of the cross-node flips.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.