acceptodds
Under review as a conference paper at ICLR 2027

Same Beliefs, Different Policies: Learning Reusable Representations for Early Stopping in LLM Reasoning

Abstract

Reasoning LLMs often continue reasoning after reaching a correct answer, wasting computation and sometimes changing it to an incorrect one. Deciding when to stop is a metareasoning problem: the model must weigh what further reasoning could achieve against its cost, given what it can already answer. We introduce BELT (BELief-based Termination), which learns beliefs about current and future answers from a frozen LLM's hidden states, then fits a linear stopping policy. We show that predicting reasoning outcomes yields reusable features for stopping control. Changing the token price changes the value of these outcomes, but leaves their prediction targets unchanged, so only the policy needs refitting. Under common policy fitting, belief-trained representations yield higher stopping utility than raw or utility-trained features. Averaged over five out-of-distribution math benchmarks, BELT reduces generated tokens by 66% on Qwen3-1.7B and 53% on OLMo-3-7B-Think at matched test-frontier accuracy relative to natural completion. The same math-trained probe transfers to coding without code-side training. In both domains, its accuracy-cost trade-offs are competitive with or better than those of end-to-end controllers. On Qwen3, it uses 6.3x fewer controller parameters than the strongest baseline and  25x less training compute across five token prices.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-Time Scaling, Large Reasoning Model, Adaptive Reasoning, Overthinking

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_378ROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BELT, a framework for early stopping in LLM reasoning that learns reusable representations by predicting four reasoning outcomes from hidden states, then fits lightweight linear stopping policies for different token costs. Experiments on mathematical reasoning and code generation demonstrate competitive accuracy–efficiency trade-offs, improved cross-domain transfer, and substantially lower controller training costs compared to existing baselines.

Critical strengths
  • The paper presents a well-motivated approach that separates outcome prediction from stopping policy optimization, enabling the reuse of learned representations across different token costs. This design substantially reduces controller training costs while maintaining competitive accuracy–efficiency trade-offs.

  • The experimental evaluation is comprehensive, covering multiple reasoning models, mathematical benchmarks, and cross-domain transfer to coding tasks. The extensive ablation studies, particularly the comparisons under a unified policy-fitting framework, provide convincing evidence for the effectiveness of outcome-based supervision.

Critical weaknesses
  • The methodological novelty is somewhat limited, as BELT builds on existing probe architectures and utility-based stopping objectives. While the separation of belief learning and policy fitting is useful, the performance gains over the strongest baseline are relatively modest (e.g., 2.2 percentage points in token savings over OSP on Qwen3 OOD at 1 pp accuracy tolerance).

  • The evaluation has some limitations in generalizability and practical applicability. The mathematical benchmarks are restricted to filtered integer-answer subsets, excluding many problems with more complex answer formats. Moreover, the main results rely on test-frontier evaluation, which selects operating points using test-set performance and may not fully reflect realistic deployment settings.

Questions for authors
  • Could the authors further clarify the novelty of the proposed belief representations compared with existing outcome-prediction probes? In particular, how much of the improvement comes from the four prediction targets rather than the additional sequence encoder capacity?

  • How well does BELT perform when stopping policies are selected entirely on the development set rather than using test-frontier operating points? Could the authors report the corresponding accuracy and token savings to clarify the practical performance gains?

  • How sensitive are the results to the filtering of mathematical benchmarks? Would BELT maintain similar advantages on problems requiring fractional, symbolic, or other non-integer answers?

  • The paper reports substantial reductions in controller training costs, but what is the actual end-to-end inference overhead of BELT, including probe execution and hidden-state processing? Does the reduction in generated tokens translate into comparable wall-clock latency improvements?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 378R (Part 1)

Thank you for the thorough review and the concrete questions. We answer them grouped by topic.


W1 and Q1. The methodological novelty is somewhat limited, as BELT builds on existing probe architectures and utility-based stopping objectives … how much of the improvement comes from the four prediction targets rather than the additional sequence encoder capacity?

Novelty of BELT lies not in the architecture of the probe, but in the belief representation learning and policy-belief disentanglement: one probe learns 4 price independent outcome beliefs (Same, Later, Now, End). Prior work mainly leverages the Now belief (current answer correctness; e.g., DEER, CoDE-Stop and Thought Calibration), which is less expressive than the multi-belief objective. Together, these beliefs give the linear policy a more informative basis for stopping decisions and allow refitting linear policies of different token prices on a token-price-agnostic representation.

targets versus encoder capacity Ablation studies in Section 6 show that The training target is the main driver of the gain. Capacity helps, but is not the main driver. (1) Linear probe trained on the BELT targets beats the OSP-1 features in utility (Table 4); (2) BELT architecture with OSP objective leads to policy collapses (App.E.2); (3) Ablating Later and Same lowers utilities (Table 6a).


W1. The performance gains over the strongest baseline are relatively modest.

While BELT is only slightly better than the OSP baseline in terms of performance, it is much more efficient and flexible.

BELT vs. a single OSP checkpoint: 6.3 times fewer parameters and about 4.1 times fewer inference FLOPs per token, over 20 times less training compute for five prices. Belief training also supervises every boundary equally, whereas utility gradients vanish at boundaries the policy rarely reaches (Eq. 11). For a larger model, the contrast in training and inference efficiency will be more extreme. For OLMo-7B, OSP checkpoint scales from 101M to 405M, while BELT scales from 16M to 19M.


W2 and Q2. the main results rely on test-frontier evaluation, which selects operating points using test-set performance and may not fully reflect realistic deployment settings.

We follow the prior work convention. Nevertheless, to address your concern, we provide additional results.

Prior work convention: Reading a frontier point requires fixing one coordinate (an accuracy tolerance or a token budget), so the test frontier is the standard Pareto comparison. It follows Thought Calibration and OS-Pruner, and is applied identically to every method.

Unseen set selection: In appendix E and Appendix B, we control accuracy/cost and select the member to be compared on half of the test set, and report the actual comparison result on the other half. So this is an unseen selection setting you have requested.

Extra Results: we did another selection on a smaller dev set. At a 1-pp accuracy drop tolerance, BELT-H is comparable to full OSP and saves 6-7 more tokens than OSP-1. We will include this in the final version. But importantly, the test frontier selection is still the convention of this field, and is the only viable approach to compare the algorithms given their multiple family members.


W2 and Q3. The mathematical benchmarks are restricted to filtered integer-answer subsets.

Thanks for the detailed reading! We did the filtering mainly for label reliability, following DAPO-Math and Wang et al. (2024), as explained in App.A. Here we add full results for math OOD test sets without integer filtering:

No-filtering OOD results: For MATH-500, OlympiadBench, and Gaokao, BELT-H outperforms the strongest baseline by +12.8, +23.2, +15.2 of token saving with no tolerance on accuracy drop. The BELT-H is the same one trained on integer-only math problems.

Further results show that this data preprocessing does not hurt generalizability. For example, the math-trained probe transfers to LiveCodeBench graded by unit tests (Sec. 5).

New generalization results: We conduct an additional experiment, where we show that BELT trained on math is applicable for agents in ALFWorld and ScienceWorld. It removes about 85% of thinking tokens at or above the natural score in ScienceWorld, and about 30% within 2 points of natural success in ALFWorld, with large seed variation.

AuthorsOpenReview ↗

Response to Reviewer 378R (Part 2)

Q4. End-to-end inference overhead and wall-clock latency.

Thanks for the question. We have added the following wall-clock results to the revised version (Qwen3-1.7B with BELT-H):

  • Belief modeling adds a small Overhead per token. Decoding time per token increases by less than 1% (16.87 instead of 17.03 ms), consistent with the arithmetic estimate in App. D.1. Half of this time is reading the hidden states and bookkeeping, each probe call at a block boundary takes only 2.7 ms.
  • BELT reduces wall-clock significantly End to end. When answering questions end-to-end, BELT saves 23.3% of the wall-clock thanks to generating 24.1% less tokens, without performance drop.

Concluding remarks: We would be grateful if you could let us know if our explanations have been satisfactory. If so, we kindly ask that you consider increasing your rating. We are also open to discussing any other questions you may have.

AuthorsOpenReview ↗

Gentle reminder and follow-up on our response

Dear reviewer 378R,

Thank you again for your time and insightful comments. We truly appreciate your constructive feedback, which has helped us significantly improve the paper.

As the discussion period is coming to a close soon, we wanted to gently follow up to ensure that our previous responses and additional experiments have adequately addressed your concerns

If you have any remaining questions or need further clarification, we would be more than happy to provide them before the deadline. Thank you very much for your consideration and effort!

Best regards, The Authors

Reviewer_syUmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BELT, an early-stopping framework that separates price-independent prediction of reasoning outcomes from price-dependent stopping decisions. A probe trained on frozen LLM hidden states predicts four outcomes—Now, End, Later, and Same—and its learned representation is reused across different token prices by fitting only a lightweight linear stopping policy. Experiments on two reasoning models show competitive or improved accuracy–cost trade-offs over existing early-stopping methods, together with substantial reductions in controller parameters and retraining cost. The paper also reports math-to-code transfer without code-side controller training.

Critical strengths
  1. The motivation is clear: directly optimizing stopping utility entangles representation learning with a particular token price, whereas the underlying reasoning outcomes remain unchanged across preferences.
  2. The separation between outcome prediction and stopping policy directly addresses this issue and provides a simple mechanism for adapting to new token prices.
  3. The experiments strongly support the main design. Under common policy fitting, belief-trained representations consistently outperform raw and utility-trained features, and target/input ablations clarify the roles of the proposed beliefs.
  4. The method achieves strong accuracy–token trade-offs while using a substantially smaller reusable controller. The additional experiments on a second backbone and math-to-code transfer further strengthen the empirical evidence.
  5. The direct-utility-training control is useful evidence that the gains are not solely due to the probe architecture.
Critical weaknesses
  1. The evaluation is based on banked reasoning trajectories rather than a live serving setup. This establishes the accuracy–token trade-off cleanly, but does not yet measure end-to-end decoding latency or the behavior of the controller under actual online stopping.
  2. On OLMo3, BELT is compared only against OSP-1 rather than the full price-specific OSP family due to its training cost. Since full OSP is stronger than OSP-1 on Qwen3, the evidence that BELT outperforms the strongest utility-trained controller on the second backbone is therefore incomplete.
Questions for authors

see weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer syUm

Thank you for the detailed and encouraging review, and for highlighting the direct-utility-training control. We address the two weaknesses below, starting with the comparator on OLMo3.


W2. On OLMo3, BELT is compared only against OSP-1 rather than the full price-specific OSP family due to its training cost.

Following your suggestion, we trained the full price-specific OS-Pruner family on OLMo3: five token prices x three seeds. Against this stronger comparator:

Extra Results

  • BELT-H has the higher mean at all seven macro readings of the OLMo3 trade-off table (token saving at the 0, 1 and 2 pp tolerances; accuracy at 70%, 60%, 50% and 40% saving), and BELT-B.
  • At the accuracy tolerances, BELT-H saves 7.0, 6.2 and 5.4 pp more tokens than full OSP (at 0, 1 and 2 pp), and BELT-B 5.2–7.9 pp more.
  • Development-selected BELT-B has a higher mean utility than full OSP at all five prices and so does BELT-H; both exceed OSP-1 at every price.
  • The BELT probe has 21x times fewer parameters than one OS-Pruner checkpoint, and covering five prices takes about 55x times fewer training GPU-hours than training the five-checkpoint family.

These new results are added to the revised version.


W1. The evaluation is based on banked reasoning trajectories rather than a live serving setup.

Extra results show that BELT also works in online serving:

We evaluated BELT on OLMo3 in live ALFWorld and ScienceWorld episodes. We ran the agent with BELT active throughout each episode, so subsequent actions and environment observations reflected the controller’s stopping decisions rather than a pre-generated trajectory. In ScienceWorld, every BELT variant removes about 85% of thinking tokens at or above the natural score. In ALFWorld, BELT-H has the highest mean saving at every reported reading, about 30% of thinking tokens within 2 points of natural success versus 11.5% for OSP-1


W1. This establishes the accuracy–token trade-off cleanly, but does not yet measure end-to-end decoding latency or the behavior of the controller under actual online stopping.

Measurements below show that BELT introduces no substantial overhead per token and reduces overall reasoning latency substantially.

Decoding time per token increases by less than 1% (16.87 instead of 17.03 ms), consistent with the arithmetic estimate in App. D.1. Of this, 0.62% is reading the hidden states and bookkeeping; each probe call at a block boundary takes only 2.7 ms.

BELT reduces wall-clock reasoning significantly End to end. When answering questions end-to-end, BELT saves 23.3% of the wall-clock thanks to generating 24.1% less tokens, without performance drop.


Concluding remarks: We would be grateful if you could let us know if our explanations have been satisfactory. If so, we kindly ask that you consider increasing your rating. We are also open to discussing any other questions you may have.

AuthorsOpenReview ↗

Gentle reminder and follow-up on our response

Dear reviewer syUm,

Thank you again for your time and insightful comments. We truly appreciate your constructive feedback, which has helped us significantly improve the paper.

As the discussion period is coming to a close soon, we wanted to gently follow up to ensure that our previous responses and additional experiments have adequately addressed your concerns

If you have any remaining questions or need further clarification, we would be more than happy to provide them before the deadline. Thank you very much for your consideration and effort!

Best regards, The Authors

Reviewer_dBxVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents BELT, a linear stopping controller which learns beliefs about current and future answers from a frozen LLM’s hidden states. A probe predicts four properties of the current reasoning prefix—current correctness, natural-completion correctness, availability of a later correct answer, and agreement with the final answer. The probe is then frozen, and a small linear policy is fitted for each token price.

The main contribution is the empirical finding that outcome supervision produces representations that support inexpensive adaptation across accuracy–computation preferences. The paper combines competitive stopping performance with useful representation comparisons and ablations. My main reservations concern the distinction between test-frontier performance and deployable policy selection, the strength of the baseline for preference adaptation, and the interpretation of the four predictions as beliefs.

Critical strengths
  1. Separating prediction from preference-dependent control is well motivated. Reusing a representation while changing only a small policy is practically attractive.
  2. Two reasoner families and math-to-code transfer provide evidence of the robustness of BELT. The manuscript also discloses important limitations, including restricted evaluation populations and estimated training costs.
Critical weaknesses
  1. Appendix A.7 selects frontier members and interpolation weights by test accuracy and cost. Although this is a reasonable descriptive comparison, its accuracy tolerance is calibrated on target-test outcomes. It does not demonstrate that a policy selected beforehand will preserve accuracy within one percentage point on a new distribution.
Questions for authors

1.The four predictions are useful features, but what if two prefixes induce the different optimal stopping decision since they could have identical outcome probabilities while differing substantially in expected time to correction. 2. Why the frontier constraint uses micro-average accuracy, whereas several headline savings are macro averages evaluated under the resulting policy mixture.

2 comments
AuthorsOpenReview ↗

Response to Reviewer dBxV

Thank you for the careful review, and acknowledging our paper as well-motivated and having robust results. We address the remaining concerns below.


W1. Appendix A.7 selects frontier members and interpolation weights by test accuracy and cost. Although this is a reasonable descriptive comparison, its accuracy tolerance is calibrated on target-test outcomes.

We would like to clarify by justifying the design choice, pinning evidence in paper, and adding extra analysis results:

Design choice justification: This is a common practice in prior work including OS-Prunner and TC because the test frontier asks which accuracy–cost trade-offs each method can attain. Comparing methods at one point of that trade-off requires holding one coordinate fixed (an accuracy tolerance or a token budget) and reading the other on the evaluated population. Therefore, we hold one point on the test set trade-off curve and compare the other.

Unseen set selection: In appendix E and Appendix B, we control accuracy/cost and select the member to be compared on half of the test set, and report the actual comparison result on the other half. So this is an unseen selection setting.

Extra Results: we did another selection on a smaller dev set. At a 1-pp accuracy drop tolerance, BELT-H is comparable to full OSP and saves 6-7 more tokens than OSP-1. We have included this in the revised version. But importantly, the test frontier selection is still the convention of this field, and is the only viable approach to compare the algorithms given their multiple family members.


Q1. What if the same belief prediction results but different future token cost expectation? The later is not directly modeled

Thanks for the sharp question! TL;DR: the cost objective is indirectly learned by our policy. We are currently adding experiments to model it explicitly.

The 4 probabilities are not a sufficient statistic for optimal stopping. Two prefixes with the same four beliefs can differ in how many tokens remain before a correct answer appears, and the optimal decision can then differ. BELT accounts for this indirectly by (i) reading history-attended hidden-states, which contain length info; (ii) We compute expected utility by summing over possible stopping positions, using the recorded correctness and cost at each position. So there is an implicit belief about the remaining computation. An explicit belief about remaining computation, as in ZIP-RC, is a natural addition, and App. F lists it as future work.


Q2. Why the frontier constraint uses micro-average accuracy, whereas several headline savings are macro averages …

Thank you for the careful reading. TL;DR: We pick one benchmark-agnostic operating point on all questions pooled (micro), and report it with every benchmark weighted equally (macro), so that small benchmarks such as AIME are not swamped by large ones.

The deployable object is one distribution-level policy (or mixture), so its frontier and tolerance are selected once on the pooled population, using micro accuracy, which weights questions equally. The per-dataset and macro values then evaluate that same policy, so that small benchmarks carry equal weight in the summary instead of being dominated by the largest set. Micro readings are reported alongside (Tables 10, 16 and 24; App. A.6). In the revised version we have stated this choice in the Metrics paragraph of App.A.6 and have noted that a constraint imposed on macro accuracy is a different deployment target.


Concluding remarks: We would be grateful if you could let us know if our explanations have been satisfactory. If so, we kindly ask that you consider increasing your rating. We are also open to discussing any other questions you may have.

AuthorsOpenReview ↗

Gentle reminder and follow-up on our response

Dear reviewer dBxV,

Thank you again for your time and insightful comments. We truly appreciate your constructive feedback, which has helped us significantly improve the paper.

As the discussion period is coming to a close soon, we wanted to gently follow up to ensure that our previous responses and additional experiments have adequately addressed your concerns

If you have any remaining questions or need further clarification, we would be more than happy to provide them before the deadline. Thank you very much for your consideration and effort!

Best regards, The Authors

Reviewer_d5dPOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

In this work, the authors study early stopping in reasoning LLMs. They introduce BELT, which learns price-independent predictions of current and future reasoning outcomes from frozen LLM hidden states, then fits lightweight, price-dependent stopping policies. It evaluates the reusability of learned belief representations across mathematical reasoning and coding tasks, comparing accuracy token cost trade-offs against utility-trained and training-free stopping baselines.

Critical strengths
  • The separation between outcome prediction and cost-sensitive stopping is well defined and cleanly implemented. BELT learns reusable representations by predicting four reasoning outcomes independently of token price, then fits a lightweight linear policy for different accuracy computation trade-offs, avoiding repeated representation training.
  • The empirical evaluation is comprehensive, with comparisons against multiple stopping baselines across ID/OOD mathematical reasoning and coding tasks.
Critical weaknesses

Evaluation covers a relatively narrow set of backbone models and settings, making it unclear how this framework would generalize to language models of different sizes. It would’ve been interesting to see how well this framework works on larger models.

Questions for authors

Please see weaknesses and clarify any questions/misunderstandings I may have.

1 comment
AuthorsOpenReview ↗

Response to Reviewer d5dP

Thank you for the positive assessment and for the suggestion to examine larger models. We agree that scale is an important open question. We have applied BELT to a substantially larger reasoning model (IFM/K2-Horizon-MoVA-36B-A4B).

New K2-Horizon results: we train BELT on this backbone and compare it with low-cost baselines including a token cap, Thought Calibration, ROM, DEER and CoDE-Stop, and evaluated on math and agentic (AlfWorld) tasks. We didn't train OS-Pruner as that is already hard for us to train on 7B models.

Extra Results: BELT outperforms the best baseline by saving 5.2 pp more tokens on a hard math set. Interestingly, the saving benefit is mostly on hard problems as K2 already answers easy questions shortly. For AlfWorld, BELT saves 28% of token while keeping the natural completing Accuracy, and saves 6 pp more tokens than token cap. It is on par with the best baseline in coding. We have included these results in the revised version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Revision Summarization

We sincerely thank all reviewers for their comments. We have uploaded a revised PDF with every change marked in blue, addressing the concerns raised by the reviewers:


syUm W1. “… does not yet measure end-to-end decoding latency or the behavior of the controller under actual online stopping.”

378R Q4. “… Does the reduction in generated tokens translate into comparable wall-clock latency improvements?”

Sec. 6.6: we conducted new experiments in which BELT controls an agent’s per-turn thinking in ALFWorld and ScienceWorld in a live-serving setting (App. G.1–G.5). Sec. 6.4: measured inside the decoding loop, BELT adds less than 1% to the time per token, and wall-clock time falls nearly as much as the token count (23% vs. 24%; App. D.4).


d5dP W1. “Evaluation covers a relatively narrow set of backbone models … It would’ve been interesting to see how well this framework works on larger models.”

Sec. 6.6: we conducted new experiments with the 36B-parameter K2 Horizon reasoner, training BELT and comparing it with the low-cost baselines (App. G.6).


syUm W2. “On OLMo3, BELT is compared only against OSP-1 rather than the full price-specific OSP family …”

Sec. 5: we trained the full OS-Pruner family for OLMo3, which underperforms BELT-H and BELT-B in Table 3. Notably, BELT is 21× smaller than one OS-Pruner checkpoint (105× smaller than five). Similar performance holds for math-to-coding transfer (Table 4).


dBxV W1. “… It does not demonstrate that a policy selected beforehand will preserve accuracy within one percentage point on a new distribution.”

378R Q2. “How well does BELT perform when stopping policies are selected entirely on the development set …?”

App. E.8: each method selects its operating point on a development split at a 1 pp tolerance, and we report the realized accuracy drop and token savings on the test sets. This provides additional evidence alongside the half-test-set selection results in the previous version. We also clarify the motivation for test-frontier selection, which follows the reporting convention in the literature.


378R W1 and Q1. “The methodological novelty is somewhat limited … how much of the improvement comes from the four prediction targets rather than the additional sequence encoder capacity?”

Sec. 6.1: a new paragraph brings together the ablation studies from the previous version, showing that the training target matters more than capacity.


378R W2 and Q3. “The mathematical benchmarks are restricted to filtered integer-answer subsets …”

App. B.12: we report results on the full math sets, including questions with non-integer answers.


378R W1. “… the performance gains over the strongest baseline are relatively modest …”

Sec. 6.4: a new subsection sets out BELT’s main strengths over OS-Pruner: parameter size, training stability, and negligible wall-clock overhead at inference.


dBxV Q2. “Why the frontier constraint uses micro-average accuracy, whereas several headline savings are macro averages …”

App. A.6: we expand the explanation from the previous version. The deployed object is one policy for the pooled questions, so the constraint uses micro accuracy; macro values evaluate that policy with every benchmark weighted equally.


We hope this revised version resolves the reviewers’ concerns. We are happy to take further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.