acceptodds
Under review as a conference paper at ICLR 2027

When Is More Reasoning Worth It? Fallback-Aware Reasoning-Strategy Routing Under Hard Token Budgets

Abstract

Reasoning strategies have heterogeneous, input-dependent token costs. Standard test-time allocation asks where compute is useful; we study the distinct problem of hard per-query output-token compliance, where realized use above a requested threshold is a violation. We separate this risk into violations when a strategy is predicted feasible and violations after no strategy is predicted feasible, then use a frozen router combining strategy-specific token prediction, estimator-fit accuracy priors, conservative resource corrections, and minimum-predicted-token fallback. In a sealed true-online evaluation of Qwen2.5-3B-Instruct and Phi-3.5-mini-instruct across four reasoning benchmarks, 2,400 routed executions precede a 4,000-execution exhaustive matrix. From low to high budgets, macro violation falls from 0.258 to 0.033 for Qwen and from 0.225 to 0.038 for Phi, while fallback falls from 0.688 to 0.123 and from 0.750 to 0.048, respectively. Yet conservatism is not uniformly beneficial: against the frozen point router, Phi’s low-budget MATH-500 violation increases by 0.25 (95% bootstrap CI [0.15, 0.35]; Holm-adjusted p = 0.00024), and low-budget fallback carries substantial model-dependent accuracy costs. Hard-budget reasoning therefore requires controlling both uncertainty within the feasible branch and the behavior of the system when predicted feasibility disappears.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time reasoning, reasoning-strategy routing, hard token budgets, resource-aware inference, fallback risk, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BBfeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies reasoning-strategy routing under per-query token budgets. It separates budget violations into those occurring when a strategy is predicted feasible and those occurring after the router enters fallback. Based on this formulation, it evaluates a router combining token-cost prediction, conservative corrections, accuracy priors, and a minimum-predicted-token fallback. The experiments show that conservative prediction can sometimes increase overall violations by increasing exposure to fallback.

Critical strengths
  1. The paper raises an interesting question: conservative resource estimates may improve compliance among predicted-feasible choices while worsening overall behavior through fallback.
  2. The authors report negative results and limitations transparently, including settings where the conservative router increases violations or substantially reduces accuracy.
Critical weaknesses
  1. The technical contribution is insufficiently developed. The theoretical analysis primarily consists of a total-probability decomposition and an algebraic comparison. These provide a useful organizing perspective, but do not establish a strong theoretical contribution. The implemented router also does not estimate or guarantee the conditional risk introduced in the conceptual formulation. Although this limitation is acknowledged, the paper needs to explain more clearly what its framework contributes beyond documenting the fallback failure mode.

  2. The evaluation does not adequately isolate the central design choices. Fallback is central to the paper’s motivation, yet the experiments do not directly compare alternative fallback policies while holding the feasibility estimator fixed. The reported ablations also show settings where simpler variants improve accuracy, compliance, and token consumption simultaneously. In addition, low- and medium-budget routing selects only Direct or Few-Shot, limiting the evidence for the broader reasoning-strategy selection problem. Missing literal prompts and predictor coefficients further hinder independent assessment.

Questions for authors
  1. With the feasibility estimator held fixed, how does the proposed fallback compare with fixed Direct, fixed Few-Shot, and a policy selected to minimize calibration-set violations?
  2. What explains the cases where removing accuracy priors improves all three metrics? Under what conditions does the full router offer a meaningful advantage?
  3. Can the authors clarify which conclusions are intended as diagnostic observations and which establish the effectiveness of the proposed method?
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer BBfe — Fallback Sensitivity and Ablation Evidence

Thank you for the thoughtful review and for recognizing the importance of understanding how conservative feasibility estimates interact with fallback behavior. We apologize for the delayed response; we were completing additional analyses to address the concerns raised during review. We have now uploaded a revised manuscript.

1. Comparing fallback policies while holding feasibility fixed

We agree that the original evaluation did not sufficiently isolate the fallback-policy choice. We therefore conducted a separately labeled post-review offline analysis that retains the frozen feasibility decisions and compares the original fallback with fixed Direct, fixed Few-Shot, and a policy selected to minimize violations on the risk-calibration data. The calibration-selected policy uses no sealed evaluation outcomes for selection.

At the low-budget tier, averaged across four benchmarks:

ModelOriginal fallbackFixed DirectFixed Few-ShotCalibration-selected
Qwen violation26.75%39.75%33.00%26.25%
Phi violation22.50%37.00%15.00%15.00%

These results show that the original minimum-predicted-token fallback is not uniformly optimal. For example, on Phi, fixed Direct increases accuracy from 46.75% to 58.50%, but violations rise from 22.50% to 37.00%. Fixed Few-Shot lowers violations to 15.00%, while accuracy falls to 41.75%.

Importantly, the original D1 online records and the exhaustive D3 records were generated independently. Thus, these are counterfactual sensitivity analyses using saved outcomes, not exact replays of the original online requests.

2. Why removing accuracy priors can improve all three metrics

We agree that this behavior is important rather than incidental. The accuracy prior is estimated from a relatively small development sample and does not explicitly depend on the budget. Consequently, it can misrank strategies on held-out examples and steer selection away from shorter, more effective outputs.

In three Qwen benchmark–budget settings (ARC at medium and high budgets, and GSM8K at high budget), removing the prior weakly improves accuracy, violation rate, and mean output length simultaneously.

For example, on ARC at the medium budget, the full router achieves 72% accuracy, 8% violations, and 128.74 mean output tokens; without the prior, these become 80% accuracy, 0% violations, and 8.96 mean output tokens.

These findings expose a genuine limitation of static accuracy priors. We do not claim they are universally beneficial.

3. What the framework establishes—and what it does not

We agree that the fallback-risk decomposition follows the law of total probability and is not a novel theorem or a statistical guarantee. Our intended contribution is primarily diagnostic and empirical: conservative resource corrections can improve compliance within predicted-feasible decisions while increasing exposure to a fallback whose conditional risk may be substantially different.

The current implementation does not directly estimate conditional fallback violation probabilities, and its safety margin provides no formal compliance guarantee. Nor do the experiments establish superiority across all budgets, models, or reasoning strategies.

We have revised the manuscript to distinguish these diagnostic observations from effectiveness claims, incorporate the fallback sensitivity results, and clarify the limitations of the accuracy-prior design. We also acknowledge the restricted strategy-selection diversity and the importance of complete prompts, predictor specifications, and reproducibility details for independent assessment.

Thank you again for these constructive questions. They helped us sharpen both the empirical analysis and the scope of our conclusions.

Thank you for the candid response and additional analysis.

While I appreciate the narrowed framing and transparency, the new results reinforce my concern that the technical contribution and empirical gains over simpler baselines remain too limited for acceptance.

I will therefore maintain my rating of 2 (Weak Rejection).

Reviewer_MVLbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how LLMs can select a suitable reasoning strategy under a strict token budget. Since different strategies may use very different numbers of tokens, the authors introduce a routing method that predicts token usage and includes a safety margin when making decisions. When none of the strategies is expected to fit within the budget, the method selects the one with the lowest predicted token usage. The authors evaluate their approach on two models and four benchmarks, showing that it can reduce budget violations. However, the results also reveal a limitation: overly conservative estimates can trigger too many fallbacks, which may hurt accuracy and, in some cases, even increase budget violations.

Critical strengths

The paper focuses on an important but often overlooked issue in budget-aware reasoning: the conservative token predictions can increase fallback frequency and mostly worsen overall budget compliance for the task. An explicit analysis of this trade-off provides a useful perspective.

The proposed router mechanism combines token prediction, conservative calibration, and accuracy-based strategy selection with an explicit fallback mechanism. The proposed model is simple, interpretable and low routing overhead makes it practical to implement.

The paper evaluates two model Qwen and Phi for the four reasoning benchmarks, using held-out online evaluations, multiple baselines, ablations, and statistical significance tests. It also reports negative results, which provides a useful insights into the limitations of conservative routing.

Critical weaknesses

The proposed method mostly combines existing budget-aware routing, token prediction, and adaptive reasoning techniques [a,b,c,d]. The fallback-risk decomposition is mathematically straightforward, making the methodological contribution incremental.

The proposed method does not enforce strict token limits, unlike budget-control approaches [e]. Its fallback minimizes predicted token usage rather than violation risk, and it lacks the statistical guarantees explored in risk-control frameworks [h,i]

The experiments are shown on the two small instruction models and four benchmarks. The small models are not much representative and read word scenarios are very different. Evaluation on larger reasoning-specialized models, unseen datasets, and different decoding settings is missing, which is a close mimic of the real scenarios.

There are few recent methods which are missing from the baselines for example the comparisons with budget forcing [e], training-free adaptive routing [f], and learned reasoning-mode selection [g] are missing, which makes it difficult to evaluate the proposed method's advantages. Also, the fallback strategies and calibration margins are not extensively evaluated. In the experiment over 95% of decisions select Direct or Few-Shot, limiting conclusions about more complex reasoning strategies, which is extensively included in the motivation

It seems that the conservative routing sometimes reduces accuracy, while low-budget MATH-500 shows 100% fallback. I believe a joint metric is more important instead of focusing only on the single directional evaluations to check whether responses are both correct and budget-compliant.

References:

[a] Chen et al., CONCUR: A Framework for Continual Constrained and Unconstrained Routing. ICLR 2026

[b] Fernandez et al., RADAR: Reasoning-Ability and Difficulty-Aware Routing for Reasoning LLMs. ICLR 2026

[c] Han et al., Token-Budget-Aware LLM Reasoning (TALE). Findings of ACL 2025

[d] Ong et al., RouteLLM: Learning to Route LLMs from Preference Data. ICLR 2025

[e] Muennighoff et al., s1: Simple Test-Time Scaling. EMNLP 2025

[f] Lee et al., DART: Draft-Agreement Routing for Training-Free Adaptive Thinking Budgets in Hybrid Reasoning Models. Findings of EMNLP 2026

[g] Kassenaar et al., Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation. arXiv, 2026

[h] Zollo et al., Prompt Risk Control: A Rigorous Framework for Responsible Deployment of Large Language Models. ICLR 2024.

[i] Wang et al., SAFER: Risk-Constrained Sample-then-Filter in Large Language Models. ICLR 2026.

Questions for authors

Please refer to the weakness section, mostly limited novelty and evaluations are key concern and I also believe that the metric is not complete to evaluate the paper's claims

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer MVLb — Additional Experiments and Revised Manuscript

Thank you for your constructive review and for recognizing the value of analyzing fallback risk. We apologize for the delayed response. We were completing additional experiments and revising the manuscript to address the substantive concerns raised during review. We have now submitted the revised PDF.

1. Joint accuracy and budget compliance. We agree that accuracy and violations should not be considered independently. We therefore report the proportion of responses that are both correct and budget-compliant, computed from the original frozen true-online records. Across the four benchmarks, the Qwen router achieves joint success rates of 45.25%, 54.75%, and 59.25% at low, medium, and high budgets, respectively; the corresponding Phi rates are 38.00%, 51.75%, and 58.00%. We have incorporated this metric into the revised results.

2. Exact budget enforcement and a capped Direct baseline. We agree that the predictive router does not guarantee strict token-limit compliance. To address the missing enforcement baseline, we conducted a separately labeled post-review experiment comprising 2,400 capped Direct generations, paired with the 2,400 original D1 router records. All 24 configurations passed integrity checks, and capped Direct produced zero measured output-token violations.

For Qwen, capped Direct achieved 49.67% accuracy versus 59.25% for routing, and 49.67% joint success versus 53.08% for routing. For Phi, capped Direct achieved 53.83% accuracy versus 53.50% for routing, and 53.83% joint success versus 49.25% for routing. Thus, routing improves Qwen's raw accuracy, but capped Direct performs better on Phi's joint metric. We do not claim universal router superiority. The new comparison is limited to capped Direct and does not cover capped Few-Shot or multi-call methods.

3. Fallback-policy sensitivity. We performed a separately identified post-review offline analysis using independently generated D3 outcomes while holding feasibility predictions fixed. At Phi's low budget, replacing the original fallback with fixed Direct raises accuracy from 46.75% to 58.50%, but violations also increase from 22.50% to 37.00%. Fixed Few-Shot reduces violations to 15.00% but lowers accuracy to 41.75%. These findings show that fallback selection creates a real quality–compliance trade-off and that no single policy is uniformly preferable.

4. Methodological novelty and limitations. We agree that the probability decomposition is elementary and does not establish a theoretical guarantee. The contribution is better characterized as an empirical diagnosis of how conservative feasibility estimation can increase exposure to a risky fallback, sometimes worsening aggregate compliance. We have clarified this scope and acknowledged that the method neither estimates conditional violation risk directly nor provides formal risk control.

We also acknowledge the limited model scale, decoding conditions, and strategy-selection diversity, as well as the absence of direct comparisons with several recent adaptive-budget methods. The revised manuscript presents these limitations more explicitly and makes the key comparisons easier to locate.

Thank you again for your feedback. We hope the new evidence and narrower claims address the central concerns, while recognizing that the remaining limitations are important.

Reviewer_S9a2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies reasoning-strategy routing under a hard per-query output-token budget, where exceeding the budget counts as a violation. Specifically, the authors argue that existing test-time compute allocation optimizes average cost, and that conservative token estimates can leave no strategy predicted feasible, so the fallback choice becomes a separate source of risk. To address this, they decompose the violation rate into feasible-branch and fallback-branch terms, and build a frozen router that predicts each strategy’s token use with a linear model plus a conservative margin, selects the feasible strategy with the highest accuracy prior, and otherwise falls back to the strategy with the lowest predicted tokens. Experiments on two small models and four benchmarks show that violation falls as budgets loosen, that fallback causes almost all violations at tight budgets, and that the conservative router does not uniformly improve over a point-estimate router.

Critical strengths
  1. The idea is straightforward and easy to understand.
Critical weaknesses
  1. The problem formulation is unclear and not well motivated.
    • Truncation already enforces the budget. A hard output-token budget can be enforced exactly by setting the generation limit to B, yet the paper deliberately does not truncate at B (Section 5) and never compares with this simple baseline.
    • No concrete scenario is given. The paper does not explain when exceeding B is a failure but truncation is not an option. This makes me confuse about the motivation.
  2. The method is simple and not well justified.
    • The safety margin provides no guarantee. The “conservative” estimate only adds a fixed margin to the prediction (Appendix D), which is equivalent to lowering the threshold. The router still violates the budget on 22–26% of queries at the low tier (Table 1). The accuracy prior ignores the budget. Strategies are ranked by a fixed prior estimated from only 60 examples per dataset, which does not depend on B. However, prior work shows that the relative performance of reasoning strategies changes with the token budget [1, 2, 3].
    • The novelty is limited. The token predictor is a linear model on word count and lexical cues, and the risk decomposition is simply the law of total probability, as the authors acknowledge (Section 4). I do not understand why do we need this predictor.
  3. The writing is poor.
    • Undefined terms. The main text relies on many undefined internal terms (e.g., V1/V2, Stage B/C/D, D1/D3, “sealed true-online”).
    • Irrelevant details. It is cluttered with protocol details such as revision hashes, fingerprints, and even “laptop shutdowns” (Section 10).
    • Key results are only in the appendix. The comparisons with fixed strategies (Tables 8–10) and the ablations (Tables 12–13) are not in the main text, and Section 8 contains no numbers, although the main text ends on page 7.

[1] Junlin Wang, Siddhartha Jain, Ben Athiwaratkun, Dejiao Zhang, Baishakhi Ray, and Varun Kumar. Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies. EMNLP 2024.
[2] Yuyang Wu, Yifei Wang, Ziyu Ye, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When More is Less: Understanding Chain-of-Thought Length in LLMs. ICLR 2026.
[3] Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token-Budget-Aware LLM Reasoning. Findings of ACL 2025.

Questions for authors

Please refer to the weaknesses.

1 comment
AuthorsOpenReview ↗

Author Response to S9a2 — Capped Direct Baseline and Revised Claims

Thank you for the detailed review. We agree that our original submission insufficiently distinguished predictive strategy selection from exact execution-time budget enforcement, and that the absence of a budget-equal capped Direct baseline was a significant experimental omission. We apologize for the delayed response; we were completing additional experiments to address this concern. We have now submitted a revised manuscript.

1. Exact truncation and the missing baseline

We agree that setting max_new_tokens=B enforces the generated-output-token limit for a single call, whereas our predictive router does not. Pre-generation routing asks which reasoning procedure is likely to yield a correct, complete response within a budget; a generation cap enforces the output limit during execution. These mechanisms address different aspects of the problem and can be combined. We do not claim that predictive routing is a substitute for exact enforcement.

To address the missing baseline, we conducted 2,400 new capped Direct generations, matched by model, benchmark, example, and budget tier to the original 2,400 routed executions. All 24 configurations passed structural, provenance, identity, and budget-integrity checks, with zero measured cap violations. The original router records were not regenerated. The capped experiment is explicitly labeled as post-review evidence.

2. Capped Direct versus routing: paired results

For Qwen, capped Direct achieves 49.67% accuracy versus 59.25% for routing: a paired difference of -9.58 percentage points (95% paired-bootstrap CI [-12.75, -6.33]). Joint correctness and compliance are 49.67% versus 53.08%, respectively (difference -3.42 points, CI [-6.42, -0.33]).

For Phi, capped Direct achieves 53.83% accuracy versus 53.50% for routing (difference +0.33 points, CI [-2.25, 3.00]). Joint correctness and compliance are 53.83% versus 49.25% (difference +4.58 points, CI [2.17, 7.00]).

As expected, capped Direct has 0% output-token violations, compared with 13.83% for the Qwen router and 11.33% for the Phi router. The evidence therefore does not support universal router superiority: Qwen routing improves raw accuracy, whereas capped Direct is better on Phi's joint metric.

The cap also has observable completion consequences: 599/2,400 capped Direct generations reach their token limit, and 1,339/2,400 do not satisfy our strict two-marker formatting-completeness criterion. We emphasize that formatting incompleteness is not equivalent to incorrectness. These findings are descriptive and do not establish that routing is preferable in deployment.

This comparison covers single-call capped Direct only; capped Few-Shot, capped routed strategies, aggregate multi-call limits, and end-to-end application utility remain untested.

3. Motivation and scope

We have clarified that the motivating use case is selecting a reasoning procedure before generation under a fixed per-response allowance, especially when completing a useful answer matters in addition to not exceeding the allowance. This is an illustrative scenario, not a separately evaluated deployment. Exact enforcement remains appropriate whenever violating a hard limit is unacceptable.

4. Method justification and novelty

We agree that a fixed additive safety margin is equivalent to tightening the predicted-feasibility threshold and provides no formal guarantee. We also agree that the fallback-risk decomposition follows the law of total probability; we do not present it as a novel theorem.

The intended contribution is an empirical diagnosis: making feasibility estimates more conservative can shift decisions toward fallback, and the resulting fallback exposure can worsen overall compliance or accuracy. The token predictor serves as an inexpensive mechanism for exploring this interaction, not as a novel prediction architecture.

We acknowledge that the accuracy prior is development-fitted, budget-independent, and may misrank strategies. Our no-prior ablations demonstrate settings where removing it improves accuracy, violations, and token consumption. We have narrowed the claims accordingly.

5. Presentation and experimental limitations

We have revised the presentation to clarify internal terminology, foreground the scientific results, and distinguish implementation and reproducibility details from the main conclusions. We also acknowledge the limitations of two small instruction models, a restricted set of reasoning strategies, and the absence of larger reasoning-specialized models and additional decoding conditions.

We appreciate the reviewer's criticism. The new capped baseline materially changes the interpretation: exact enforcement eliminates measured token-limit violations, but its effect on answer quality is model-dependent. We present this as a more precise and limited empirical finding, not as evidence that predictive routing should replace mechanical budget enforcement.

Comments on the paper

2 comments

Thank you for your follow-up, and we apologize for the delayed response. We were completing additional experiments and revising the manuscript to address the concerns raised during review.

We have now uploaded the revised manuscript and provided a detailed point-by-point response under your original review, including new capped Direct comparisons, joint correctness/compliance results, and fallback-policy sensitivity analyses.

We appreciate your time and consideration, and we hope the additional evidence helps clarify the contributions and limitations of the work.

Thanks for the rebuttal.

I appreciate the author for the extra experiment and responce.

The limited experiment and novelty is the main concern, which remains after rebuttal.

I will keep my rating.

Thanks

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.