acceptodds
Under review as a conference paper at ICLR 2027

When Is More Reasoning Worth It?

Abstract

The next generation of reasoning models will not be the ones that think longest, but the ones that know when to think. Adapting reasoning effort to each question is a central open problem, and solving it would move the frontier between cost and accuracy. This work is a step in this direction. We build a controlled evaluation of effort selection and release its data: repeated responses from open models at every effort setting on the same questions. We first show that adapting effort to the question can save nearly half of the generated tokens with almost no loss in accuracy. We then compare two practical strategies, predicting from the question text and checking agreement among cheap-model answers, with every model call charged. We propose question-level cost forecasts that improve an existing selector without extra generation. Finally, we identify where further gains lie: knowing where extra reasoning helps matters beyond knowing where it is expensive, but this benefit is model specific and current forecasts capture little of it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reasoning LLMs, reasoning effort, adaptive computation, test-time compute, inference-time scaling, cost-aware inference, token efficiency, adaptive reasoning, model routing, question-specific computation, LLM efficiency, selective computation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NZHAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how to dynamically select Reasoning Effort based on individual questions to reduce inference costs while maintaining accuracy. Through an analysis of nearly 600,000 inference records across 9 models and 10 benchmarks, the authors demonstrate that appropriate effort allocation can reduce generated tokens by approximately 48% with only a 0.29 percentage-point decrease in accuracy. The authors propose two approaches: (1) an embedding-based two-stage regression model to predict accuracy gains (Gain) and token costs (Cost) under different reasoning effort levels; and (2) a training-free approach that uses answer agreement across multiple attempts by a smaller model to guide effort selection, while improving RADAR through question-level cost prediction. The experiments suggest that accurately predicting Gain is more valuable than predicting Cost alone. However, Gain exhibits strong model specificity and remains difficult to predict accurately with existing methods, making it an important direction for future adaptive reasoning optimization.

Critical strengths
  1. The motivation of this paper is valuable. The authors aim to dynamically allocate Reasoning Effort based on individual questions to reduce inference costs while maintaining accuracy. This is a practically important research problem with significant potential for real-world applications.
  2. The authors propose two approaches that cover both training-based and training-free settings. Both methods demonstrate certain improvements in experimental metrics.
Critical weaknesses
  1. Concerns about the effectiveness of embedding-based prediction and its generalization ability.

    The authors use Qwen3-Embedding-0.6B to convert each question into an embedding vector and then train a first-stage predictor to estimate accuracy and token consumption under different reasoning effort levels. The second-stage predictor is further calibrated using a small amount of data from the target model to adapt the predictions to its capabilities.

    My main concern is: How can the authors ensure that a test question can be reliably matched to patterns in the training data through embedding similarity?

    In my view, the first-stage predictor essentially learns a mapping from question representations to expected accuracy and token consumption based on the training data. However, a key factor in determining whether a question benefits from a higher reasoning budget is whether the task fundamentally requires multi-step reasoning. For example, mathematical proofs, symbolic manipulation, and coding tasks are generally more likely to benefit from additional reasoning.

    The proposed predictor relies on semantic embeddings, which means that its generalization largely depends on the relationship between semantic similarity and the patterns in the training data. However, high semantic similarity does not necessarily imply similar question difficulty or comparable reasoning requirements. Two semantically similar questions may differ substantially in their reasoning complexity and their potential benefits from increased reasoning effort.

    Therefore, I doubt whether this training approach can accurately predict the performance of an individual question under a specific reasoning budget. The predictor operates on a relatively limited amount of data and attempts to establish a mapping from question embeddings to reasoning-related performance through a lightweight regression model. However, understanding question difficulty and reasoning requirements is a challenging task that typically requires the capabilities of LLMs developed through large-scale pretraining.

    Why do the authors not directly use a pretrained LLM to model the two-stage prediction process as a text generation task, or attach a regression head to an LLM and train it for direct performance prediction? Such approaches may provide stronger semantic understanding and reasoning-aware representations than a lightweight predictor built on fixed embeddings.

  2. Potential reliability issues with the agreement-based effort allocation method.

    I have concerns about the method described around Line 400, "Agreement guides effort without target calibration."

    The authors assume that if a smaller model produces consistent answers across four attempts, a larger model can solve the question using a low reasoning budget. Otherwise, a higher reasoning budget is assigned. However, this approach relies on several assumptions that may not always hold:

    (1) Higher answer agreement from a smaller model indicates greater confidence in its answer. First, answer consistency depends on generation parameters such as temperature. More importantly, consistent answers do not necessarily indicate correctness or well-calibrated confidence. A model may consistently produce the same incorrect answer.

    (2) Questions with higher agreement among smaller-model responses require less reasoning effort from a larger model. The capability boundaries of smaller and larger models cannot necessarily be estimated through answer agreement. For example, questions that lead to inconsistent answers from a smaller model may still be straightforward for a more capable model and may not require additional reasoning effort.

    In other words, the relationship between smaller-model agreement and larger-model reasoning requirements may be weak or unstable. This relationship may become even less reliable as the gap in model size and capability increases, especially when the target model is substantially more capable, such as Claude 5.5 or GPT-6.

    Furthermore, the authors acknowledge that introducing a smaller model increases the total number of generated tokens. This additional computation may also introduce inference latency, potentially offsetting some of the practical benefits of adaptive effort allocation.

  3. Incomplete model specifications in the experimental tables.

    The experimental tables lack sufficient details about the evaluated models, including specific model versions and parameter counts. For example, Table 4 does not specify the parameter count of GLM, while the exact versions of Qwen, GLM, Gemma, and several other models are unclear. These omissions make it difficult to identify the precise model configurations and may negatively affect the reproducibility of the experiments.

Questions for authors

Please refer to the issues raised in the Critical Weaknesses section.

Reviewer_YBUjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors examine how the reasoning effort argument to a language model affects its performance and token output, across models and tasks. They train predictors of these metrics (called “gain” and “cost”), using four signals: (1) other attempts on the same problem by the same model, (2) attempts of problems from the same dataset, (3) the raw text of the problem, and (4) attempts on the problem from a smaller, cheaper model. They conduct analysis of these signals, showing how predictors can use them to substantially reduce cost at very little degradation to accuracy.

Critical strengths

The question of how the reasoning effort parameter affects performance and cost is an important one, and a strong study that provides insight into these relatively opaque settings could be valuable for informing people’s usage of reasoning models.

The resource of nearly 600,000 reasoning traces generated from each of these models under different effort parameters is expensive to produce and a meaningful contribution in its own right, although it is limited by the omission of any frontier-level open or closed models (where it could include the total token cost, if not the reasoning trace)

Critical weaknesses

The presentation is very unclear. Some terms are introduced but never used (e.g. δ\delta on page two, which also has notational abuse of three subscripts). A tiny portion of the paper is dedicated to related work, which is inadequate given how many papers have studied effort allocation and routing in language models (e.g. “Just Enough Thinking” targets this exact question and is used in the training of the MAI-Thinking series of models, but is not mentioned). Paragraph headers disrupt rather than help direct the reader’s flow. Asides are peppered throughout (e.g. line 104). Figure text is too small to read easily. Confidence intervals are presented in a difficult to parse fashion.

The methods used are of too limited novelty to be a contribution in themselves. Although the analysis of them and associated resource is still interesting and a meaningful contribution, this is limited by the lack of clarity and focus of this analysis, which makes it difficult to learn much from the paper despite thorough experimentation.

Questions for authors

Clarifications:

  • What does “embedding-based costs” mean? (line 350)
  • What does “costs are tokenizer-specific generated tokens” mean? Aren’t all generated tokens tokenizer-specific? (line 111)
  • What is the “prompt-length cost model”? (326)

Future Work:

  • Why is so much of the improvement over RADAR 4.2 delivered only on GLM and Nemotron?
  • Likewise, why are ZebraLogic and CogniLoad the main subjects of this improvement? This undermines the generality of the method’s contribution, and also points in a direction for possible interesting research exploring why these are particularly sensitive to your method.
Reviewer_2djxOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper studies when increasing a reasoning model’s effort is worth the additional token cost and whether effort can be predicted and selected for each question. They use repeated generations across nine models and ten benchmarks to first measure the potential benefits of question-specific effort selection and then evaluate predictors of accuracy gains and token costs. They find that token costs are easier to predict than accuracy gains, and adding these cost predictions into an existing selector improves the accuracy-token trade-off.

Critical strengths

Not a critical strength but a positive aspect of the paper is the scale of the evaluation across nine models and ten benchmarks, including repeated generations and comparisons with simple baselines.

Critical weaknesses
  1. The main issue with the paper is that the practical improvements are modest. In Section 4, the cost-prediction modification to RADAR improves average accuracy by 0.69 percentage points at matched token use, but this drops to 0.08 points when GLM and Nemotron are excluded. They also report 8% token savings at the midpoint of the fixed-setting accuracy range. These results do not convincingly show a broad advantage over simpler alternatives. For example, random mixtures of medium and high effort match the learned selectors for GPT-OSS at the midpoint token budget. Also, the learned predictors are evaluated on held-out questions from the same benchmarks used for training and calibration. Their effectiveness on samples from unseen distributions is not tested, which is an important limitation for practical deployment.

  2. The second problem is the presentation, which I found hard to follow. The introduction contains many numerical results and experimental details before clearly establishing the problem and main contributions. The paper then moves directly into the experimental setup, and even the contents of these experimental sections are not well organized.

Questions for authors

My main question is how well do the learned predictors generalize to an unseen benchmark without calibration on that benchmark?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.