- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates how to dynamically select Reasoning Effort based on individual questions to reduce inference costs while maintaining accuracy. Through an analysis of nearly 600,000 inference records across 9 models and 10 benchmarks, the authors demonstrate that appropriate effort allocation can reduce generated tokens by approximately 48% with only a 0.29 percentage-point decrease in accuracy. The authors propose two approaches: (1) an embedding-based two-stage regression model to predict accuracy gains (Gain) and token costs (Cost) under different reasoning effort levels; and (2) a training-free approach that uses answer agreement across multiple attempts by a smaller model to guide effort selection, while improving RADAR through question-level cost prediction. The experiments suggest that accurately predicting Gain is more valuable than predicting Cost alone. However, Gain exhibits strong model specificity and remains difficult to predict accurately with existing methods, making it an important direction for future adaptive reasoning optimization.
Critical strengths
- The motivation of this paper is valuable. The authors aim to dynamically allocate Reasoning Effort based on individual questions to reduce inference costs while maintaining accuracy. This is a practically important research problem with significant potential for real-world applications.
- The authors propose two approaches that cover both training-based and training-free settings. Both methods demonstrate certain improvements in experimental metrics.
Critical weaknesses
-
Concerns about the effectiveness of embedding-based prediction and its generalization ability.
The authors use Qwen3-Embedding-0.6B to convert each question into an embedding vector and then train a first-stage predictor to estimate accuracy and token consumption under different reasoning effort levels. The second-stage predictor is further calibrated using a small amount of data from the target model to adapt the predictions to its capabilities.
My main concern is: How can the authors ensure that a test question can be reliably matched to patterns in the training data through embedding similarity?
In my view, the first-stage predictor essentially learns a mapping from question representations to expected accuracy and token consumption based on the training data. However, a key factor in determining whether a question benefits from a higher reasoning budget is whether the task fundamentally requires multi-step reasoning. For example, mathematical proofs, symbolic manipulation, and coding tasks are generally more likely to benefit from additional reasoning.
The proposed predictor relies on semantic embeddings, which means that its generalization largely depends on the relationship between semantic similarity and the patterns in the training data. However, high semantic similarity does not necessarily imply similar question difficulty or comparable reasoning requirements. Two semantically similar questions may differ substantially in their reasoning complexity and their potential benefits from increased reasoning effort.
Therefore, I doubt whether this training approach can accurately predict the performance of an individual question under a specific reasoning budget. The predictor operates on a relatively limited amount of data and attempts to establish a mapping from question embeddings to reasoning-related performance through a lightweight regression model. However, understanding question difficulty and reasoning requirements is a challenging task that typically requires the capabilities of LLMs developed through large-scale pretraining.
Why do the authors not directly use a pretrained LLM to model the two-stage prediction process as a text generation task, or attach a regression head to an LLM and train it for direct performance prediction? Such approaches may provide stronger semantic understanding and reasoning-aware representations than a lightweight predictor built on fixed embeddings.
-
Potential reliability issues with the agreement-based effort allocation method.
I have concerns about the method described around Line 400, "Agreement guides effort without target calibration."
The authors assume that if a smaller model produces consistent answers across four attempts, a larger model can solve the question using a low reasoning budget. Otherwise, a higher reasoning budget is assigned. However, this approach relies on several assumptions that may not always hold:
(1) Higher answer agreement from a smaller model indicates greater confidence in its answer. First, answer consistency depends on generation parameters such as temperature. More importantly, consistent answers do not necessarily indicate correctness or well-calibrated confidence. A model may consistently produce the same incorrect answer.
(2) Questions with higher agreement among smaller-model responses require less reasoning effort from a larger model. The capability boundaries of smaller and larger models cannot necessarily be estimated through answer agreement. For example, questions that lead to inconsistent answers from a smaller model may still be straightforward for a more capable model and may not require additional reasoning effort.
In other words, the relationship between smaller-model agreement and larger-model reasoning requirements may be weak or unstable. This relationship may become even less reliable as the gap in model size and capability increases, especially when the target model is substantially more capable, such as Claude 5.5 or GPT-6.
Furthermore, the authors acknowledge that introducing a smaller model increases the total number of generated tokens. This additional computation may also introduce inference latency, potentially offsetting some of the practical benefits of adaptive effort allocation.
-
Incomplete model specifications in the experimental tables.
The experimental tables lack sufficient details about the evaluated models, including specific model versions and parameter counts. For example, Table 4 does not specify the parameter count of GLM, while the exact versions of Qwen, GLM, Gemma, and several other models are unclear. These omissions make it difficult to identify the precise model configurations and may negatively affect the reproducibility of the experiments.
Questions for authors
Please refer to the issues raised in the Critical Weaknesses section.