acceptodds
Under review as a conference paper at ICLR 2027

THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Models

Abstract

Large reasoning models (LRMs) have achieved impressive performance in complex tasks, often outperforming conventional large language models (LLMs). However, the prevalent issue of overthinking severely limits their computational efficiency. Overthinking occurs when models generate excessive and redundant tokens that contribute little to accurate outcomes, especially in simple tasks, resulting in a significant waste of computational resources. To systematically investigate this issue, we introduce Think-Bench, a benchmark designed to evaluate the reasoning efficiency of LRMs. We also propose novel efficiency metrics and conduct a comprehensive evaluation of various LRMs across multiple dimensions, including the reasoning process, outcome quality, and chain-of-thought (CoT) characteristics. Our analysis reveals that most LRMs exhibit overthinking in handling easy questions, generating unnecessarily lengthy reasoning chains. While many LRMs demonstrate high CoT quality, several suffer from low efficiency. We hope that Think-Bench can serve as a robust foundation for advancing research into LRMs.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Large Reasoning Model, Thinking Efficiency, Chain-of-thought

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EmyoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Think-Bench, a benchmark for evaluating reasoning efficiency and chain-of-thought quality. It contains 1,375 mathematics, physics, and chemistry problems with human-reviewed key reasoning steps. The authors evaluate 14 reasoning models using metrics covering accuracy, reasoning quality, and reflection behavior, and report substantial overthinking across models.

Critical strengths
  1. The benchmark includes human-reviewed reasoning-step annotations across three disciplines, supporting evaluation beyond final-answer accuracy.
  2. The authors evaluate multiple models and include human agreement and judge reliability studies, providing useful checks on the automated evaluation pipeline.
Critical weaknesses
  1. The efficiency metric does not adequately measure computational efficiency. A model that takes 2000 tokens to answer correctly and immediately stops scores higher than one that answers after 700 tokens and verifies for another 300. The metric penalizes post-answer generation while overlooking wasted reasoning before the answer. It also does not establish whether subsequent verification is unnecessary. I would be happy to raise my score if the authors can address this concern and provide evidence that the metric reliably captures reasoning efficiency.
  2. Several reported results are inconsistent. Claude’s overall reflection quality in Table 2 is lower than both difficulty-specific scores in Table 3. DeepSeek-R1-distill-Qwen-7B’s overall token count exceeds all three subject averages in Table 4. Appendix E.1 also gives conflicting annotation-agreement counts. These discrepancies need clarification before the model comparisons can be considered reliable.
Questions for authors
  1. How should Equation 1 be interpreted when a much longer trace receives a higher efficiency score simply because the correct answer appears later? Does this score agree with accuracy-cost comparisons under controlled stopping or compression experiments?
  2. Can you reconcile the aggregate and subgroup results in Tables 2–4 and the annotation-agreement counts in Appendix E.1? Were the same samples and aggregation rules used throughout?
  3. How are Reasonable and N/A judgments handled in the precision calculation? How do the limits of 35 reasoning steps and 30 reflection steps in the judging prompts affect evaluations of long traces?
1 comment

Response to Q1:

We use Equation (1) to describe the relative position of the first correct answer within the entire trajectory. In cases where a longer trajectory receives a higher ratio, Total Tokens and First Correct Tokens will both indicate its longer generation process, while final accuracy provides a measure of answer quality. We interpret model performance using these metrics jointly.

Table 2 contains a comparison relevant to your question: Grok-3-mini-beta and DeepSeek-R1 have accuracies of 93.96% and 93.82%, Total Tokens of 1,891.34 and 3,795.19, First Correct Tokens of 1,169.05 and 1,912.12, and corresponding ratios of 61.69% and 48.96%, respectively. In this specific comparison, Grok has a shorter total generation length, an earlier first correct answer, and less subsequent generation, supporting its advantage in token usage at similar accuracy.

Regarding controlled stopping or compression, we have not yet conducted controlled experiments that stop at the first correct answer or compress trajectories. The available intervention results are the GPT-5 token-budget experiments in Table 8 (page 15):

Token budgetAccuracy
3,00075.85%
5,00084.87%
Without this token limit93.16%

These results show that reducing the generation budget affects final accuracy and that outcome quality must also be considered when evaluating reasoning length.

Response to Q3:

In our original implementation, both Match and Reasonable count as valid reasoning steps, Wrong counts as an invalid step, and N/A is excluded. The actual formula used to calculate Precision is:

Precision=nMatch+nReasonablenMatch+nReasonable+nWrong.\mathrm{Precision}=\frac{n_{\mathrm{Match}}+n_{\mathrm{Reasonable}}}{n_{\mathrm{Match}}+n_{\mathrm{Reasonable}}+n_{\mathrm{Wrong}}}.

Here, each nn is the number of steps with the corresponding label. Reasonable denotes reasoning that is logically valid but does not appear directly in the reference annotations, so valid alternative solutions can receive credit. The definitions of the four labels are provided on page 24, lines 1292–1295, and page 25, line 1297.

The notation in the original Equation (4) does not fully express this counting procedure. We plan to replace the definition of Precision on page 6, lines 291–303, with this formula and explicitly state:

nMatchn_{\mathrm{Match}}, nReasonablen_{\mathrm{Reasonable}}, and nWrongn_{\mathrm{Wrong}} denote the numbers of reasoning steps that match the reference, are valid but do not match the reference, and are invalid, respectively. Both Match and Reasonable count as valid steps, and N/A is excluded from the calculation of Precision. This formula applies to samples with a nonzero denominator; Recall is calculated based on the coverage of reference key steps.

The limits of 35 and 30 steps are the respective maximum numbers of reasoning-annotation and reflection-annotation records returned by the judge. Page 25, lines 1301–1302, require highly repetitive reasoning patterns to be merged while retaining the final-answer step; page 27, line 1405, specifies a maximum of 30 reflection events in the output. Repetition in reflection is itself a behavior that needs to be evaluated, and its quality is still assessed according to the reflection rules.

Reviewer_njDGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces THINK-Bench, a benchmark designed to evaluate the reasoning efficiency and Chain-of-Thought (CoT) quality of large reasoning models (LRMs). Motivated by the overthinking phenomenon, where models generate excessive reasoning tokens without corresponding improvements in accuracy, the authors argue that conventional outcome-based benchmarks fail to adequately characterize reasoning efficiency and intermediate reasoning quality.

THINK-Bench comprises 1,375 questions across mathematics, physics, and chemistry, accompanied by human-reviewed key reasoning steps. The authors introduce evaluation metrics covering token consumption, first-correct-answer position, reflection behavior, and step-level reasoning quality, including recall and precision. Experiments on 14 LRMs, using Claude 3.7 Sonnet as an automatic judge, suggest that stronger reasoning performance frequently comes at the expense of longer reasoning traces. The authors conclude that reasoning quality and efficiency should be evaluated jointly, highlighting the need for more adaptive and computationally efficient reasoning strategies.

Critical strengths
  1. Relevant Problem and Evaluation Perspective.

The paper addresses the important problem of overthinking in LRMs. Evaluating reasoning efficiency alongside answer correctness is relevant to understanding and optimizing inference-time computation, particularly as extended reasoning becomes increasingly common.

  1. Fine-Grained Benchmark Construction and Broad Model Coverage.

THINK-Bench incorporates human-reviewed reasoning-step annotations across three scientific disciplines and evaluates 14 representative LRMs. The combination of step-level annotations and cross-model comparisons provides a useful basis for examining differences in reasoning behavior beyond final-answer accuracy.

Critical weaknesses
  1. Conceptual Flaws in the Proposed Efficiency Metric (Section 3.1).

The central efficiency metric is defined as the ratio of First Correct Tokens to Total Tokens. However, for a fixed total token count, this metric assigns higher efficiency to models that reach the first correct answer later, contradicting the stated objective of efficient reasoning. Furthermore, identifying an intermediate correct answer does not guarantee final-answer correctness, as models may subsequently revise their conclusions. The distinction between "useful tokens" and "reflection tokens" is also questionable: reasoning before the first correct answer may be redundant or incorrect, while subsequent verification may be essential. These issues fundamentally undermine the validity and interpretation of the proposed efficiency measurements.

  1. Insufficient Consideration of Computational Cost and Cross-Model Comparability.

The paper evaluates reasoning efficiency primarily through token-based metrics without adequately accounting for differences in computational cost across model architectures and scales. Token generation costs depend on model size and architecture; a more appropriate computational-efficiency evaluation should consider quantities such as generated tokens multiplied by active model parameters, FLOPs, or inference latency. Additionally, the paper does not sufficiently clarify how reasoning traces are obtained and standardized for proprietary models, particularly Claude 3.7 Sonnet. These limitations weaken the reliability and comparability of the reported cross-model efficiency rankings.

Questions for authors
  1. First Correct Answer vs. Final Correctness. How does the evaluation handle cases where a model initially produces a correct answer but subsequently changes it to an incorrect final answer? Does the First Correct Tokens metric explicitly account for final-answer correctness?

  2. Validity of the Efficiency Metric. Why is efficiency defined as First Correct Tokens divided by Total Tokens, given that this ratio increases when the first correct answer occurs later under a fixed token budget? How do the authors justify treating tokens before the first correct answer as useful and subsequent tokens as reflection overhead?

  3. Computational Cost. How can the proposed token-based metrics support meaningful efficiency comparisons across models with substantially different computational costs? Could the authors provide cost-aware measurements, such as FLOPs, active parameter counts, or inference latency?

  4. Reasoning Trace Extraction. How were reasoning traces collected from Claude 3.7 Sonnet? Was extended thinking enabled, and what inference provider, thinking budget, and generation settings were used? In particular, how does this configuration relate to the stated temperature of 0?

  5. Annotation Agreement Inconsistency (Appendix E.1). The paper reports 252 agreed instances out of 300 but claims an agreement rate of 94% (282/300). The confusion matrix supports 282 agreements and the reported Cohen's kappa. Could the authors clarify and correct these inconsistent statistics?

Reviewer_mfKJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Think-Bench, a multidisciplinary benchmark with fine-grained step annotations for evaluating reasoning efficiency and CoT quality. The experiments provide insights into models’ reasoning behavior.

Critical strengths
  • The evaluation motivation is intuitive and easy to understand.

  • The benchmark covers multiple domains and evaluates a range of models.

Critical weaknesses
  1. The claimed evaluation gap needs clearer justification. Step-level reasoning correctness has been studied in PRM800K, REVEAL, and ProcessBench, while MME-CoT already evaluates both CoT quality and efficiency. The paper should clarify what additional evaluation capabilities Think-Bench provides beyond these frameworks.

  2. Claude 3.7 Sonnet is used for both reference generation and evaluation, which may introduce correlated biases. Although human validation is provided, it remains unclear whether the model rankings are robust to the choice of judge.

  3. The efficiency metric measures a ratio rather than absolute token cost. Models using very different numbers of tokens can receive the same score, so its interpretation as reasoning efficiency needs further justification.

Questions for authors

NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.