- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces Think-Bench, a benchmark for evaluating reasoning efficiency and chain-of-thought quality. It contains 1,375 mathematics, physics, and chemistry problems with human-reviewed key reasoning steps. The authors evaluate 14 reasoning models using metrics covering accuracy, reasoning quality, and reflection behavior, and report substantial overthinking across models.
Critical strengths
- The benchmark includes human-reviewed reasoning-step annotations across three disciplines, supporting evaluation beyond final-answer accuracy.
- The authors evaluate multiple models and include human agreement and judge reliability studies, providing useful checks on the automated evaluation pipeline.
Critical weaknesses
- The efficiency metric does not adequately measure computational efficiency. A model that takes 2000 tokens to answer correctly and immediately stops scores higher than one that answers after 700 tokens and verifies for another 300. The metric penalizes post-answer generation while overlooking wasted reasoning before the answer. It also does not establish whether subsequent verification is unnecessary. I would be happy to raise my score if the authors can address this concern and provide evidence that the metric reliably captures reasoning efficiency.
- Several reported results are inconsistent. Claude’s overall reflection quality in Table 2 is lower than both difficulty-specific scores in Table 3. DeepSeek-R1-distill-Qwen-7B’s overall token count exceeds all three subject averages in Table 4. Appendix E.1 also gives conflicting annotation-agreement counts. These discrepancies need clarification before the model comparisons can be considered reliable.
Questions for authors
- How should Equation 1 be interpreted when a much longer trace receives a higher efficiency score simply because the correct answer appears later? Does this score agree with accuracy-cost comparisons under controlled stopping or compression experiments?
- Can you reconcile the aggregate and subgroup results in Tables 2–4 and the annotation-agreement counts in Appendix E.1? Were the same samples and aggregation rules used throughout?
- How are Reasonable and N/A judgments handled in the precision calculation? How do the limits of 35 reasoning steps and 30 reflection steps in the judging prompts affect evaluations of long traces?
Response to Q1:
We use Equation (1) to describe the relative position of the first correct answer within the entire trajectory. In cases where a longer trajectory receives a higher ratio, Total Tokens and First Correct Tokens will both indicate its longer generation process, while final accuracy provides a measure of answer quality. We interpret model performance using these metrics jointly.
Table 2 contains a comparison relevant to your question: Grok-3-mini-beta and DeepSeek-R1 have accuracies of 93.96% and 93.82%, Total Tokens of 1,891.34 and 3,795.19, First Correct Tokens of 1,169.05 and 1,912.12, and corresponding ratios of 61.69% and 48.96%, respectively. In this specific comparison, Grok has a shorter total generation length, an earlier first correct answer, and less subsequent generation, supporting its advantage in token usage at similar accuracy.
Regarding controlled stopping or compression, we have not yet conducted controlled experiments that stop at the first correct answer or compress trajectories. The available intervention results are the GPT-5 token-budget experiments in Table 8 (page 15):
These results show that reducing the generation budget affects final accuracy and that outcome quality must also be considered when evaluating reasoning length.
Response to Q3:
In our original implementation, both
MatchandReasonablecount as valid reasoning steps,Wrongcounts as an invalid step, andN/Ais excluded. The actual formula used to calculate Precision is:Precision=nMatch+nReasonable+nWrongnMatch+nReasonable.
Here, each n is the number of steps with the corresponding label.
Reasonabledenotes reasoning that is logically valid but does not appear directly in the reference annotations, so valid alternative solutions can receive credit. The definitions of the four labels are provided on page 24, lines 1292–1295, and page 25, line 1297.The notation in the original Equation (4) does not fully express this counting procedure. We plan to replace the definition of Precision on page 6, lines 291–303, with this formula and explicitly state:
The limits of 35 and 30 steps are the respective maximum numbers of reasoning-annotation and reflection-annotation records returned by the judge. Page 25, lines 1301–1302, require highly repetitive reasoning patterns to be merged while retaining the final-answer step; page 27, line 1405, specifies a maximum of 30 reflection events in the output. Repetition in reflection is itself a behavior that needs to be evaluated, and its quality is still assessed according to the reflection rules.