- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues overthinking should be studied not via accuracy and average token length of reasoning traces, but with respect to instability of decoding. Then the authors introduce several metrics to quantify overthinking.
Critical strengths
I agree with the general idea: only looking at Acc, token length is not the best way to quantify reasoning efficiency. However, this is the norm currently in the community for reasoning efficiency.
Critical weaknesses
-
No intuition. The paper starts right away by defining several metrics. There is no motivation for why the authors choose these specific metrics over other choices.
-
Limited insights. The paper evaluates the framework with seven models on three math benchmarks, but provides little insight. It is still unclear why one should adopt this framework over traditional accuracy and token-count metrics. The good thing about accuracy and token count is that they provide an easy way to compare checkpoints: at the same accuracy, if one checkpoint uses fewer tokens, it is clearly better. However, the several metrics introduced in the paper do not seem as interpretable or meaningful.
-
Expensive. To reliably estimate the introduced metrics, the authors generate 100 samples per problem. That is very expensive on hard benchmarks such as AIME or LiveCodeBench. This is probably why the framework’s main experiments are limited to easier benchmarks such as MATH500 and GSM8K.
Questions for authors
I seem unable to draw any conclusions from the results in Table 1, and I would appreciate it if the authors could clarify how to interpret them. For example, in Section 5.2, the authors argue that “Instability costs accuracy, not only compute, since on the same problem a run that goes unusually long is more likely to be wrong,” based on the value of W-corr. But for many models, the values are around −0.1 or even smaller in magnitude. These negative correlations seem too weak for the authors to draw such a concrete conclusion.
Also, one suggestion that I think could make this framework more interesting: there are several early-stopping and post-training methods for reasoning efficiency ([1,2,3,4] and several others) that are evaluated mainly through accuracy and token length, but these metrics do not explain exactly how the methods differ. Applying this framework to these methods could provide more insight into how each method works differently.
[1] Yang, Chenxu, et al. "Dynamic Early Exit in Reasoning Models."
[2] Hosseini, Parsa, et al. "Early Stopping for Large Reasoning Models via Confidence Dynamics."
[3] Zhao, Anhao, et al. "On-policy supervised fine-tuning for efficient reasoning."
[4] Arora, Daman, and Andrea Zanette. "Training language models to reason efficiently."