- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for understanding why reasoning models differ in accuracy-per-token efficiency. It first separates answer availability, correctness, and generation cost, while including the tokens spent on failed and truncated responses. It then examines how generation cost changes with task workload, using benchmark-specific measures and a solver-derived proxy for tasks without workload annotations.
The experiments cover 785,541 responses across twelve benchmarks. The main findings are that models can use substantially different numbers of tokens even when answering the same questions correctly, that relative cost can change or reverse as workload increases, and that greater reasoning effort usually improves accuracy while reducing accuracy per token.
Critical strengths
Careful accounting of generation cost. The paper includes failed and truncated responses in its cost calculations and compares successful responses on shared questions. These choices make the comparisons more informative: they account for the cost of failure while also showing whether token differences remain when both models succeed.
Broad experiments with strong statistical support. The study covers many models, benchmarks, and reasoning settings. Paired cluster bootstraps and simultaneous confidence intervals strengthen the comparisons, while the split-half and scoring-sensitivity analyses provide additional evidence that the main patterns are stable.
Critical weaknesses
1.The workload-dependent results need a clearer interpretation. These comparisons include incorrect and truncated outputs, so changes in relative cost may reflect a mixture of successful reasoning, changing failure rates, and generation limits. For example, the ZebraLogic reversal is an interesting result, but it is unclear how much of it remains when both models answer correctly. A corresponding analysis of correct responses on shared-success questions, together with a check of length-stopped responses, would help explain what drives the result.
2.The contribution needs clearer positioning against related work. The paper’s main value appears to be its measurement framework and extensive empirical analysis. The distinction from existing work on token efficiency and difficulty-dependent behavior should be explained more directly, particularly in relation to OckBench, ThoughtTerminator, and OptimalThinkingBench. Showing how the decomposition changes an evaluation conclusion or practical decision would strengthen the case for its added value.
Questions for authors
1.Do the ZebraLogic reversal and the broader workload-dependent cost patterns persist when comparing correct responses on shared-success questions within each workload group? How sensitive are these results to length-stopped responses?
2.What additional conclusions does the framework support beyond existing accuracy–cost and difficulty-conditioned analyses? Please clarify this distinction in the comparison with related work.
3.For the Qwen/Gemma comparison, does the reported cost gap remain similar when output length is measured in characters or bytes? This would help establish how much the cross-model differences depend on tokenization.