acceptodds
Under review as a conference paper at ICLR 2027

Beyond Accuracy: Decomposing the Reasoning Efficiency of LLMs

Abstract

Accuracy per token summarizes reasoning efficiency but does not reveal whether differences arise from failed attempts, costly successful answers, or disproportionately long reasoning relative to the effort a problem demands. We introduce a two-layer protocol that connects these distinctions to the same efficiency score. The outcome layer separates answer availability, correctness among available answers, and generation cost. The workload layer relates cost to task structure, separating mean generated tokens per workload unit from their association with workload. Across 785,541 responses on twelve benchmarks, cost gaps persist among successful answers: Qwen3.5-27B generates 1.79 to 3.46 times as many tokens as Gemma-4-31B for correct outputs on shared problems across ten benchmarks. Relative costs also change with workload. On ZebraLogic, Qwen3.5-27B's cost relative to GPT-OSS-120B changes from 4.49 at low workload to 0.78 at high workload, despite similarly long responses on average. Across nine checkpoints on four annotated benchmarks, simultaneous 95% confidence intervals support relative-cost changes in 82 of 144 model-pair comparisons and effort-induced changes in 21 of 36 within-checkpoint comparisons. These results show that successful-answer cost and workload response are complementary properties of reasoning models: cost gaps can persist on shared successful questions while changing substantially with task structure and inference settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reasoning efficiency, test-time compute, reasoning evaluation, token efficiency, workload-aware evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MMWVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for understanding why reasoning models differ in accuracy-per-token efficiency. It first separates answer availability, correctness, and generation cost, while including the tokens spent on failed and truncated responses. It then examines how generation cost changes with task workload, using benchmark-specific measures and a solver-derived proxy for tasks without workload annotations.

The experiments cover 785,541 responses across twelve benchmarks. The main findings are that models can use substantially different numbers of tokens even when answering the same questions correctly, that relative cost can change or reverse as workload increases, and that greater reasoning effort usually improves accuracy while reducing accuracy per token.

Critical strengths

Careful accounting of generation cost. The paper includes failed and truncated responses in its cost calculations and compares successful responses on shared questions. These choices make the comparisons more informative: they account for the cost of failure while also showing whether token differences remain when both models succeed.

Broad experiments with strong statistical support. The study covers many models, benchmarks, and reasoning settings. Paired cluster bootstraps and simultaneous confidence intervals strengthen the comparisons, while the split-half and scoring-sensitivity analyses provide additional evidence that the main patterns are stable.

Critical weaknesses

1.The workload-dependent results need a clearer interpretation. These comparisons include incorrect and truncated outputs, so changes in relative cost may reflect a mixture of successful reasoning, changing failure rates, and generation limits. For example, the ZebraLogic reversal is an interesting result, but it is unclear how much of it remains when both models answer correctly. A corresponding analysis of correct responses on shared-success questions, together with a check of length-stopped responses, would help explain what drives the result.

2.The contribution needs clearer positioning against related work. The paper’s main value appears to be its measurement framework and extensive empirical analysis. The distinction from existing work on token efficiency and difficulty-dependent behavior should be explained more directly, particularly in relation to OckBench, ThoughtTerminator, and OptimalThinkingBench. Showing how the decomposition changes an evaluation conclusion or practical decision would strengthen the case for its added value.

Questions for authors

1.Do the ZebraLogic reversal and the broader workload-dependent cost patterns persist when comparing correct responses on shared-success questions within each workload group? How sensitive are these results to length-stopped responses?

2.What additional conclusions does the framework support beyond existing accuracy–cost and difficulty-conditioned analyses? Please clarify this distinction in the comparison with related work.

3.For the Qwen/Gemma comparison, does the reported cost gap remain similar when output length is measured in characters or bytes? This would help establish how much the cross-model differences depend on tokenization.

Reviewer_SDYbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a two-layer framework for analyzing differences in LLM reasoning efficiency: the outcome layer separates answer availability, conditional correctness, and generation cost, while the workload layer further distinguishes mean tokens per workload unit from their coupling with workload. Through experiments covering twelve benchmarks and approximately 786,000 responses, the authors find that cost differences persist on questions both models answer correctly, relative costs can change or even reverse with workload, and increased reasoning effort typically improves accuracy at the expense of token efficiency. By combining exact decomposition with comparisons on shared questions, the framework provides a detailed diagnosis of aggregate efficiency differences.

Critical strengths
  • The exact decomposition provides an interpretable account of reasoning efficiency.
  • Experiments spanning twelve benchmarks provide broad empirical support.
Critical weaknesses
  • The paper does not clearly establish the added diagnostic value of its algebraic decomposition over simpler accuracy–cost analyses and workload-stratified response-length comparisons.
  • The conclusion that higher reasoning effort reduces efficiency is based primarily on endpoint comparisons under accuracy-per-token, without establishing whether it remains inefficient at matched accuracy or along the attainable accuracy–cost frontier.
  • Although the paper includes workload sensitivity analyses, it provides limited direct evidence that the separate overhead and coupling contributions remain stable under plausible alternative workload definitions. Since these definitions can redistribute the two contributions without changing their sum, the robustness of the component-level interpretation remains unclear.
Questions for authors

How robust are the separate overhead and coupling contributions to alternative workload definitions, such as using the total number of proof steps instead of proof depth?

Reviewer_JvxqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a two-layer evaluation framework for analyzing the reasoning efficiency of LLMs beyond conventional accuracy-per-token metrics. The outcome layer decomposes efficiency into answer availability, conditional correctness, and generation cost, while the workload layer further analyzes token consumption relative to task difficulty, including mean tokens per workload unit and workload coupling. Through experiments across multiple models and benchmarks, the authors demonstrate that token cost differences persist even among correctly answered questions, that the relative efficiency of different models can vary substantially with question workload, and that increasing reasoning effort generally improves accuracy at the expense of token efficiency.

Overall, I find this paper to be a borderline paper. I appreciate the systematic decomposition of reasoning efficiency and believe that some of the empirical findings provide useful insights into differences between reasoning models. However, several findings are relatively intuitive, and the reliance on workload estimates limits the general applicability of the proposed framework. While the work has value as an empirical analysis, I am not fully convinced that its novelty and broader utility are sufficiently strong for clear acceptance.

Critical strengths
  1. Interesting and informative evaluation framework. I appreciate the proposed decomposition of reasoning efficiency along multiple dimensions, including token consumption, answer availability, correctness, and question workload. This framework provides a more detailed understanding of model efficiency than simply reporting accuracy and average token usage.

  2. Interesting empirical observations. Some experimental findings are particularly interesting. For example, the comparison between Qwen3.5-27B and GPT-OSS-120B on ZebraLogic reveals that their relative token costs change significantly across different workload levels, even reversing which model is more efficient. This observation highlights the limitations of evaluating reasoning efficiency solely through aggregate statistics.

Critical weaknesses
  1. Some findings are relatively intuitive. Although the paper reports several interesting observations, some of its conclusions are rather expected. For example, the finding that increasing reasoning effort generally improves accuracy while reducing accuracy-per-token efficiency is not particularly surprising, as higher reasoning effort naturally incurs additional token costs. While the extensive experiments validate this phenomenon, I am not convinced that this observation itself provides substantial new insights into LLM reasoning efficiency.

  2. Limited applicability when workload annotations are unavailable. The proposed workload-based evaluation relies on estimates of the reasoning workload required by each question. However, such annotations are unavailable for many real-world tasks and existing benchmarks. Although the authors introduce a solver-derived workload as an alternative, this approach depends on successful reference-model outputs and may not reliably reflect the intrinsic difficulty of each question. Consequently, the applicability and reliability of the proposed evaluation framework in more general settings remain unclear. This limitation makes it difficult to regard the framework as a broadly applicable new evaluation metric.

Questions for authors

I don't have more questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.