acceptodds
Under review as a conference paper at ICLR 2027

AI is Becoming an Expert Reasoner

Abstract

Across physics and chess experiments, the expertise literature shows that what differentiates experts from novices is not more search but less: experts take the necessary steps while novices employ a more circuitous trial-and-error approach. We apply this lens to large reasoning models and ask how their token efficiency has changed over time. We introduce the minimal human derivation, the shortest published human solution to a problem, as a problem-specific benchmark against which excess reasoning can be measured across problems and time. On 40 problems from the 2026 AIME and HMMT competitions, evaluated across 15 frontier models with repeated sampling, mean tokens per correct solution fell 8.5x across OpenAI models (o1 to GPT-6-Astra) and 4.8x across Anthropic releases (Opus 4.5 to Fable 5.1), while accuracy rose in both families. Excess reasoning above the human benchmark falls 22.3-32.6% for OpenAI and 31.7-45.4% for Anthropic per quarter. Extrapolating within family, average frontier traces come within 10% of the human benchmark by late 2027 to late 2028. Matched pairs of open-weight models suggest efficiency gains come from both pre-training scale and post-training. We discuss implications for forecasting, the token economics of AI, and chain-of-thought monitorability as reasoning approaches the length of the shortest human derivation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reasoning models, chain-of-thought, test-time compute, reasoning efficiency, LLM evaluation, benchmarks, chain-of-thought monitorability, economics of AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jYsuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper measures how many tokens frontier reasoning models spend per correct solution and how that has changed across releases. It proposes the minimal human derivation (MHD), the shortest human solution posted for each problem, as a per-problem floor, and calls tokens above it excess reasoning. On 40 problems from the 2026 AIME and HMMT and 15 OpenAI and Anthropic models, tokens per correct solution fall 8.5x (o1 to GPT-6-Astra) and 4.8x (Opus 4.5 to Fable 5.1) while accuracy rises; a log-linear fit puts the quarterly decay of excess at 22-45% and extrapolates arrival near the floor to 2027-28. Two open-weight pairs separate scale from post-training, with LLM judges counting verifications and backtracks.

Critical strengths

The design is careful: a fixed problem set postdating most training cutoffs, repeated sampling, per-model tokenization of the floor, clustered bootstrap intervals. The release-by-release series is the part I found valuable. The open-weight decomposition in Section 6, where scale removes detours and post-training mostly removes the prose around them, is the most interesting result.

Critical weaknesses
  1. The contribution is a measurement: tokens per correct solution fell, established cleanly. The conceptual claim rests on the MHD, and I did not come away with a new understanding of why models got shorter.

  2. "Excess reasoning" names a conclusion the paper does not reach. The MHD is a post-hoc write-up that keeps the search inside a derivation and drops the search for one; the human solutions for one problem already differ by up to 4.9x in length, so the floor is set by whoever happened to post the shortest; Table A2 shows the rate is not sensitive to that choice, but the meaning of E is. Nothing shows how much of the gap above the floor is removable. The judge counts in Section 6 are the right instrument for that, but E is never decomposed with them; and while the scale pair behaves as the framework predicts, the authors' own reading of the GLM pair is that 5.3 searches as much as 5.2 and writes less around it, with reasoning tokens down 46% while verifications per trace fall only 24% and backtracks rise. Section 7 concedes that token counts cannot separate fewer detours from terser prose, which is the distinction the title rests on.

  3. The per-problem-optimum idea has prior work to compare against: THOUGHTTERMINATOR's problem-level optimal token spend (arxiv.org/abs/2504.13367) and the reasoning-efficiency frontier of arxiv.org/abs/2506.07104. Both are model-side floors; the paper's case for a human one is legibility and a fixed target, not that it is a better floor. Estimating only on correct attempts also sits on a known length-correctness dependence (arxiv.org/abs/2505.00127) that the paper does not relate to its conditioning.

  4. The 40 problems were picked so that o1 can solve most of them, and the paper shows that newer models spend fewer tokens on problems older models could already solve. Whether they also spend fewer tokens on problems near the limit of what they can do now is not measured. Though the authors admit it may not be falling, that is the case the title is about, and it is also the case that determines what users pay, since people use a model on the hardest things it can handle. I would also want to see a second domain before accepting a title this broad, for example, competitive programming.

Questions for authors
  1. Open-weight families (DeepSeek, Qwen, GLM) have release series over a comparable period with visible traces. Does the same decay appear there, and when the judges are run across releases, does excess fall through fewer detours or shorter prose?

  2. The backtracking judge already returns line numbers (Figure A10). For the open-weight traces, add the same for verification, remove both kinds of span, and check that the solution still holds. How much of E survives, and how far is the remainder from the MHD?

  3. On competitive programming problems with published editorials, what plays the role of the MHD, and does the same decay appear?

  4. On a moving set of problems at each model's own ~50% solve rate, does tokens per correct solution fall at all?

Reviewer_2SasOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work analyzes the change in token usage of frontier reasoning models across releases. Multiple reasoning models (mainly from OpenAI, from o1 to GPT-6-Astra and Anthropic, from Opus 4.5 to Fable 5.1) are tasked to solve a set of 40 mathematical problems (from AIME and HMMT 2026). Total completion tokens are compared, including reasoning and the final answer, w.r.t. the "minimal human derivation" (MHD), i.e. the corresponding shortest human solution, collected from a set of published solutions from human experts. Tokens used per correct solution fell by 8.5x (for OpenAI) and 4.8x (for Anthropic) across releases, while accuracy improved. Differences between tokens used and MHD are fit by an exponential decline to predict when models will come within 10% of the human reference. Experiments with open weight models are reported to analyze the roles of model scale and post-training.

Critical strengths
  • The reported drops of token usage per correct solution while accuracy increased is an interesting analysis, which includes different frontier models, with repeated evaluation, and reports accuracy as well. This study matters for both inference cost and monitoring of reasoning traces.

  • The "minimal human derivation" is intuitive and acts as a more interpretable reference than just token counts.

Critical weaknesses
  • Shorter solutions do not directly mean expert reasoning. For example, as stated in the text GLM 5.3 uses fewer reasoning tokens but backtracks 45% more often than GLM 5.2 (Table 2). Moreover, token usage depends also on the "effort", whose meaning may change across releases. Figure A5 reports falling token usage at the three effort levels, but does not report accuracy. It could be useful to compare token usage at the same level of accuracy, to help separate model improvements from changes in what the effort levels mean (which could affect the claim and framing of the paper around expert reasoning).

  • Minimal human derivation is a useful metric, but the model could develop a solution different from the shortest published human solution. Subtracting the length of the human solution is not guaranteed to tell which steps were unnecessary (as Section 3 would hint at). Figure 2 shows variation across attempts falling alongside average length. However, this could also be due simply to the attempts being shorter; does this behavior remain after normalizing by average length (as done in the GLM comparison in Section 6, where run-to-run variability becomes proportionally larger even as the mean shortens)?

  • The forecasts for reaching within 10% of MHD (also reported in the abstract) are given without uncertainty intervals, even if Figure 3 reports confidence bands. When the newest model are removed, the forecast dates are delayed by 8 to 11 months. These models may have been exposed to published solutions; removing them changes the forecast, but this would not show whether the shortening is due to model improvements, exposure to solution, or both.

Overall, the work reports a useful trend, but I find the contribution beyond the observed reduction in completion token usage somewhat limited. The interpretation in terms of expertise is not sufficiently supported, and the comparison with MHD and date forecasts require stronger validation to justify the stated conclusions.

Questions for authors
  • It would be interesting to add accuracy to the effort comparison. Do models use fewer tokens to reach the same accuracy? Or what is the accuracy reached with a fixed token budget? This would help distinguish model ability from internal provider settings.
  • What does the "medium" setting mean for each model? Was adaptive thinking on?
  • Were solutions below MHD checked for correctness of the reasoning (other than of the final answer)?
  • Does the variation across attempts in Figure 2 still fall when normalized by shorter average length?
  • How many samples were used to validate the judges for the reasoning traces? How much were they agreeing with human counts?
  • Please report uncertainty intervals for the forecast dates.
Reviewer_MEaYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper mainly examines how reasoning length on the 2026 AIME and HMMT changes with model release date across OpenAI and Anthropic models, finding a rapid decline. It measures the number of tokens exceeding the shortest human-written solution and fits an exponential trend to estimate the rate of decline. The authors interpret this reduction as evidence that models are becoming more expert-like reasoners, solving the same problems with less trial and error and more concise language. The paper also analyzes the reasoning traces of open-source model pairs and finds that, under the settings considered, both scaling in pretraining and post-training contribute to improved reasoning efficiency.

Critical strengths

The paper is clearly written and addresses a meaningful topic. The experiments examining how pretraining scale and post-training affect model reasoning traces are interesting, particularly the measurements of verification and backtracking.

Critical weaknesses

The evaluation focuses mainly on models from OpenAI and Anthropic and uses only problems from the 2026 AIME and HMMT, resulting in limited diversity and scale. The analysis of what happens inside the traces is also relatively limited. Moreover, the rationale for using an exponential form for the main extrapolation is not sufficiently justified, which lowers my confidence in the paper’s main results.

Questions for authors

What evidence would provide stronger support for “expert-like reasoning” while ruling out changes in expression strategy? The decline in the mean and variance of reasoning length could also result from training for more concise expression or adjustments to the reasoning budget. Could the authors directly compare erroneous branches, productive steps, and the number of backtracking events?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.