- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper measures how many tokens frontier reasoning models spend per correct solution and how that has changed across releases. It proposes the minimal human derivation (MHD), the shortest human solution posted for each problem, as a per-problem floor, and calls tokens above it excess reasoning. On 40 problems from the 2026 AIME and HMMT and 15 OpenAI and Anthropic models, tokens per correct solution fall 8.5x (o1 to GPT-6-Astra) and 4.8x (Opus 4.5 to Fable 5.1) while accuracy rises; a log-linear fit puts the quarterly decay of excess at 22-45% and extrapolates arrival near the floor to 2027-28. Two open-weight pairs separate scale from post-training, with LLM judges counting verifications and backtracks.
Critical strengths
The design is careful: a fixed problem set postdating most training cutoffs, repeated sampling, per-model tokenization of the floor, clustered bootstrap intervals. The release-by-release series is the part I found valuable. The open-weight decomposition in Section 6, where scale removes detours and post-training mostly removes the prose around them, is the most interesting result.
Critical weaknesses
-
The contribution is a measurement: tokens per correct solution fell, established cleanly. The conceptual claim rests on the MHD, and I did not come away with a new understanding of why models got shorter.
-
"Excess reasoning" names a conclusion the paper does not reach. The MHD is a post-hoc write-up that keeps the search inside a derivation and drops the search for one; the human solutions for one problem already differ by up to 4.9x in length, so the floor is set by whoever happened to post the shortest; Table A2 shows the rate is not sensitive to that choice, but the meaning of E is. Nothing shows how much of the gap above the floor is removable. The judge counts in Section 6 are the right instrument for that, but E is never decomposed with them; and while the scale pair behaves as the framework predicts, the authors' own reading of the GLM pair is that 5.3 searches as much as 5.2 and writes less around it, with reasoning tokens down 46% while verifications per trace fall only 24% and backtracks rise. Section 7 concedes that token counts cannot separate fewer detours from terser prose, which is the distinction the title rests on.
-
The per-problem-optimum idea has prior work to compare against: THOUGHTTERMINATOR's problem-level optimal token spend (arxiv.org/abs/2504.13367) and the reasoning-efficiency frontier of arxiv.org/abs/2506.07104. Both are model-side floors; the paper's case for a human one is legibility and a fixed target, not that it is a better floor. Estimating only on correct attempts also sits on a known length-correctness dependence (arxiv.org/abs/2505.00127) that the paper does not relate to its conditioning.
-
The 40 problems were picked so that o1 can solve most of them, and the paper shows that newer models spend fewer tokens on problems older models could already solve. Whether they also spend fewer tokens on problems near the limit of what they can do now is not measured. Though the authors admit it may not be falling, that is the case the title is about, and it is also the case that determines what users pay, since people use a model on the hardest things it can handle. I would also want to see a second domain before accepting a title this broad, for example, competitive programming.
Questions for authors
-
Open-weight families (DeepSeek, Qwen, GLM) have release series over a comparable period with visible traces. Does the same decay appear there, and when the judges are run across releases, does excess fall through fewer detours or shorter prose?
-
The backtracking judge already returns line numbers (Figure A10). For the open-weight traces, add the same for verification, remove both kinds of span, and check that the solution still holds. How much of E survives, and how far is the remainder from the MHD?
-
On competitive programming problems with published editorials, what plays the role of the MHD, and does the same decay appear?
-
On a moving set of problems at each model's own ~50% solve rate, does tokens per correct solution fall at all?