- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies how the reasoning-effort setting exposed by commercial and open-weight LLMs affects accuracy, cost, and agent behavior. It evaluates eight models on AIME 2026, GPQA Diamond, Terminal-Bench 2.1, and a 44-instance SWE-Bench Pro pilot, and reports four findings. First, solved sets at different effort levels are not nested, so higher effort “regresses” on some tasks. Second, on tasks solved at every level, higher effort costs more (up to 42x), and none can cost more than low because trajectories get longer. Third, higher effort shifts agents toward more self-verification and toward retrieving reference solutions, including leaked gold commits on SWE-Bench Pro. Fourth, a “naive laddering” policy (run at low effort, retry at the highest effort after an externally verified failure) matches or beats fixed top effort, often at lower cost, especially once a wage-based waiting cost is added. The authors frame reasoning effort as a credence good and argue that providers should make reasoning charges verifiable.
Critical strengths
- Breadth of experiments with 8 closed source and open-weight providers and single and agentic benchmarks
- SWE-Bench Pro gold-commit analysis and DeepSWE ablation are interesting observations
- The observation that no reasoning can be more expensive than low in agentic settings is a useful practical guideline
Critical weaknesses
- Extremely limited novelty with the main contributions already established in literature. Overthinking on easy problems and in agentic SWE tasks is documented (Chen et al., 2025; Cuadron et al., 2025). The Holistic Agent Leaderboard already found that higher reasoning effort reduced accuracy in most of its runs (Kapoor et al., 2026). Input tokens dominating agentic cost is shown by Bai et al. (2026). Listed prices misranking realized cost is the subject of Chen et al. (2026). The unauditability of hidden reasoning tokens is covered by Sun et al. (2025) and Velasco et al. (2025). Moon (2026) studies effort as an API contract directly. That higher effort costs more on easy tasks, and that a stochastic sampler’s solved sets are not nested across settings, are close to expected by construction. The paper repackages these known observations under a new framing rather than offering a new method, theory, or a surprising result. The gold-patch leakage finding is the clearest exception, but it builds directly on Jain (2026) and is more a benchmark-validity result than a finding about reasoning effort.
- The waiting-cost model is unreasonable as it assumes the user is fully blocked during decoding, prices time at the BLS all-occupation mean wage, and ignores prefill, queueing, and tool latency
- The credence-good framing is rhetorical. No economic model is developed, no information asymmetry is measured, and the policy recommendation does not follow from the experiments.
Questions for authors
N/A
2 comments
The gold-patch leakage finding is the clearest exception ... is more a benchmark-validity result than a finding about reasoning effort.
Jain identified the SWE-Bench Pro evaluation risk. Our finding is that raising reasoning effort made the same agent more likely to seek that fix under identical access rules. These blocked attempts are especially informative because a pass/fail score alone would miss the change in behavior.
This is fundamentally different from an evaluation validity question - it is rather a safety implication for reward hacking. A patch can pass the tests whether the agent derived it or copied the answer, so the success criterion leaves the route to that outcome underspecified. In the real world, this kind of thing due to underspecification happens all the time (e.g., during OpenAI’s ExploitGym evaluations, agents break into Hugging Face systems and access private datasets). Our result gives a specific reason and suggestion to test safety constraints at the effort level intended for deployment.
The waiting-cost model is unreasonable as it assumes the user is fully blocked during decoding, prices time at the BLS all-occupation mean wage, and ignores prefill, queueing, and tool latency
We agree that a user can start other work during a run, but if the agent needs input to proceed, the user must interrupt that work and then return to it. Several works we cited found that interruptions increased stress and perceived time pressure, and that attention lingering on an unfinished task impaired performance on the next one. We do not count the monetary cost of this value.
At a given hourly rate, the decoding-only calculation gives a lower bound on the cost of waiting through completion because it omits prefill, queueing, and tool execution. Even at the BLS all-occupation mean, estimated decoding time alone adds two or even much more times the mean API bill for many setups. The wage is an explicit reference rate. Using the software-developer mean average, the cost increases even more. The goal of this estimation is to provide a lower bound of the cost, and even the lower bound is expensive enough to raise the attention of the community.
The credence-good framing is rhetorical. No economic model is developed, no information asymmetry is measured, and the policy recommendation does not follow from the experiments.
We believe the credence-good framing accurately captures the difficulty users face: they must purchase reasoning effort before knowing how much their task needs, and a successful outcome does not establish that the additional effort was worth its cost. This difficulty in assessing whether a purchased service was necessary is central to the credence-good literature (Dulleck and Kerschbamer, 2006), which is well-established but often neglected or unknown in the field of AI.
Our practical recommendations are that users should evaluate effort against their task requirements, expected performance, and total cost, and that model developers should be more transparent about what each effort level delivers, including its measured benefits, costs, and behavioral effects across task types. Our experiments motivate these recommendations by showing that higher effort can change both spending and solution strategies, and may not bring sufficient marginal gain. We see these recommendations as consistent with the credence-good literature's concern with helping buyers assess whether the service supplied matches their needs.
We thank the reviewer for their helpful review. Below is our response:
Chen et al. (https://proceedings.mlr.press/v267/chen25bx.html) measure how often a model repeats a fution within a single LLM call. This is fundamentally different from what we are studying. We study how the level of reasoning effort affects the model's behavior in long-horizon agentic executions. Our experiments, in fact, add up to the study of single language model calls and find that both the cost and the behavior can compound, leading to different results.
Kapoor et al. (https://openreview.net/pdf?id=vUaY1t64ZZ). has an experiment that compares aggregate accuracy across effort settings. Those analyses leave open the behavioral differences between different reasoning efforts and for example, do not answer whether raising effort makes it more frequent for the models to pursue the shortcut solution.
Cuadron et al. compare low and high effort for o1 on SWE-Bench Verified and find that low effort scores higher on their measure of overthinking. That score cannot tell us whether raising effort makes an agent more likely to seek an existing solution. Our trajectories show that models tend to read the reference fix from online or git history more at higher effort. We also measure the cost on tasks solved at every setting and break them down for further analysis, which their aggregate cost comparison leaves open.
Bai et al. identify which tokens dominate coding-agent runs. Chen et al. hold each model at one reasoning configuration to study price reversals across models. We hold the model and its token prices fixed and change only the requested effort. This measures how much extra context processing the user's choice of effort buys, and are not covered by the previous two works.
Sun et al. audit hidden tokens and Artola Velasco, et al. study incentives to misreport them. Even with perfectly accurate billing, the buyer still cannot see what a cheaper setting would have achieved on the same task. Moon directly tests explicit high versus omitted effort for one model on AIME. Our multi-turn experiments capture a consequence that single-call design cannot measure: disabling reasoning can lengthen the trajectory enough to raise the total bill.
We disagree that these findings follow by construction. Sampling can explain why two runs on one task disagree, but it does not predict how changing the inference setting will affect the actions an agent takes or the cost of reaching the same outcome. When everything else is fixed, the cost can still vary from two times to multiple times the lower effort.
There is a concrete reason users might turn effort up by default. In OpenAI’s o1 test-time scaling results, additional test-time compute improved aggregate benchmark performance. Our experiments test how well that expectation carries over to individual agent tasks. A single run cannot determine a task's underlying success probability. For a user choosing a setting before execution, the practical implication is that an aggregate scaling gain is insufficient grounds for blindly raising effort on a particular task.