acceptodds
Under review as a conference paper at ICLR 2027

What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents

Abstract

Generative models increasingly expose a reasoning effort setting that lets users scale inference-time compute, trading higher cost for potentially improved performance. Users must choose an effort level before knowing whether extra reasoning will help, and even a successful run does not reveal whether the effort purchased was necessary. Reasoning effort is therefore a credence good: buyers observe the outcome but cannot assess the marginal value of the computation they paid for. In this work, we measure the counterfactual effect of effort levels on both proprietary and open-weight models on four benchmarks, spanning single-call reasoning and long-horizon agentic tasks. Effort, it turns out, does not consistently preserve tasks that lower effort solves, can overspend, and can change trajectory behavior. First, we find that although higher effort can raise accuracy in aggregate, it frequently fails on tasks that lower effort solves. Second, on the subset of easy tasks that every effort level solves, higher effort is generally more costly, with the highest effort level costing up to 42xas much as the lowest. Lastly, effort changes not only how many tokens are used, but also how agents solve tasks: at higher effort, agents verify their own work more heavily and more often search for solutions than work through problems directly, both of which lengthen trajectories without necessarily altering the outcome. These findings show that naively choosing the highest level is not only cost-inefficient but also offers no guarantee of the best instance-level performance. A simple alternative, starting at the lowest effort and escalating only after a verified failure, matches or improves on fixed top effort across all models and benchmarks at lower cost for most proprietary models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Reasoning, Agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_etDFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how the reasoning-effort setting exposed by commercial and open-weight LLMs affects accuracy, cost, and agent behavior. It evaluates eight models on AIME 2026, GPQA Diamond, Terminal-Bench 2.1, and a 44-instance SWE-Bench Pro pilot, and reports four findings. First, solved sets at different effort levels are not nested, so higher effort “regresses” on some tasks. Second, on tasks solved at every level, higher effort costs more (up to 42x), and none can cost more than low because trajectories get longer. Third, higher effort shifts agents toward more self-verification and toward retrieving reference solutions, including leaked gold commits on SWE-Bench Pro. Fourth, a “naive laddering” policy (run at low effort, retry at the highest effort after an externally verified failure) matches or beats fixed top effort, often at lower cost, especially once a wage-based waiting cost is added. The authors frame reasoning effort as a credence good and argue that providers should make reasoning charges verifiable.

Critical strengths
  • Breadth of experiments with 8 closed source and open-weight providers and single and agentic benchmarks
  • SWE-Bench Pro gold-commit analysis and DeepSWE ablation are interesting observations
  • The observation that no reasoning can be more expensive than low in agentic settings is a useful practical guideline
Critical weaknesses
  • Extremely limited novelty with the main contributions already established in literature. Overthinking on easy problems and in agentic SWE tasks is documented (Chen et al., 2025; Cuadron et al., 2025). The Holistic Agent Leaderboard already found that higher reasoning effort reduced accuracy in most of its runs (Kapoor et al., 2026). Input tokens dominating agentic cost is shown by Bai et al. (2026). Listed prices misranking realized cost is the subject of Chen et al. (2026). The unauditability of hidden reasoning tokens is covered by Sun et al. (2025) and Velasco et al. (2025). Moon (2026) studies effort as an API contract directly. That higher effort costs more on easy tasks, and that a stochastic sampler’s solved sets are not nested across settings, are close to expected by construction. The paper repackages these known observations under a new framing rather than offering a new method, theory, or a surprising result. The gold-patch leakage finding is the clearest exception, but it builds directly on Jain (2026) and is more a benchmark-validity result than a finding about reasoning effort.
  • The waiting-cost model is unreasonable as it assumes the user is fully blocked during decoding, prices time at the BLS all-occupation mean wage, and ignores prefill, queueing, and tool latency
  • The credence-good framing is rhetorical. No economic model is developed, no information asymmetry is measured, and the policy recommendation does not follow from the experiments.
Questions for authors

N/A

2 comments

We thank the reviewer for their helpful review. Below is our response:

Extremely limited novelty with the main contributions already established in literature. Overthinking on easy problems and in agentic SWE tasks is documented (Chen et al., 2025; Cuadron et al., 2025). The Holistic Agent Leaderboard already found that higher reasoning effort reduced accuracy in most of its runs (Kapoor et al., 2026).

  • Chen et al. (https://proceedings.mlr.press/v267/chen25bx.html) measure how often a model repeats a fution within a single LLM call. This is fundamentally different from what we are studying. We study how the level of reasoning effort affects the model's behavior in long-horizon agentic executions. Our experiments, in fact, add up to the study of single language model calls and find that both the cost and the behavior can compound, leading to different results.

  • Kapoor et al. (https://openreview.net/pdf?id=vUaY1t64ZZ). has an experiment that compares aggregate accuracy across effort settings. Those analyses leave open the behavioral differences between different reasoning efforts and for example, do not answer whether raising effort makes it more frequent for the models to pursue the shortcut solution.

  • Cuadron et al. compare low and high effort for o1 on SWE-Bench Verified and find that low effort scores higher on their measure of overthinking. That score cannot tell us whether raising effort makes an agent more likely to seek an existing solution. Our trajectories show that models tend to read the reference fix from online or git history more at higher effort. We also measure the cost on tasks solved at every setting and break them down for further analysis, which their aggregate cost comparison leaves open.

Input tokens dominating agentic cost is shown by Bai et al. (2026). Listed prices misranking realized cost is the subject of Chen et al. (2026).

Bai et al. identify which tokens dominate coding-agent runs. Chen et al. hold each model at one reasoning configuration to study price reversals across models. We hold the model and its token prices fixed and change only the requested effort. This measures how much extra context processing the user's choice of effort buys, and are not covered by the previous two works.

The unauditability of hidden reasoning tokens is covered by Sun et al. (2025) and Velasco et al. (2025). Moon (2026) studies effort as an API contract directly.

Sun et al. audit hidden tokens and Artola Velasco, et al. study incentives to misreport them. Even with perfectly accurate billing, the buyer still cannot see what a cheaper setting would have achieved on the same task. Moon directly tests explicit high versus omitted effort for one model on AIME. Our multi-turn experiments capture a consequence that single-call design cannot measure: disabling reasoning can lengthen the trajectory enough to raise the total bill.

That higher effort costs more on easy tasks, and that a stochastic sampler’s solved sets are not nested across settings, are close to expected by construction. The paper repackages these known observations under a new framing rather than offering a new method, theory, or a surprising result.

We disagree that these findings follow by construction. Sampling can explain why two runs on one task disagree, but it does not predict how changing the inference setting will affect the actions an agent takes or the cost of reaching the same outcome. When everything else is fixed, the cost can still vary from two times to multiple times the lower effort.

There is a concrete reason users might turn effort up by default. In OpenAI’s o1 test-time scaling results, additional test-time compute improved aggregate benchmark performance. Our experiments test how well that expectation carries over to individual agent tasks. A single run cannot determine a task's underlying success probability. For a user choosing a setting before execution, the practical implication is that an aggregate scaling gain is insufficient grounds for blindly raising effort on a particular task.

The gold-patch leakage finding is the clearest exception ... is more a benchmark-validity result than a finding about reasoning effort.

Jain identified the SWE-Bench Pro evaluation risk. Our finding is that raising reasoning effort made the same agent more likely to seek that fix under identical access rules. These blocked attempts are especially informative because a pass/fail score alone would miss the change in behavior.

This is fundamentally different from an evaluation validity question - it is rather a safety implication for reward hacking. A patch can pass the tests whether the agent derived it or copied the answer, so the success criterion leaves the route to that outcome underspecified. In the real world, this kind of thing due to underspecification happens all the time (e.g., during OpenAI’s ExploitGym evaluations, agents break into Hugging Face systems and access private datasets). Our result gives a specific reason and suggestion to test safety constraints at the effort level intended for deployment.

The waiting-cost model is unreasonable as it assumes the user is fully blocked during decoding, prices time at the BLS all-occupation mean wage, and ignores prefill, queueing, and tool latency

We agree that a user can start other work during a run, but if the agent needs input to proceed, the user must interrupt that work and then return to it. Several works we cited found that interruptions increased stress and perceived time pressure, and that attention lingering on an unfinished task impaired performance on the next one. We do not count the monetary cost of this value.

At a given hourly rate, the decoding-only calculation gives a lower bound on the cost of waiting through completion because it omits prefill, queueing, and tool execution. Even at the BLS all-occupation mean, estimated decoding time alone adds two or even much more times the mean API bill for many setups. The wage is an explicit reference rate. Using the software-developer mean average, the cost increases even more. The goal of this estimation is to provide a lower bound of the cost, and even the lower bound is expensive enough to raise the attention of the community.

The credence-good framing is rhetorical. No economic model is developed, no information asymmetry is measured, and the policy recommendation does not follow from the experiments.

We believe the credence-good framing accurately captures the difficulty users face: they must purchase reasoning effort before knowing how much their task needs, and a successful outcome does not establish that the additional effort was worth its cost. This difficulty in assessing whether a purchased service was necessary is central to the credence-good literature (Dulleck and Kerschbamer, 2006), which is well-established but often neglected or unknown in the field of AI.

Our practical recommendations are that users should evaluate effort against their task requirements, expected performance, and total cost, and that model developers should be more transparent about what each effort level delivers, including its measured benefits, costs, and behavioral effects across task types. Our experiments motivate these recommendations by showing that higher effort can change both spending and solution strategies, and may not bring sufficient marginal gain. We see these recommendations as consistent with the credence-good literature's concern with helping buyers assess whether the service supplied matches their needs.

Reviewer_jSVkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper audits reasoning effort settings in LLMs by measuring task success, inference cost, and agent behavior across several models and benchmarks.

It also evaluates self routing and a simple laddering policy that retries failed low effort executions at the highest effort.

Critical strengths

The evaluation covers several proprietary and open models across both single call and long horizon agent tasks.

The paper provides useful cost and trajectory analyses and identifies substantial leakage in the original SWE Bench Pro environment, which it then evaluates in a corrected setting.

Critical weaknesses

The central qualitative findings have limited novelty because prior work already reports overthinking and nonmonotonic performance across reasoning effort settings [1].

The naive laddering evaluation confounds effort allocation with the benefit of a second independent attempt, since it lacks retry matched baselines.

[1] Do NOT Think That Much for 2+3=? On the Overthinking of Long Reasoning Models, ICML 2025.

Questions for authors

Can the authors compare low followed by high laddering with retry matched policies such as low followed by low and high followed by high under the same attempt budget?

2 comments

We thank the reviewer for the review and the helpful suggestion to compare retry-matched policies. Below are our response and additional results:

The central qualitative findings have limited novelty because prior work already reports overthinking and nonmonotonic performance across reasoning effort settings [1].

  • Chen et al. measure how often a model repeats a solution within a single LLM call. This is fundamentally different from what we are studying. We study how the level of reasoning effort affects the model's behavior in long-horizon agentic executions. Our experiments, in fact, add up to the study of single language model calls and find that both the cost and the behavior can compound, leading to different results.

The naive laddering evaluation confounds effort allocation with the benefit of a second independent attempt, since it lacks retry matched baselines.

Can the authors compare low followed by high laddering with retry matched policies such as low followed by low and high followed by high under the same attempt budget?

We agree that we should include low→low and high→high as baselines for naïve laddering. Below are the corresponding results on AIME, GPQA, and DeepSWE v1.1 (we show three models in the table for easier reading, and will include the full table in the updated manuscript).

Dataset / modelL→LL→HH→H
AIME 2026 / GPT-5.6-sol97.33% / $0.033911100.00% / $0.049930100.00% / $0.085918
AIME 2026 / Claude Sonnet 583.33% / $0.06200697.50% / $0.19056998.44% / $0.302174
AIME 2026 / GLM-5.291.26% / $0.05718892.50% / $0.06738393.41% / $0.107113
GPQA Diamond / GPT-5.6-sol93.52% / $0.01985194.70% / $0.03876995.03% / $0.081201
GPQA Diamond / Claude Sonnet 585.69% / $0.01446893.66% / $0.07875292.42% / $0.200940
GPQA Diamond / GLM-5.294.36% / $0.06166594.70% / $0.07276095.03% / $0.124630
DeepSWE v1.1 / GPT-5.6-sol59.59% / $1.68848076.83% / $3.79634779.94% / $6.044053
DeepSWE v1.1 / Claude Sonnet 543.81% / $3.80556859.02% / $10.99411164.16% / $17.607266
DeepSWE v1.1 / GLM-5.252.65% / $4.78054957.47% / $5.49677261.50% / $6.135514

Each cell reports pass rate / mean API cost per task. L and H denote the lowest and highest reasoning effort level available to that model. The results suggest that simply retrying cannot recover the performance benefit brought up by naive laddering. Additionally, running the highest effort level two times, although achieves performance as high as naive laddering, the cost increase drastically, with the maximum of 2 times the cost. We believe these results defend and strengthen our claim of the effectiveness of naive laddering.

Thank you for the rebuttal. While the additional comparisons address my second concern, my main concern regarding limited novelty remains unresolved, so I will maintain my original rating.

Reviewer_sKkKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents an empirical study of how reasoning-effort settings affect task performance, execution costs, and agent behavior. It evaluates eight models across four benchmarks, with seven models included in the agentic evaluations, while holding tasks, prompts, and harnesses fixed within each model–benchmark configuration. The analyses examine observed changes in task coverage, complete execution costs, and behavioral differences such as self-verification and reference-solution retrieval. The paper also evaluates self-routing and naïve laddering, which retries verified failures at higher effort. Validation includes benchmark evaluations, trajectory analysis, comparisons between original and corrected SWE-Bench Pro environments, and supplementary analysis of repeated DeepSWE runs. Its main contribution is an empirical characterization of effort-related tradeoffs and their implications for effort allocation and agent evaluation.

Critical strengths

The paper examines how reasoning effort affects task performance, complete execution costs, and agent behavior across multiple models and benchmarks, covering both single-call reasoning and multi-step agent interactions. Its execution-level accounting captures model-call frequency, trajectory length, and repeated context processing rather than relying solely on reasoning-token counts. The observation that disabling reasoning can increase total expenditure through longer trajectories is particularly informative. Together, these analyses provide useful evidence for understanding the practical consequences of different effort settings.

Critical weaknesses

1.Several central takeaways, such as additional reasoning being wasteful on easier tasks and more reasoning not consistently improving outcomes, substantially overlap with existing research on overthinking and inference efficiency. Although the broader evaluation and cost and behavioral analyses are useful, the paper does not sufficiently develop them into substantial new explanatory insights or a reliable effort-selection solution. Self-routing offers no consistent advantage, while naïve laddering depends on external correctness feedback and fresh executions after failure (§3.4). Consequently, the work remains primarily descriptive and provides limited guidance for settings without reliable verification. In my assessment, its practical contribution are insufficient for publication at ICLR.

2.The always-solved cost comparison in §3.2 does not establish equal success probabilities across effort levels. In particular, AIME and GPQA use pass@k to select this subset, so higher effort could improve single-execution reliability even when the observed task coverage remains unchanged. Higher expenditure therefore does not by itself establish “overspending.” Separately, naïve laddering introduces both verified retries and effort escalation, but the evaluation lacks same-effort retry baselines, such as low→low and high→high, under equivalent feedback and matched cost or accuracy targets. Although both execution costs are accounted for, the results do not isolate the benefit of changing effort from that of retrying. These gaps weaken the paper’s central economic conclusions and practical recommendation.

Questions for authors

See Weaknesses.

3 comments

We thank the reviewer for the careful review and suggestions. Below are our response and additional results:

1. Novelty and practical guidance

Several central takeaways, such as additional reasoning being wasteful on easier tasks and more reasoning not consistently improving outcomes, substantially overlap with existing research on overthinking and inference efficiency.

Our contribution is the scale of the within-task effort comparison and the findings of behavioral differences across effort levels. We vary effort across eight models and five benchmarks. This lets us price higher effort on tasks solved at every setting and trace how the same agent changes its strategy as effort rises.

We also note that we cannot find works in overthinking and inference efficiency that substantially overlap with our work. Below, we list some of the closest ones and clarify our differences:

  • Chen et al. (https://proceedings.mlr.press/v267/chen25bx.html) measure how often a model repeats a solution within a single LLM call. This is fundamentally different from what we are studying. We study how the level of reasoning effort affects the model's behavior in long-horizon agentic executions. Our experiments, in fact, add up to the study of single language model calls and find that both the cost and the behavior can compound, leading to different results.

  • Kapoor et al. (https://openreview.net/pdf?id=vUaY1t64ZZ). has an experiment that compares aggregate accuracy across effort settings. Those analyses leave open the behavior differences between different reasoning efforts and do not answer whether raising effort makes shortcut behavior more frequent.

the work remains primarily descriptive and provides limited guidance for settings without reliable verification.

Even when a user cannot verify a live result, they must still choose a default. Our results give a concrete reason to benchmark the effort level before choosing the cheaper setting (as both None reasoning effort and higher reasoning effort can cause significantly higher cost). This guidance can benefit the users and requires no verifier at runtime, so the contribution should not be neglected.

2. Reliability and retry controls

The always-solved cost comparison in §3.2 does not establish equal success probabilities across effort levels. In particular, AIME and GPQA use pass@k to select this subset, so higher effort could improve single-execution reliability even when the observed task coverage remains unchanged.

Below is the pass@1 at each effort setting. We sampled one complete run per benchmark and used that run for every model and effort. The table reports changes in accuracy (percentage points) and mean API cost (US cents per question) from each model’s lowest reasoning effort to its highest.

ModelEffortAIME Δ accuracy (pp)AIME Δ cost (¢)GPQA Δ accuracy (pp)GPQA Δ cost (¢)
GPT-5.6-sollow → xhigh+3.3 [0.0, 10.0]+4.55 [2.29, 7.46]+2.0 [-0.5, 5.1]+4.78 [3.13, 6.78]
GPT-5.6-terralow → xhigh+6.7 [-6.7, 20.0]+2.47 [0.73, 4.98]+3.5 [-1.0, 8.1]+1.66 [1.13, 2.26]
GPT-5.6-lunalow → xhigh+10.0 [-3.3, 23.3]+0.56 [0.25, 0.96]+7.6 [2.0, 13.1]+0.63 [0.46, 0.81]
Claude Sonnet 5low → xhigh+26.7 [13.3, 43.3]+20.01 [6.74, 36.62]+12.6 [7.1, 18.2]+13.61 [9.66, 18.09]
GLM-5.2high → max+10.0 [0.0, 23.3]+4.19 [2.64, 5.67]+2.0 [-2.0, 6.1]+5.59 [4.53, 6.71]
Inklinglow → max+13.3 [3.3, 26.7]+7.65 [5.26, 10.40]+19.7 [12.6, 26.8]+7.82 [6.72, 8.97]
Kimi K3low → max+33.3 [16.7, 50.0]+12.91 [3.60, 27.75]+11.6 [7.1, 16.7]+10.15 [7.79, 12.81]
DeepSeek V4 Flashlow → max-3.3 [-13.3, 6.7]+0.22 [0.12, 0.35]+1.5 [-2.0, 5.1]+0.08 [0.05, 0.12]

Brackets give 95% paired bootstrap intervals from 20,000 resamples of questions within the sampled run. We will revise the manuscript to report single-run accuracy, so the effort comparisons use the same metrics.

Separately, naïve laddering introduces both verified retries and effort escalation, but the evaluation lacks same-effort retry baselines, such as low→low and high→high, under equivalent feedback and matched cost or accuracy targets.

We agree that we should include low→low and high→high as baselines for naïve laddering. Below are the corresponding results on AIME, GPQA, and DeepSWE v1.1 (we show three models in the table for easier reading, and will include the full table in the updated manuscript).

Dataset / modelL→LL→HH→H
AIME 2026 / GPT-5.6-sol97.33% / $0.033911100.00% / $0.049930100.00% / $0.085918
AIME 2026 / Claude Sonnet 583.33% / $0.06200697.50% / $0.19056998.44% / $0.302174
AIME 2026 / GLM-5.291.26% / $0.05718892.50% / $0.06738393.41% / $0.107113
GPQA Diamond / GPT-5.6-sol93.52% / $0.01985194.70% / $0.03876995.03% / $0.081201
GPQA Diamond / Claude Sonnet 585.69% / $0.01446893.66% / $0.07875292.42% / $0.200940
GPQA Diamond / GLM-5.294.36% / $0.06166594.70% / $0.07276095.03% / $0.124630
DeepSWE v1.1 / GPT-5.6-sol59.59% / $1.68848076.83% / $3.79634779.94% / $6.044053
DeepSWE v1.1 / Claude Sonnet 543.81% / $3.80556859.02% / $10.99411164.16% / $17.607266
DeepSWE v1.1 / GLM-5.252.65% / $4.78054957.47% / $5.49677261.50% / $6.135514

Each cell reports pass rate / mean API cost per task. L and H denote the lowest and highest reasoning effort level available to that model. The results suggest that simply retrying cannot recover the performance benefit brought up by naive laddering. Additionally, running the highest effort level two times, although achieves performance as high as naive laddering, the cost increase drastically, with the maximum of 2 times the cost. We believe these results defend and strengthen our claim of the effectiveness of naive laddering.

Thank you for the additional experiments. Although the supplementary experiments have addressed some of my concerns, I still have concerns about the novelty and practical application value of this work to the community. Therefore, I will keep my rating unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.