acceptodds
Under review as a conference paper at ICLR 2027

How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks

10 views

Abstract

Wide adoption of AI agents in complex human workflows drives rapid growth of LLM token consumption. When agents are deployed on tasks that can require a large amount of tokens, three questions naturally arise: (1) How do AI agents spend the tokens? (2) What models are more token efficient? and (3) Can LLMs anticipate the token usage before task execution? In this paper, we present a systematic study of token consumption in AI agents, analyzing up to twelve frontier LLMs across two coding-agent harnesses and four agentic environments. We find that: (1) agentic tasks are uniquely expensive, with input tokens rather than output tokens accounting for most of the consumption. (2) Token usage is highly variable: on the same task, the most expensive run typically consumes twice as many tokens as the cheapest, and higher usage does not guarantee higher accuracy. (3) Human-labeled task difficulty only weakly aligns with actual resource expenditure. (4) Models differ systematically in token efficiency, with similarly accurate models consuming up to 3× different amounts, but the harness can change consumption several-fold and even reorder models. (5) Finally, agents show promise in anticipating their own cost: the strongest models' pre-execution predictions correlate with actual usage at up to 0.77, a potential signal for budget alerts, although they tend to underestimate input tokens. Our study reveals important insights regarding the economics of AI agents and can inspire new studies in this direction.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI agents, LLM token consumption, agentic coding, token efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zEi3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a broad empirical study of token consumption in agentic workflows, covering twelve models, two coding-agent harnesses, and additional environments. It examines consumption variability, model efficiency, and the feasibility of predicting token usage before task execution.

Critical strengths

1.The study addresses an important practical problem with substantial experimental coverage. Repeated runs help distinguish differences across tasks from variability within the same task. 2.The cross-harness comparison is particularly informative, showing that token consumption and model rankings can change considerably with the execution framework. 3.The self-prediction results offer an interesting direction for anticipating resource requirements. The paper is clearly organized and acknowledges important limitations.

Critical weaknesses

1.The self-prediction study would benefit from simple prediction baselines and quantitative error or calibration metrics beyond correlation.

2.The interpretation of cross-harness differences should remain cautious because prompts and model configurations also change. Clearer reporting of these settings would strengthen reproducibility.

Questions for authors

1.How does self-prediction compare with an inexpensive predictor using task features or historical consumption?

2.Which model configurations differ between the two harnesses, and how might these differences affect the efficiency comparison?

4 comments

Thank you for your response. I decide to maintain my scores because of the pending results.

AuthorsOpenReview ↗

Response to Reviewer zEi3 (Part 2): Results on Baselines and Cross Harness Configurations

Thank you for your reply. Our server access has been restored, and all pending results are now available: below, we compare self-prediction against inexpensive baselines on additional error and calibration metrics (W1&Q1), and report the full configurations for both harnesses (W2&Q2).

(W1&Q1) Baselines and error metrics

Setup. We evaluate the four frontier models (GPT-6 Astra, Kimi K3, Opus 5, Sonnet 5) on every SWE-bench Verified task, predicting actual input and output tokens with the same task issue as input. Self-prediction performs best compared with other baselines on all metrics, and the bias can be removed with a two parameter calibration.

  • Additional Baselines: (1) historical average: the mean cost of past runs, a constant prediction; (2) lightweight task-feature predictors: ridge and MLP regressors (5-fold cross-fitted) on an embedding of the issue text; (3) single-turn LLM: the same model predicts its cost in one call from the issue, without exploring the repository.
  • Metrics: nMAE (MAE divided by mean actual tokens, comparable across models); MdAPE (median relative error); bias (median predicted/actual, 1 = calibrated); and Pearson r.
  • Calibration: LLM- and agent self-predictions are biased low, so we recalibrate them with (actual)=a+b(pred).

Results (mean over four models, input / output tokens; nMAE are MdAPE are after calibration):

MethodnMAE ↓MdAPE ↓Pearson r ↑Bias, raw → calibrated
Historical average0.51 / 0.450.47 / 0.42–1.30 / 1.22
MLP (issue embedding)0.46 / 0.410.39 / 0.370.32 / 0.341.02 / 1.00
Ridge (issue embedding)0.43 / 0.390.38 / 0.350.41 / 0.391.01 / 1.00
Single-turn LLM0.45 / 0.370.38 / 0.320.43 / 0.520.11 → 0.97 / 0.44 → 0.98
Self-prediction0.38 / 0.340.34 / 0.310.61 / 0.590.22 → 0.98 / 0.71 → 0.98

Findings
Self-prediction is the most accurate predictor on all metrics. Raw self-predictions and LLM predictions can underestimate cost, especially input tokens, but a two-parameter recalibration on past runs removes this bias, so we recommend calibrated predictions for absolute cost estimates.

In the revised version, we will add this table to Section 7, the per-model results and the information-level ablation to the appendix, and separate ranking ability (correlation) from absolute accuracy (error after calibration).

(W2&Q2) Cross-harness configurations

The model configurations are aligned across both harnesses, except for the per-call output cap, which the OpenHands SDK requires; the remaining differences are part of each harness's design:

SettingOpenHandsmini-SWE-agent
Model configuration
ModelsGPT-6 Astra, Sonnet 5, Opus 5, Kimi K3Same
Reasoning effortHighHigh
Temperature / top-pProvider defaultProvider default
Max output tokens per call64K (required by the OpenHands SDK)Provider default
Max context1.0M (Claude), 1.05M (Astra), 936K (Kimi)Same
Prompt cachingOnOn
EndpointOpenAI API (Astra); OpenRouter and Anthropic API (Claude); OpenAI-compatible endpoint (Kimi)Same
Harness design
ToolsSpecialized file-editing and shell toolsSingle bash tool
PromptOpenHands SWE-bench prompt (3,058-token system prompt)Minimal system prompt; instructions in the first user message
Step limit100 iterations250 steps

The output cap was never reached, and mini-SWE-agent’s higher step limit reflects its smaller, single-command steps, so neither should affect the results comparison. The remaining differences (tools and prompt) are part of each harness’s design; since they are bundled, the comparison characterizes the harnesses as deployed rather than isolating any single choice. In the revised version, we will add this table to Appendix A.1 and revise Section 5.1 to state precisely how the two setups differ.

We hope these results address the reviewer's remaining concerns, and we would be happy to clarify anything further.

AuthorsOpenReview ↗

Response to Reviewer zEi3 (Part 3): Summary of Paper Revisions

We apologize that the server maintenance delayed our results and revision, leaving less time for discussion than we had hoped, and we sincerely appreciate the reviewer's patience.

We have uploaded the revised paper, with all new or rewritten text marked in blue, and a summary of changes in the Supplementary Material. Following the reviewer's suggestions, the main changes are:

  • Baselines and metrics (W1&Q1): Section 7.1 (Table 2) compares self-prediction with a historical average, ridge regression, and a single-turn LLM on error, calibration, and overhead, with predictor details, per-model results, and a run-to-run noise limit in Appendix H.1–H.3.
  • Cross-harness configurations (W2&Q2): Appendix A.1 (Table 5) lists the model configuration and harness design for both harnesses, and Section 5.1 now states that each harness bundles several design choices, so the comparison characterizes the harnesses as deployed.

We thank the reviewer again for the time and constructive feedback. We hope the revision addresses the reviewer's concerns and would be happy to clarify anything further.

Thank you for your response. I decide to maintain my positive score

Reviewer_rqAoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how LLM agents consume tokens. The main setting runs 12 models with OpenHands on all 500 SWE-bench Verified tasks, four runs per task. Part of the analysis is repeated with a second harness (mini-SWE-agent) and in τ2\tau^2-bench and ClawEval. The main findings are: (1) input tokens make up almost all consumption because context is resent every turn; (2) consumption varies widely across tasks and across runs of the same task; (3) within a task, costlier runs are not more accurate; (4) human-labeled difficulty only weakly predicts consumption; (5) token efficiency differs across models, but the harness can change consumption several-fold and reorder models. Finally, the paper asks agents to predict their own token usage before execution. The best model reaches a correlation of 0.77, but all models underestimate input tokens. The authors will release over 34,000 trajectories.

Critical strengths
  1. Large-scale data with repeated runs. The study covers 12 models, all 500 tasks, and four runs per task, plus a second harness and two other environments. Repeated runs let the paper measure how much token consumption varies across runs of the same task and separate this from task-level variation. Prior cost studies on SWE-bench mostly use single runs. The released trajectories, with per-round token logs by type, are a useful resource.

  2. Careful statistics. The paper uses task-level bootstrap intervals, multiple-comparison correction, and a between-within regression to separate task difficulty from extra spending on the same task. Limitations are stated honestly. This makes the reported patterns more trustworthy than single-run cost comparisons.

Critical weaknesses
  1. Limited novelty relative to prior work, and the cost framing is not fully supported. Several main findings are already reported or expected from prior work. SWE-Effi (Fan et al., 2025) already describes the "token snowball" effect and shows that efficiency depends on how well the scaffold fits the base model. HAL (Kapoor et al., 2025; ICLR 2026) shows that the scaffold strongly affects both accuracy and cost for the same model. Bjarnason et al. ("On Randomness in Agentic Evals", 2026) show that repeated runs on SWE-bench Verified diverge early and lead to different outcomes, which makes large run-to-run cost variation expected, although they do not measure token consumption directly. The paper does not discuss these overlapping findings, so its increment is mainly scale and the repeated-run cost measurements. In addition, the title and abstract are about money, but the analysis is mostly about token counts. The appendix shows that token share and dollar share differ a lot, and that model rankings by dollars differ from rankings by tokens. Dollar data is also missing for several models and settings.

  2. The self-prediction part does not show practical value. Run-to-run noise limits how well any pre-execution estimate can match a single run, but the paper does not report results relative to this limit. The method only predicts a model's own cost, so it cannot help choose between models, which is a main use case for cost estimates. Prediction is also not cheap (0.34× task cost for the best predictor, over 2× for some models). Calibration and downstream use (budget alerts, routing) are proposed but never tested. So the claim that self-prediction is a "potential signal for budget alerts" is not supported by evidence.

Questions for authors
  1. Which main findings are new compared with SWE-Effi, HAL, and Bjarnason et al. (2026)? I'd appreciate it if the authors can summarize them explicitly.
  2. For the self-prediction part, how do the authors expect it to help in practice?
Reviewer_PTF7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an empirical study of token consumption in LLM-based agents, covering twelve frontier models, two coding-agent harnesses, and multiple agentic environments. Through analyzing over 34,000 agent trajectories, the authors investigate how token consumption varies across tasks, models, harnesses, and repeated runs. Furthermore, the authors investigate whether LLM agents can predict their own token consumption before execution.

Critical strengths

Large-scale empirical evaluation across diverse models and agentic settings. The paper conducts an extensive empirical study involving twelve frontier LLMs and over 34,000 agent trajectories.

Critical weaknesses
  • Limited novelty and insufficient depth of empirical insights. My main concern is that several key observations in this paper have already been discussed in prior studies. For example, SWE-Effi, Efficient Agents, More with Less, and Tokenomics have investigated agent resource consumption, cost–performance trade-offs, and inefficiencies associated with agent execution. Although this paper provides broader model coverage and repeated evaluations, it remains unclear what fundamentally new insights are established beyond these existing findings. Moreover, the paper primarily reports descriptive statistical observations without providing sufficiently deep explanations and insights of the underlying mechanisms. As a result, the overall narrative feels fragmented, and the work reads more like a collection of empirical findings than a cohesive research study with clearly differentiated contributions.
  • Insufficiently motivated and evaluated token self-prediction. The introduction identifies pre-execution token prediction as a central research question, and Sections 4–6 provide extensive empirical analyses of how token consumption varies across models, tasks, and agent harnesses. However, these findings are not leveraged to inform or improve the design of the self-prediction approach in Section 7, leaving the empirical analysis and prediction study largely disconnected. Moreover, the evaluation relies primarily on Pearson correlation, without comparisons against simple yet meaningful baselines, such as historical token consumption statistics or lightweight predictors based on task characteristics.
Questions for authors
  • Novelty compared with existing studies. Could the authors clarify the key new insights of this work compared with closely related studies such as SWE-Effi, Efficient Agents, More with Less, and Tokenomics, beyond the increased scale and diversity of the empirical evaluation?

  • Baselines for token self-prediction. Could the authors compare the proposed self-prediction approach against simpler baselines, such as historical average token consumption and lightweight predictors based on task characteristics? Additionally, could the authors report absolute or relative prediction errors and calibration metrics rather than relying primarily on Pearson correlation?

  • Practical effectiveness of self-prediction. Could the authors provide an end-to-end evaluation demonstrating whether self-prediction can effectively support budget alerts or cost-aware model/harness selection?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Reviews, Additional Experiments, and Revisions

We sincerely thank all reviewers for their thoughtful and constructive feedback, which has substantially improved our paper.

We apologize that an emergency server maintenance delayed our additional results and revision, leaving less time for discussion than we had hoped, and we appreciate everyone's time and patience. For the convenience of the reviewers and the AC, we summarize the reviews, additional experiments, and revisions below. The revised paper is uploaded, with all new or rewritten text marked in blue and a summary of changes in the Supplementary Material.

Strengths noted by the reviewers

StrengthReviewers
Scale and repeated runs: twelve frontier models, 34,000+ trajectories, and four runs per task, separating variation across tasks from variation within the same taskPTF7, rqAo, zEi3
Statistics Rigor: task-level bootstrap intervals, multiple-comparison correction, a between–within regression, and honestly stated limitationsrqAo, zEi3
Informative cross-harness comparison: consumption and model rankings change with the harnesszEi3
Practical relevance and resources: an important practical problem, a promising self-prediction direction, clear presentation, and released trajectories with per-round token logsrqAo, zEi3

Main concerns, additional experiments, and revisions

ConcernRaised byResponsePaper locationDiscussion
Novelty relative to prior workPTF7, rqAoPaper revision: state the new insights and deeper analyses enabled by repeated runs, and clarify the gap they fill relative to closely related work.Abstract; Sections 1–2PTF7 Part 1; rqAo Part 1
Prediction baselines and error metricsPTF7, zEi3New experiment: compare self-prediction with three baselines on error, correlation, calibration, and overhead. Self-prediction is the most accurate on every metric.Section 7.1 (Table 2); Appendix H.1–H.2PTF7 Part 3; zEi3 Part 2
Run-to-run noise limitrqAoNew analysis: compare predictors with a run-to-run noise limit. Self-prediction closes the largest share of the achievable gap.Section 7.1; Appendix H.3rqAo Part 4
Practical value of self-predictionPTF7, rqAoNew analysis: end-to-end budget alerts, where pre-execution prediction flags over-budget runs before execution and saves about 31% of execution cost; self-prediction detects them best, and a cascade makes it affordable.Section 7.2 (Table 3); Appendix H.4PTF7 Part 4; rqAo Parts 2–3
Monetary framingrqAoNew analysis: price every run at published rates; within-model findings carry over to dollars. Paper revision: explain why tokens are the primary unit of the main analysis.Abstract; Section 1; Section 6.3; Appendix GrqAo Parts 1 and 3
Cross-harness configurationszEi3Paper revision: report both harnesses' configurations, and clarify that the comparison characterizes the harnesses as deployed.Section 5.1; Appendix A.1 (Table 5)zEi3 Part 2

We thank the reviewers again for their valuable feedback and would be happy to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.