- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a broad empirical study of token consumption in agentic workflows, covering twelve models, two coding-agent harnesses, and additional environments. It examines consumption variability, model efficiency, and the feasibility of predicting token usage before task execution.
Critical strengths
1.The study addresses an important practical problem with substantial experimental coverage. Repeated runs help distinguish differences across tasks from variability within the same task. 2.The cross-harness comparison is particularly informative, showing that token consumption and model rankings can change considerably with the execution framework. 3.The self-prediction results offer an interesting direction for anticipating resource requirements. The paper is clearly organized and acknowledges important limitations.
Critical weaknesses
1.The self-prediction study would benefit from simple prediction baselines and quantitative error or calibration metrics beyond correlation.
2.The interpretation of cross-harness differences should remain cautious because prompts and model configurations also change. Clearer reporting of these settings would strengthen reproducibility.
Questions for authors
1.How does self-prediction compare with an inexpensive predictor using task features or historical consumption?
2.Which model configurations differ between the two harnesses, and how might these differences affect the efficiency comparison?
4 comments
Response to Reviewer zEi3 (Part 2): Results on Baselines and Cross Harness Configurations
Thank you for your reply. Our server access has been restored, and all pending results are now available: below, we compare self-prediction against inexpensive baselines on additional error and calibration metrics (W1&Q1), and report the full configurations for both harnesses (W2&Q2).
(W1&Q1) Baselines and error metrics
Setup. We evaluate the four frontier models (GPT-6 Astra, Kimi K3, Opus 5, Sonnet 5) on every SWE-bench Verified task, predicting actual input and output tokens with the same task issue as input. Self-prediction performs best compared with other baselines on all metrics, and the bias can be removed with a two parameter calibration.
- Additional Baselines: (1) historical average: the mean cost of past runs, a constant prediction; (2) lightweight task-feature predictors: ridge and MLP regressors (5-fold cross-fitted) on an embedding of the issue text; (3) single-turn LLM: the same model predicts its cost in one call from the issue, without exploring the repository.
- Metrics: nMAE (MAE divided by mean actual tokens, comparable across models); MdAPE (median relative error); bias (median predicted/actual, 1 = calibrated); and Pearson r.
- Calibration: LLM- and agent self-predictions are biased low, so we recalibrate them with (actual)=a+b(pred).
Results (mean over four models, input / output tokens; nMAE are MdAPE are after calibration):
| Method | nMAE ↓ | MdAPE ↓ | Pearson r ↑ | Bias, raw → calibrated |
|---|---|---|---|---|
| Historical average | 0.51 / 0.45 | 0.47 / 0.42 | – | 1.30 / 1.22 |
| MLP (issue embedding) | 0.46 / 0.41 | 0.39 / 0.37 | 0.32 / 0.34 | 1.02 / 1.00 |
| Ridge (issue embedding) | 0.43 / 0.39 | 0.38 / 0.35 | 0.41 / 0.39 | 1.01 / 1.00 |
| Single-turn LLM | 0.45 / 0.37 | 0.38 / 0.32 | 0.43 / 0.52 | 0.11 → 0.97 / 0.44 → 0.98 |
| Self-prediction | 0.38 / 0.34 | 0.34 / 0.31 | 0.61 / 0.59 | 0.22 → 0.98 / 0.71 → 0.98 |
Findings
Self-prediction is the most accurate predictor on all metrics. Raw self-predictions and LLM predictions can underestimate cost, especially input tokens, but a two-parameter recalibration on past runs removes this bias, so we recommend calibrated predictions for absolute cost estimates.
In the revised version, we will add this table to Section 7, the per-model results and the information-level ablation to the appendix, and separate ranking ability (correlation) from absolute accuracy (error after calibration).
(W2&Q2) Cross-harness configurations
The model configurations are aligned across both harnesses, except for the per-call output cap, which the OpenHands SDK requires; the remaining differences are part of each harness's design:
| Setting | OpenHands | mini-SWE-agent |
|---|---|---|
| Model configuration | ||
| Models | GPT-6 Astra, Sonnet 5, Opus 5, Kimi K3 | Same |
| Reasoning effort | High | High |
| Temperature / top-p | Provider default | Provider default |
| Max output tokens per call | 64K (required by the OpenHands SDK) | Provider default |
| Max context | 1.0M (Claude), 1.05M (Astra), 936K (Kimi) | Same |
| Prompt caching | On | On |
| Endpoint | OpenAI API (Astra); OpenRouter and Anthropic API (Claude); OpenAI-compatible endpoint (Kimi) | Same |
| Harness design | ||
| Tools | Specialized file-editing and shell tools | Single bash tool |
| Prompt | OpenHands SWE-bench prompt (3,058-token system prompt) | Minimal system prompt; instructions in the first user message |
| Step limit | 100 iterations | 250 steps |
The output cap was never reached, and mini-SWE-agent’s higher step limit reflects its smaller, single-command steps, so neither should affect the results comparison. The remaining differences (tools and prompt) are part of each harness’s design; since they are bundled, the comparison characterizes the harnesses as deployed rather than isolating any single choice. In the revised version, we will add this table to Appendix A.1 and revise Section 5.1 to state precisely how the two setups differ.
We hope these results address the reviewer's remaining concerns, and we would be happy to clarify anything further.
Response to Reviewer zEi3 (Part 3): Summary of Paper Revisions
We apologize that the server maintenance delayed our results and revision, leaving less time for discussion than we had hoped, and we sincerely appreciate the reviewer's patience.
We have uploaded the revised paper, with all new or rewritten text marked in blue, and a summary of changes in the Supplementary Material. Following the reviewer's suggestions, the main changes are:
- Baselines and metrics (W1&Q1): Section 7.1 (Table 2) compares self-prediction with a historical average, ridge regression, and a single-turn LLM on error, calibration, and overhead, with predictor details, per-model results, and a run-to-run noise limit in Appendix H.1–H.3.
- Cross-harness configurations (W2&Q2): Appendix A.1 (Table 5) lists the model configuration and harness design for both harnesses, and Section 5.1 now states that each harness bundles several design choices, so the comparison characterizes the harnesses as deployed.
We thank the reviewer again for the time and constructive feedback. We hope the revision addresses the reviewer's concerns and would be happy to clarify anything further.
Thank you for your response. I decide to maintain my positive score
Thank you for your response. I decide to maintain my scores because of the pending results.