- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EvoCost, a framework for reducing the inference cost of LLM agents by evolving the agent harness instead of changing the model parameters. The method analyzes execution traces, constructs an LLM-annotated support DAG, and uses it to identify interactions that may be skipped, merged, collapsed, or redirected after repeated failures. LLMs then generate patches to the instructions, tool interfaces, and executor code. Candidate patches are merged and retained only after rollout-based cost and performance checks. On four Harbor benchmarks and three model backbones, the authors report lower rollout cost in all 12 settings, with an average reduction of 22.69%, while task performance is preserved or improved in 10 settings.
Critical strengths
The paper targets a practical problem that is not fully addressed by trajectory compression or early stopping. Its main idea—changing the mechanism that generates future interactions rather than only shortening an already generated trajectory—is reasonable and useful. The implementation also goes beyond prompt editing. Some of the proposed changes coordinate instructions, tool schemas, and executor code, which is necessary if the agent is expected to batch operations or replace several model-tool rounds with a composite operation. The paper makes a serious effort to evaluate the method across different task pools and models, and the ablations on parallel patch generation, hybrid gating, and dependency-aware evidence are helpful. The cost decomposition and break-even analysis are also useful additions.
Critical weaknesses
My main concern is that the support DAG is treated as evidence for safe intervention, although it is only an LLM-generated interpretation of a single observed trajectory. An edge indicates that the annotator believes one interaction was directly used by another; the absence of an edge does not establish that the earlier interaction was unnecessary under counterfactual execution. This is especially important for merging and collapsing operations, where execution order, hidden state, tool side effects, and failure behavior may affect the result. The paper does not report dependency-labeling accuracy, human agreement, robustness to noisy edges, or systematic counterfactual tests. The validation protocol also seems vulnerable to overfitting. Each benchmark has only 16 reserved tasks, and the same tasks are repeatedly used for trajectory discovery, patch generation, and iterative acceptance. The final held-out evaluation is useful, but it does not show whether the intermediate gating decisions generalize. Most settings are also reported from a single evolution run, without task-level confidence intervals or significance testing. In addition, the acceptance rule is relative to the current harness, so small performance losses can accumulate over multiple accepted updates.
The cost claim should be interpreted more narrowly than the paper sometimes suggests. The 22.69% figure is an unweighted reduction in held-out rollout cost; it excludes the substantial cost of annotation, patch generation, judging, merging, and validation. The appendix shows that hundreds of future tasks may be needed to amortize this search cost. Also, the number of model calls decreases much less than the token cost, suggesting that a substantial fraction of the gain comes from shorter inputs and different token/cache usage rather than from avoiding interactions themselves. The identities and configurations of the auxiliary LLMs are also not specified in enough detail to assess reproducibility and budget fairness.
Questions for authors
How accurate are the dependency annotations? Have the authors conducted human audits, multi-annotator agreement studies, or controlled experiments with corrupted DAG edges?
What models and prompts are used for the dependency annotator, patch generator, merger, and semantic judge? Are all of their calls included in the reported search cost, and can the authors provide results under an equal total API-token or dollar budget?
Can the authors separate the contribution of skip, collapse, merge, and failure-to-pivot interventions? In particular, how many accepted patches actually reduce interaction count, and how much of the reported saving comes from reduced context length or cache-related token charges?
Can the authors report total cost as a function of the number of future deployment tasks, including the one-time evolution cost, and evaluate whether the learned harness transfers across models or benchmark domains?
9 comments
Part2: Address Q2 of Reviewer 1KCk
Q2: What models and prompts are used for the dependency annotator, patch generator, merger, and semantic judge? Are all of their calls included in the reported search cost, and can the authors provide results under an equal total API-token or dollar budget?
Related weakness (W3, excerpts):
The 22.69% figure is an unweighted reduction in held-out rollout cost. It excludes the substantial cost of annotation, patch generation, judging, merging, and validation.
The identities and configurations of the auxiliary LLMs are also not specified in enough detail to assess reproducibility and budget fairness.
A2: Thank you for highlighting reproducibility and budget fairness. We use the same LLM for execution and all auxiliary roles within each main experimental setting, and include all evolution-stage API calls in the search cost. Our comparisons use a common evolution-cycle limit, and we report each method's actual API expenditure.
Models and prompts. The three model identifiers are gpt-5.6-terra, DeepSeek-V4-Flash through its official API, and gpt-5.5-2026-04-24. Full prompts are available in the following supplementary source files, with an overview in the Prompt Structures appendix.
| Role | Prompt implementation |
|---|---|
| Dependency annotation | SYSTEM_PROMPT in src/evolve/_coat_annotator.py |
| Patch generation | src/evolve/_coat_prompt.py |
| Patch merging | src/evolve/_coat_patch_merger.py |
| Semantic judging | src/evolve/_coat_runtime.py |
Cost accounting. We count initial development rollouts, annotation, generation, judging, merging, installation checks, and validation, including rejected proposals and unsuccessful attempts. We report search costs in the Evolution Cost and Break-even Analysis appendix, and final rollout charges in the Inference Cost Decomposition appendix.
Search budget. All methods share the initial harness, 16 development tasks, editable artifacts, and maximum evolution cycles. Token and dollar costs vary with each method's proposals, retries, and validation. We include these measured search costs in the total-cost analysis in A4.
We hope this clarifies the comparison and cost accounting. Thank you for helping us make these details explicit. We will add them to the experimental setup.
Part3: Address Q3 of Reviewer 1KCk
Q3: Can the authors separate the contribution of skip, collapse, merge, and failure-to-pivot interventions? In particular, how many accepted patches actually reduce interaction count, and how much of the reported saving comes from reduced context length or cache-related token charges?
Related weakness (W3, excerpt):
Also, the number of model calls decreases much less than the token cost, suggesting that a substantial fraction of the gain comes from shorter inputs and different token/cache usage rather than from avoiding interactions themselves.
A3: Thank you for suggesting this breakdown. We find two sources of savings: fewer model calls and lower token charges per call. The largest absolute saving comes from uncached-input charges on all four benchmarks.
Intervention ablations (in progress). We are separately disabling skip, collapse, merge, and failure-to-pivot with the initial harness, development tasks, and search budgets fixed. We will report scores, model-call counts, and API costs when these experiments finish.
Audit of candidate and accepted edits. In a GPT-6-Astra review of 262 changed candidates, 237 were judged likely to save cost, 106 were flagged as risky, and 8 had uncertain correctness. Cost-saving and risk labels can overlap. All 12 accepted merged updates received positive cost-saving judgments, and one was flagged as risky. We plan paired executions before and after each accepted update to measure how many actually reduce calls.
Measured call and cost reductions. The following results use GPT-5.6-Terra. All reductions are relative to Original.
Each cell reports reduction in model calls / reduction in API cost per task.
| Method | SWE-bench | DeepSWE | DAB | Terminal-Bench 2.1 |
|---|---|---|---|---|
| EvoCost | 14.8% / 43.9% | 11.1% / 40.8% | 6.3% / 24.4% | 0.9% / 20.2% |
On SWE-bench, we reduce model calls by 14.8% and average cost per call by 34.2%, yielding a 43.9% total cost reduction. Shorter interaction histories also reduce the input carried into subsequent calls, explaining why cost savings exceed the reduction in call counts.
A concrete example: sphinx-doc__sphinx-7757. Following your question, we examined a Sphinx signature-rendering bug using the same GPT-5.6-Terra snapshot and API prices before and after evolution. EvoCost uses bounded repository queries, batches code and test edits, and switches to direct function checks after encountering missing test dependencies. These changes reduce long search outputs and repeated installation logs. Both runs resolve the issue.
| Measurement | Original | EvoCost |
|---|---|---|
| Model calls, including submission | 13 | 11 |
| Tool calls | 17 | 19 |
| Total input tokens | 240,053 | 144,364 |
| Uncached input tokens | 122,272 | 51,180 |
| Cached input tokens | 117,781 | 93,184 |
| Output tokens | 4,480 | 4,961 |
| Rollout API cost (USD) | 0.3218602 | 0.1805288 |
| Resolved by the benchmark evaluator | Yes | Yes |
We reduce API cost by 43.91% on this task, mainly through lower uncached-input charges (0.2445 dollars → 0.1024 dollars). The 19 tool calls fit into 11 model calls because a response can invoke multiple tools. We provide the original trajectory and evolved trajectory for inspection.
Thanks again for drawing attention to this distinction. We hope the breakdown makes the source of the savings clearer. We will include it in the revision and update the intervention results when available.
Part1: Address Q1 of Reviewer 1KCk
Thank you for your careful review and valuable suggestions. Following your feedback, we added experiments on dependency quality, evolution stability, and harness transfer. We address your questions below and will incorporate these results and clarifications into the revised paper.
Q1: How accurate are the dependency annotations? Have the authors conducted human audits, multi-annotator agreement studies, or controlled experiments with corrupted DAG edges?
Related weakness (W1):
A1: Thank you for this important question. Following your suggestion, we added a human audit and a controlled edge-perturbation experiment. We use the DAG to identify candidate edits, then screen the edits and execute the candidate harness on validation tasks. We retain an update only when it meets our performance and cost criteria.
Human audit. Two reviewers independently audited eight trajectories from four benchmarks, covering 148 action steps. Their initial disagreement rate was 8.41%, with approximately 2.03% of judgments unresolved after joint review.
Type and State are operation-type and execution-status accuracies over 148 steps. Recall is the fraction of 194 reference dependencies connected by a predicted path. Complete is the fraction of 136 dependent steps retaining all required evidence. Edges is the predicted edge count.
We find that all three annotators retain most required dependencies, with recall of 84.5%–99.0%. GPT-5.5 performs best, achieving 99.0% recall and 98.5% completeness.
Controlled edge perturbations. We randomly added 10% more edges to the DAG and evaluated the evolved harnesses with GPT-5.6-Terra. Each cell reports score (%) / cost (USD per task).
With 10% added edges, we still reduce cost by 17.85%–42.09% across all four benchmarks. Scores remain within 1.37 percentage points of the unperturbed results. These results show that EvoCost preserves its cost savings under the tested edge noise.
Thanks again for suggesting these checks. We hope they clarify the quality of the dependency evidence and how we validate the resulting edits. We will include both experiments in the revision.