acceptodds
Under review as a conference paper at ICLR 2027

EvoCost: Cost-Effective Agents via Harness Self-Evolution

Abstract

LLM agents can incur substantial inference cost on complex tasks that require long interaction trajectories. Existing cost-saving methods mainly compress accumulated history or control execution stopping, but do not change the behavior that generates unnecessary interactions. We introduce \sys, a framework for cost-effective harness self-evolution that learns from execution experience to reduce unnecessary interactions in future executions while preserving task performance. \sys first performs cost-saving opportunity discovery by reconstructing execution dependencies and combining them with execution outcomes to identify interactions that can be safely skipped, collapsed, batched, or redirected. It then performs validated harness evolution, translating these opportunities into reusable harness edits and retaining updates that reduce inference cost while meeting stage-dependent task-performance requirements. Across four benchmarks and three model backbones, \sys reduces inference cost relative to the original harness in all 12 settings by 22.7% on average, while improving or preserving task performance in 10 of them. These results demonstrate that harness self-evolution can provide a practical path toward more cost-effective agent execution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent harness, cost efficiency, harness self-evolution, tool use

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1KCkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvoCost, a framework for reducing the inference cost of LLM agents by evolving the agent harness instead of changing the model parameters. The method analyzes execution traces, constructs an LLM-annotated support DAG, and uses it to identify interactions that may be skipped, merged, collapsed, or redirected after repeated failures. LLMs then generate patches to the instructions, tool interfaces, and executor code. Candidate patches are merged and retained only after rollout-based cost and performance checks. On four Harbor benchmarks and three model backbones, the authors report lower rollout cost in all 12 settings, with an average reduction of 22.69%, while task performance is preserved or improved in 10 settings.

Critical strengths

The paper targets a practical problem that is not fully addressed by trajectory compression or early stopping. Its main idea—changing the mechanism that generates future interactions rather than only shortening an already generated trajectory—is reasonable and useful. The implementation also goes beyond prompt editing. Some of the proposed changes coordinate instructions, tool schemas, and executor code, which is necessary if the agent is expected to batch operations or replace several model-tool rounds with a composite operation. The paper makes a serious effort to evaluate the method across different task pools and models, and the ablations on parallel patch generation, hybrid gating, and dependency-aware evidence are helpful. The cost decomposition and break-even analysis are also useful additions.

Critical weaknesses

My main concern is that the support DAG is treated as evidence for safe intervention, although it is only an LLM-generated interpretation of a single observed trajectory. An edge indicates that the annotator believes one interaction was directly used by another; the absence of an edge does not establish that the earlier interaction was unnecessary under counterfactual execution. This is especially important for merging and collapsing operations, where execution order, hidden state, tool side effects, and failure behavior may affect the result. The paper does not report dependency-labeling accuracy, human agreement, robustness to noisy edges, or systematic counterfactual tests. The validation protocol also seems vulnerable to overfitting. Each benchmark has only 16 reserved tasks, and the same tasks are repeatedly used for trajectory discovery, patch generation, and iterative acceptance. The final held-out evaluation is useful, but it does not show whether the intermediate gating decisions generalize. Most settings are also reported from a single evolution run, without task-level confidence intervals or significance testing. In addition, the acceptance rule is relative to the current harness, so small performance losses can accumulate over multiple accepted updates.

The cost claim should be interpreted more narrowly than the paper sometimes suggests. The 22.69% figure is an unweighted reduction in held-out rollout cost; it excludes the substantial cost of annotation, patch generation, judging, merging, and validation. The appendix shows that hundreds of future tasks may be needed to amortize this search cost. Also, the number of model calls decreases much less than the token cost, suggesting that a substantial fraction of the gain comes from shorter inputs and different token/cache usage rather than from avoiding interactions themselves. The identities and configurations of the auxiliary LLMs are also not specified in enough detail to assess reproducibility and budget fairness.

Questions for authors

How accurate are the dependency annotations? Have the authors conducted human audits, multi-annotator agreement studies, or controlled experiments with corrupted DAG edges?

What models and prompts are used for the dependency annotator, patch generator, merger, and semantic judge? Are all of their calls included in the reported search cost, and can the authors provide results under an equal total API-token or dollar budget?

Can the authors separate the contribution of skip, collapse, merge, and failure-to-pivot interventions? In particular, how many accepted patches actually reduce interaction count, and how much of the reported saving comes from reduced context length or cache-related token charges?

Can the authors report total cost as a function of the number of future deployment tasks, including the one-time evolution cost, and evaluate whether the learned harness transfers across models or benchmark domains?

9 comments
AuthorsOpenReview ↗

Part1: Address Q1 of Reviewer 1KCk

Thank you for your careful review and valuable suggestions. Following your feedback, we added experiments on dependency quality, evolution stability, and harness transfer. We address your questions below and will incorporate these results and clarifications into the revised paper.

Q1: How accurate are the dependency annotations? Have the authors conducted human audits, multi-annotator agreement studies, or controlled experiments with corrupted DAG edges?

Related weakness (W1):

My main concern is that the support DAG is treated as evidence for safe intervention, although it is only an LLM-generated interpretation of a single observed trajectory. An edge indicates that the annotator believes one interaction was directly used by another. The absence of an edge does not establish that the earlier interaction was unnecessary under counterfactual execution. This is especially important for merging and collapsing operations, where execution order, hidden state, tool side effects, and failure behavior may affect the result. The paper does not report dependency-labeling accuracy, human agreement, robustness to noisy edges, or systematic counterfactual tests.

A1: Thank you for this important question. Following your suggestion, we added a human audit and a controlled edge-perturbation experiment. We use the DAG to identify candidate edits, then screen the edits and execute the candidate harness on validation tasks. We retain an update only when it meets our performance and cost criteria.

Human audit. Two reviewers independently audited eight trajectories from four benchmarks, covering 148 action steps. Their initial disagreement rate was 8.41%, with approximately 2.03% of judgments unresolved after joint review.

AnnotatorEdgesType (%)State (%)Recall (%)Complete (%)
GPT-5.541888.598.099.098.5
DeepSeek-V4-Flash42075.798.092.889.7
GPT-5.6-Terra23581.197.384.584.6

Type and State are operation-type and execution-status accuracies over 148 steps. Recall is the fraction of 194 reference dependencies connected by a predicted path. Complete is the fraction of 136 dependent steps retaining all required evidence. Edges is the predicted edge count.

We find that all three annotators retain most required dependencies, with recall of 84.5%–99.0%. GPT-5.5 performs best, achieving 99.0% recall and 98.5% completeness.

Controlled edge perturbations. We randomly added 10% more edges to the DAG and evaluated the evolved harnesses with GPT-5.6-Terra. Each cell reports score (%) / cost (USD per task).

MethodSWE-benchDeepSWE v1.0DABTerminal-Bench 2.1
Original76.65 / 0.347930.93 / 1.021045.45 / 0.224364.38 / 0.3272
EvoCost72.93 / 0.195031.96 / 0.604245.45 / 0.169567.12 / 0.2613
DAG with 10% added edges73.14 / 0.202730.93 / 0.591344.32 / 0.176765.75 / 0.2688

With 10% added edges, we still reduce cost by 17.85%–42.09% across all four benchmarks. Scores remain within 1.37 percentage points of the unperturbed results. These results show that EvoCost preserves its cost savings under the tested edge noise.

Thanks again for suggesting these checks. We hope they clarify the quality of the dependency evidence and how we validate the resulting edits. We will include both experiments in the revision.

AuthorsOpenReview ↗

Part2: Address Q2 of Reviewer 1KCk

Q2: What models and prompts are used for the dependency annotator, patch generator, merger, and semantic judge? Are all of their calls included in the reported search cost, and can the authors provide results under an equal total API-token or dollar budget?

Related weakness (W3, excerpts):

The 22.69% figure is an unweighted reduction in held-out rollout cost. It excludes the substantial cost of annotation, patch generation, judging, merging, and validation.

The identities and configurations of the auxiliary LLMs are also not specified in enough detail to assess reproducibility and budget fairness.

A2: Thank you for highlighting reproducibility and budget fairness. We use the same LLM for execution and all auxiliary roles within each main experimental setting, and include all evolution-stage API calls in the search cost. Our comparisons use a common evolution-cycle limit, and we report each method's actual API expenditure.

Models and prompts. The three model identifiers are gpt-5.6-terra, DeepSeek-V4-Flash through its official API, and gpt-5.5-2026-04-24. Full prompts are available in the following supplementary source files, with an overview in the Prompt Structures appendix.

RolePrompt implementation
Dependency annotationSYSTEM_PROMPT in src/evolve/_coat_annotator.py
Patch generationsrc/evolve/_coat_prompt.py
Patch mergingsrc/evolve/_coat_patch_merger.py
Semantic judgingsrc/evolve/_coat_runtime.py

Cost accounting. We count initial development rollouts, annotation, generation, judging, merging, installation checks, and validation, including rejected proposals and unsuccessful attempts. We report search costs in the Evolution Cost and Break-even Analysis appendix, and final rollout charges in the Inference Cost Decomposition appendix.

Search budget. All methods share the initial harness, 16 development tasks, editable artifacts, and maximum evolution cycles. Token and dollar costs vary with each method's proposals, retries, and validation. We include these measured search costs in the total-cost analysis in A4.

We hope this clarifies the comparison and cost accounting. Thank you for helping us make these details explicit. We will add them to the experimental setup.

AuthorsOpenReview ↗

Part3: Address Q3 of Reviewer 1KCk

Q3: Can the authors separate the contribution of skip, collapse, merge, and failure-to-pivot interventions? In particular, how many accepted patches actually reduce interaction count, and how much of the reported saving comes from reduced context length or cache-related token charges?

Related weakness (W3, excerpt):

Also, the number of model calls decreases much less than the token cost, suggesting that a substantial fraction of the gain comes from shorter inputs and different token/cache usage rather than from avoiding interactions themselves.

A3: Thank you for suggesting this breakdown. We find two sources of savings: fewer model calls and lower token charges per call. The largest absolute saving comes from uncached-input charges on all four benchmarks.

Intervention ablations (in progress). We are separately disabling skip, collapse, merge, and failure-to-pivot with the initial harness, development tasks, and search budgets fixed. We will report scores, model-call counts, and API costs when these experiments finish.

Audit of candidate and accepted edits. In a GPT-6-Astra review of 262 changed candidates, 237 were judged likely to save cost, 106 were flagged as risky, and 8 had uncertain correctness. Cost-saving and risk labels can overlap. All 12 accepted merged updates received positive cost-saving judgments, and one was flagged as risky. We plan paired executions before and after each accepted update to measure how many actually reduce calls.

Measured call and cost reductions. The following results use GPT-5.6-Terra. All reductions are relative to Original.

Each cell reports reduction in model calls / reduction in API cost per task.

MethodSWE-benchDeepSWEDABTerminal-Bench 2.1
EvoCost14.8% / 43.9%11.1% / 40.8%6.3% / 24.4%0.9% / 20.2%

On SWE-bench, we reduce model calls by 14.8% and average cost per call by 34.2%, yielding a 43.9% total cost reduction. Shorter interaction histories also reduce the input carried into subsequent calls, explaining why cost savings exceed the reduction in call counts.

A concrete example: sphinx-doc__sphinx-7757. Following your question, we examined a Sphinx signature-rendering bug using the same GPT-5.6-Terra snapshot and API prices before and after evolution. EvoCost uses bounded repository queries, batches code and test edits, and switches to direct function checks after encountering missing test dependencies. These changes reduce long search outputs and repeated installation logs. Both runs resolve the issue.

MeasurementOriginalEvoCost
Model calls, including submission1311
Tool calls1719
Total input tokens240,053144,364
Uncached input tokens122,27251,180
Cached input tokens117,78193,184
Output tokens4,4804,961
Rollout API cost (USD)0.32186020.1805288
Resolved by the benchmark evaluatorYesYes

We reduce API cost by 43.91% on this task, mainly through lower uncached-input charges (0.2445 dollars → 0.1024 dollars). The 19 tool calls fit into 11 model calls because a response can invoke multiple tools. We provide the original trajectory and evolved trajectory for inspection.

Thanks again for drawing attention to this distinction. We hope the breakdown makes the source of the savings clearer. We will include it in the revision and update the intervention results when available.

Reviewer_EyAQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EVOCOST, a cost-oriented harness self-evolution framework for LLM agents. Instead of reducing inference cost through post-hoc trajectory compression or early stopping, the paper focuses on proactively modifying the agent harness — including instructions, reusable skills, tool schemas, and executor code — so that unnecessary interactions are avoided during future executions. The framework analyzes execution trajectories using a support DAG that captures dependency relations between actions, identifies potential cost-saving opportunities such as skippable operations, collapsible action chains, mergeable calls, and failure-to-pivot patterns, and converts them into reusable harness edits. Candidate edits are generated in parallel, merged for compatibility, and validated through a hybrid gating mechanism balancing task performance and inference cost reduction. Experiments on different datasets across multiple model backbones show consistent reductions in inference cost while largely preserving downstream task performance.

Critical strengths
  1. The paper studies an important and highly practical problem. As agentic systems become increasingly expensive to deploy, optimizing the harness itself for lower inference cost is a meaningful direction beyond standard context compression or stopping heuristics.

  2. The proposed dependency-aware trajectory analysis is interesting. Using a support DAG to distinguish merely expensive operations from operations that are actually useful for downstream reasoning is a thoughtful design and helps motivate more targeted harness modifications.

  3. The writing is clear and easy to follow.

Critical weaknesses
  1. The framework is fairly complex and heavily engineered. EVOCOST involves dependency labeling, DAG reconstruction, multiple opportunity heuristics, patch generation, merging, semantic filtering, staged gating, and rollout validation. It is therefore difficult to isolate which components are truly responsible for the observed gains.

  2. The evaluation mainly focuses on SWE-style and tool-using agent benchmarks. It remains unclear whether the dependency-analysis framework would generalize to broader agent settings where execution dependencies are less structured or harder to recover reliably.

Questions for authors

Please refer to weaknesses.

4 comments
AuthorsOpenReview ↗

Part1: Address the First Question of Reviewer EyAQ

Thank you for your thoughtful review and valuable suggestions, and we will refine the paper following your suggestions. We added controlled comparisons to identify the contribution of each design and clarified the scope of our evaluation. We respond to your questions below.

Q1: Which components are responsible for the observed gains in this complex framework?

Related weakness (W1):

The framework is fairly complex and heavily engineered. EVOCOST involves dependency labeling, DAG reconstruction, multiple opportunity heuristics, patch generation, merging, semantic filtering, staged gating, and rollout validation. It is therefore difficult to isolate which components are truly responsible for the observed gains.

A1: Thank you for this valuable suggestion. We added ablations showing that each of our three designs improves both score and cost over its tested alternative. Our goal is cost-effective harness self-evolution: we improve reusable skills, instructions, tools, and executor code while keeping model parameters fixed. The three designs serve distinct roles:

  1. Dependency-aware opportunity discovery. We use dependencies and execution outcomes to identify avoidable work and preserve evidence needed downstream.
  2. Parallel Patch Generation. We generate independent proposals from a shared harness snapshot and merge compatible edits.
  3. Hybrid Gating. We screen individual patches and validate the merged harness against staged cost and performance criteria.

Following your suggestion, we compare these designs using GPT-5.5 on SWE-bench and DeepSWE, with a common initial harness, development tasks, and proposal budget. Each variant has one evolution run per benchmark, followed by held-out evaluation. Performance tables report score (%) / cost (USD per task).

1. Dependency evidence: Raw / StrucAno / DAG. Raw provides chronological actions and observations. StrucAno adds operation types, status, resources, outcomes, costs, and ordinary trajectory analysis. DAG further adds explicit dependencies and their derived optimization patterns. We hold all other components fixed and run each variant for ten cycles.

MethodSWE-benchDeepSWE
Raw72.93 / 1.183144.33 / 6.3897
StrucAno72.73 / 1.107845.36 / 5.8816
DAG76.03 / 1.066348.45 / 5.6965

We improve both score and cost by adding dependency evidence. Compared with StrucAno, DAG raises scores by 3.30 and 3.09 points and reduces cost by 3.74% and 3.15% on SWE-bench and DeepSWE, respectively. This shows the additional value of explicit dependencies and their derived patterns for harness optimization.

2. Proposal organization: Parallel / Sequential. We replace parallel generation and joint merging with sequential editing, where each retained edit becomes the starting point for the next proposal. Hybrid Gating stays fixed.

3. Modification validation: Hybrid / Hard. We replace Hybrid Gating with a hard rule requiring a non-decreasing score and strictly lower cost, removing semantic screening and staged acceptance margins. Parallel Patch Generation stays fixed.

MethodSWE-benchDeepSWE
Full method76.03 / 1.066348.45 / 5.6965
Sequential71.28 / 1.093546.39 / 5.8180
Hard74.79 / 1.290647.42 / 6.0101

We achieve higher scores and lower costs than both alternatives on both benchmarks. On SWE-bench, parallel generation with joint merging improves the score by 4.75 points over Sequential, and Hybrid Gating reduces cost by 17.38% relative to Hard with a higher score. Both designs contribute to the final gains.

Thank you also for encouraging us to examine how these designs affect update selection. The table pools 20 cycles across the two benchmarks. Gains are averages over accepted updates relative to the preceding harness. Parallel / Hybrid denotes the full method.

VariantAccepted updates / cyclesAcceptance rateMean score gain (points)Mean cost reduction
Parallel / Hybrid5 / 2025%5.0026.01%
Sequential3 / 2015%4.1715.53%
Hard7 / 2035%1.799.99%

Our full method selects updates with the largest average gains in both score and cost. Hard accepts the most updates, with smaller gains per update. This helps explain the full method's stronger final results.

We hope these ablations clarify the role of each design. Thanks again for this suggestion. We will organize the revised method and ablation discussion around opportunity discovery, patch generation, and validation, linking each to its measured contribution.

AuthorsOpenReview ↗

Part2: Address the Second Question of Reviewer EyAQ

Q2: Does the dependency-analysis framework generalize beyond SWE-style and tool-using benchmarks?

Related weakness (W2):

The evaluation mainly focuses on SWE-style and tool-using agent benchmarks. It remains unclear whether the dependency-analysis framework would generalize to broader agent settings where execution dependencies are less structured or harder to recover reliably.

A2: Thank you for asking us to clarify the range of agent settings. We evaluate tool-using agents across repository repair, long-horizon software engineering, data analysis, and terminal workflows. Our dependency analysis uses recorded actions, outputs, and outcomes, and we validate harness edits through task execution.

BenchmarkTask and operating environmentWhat this contributes to the evaluation
SWE-bench VerifiedResolve repository issues by inspecting and editing code, with instance-specific tests used for evaluation.Dependencies among code inspection, implementation changes, and regression testing in issue-resolution workflows.
DeepSWE v1Long-horizon software-engineering tasks drawn from 91 repositories and five programming languages.Variation in repository structures, languages, and longer execution sequences within software engineering.
DABAnswer questions over heterogeneous data systems, evaluated using hidden reference answers and validators.Dependencies among data access, intermediate results, and the final answer in data-oriented workflows.
Terminal-Bench 2.1Complete containerized terminal workflows spanning software engineering, system administration, and data science.Dependencies among commands and the environment state they read or modify across several task families.

We apply the same DAG construction and harness-evolution process across these task families, covering dependencies in code, data, and environment state. Our current evaluation therefore extends beyond repository repair to several tool-using workflows.

We appreciate your suggestion to study less explicit dependencies. We will prioritize such settings in future evaluations and measure both dependency recovery and the quality of the resulting edits. We hope this clarifies our current coverage and next steps. In the revision, we will make the requirements for observable traces and executable validation explicit.

Thank you again for your careful review. We hope these experiments and clarifications address your questions about the design choices and evaluation scope.

AuthorsOpenReview ↗

Summary of our modification, corresponding to questions of the reviewer EyAQ

Thank you again for asking us to explain the contribution of each design and the scope of our evaluation more clearly. We have revised the method overview and benchmark descriptions to address these points. Below, we point to the revised passages and the experiments that address each question.

For modification in A1

Thank you for helping us clarify the role of each design. We have added a short explanation of opportunity discovery, Parallel Patch Generation, and Hybrid Gating to the opening of EvoCost (Section 3).

Dependency-aware opportunity discovery. Please refer to Evidence Representation in Section 4.3.1 and Figure 5. The existing Raw, StrucAno, and DAG comparison isolates the additional contribution of explicit dependencies and their derived patterns.

Parallel Patch Generation. Please refer to Effect of Parallel Patch Generation in Section 4.3.1, Table 3, and Figure 4. These report final held-out performance and cost, together with update acceptance and gains during evolution.

Hybrid Gating. Please refer to Effect of Hybrid Gating in Section 4.3.1, Table 3, and Figure 4. The comparison with Hard reports both final outcomes and the gains among accepted updates.

For modification in A2

Thank you for asking us to make the evaluation scope explicit. We have expanded Benchmarks in Experimental Setup (Section 4.1) to describe dependencies in code, data, and environment state. The added text explains that the same analysis and evolution process is used across these workflows, with observable execution traces and executable task validation. The benchmark descriptions and task counts remain unchanged.

AuthorsOpenReview ↗

Follow-up on Our Responses

Thank you again for your constructive feedback, which has helped us improve the manuscript. As the discussion period is nearing its deadline, we would greatly appreciate hearing whether our responses and revisions have addressed your concerns. We would be happy to clarify anything that remains unclear.

Reviewer_MrmCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EvoCost, a framework for automatically improving the cost efficiency of LLM agents by evolving their harnesses. It first analyzes execution trajectories to identify redundant or inefficient operations, then generates and evaluates harness modifications using a combination of semantic screening and rollout-based validation. Experiments on four benchmarks and three backbone models report an average inference cost reduction of 22.69%, with task performance maintained or improved in 10 of the 12 evaluated settings.

Critical strengths

S1. A practical and relevant problem with a reasonable approach.

Reducing agent inference costs is important, particularly for multi-step tasks involving repeated tool calls. I like that the method considers dependencies between trajectory steps, since simply removing apparently redundant actions could break subsequent execution. The proposed analysis provides a useful way to identify potential optimization opportunities.

S2. Encouraging results across different benchmarks and models.

The experiments cover four benchmarks and three backbone models, with fairly consistent cost reductions. The comparison with existing harness optimization methods is useful, and I appreciate that the authors also report the cost of the evolution process and estimate when the optimization would pay for itself. This makes the practical implications easier to assess.

Critical weaknesses

W1. The robustness of the evolutionary optimization is not sufficiently established.

Each benchmark uses only 16 development tasks for both identifying optimization opportunities and validating candidate modifications. Repeatedly selecting changes on such a small set raises concerns about overfitting to those tasks. Although the final evaluation uses held-out tasks and the paper includes some sensitivity experiments, the main results do not consistently report variability across independent evolution runs. This matters because the method is based on iterative search, where the final harness may depend considerably on the sampled tasks and search trajectory.

W2. The contribution of dependency-aware analysis is not convincingly isolated.

A central idea of the paper is to construct a dependency graph from agent trajectories and use it to guide harness modifications. However, the paper provides limited evidence about the accuracy of these inferred dependencies or how reliably they identify steps that can be removed or modified. The relevant ablations are also limited in scope. Given that prior harness optimization approaches already use execution feedback to improve agent behavior, stronger evidence is needed to establish how much of the reported improvement comes specifically from the proposed dependency-aware analysis.

Questions for authors

Q1. How sensitive are the results to the 16 development tasks used during evolution? In particular, could the authors report results from multiple independent evolution runs and examine whether separating the tasks used for opportunity discovery from those used for candidate validation changes the conclusions?

Q2. Is there a way to directly evaluate the quality of the inferred dependency graphs, or to compare the proposed dependency-aware optimization with a simpler cost-aware optimizer using the same search budget? This would help clarify whether the dependency analysis is responsible for the observed improvements.

4 comments
AuthorsOpenReview ↗

Part1: Address the First Question of Reviewer MrmC

Thank you for your careful review and concrete experimental suggestions, and we will refine the paper following your suggestions. Following your feedback, we added repeated runs, disjoint discovery/validation splits, a human audit, and a controlled comparison with a simpler cost-aware optimizer. We respond to your questions below.

Q1: How sensitive are the results to the 16 development tasks? What happens across independent evolution runs and when opportunity discovery and candidate validation use separate tasks?

Related weakness (W1):

The robustness of the evolutionary optimization is not sufficiently established. Each benchmark uses only 16 development tasks for both identifying optimization opportunities and validating candidate modifications. Repeatedly selecting changes on such a small set raises concerns about overfitting to those tasks. Although the final evaluation uses held-out tasks and the paper includes some sensitivity experiments, the main results do not consistently report variability across independent evolution runs. This matters because the method is based on iterative search, where the final harness may depend considerably on the sampled tasks and search trajectory.

A1: Thank you for suggesting these robustness checks. We find cost savings in every tested run, and higher mean scores with a larger development split. Sixteen development tasks are a configurable default. We examine task selection, search randomness, and development-set size below.

Repeated evolution runs. With GPT-5.6-Terra, we vary the evolution seed with either fixed or resampled development tasks, using three runs per setting. Each cell reports held-out score (%) / cost reduction (%) relative to Original as mean ± sample standard deviation.

SettingSWE-benchDeepSWEDABTerminal-Bench 2.1
Fixed tasks72.72 ± 0.95 / (44.7 ± 3.4)%31.62 ± 0.59 / (40.9 ± 1.4)%46.59 ± 1.97 / (20.3 ± 5.0)%66.66 ± 0.79 / (18.6 ± 6.0)%
Resampled tasks75.00 ± 2.44 / (41.4 ± 4.5)%29.90 ± 2.73 / (36.5 ± 5.5)%44.70 ± 2.36 / (22.9 ± 3.6)%66.66 ± 0.79 / (18.0 ± 8.8)%

We obtain mean cost reductions of 18.0%–44.7% across these settings. Resampling increases score variability on SWE-bench, DeepSWE, and DAB, showing that task selection affects final performance. On SWE-bench with fixed tasks, the 44.7% cost reduction accompanies a mean score of 72.72%, compared with Original's 76.65%.

Separate discovery and validation tasks. Following your suggestion, we use disjoint splits with 8 or 16 tasks per stage, each repeated three times. All development tasks remain separate from the original held-out test set. We report score (%) / cost reduction (%) as mean ± sample standard deviation.

SettingSWE-benchDeepSWEDABTerminal-Bench 2.1
8+872.93 ± 1.36 / (44.7 ± 3.3)%31.62 ± 2.59 / (39.3 ± 1.7)%43.56 ± 2.62 / (18.5 ± 3.8)%65.29 ± 0.79 / (18.9 ± 5.2)%
16+1674.52 ± 0.86 / (46.5 ± 2.2)%33.68 ± 0.59 / (42.9 ± 1.9)%49.62 ± 0.66 / (19.9 ± 5.8)%69.40 ± 0.79 / (18.4 ± 5.8)%

We retain cost savings after separating discovery and validation. Expanding from 8+8 to 16+16 improves mean scores by 1.59–6.06 points on all four benchmarks and reduces score variability on three. Its evolution API cost is approximately 97% higher than the default sixteen-task setting. We therefore recommend that users with sufficient evolution budgets increase the number of development tasks, a configurable hyperparameter, to improve the final harness's performance.

Thanks again for helping us examine this choice directly. We hope these results clarify the effects of task selection and development budget. The sensitivity results are included in the main text and Figure 6. We will make the run counts and task splits explicit in the revised setup.

AuthorsOpenReview ↗

Part2: Address the Second Question of Reviewer MrmC

Q2: Can dependency-graph quality be evaluated directly, and does dependency-aware optimization outperform a simpler cost-aware optimizer under the same search budget?

Related weakness (W2):

The contribution of dependency-aware analysis is not convincingly isolated. A central idea of the paper is to construct a dependency graph from agent trajectories and use it to guide harness modifications. However, the paper provides limited evidence about the accuracy of these inferred dependencies or how reliably they identify steps that can be removed or modified. The relevant ablations are also limited in scope. Given that prior harness optimization approaches already use execution feedback to improve agent behavior, stronger evidence is needed to establish how much of the reported improvement comes specifically from the proposed dependency-aware analysis.

A2: Thank you for proposing these complementary evaluations. We added both and find that the graphs retain most required dependencies. Using this evidence also improves score and cost over the structured cost-aware alternative on both benchmarks.

Direct evaluation of dependency evidence. Two reviewers independently audited eight trajectories from four benchmarks, covering 148 action steps. Their initial disagreement rate was 8.41%, with approximately 2.03% of judgments unresolved after joint review.

AnnotatorEdgesType (%)State (%)Recall (%)Complete (%)
GPT-5.541888.598.099.098.5
DeepSeek-V4-Flash42075.798.092.889.7
GPT-5.6-Terra23581.197.384.584.6

Type and State are operation-type and execution-status accuracies over 148 steps. Recall is the fraction of 194 reference dependencies connected by a predicted path. Complete is the fraction of 136 dependent steps retaining all required evidence. Edges is the predicted edge count.

We find that all three annotators retain most required dependencies, with recall of 84.5%–99.0%. GPT-5.5 performs best, achieving 99.0% recall and 98.5% completeness.

Comparison with a structured cost-aware optimizer. Following your suggestion, we compare three evidence representations using GPT-5.5 on SWE-bench and DeepSWE. Raw provides chronological actions and observations. StrucAno adds operation types, status, resources, outcomes, costs, and ordinary trajectory analysis. DAG further adds explicit dependencies and their derived optimization patterns. We hold the remaining setup fixed, including the initial harness, development tasks, proposal and retry limits, and validation. Each variant has one ten-cycle evolution run per benchmark. Each cell reports held-out score (%) / cost (USD per task).

MethodSWE-benchDeepSWE
Raw72.93 / 1.183144.33 / 6.3897
StrucAno72.73 / 1.107845.36 / 5.8816
DAG76.03 / 1.066348.45 / 5.6965

We improve both score and cost by adding dependency evidence. Compared with StrucAno, DAG raises scores by 3.30 and 3.09 points and reduces cost by 3.74% and 3.15% on SWE-bench and DeepSWE, respectively. This shows the additional value of explicit dependencies and their derived patterns for harness optimization.

We hope the audit and controlled comparison address your question about the contribution of dependency analysis. Thank you again for suggesting them. We will include both in the revised paper.

Thank you again for your constructive review. Your suggestions helped us strengthen the evidence for evolution stability and dependency-aware optimization. We hope our responses address your concerns.

AuthorsOpenReview ↗

Summary of our modification, corresponding to questions of the reviewer MrmC

Thank you again for your suggestions on evolution stability and dependency quality. We have added the corresponding analyses to the paper and clarified the experimental settings. Below, we explain how we addressed each point and where to find the results in the revised paper.

For modification in A1

Thank you for suggesting these checks of task selection and evolution stability.

Repeated evolution runs. Please refer to Stability Across Evolution Seeds and Sensitivity to Evolution Cases in Section 4.3.2, with Figure 6(a) and (b). These existing experiments use three runs per setting and show both score and cost variation under fixed and resampled development tasks.

Separate discovery and validation tasks. Please refer to Separate Discovery and Validation Tasks in Section 4.3.2, with Figure 6(c) and (d). We retain the existing panels and results. We have added an explicit description of the disjoint task splits, their separation from the held-out test set, three repetitions, and the approximately 97% higher evolution expenditure for 16+16.

For modification in A2

Thank you for proposing direct evaluation and a controlled comparison of dependency evidence.

Direct evaluation of dependency evidence. We have reorganized the audit into Dependency Evidence Reliability (Section 4.3.4), after the sensitivity and transfer analyses. Table 4 reports the three annotators and defines the metric denominators. The accompanying text reports the two reviewers' disagreement rates. We have also added the random edge-perturbation experiment in Table 5, with scores and costs across all four benchmarks.

Comparison with a structured cost-aware optimizer. Please refer to Evidence Representation in Section 4.3.1 and Figure 5. We retain the existing controlled Raw, StrucAno, and DAG comparison, including its ten-cycle budget and the reported score and cost improvements over StrucAno.

AuthorsOpenReview ↗

Follow-up on Our Responses

Thank you again for your detailed review and valuable suggestions. As the discussion period draws to a close, we would be grateful if you could let us know whether our responses have addressed your concerns. If anything remains unclear, we would be happy to discuss it further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.