acceptodds
Under review as a conference paper at ICLR 2027

Beneath the Surface: What We Overlook When We Talk About Agent Evolution

10 views

Abstract

Agent harness evolution enables recursive self-improvement (RSI) without updating model weights. We examine whether higher benchmark scores yield reliable, efficient, and transferable improvement. We evaluate four methods across harnesses, general and coding tasks. Repeated evaluations, controlled agent expansion, component interventions, and admission replays reveal recurring patterns. All sixteen evolved agents gain previously uncovered tasks while losing previously covered ones. In one general-task case, reliability rises by 5.2 points despite losing ten previously covered tasks. Most reliability gains also increase deployment cost. Specific changes usually outperform generic reminders, but their benefits can change when combined. The highest-scoring agent does not yield the largest mean improvement in any of our four pools, and stricter admission can discard useful alternatives. Transfer selectively preserves gains: coverage usually falls on the general-task target across both harness types and paradigms, but rises on the coding target with the outer-layer-only harness. These findings motivate preservation-aware evaluation, tested decision rules, and explicit applicability conditions. As a constructive follow-up, EvoAudit turns these diagnostics into revisions of select, analyze, mutate, and gate, achieving the best performance. Our code is available at https://anonymous.4open.science/r/evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Evolution, Recursive Self-Improvement, Agent Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_46YAOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether improving an agent’s benchmark score actually gives us a better agent. The authors compare four harness-evolution methods across two harnesses and two task domains, with the model weights held fixed. They examine average success, gained and lost tasks, cost, and transfer, then use those observations to revise A-Evolve into AGE. The main finding is that improvement often comes with regressions, and that gains do not transfer uniformly. AGE improves reliability and coverage over A-Evolve in all four evaluated settings.

The study separates average success across repeated attempts from coverage of tasks solved at least once, and supplements both with task-level turnover. This is a useful distinction: a champion can become more consistent on a smaller set of tasks, or cover new tasks while becoming less consistent elsewhere. The frozen-pool selection and admission experiments examine decisions within the search procedure, while the component interventions examine whether useful changes combine additively. I understand the paper’s contribution as an empirical diagnosis of these archived evolution processes, followed by a constructive redesign, rather than a general demonstration of open-ended recursive improvement.

Critical strengths
  1. I find the distinction between gaining new tasks and preserving old ones useful. A higher average score can conceal the loss of behavior that already worked. The task-level comparisons and trace examples make this problem concrete, and the cost and transfer results help explain why it matters in practice.

    The paper deserves credit for preserving the task identities and examining which outcomes changed. A gain in the aggregate is much easier to interpret when the reader can see the tasks gained, tasks lost and accompanying resource changes. The trace audit also offers plausible behavioral explanations, including state preservation, grounding and completion, rather than treating every success-count change as a new capability. This is a promising direction for evaluating agent updates, especially where regressions on a small set of important workflows would matter.

  2. The paper goes beyond pointing out problems. It uses the component experiments to motivate changes and then evaluates the revised method. The gains across both harnesses and domains give some support to the proposed approach, even though the experiments do not tell us which revision is responsible.

    The mechanism experiments make the paper more interesting than a catalogue of benchmark gains. The subset comparisons test whether changes remain useful in combination, and the admission replay distinguishes retaining a good candidate from retaining a large set of candidates. I appreciate that the authors do not interpret the size-normalized retention measure in isolation. The improvement of AGE over its reference recipe across both harnesses and domains is encouraging evidence for further work on these decision-specific interventions.

Critical weaknesses
  1. My main concern is the lack of repeated evolution runs. Appendix C.1 says each configuration contributes one canonical lineage, and AGE uses search seed 0 in Appendix F.2. Running the selected agent eight times tells us how variable its execution is. It does not tell us how variable the outcome of the evolutionary search is. I therefore find the method-level conclusions stronger than this design supports.

    This distinction also affects the interpretation of the claimed preservation benefits. Repeated execution of a fixed champion can estimate its performance, but the champion itself is the product of a stochastic search and selection process. A favorable search path may produce both the chosen artifact and the apparent benefit of a particular operator. I would therefore separate uncertainty conditional on the selected artifact from uncertainty over the procedure that produces it. Even a modest number of independent lineages, or a clear limitation of the conclusions when those are unavailable, would make the evidence easier to evaluate.

  2. I would also like a clearer explanation of what was held out during the redesign. The mechanism analyses use test results, whereas the online AGE search excludes them. But if the test results informed the choice of the AGE operators, those tasks have already influenced the method at another stage. That would make the final comparison useful development evidence, but not an independent test of the redesign.

    There is a second distinction I would like the paper to make more explicitly. A valid benchmark score can measure average success correctly without guaranteeing lower cost, preservation of every previously solved task or performance on a different distribution. Those are different objectives. ABC recommends matching the measured outcome to the intended capability and explaining its interpretation. The observation that aggregate gains coexist with regressions does not, by itself, show that the benchmark is invalid or that its score lacks meaning. The authors should define the additional notion of usefulness they require and identify what their experiments establish beyond ordinary aggregation and optimization tradeoffs.

    The proposed revisions are best assessed against that definition. For example, cost-aware tie-breaking addresses one particular decision, while avoiding regressions addresses another; neither necessarily produces the best agent for every deployment objective. Likewise, a parent with the highest current score need not have the largest remaining improvement potential, but the important contribution is evidence about which selection rule helps under specified budgets. I would be more convinced by a clear explanation of how the revised decisions are justified independently of the same evidence used to discover their weaknesses.

Overall Assessment

Presentation: Generally understandable, but some of the definitions and experimental stages take effort to disentangle.

A compact explanation of which experiments use whole-benchmark reporting, held-out test tasks, decision panels and external transfer would help readers follow the evidence. These distinctions matter to the claims, not just to presentation.

Informativeness: The task-preservation and transfer analyses provide a useful account of what the headline scores miss.

Soundness: The single-lineage design and the separation of development from evaluation need clarification.

I think this is a reasonable paper to accept for its analysis of how harness evolution succeeds and fails. My reservation is that the current experiment supports conclusions about these runs more readily than conclusions about the methods in general. A clear account of the redesign holdouts would help. If the final evaluation informed the redesign itself, or the gains proved unstable across search runs, I would lower my recommendation.

The authors have put substantial effort into examining the process rather than simply reporting a final winner, and I encourage that direction. My support depends on presenting the result at the level its experimental design supports. The strongest version of the paper would explain a concrete deployment/evaluation objective, show why a particular search decision is inadequate for that objective, and validate the proposed correction on evidence not used to design it. That would make the contribution more precise than a general warning about benchmark scores.

Questions for authors
  1. What outcomes could EvoAudit access while designing the revisions? Were the final AGE test and transfer results unavailable until those revisions were fixed?

  2. How stable are the AGE gains across independent 20-generation runs at the same budget? If those runs are not available, can the authors restrict the conclusion to the evaluated lineages and separate execution uncertainty from search uncertainty?

  3. What explicitly defined notion of usefulness is not captured by the benchmark score? Which findings demonstrate a limitation specific to harness evolution, rather than the familiar fact that averages, costs, preservation and transfer are different objectives?

  4. Which of the four AGE revisions contributes most to the observed improvement? Can the existing diagnostic and evaluation records separate the benefit of a particular revision from that of changing the bundle, and show the sensitivity of preservation decisions to execution noise?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 46YA (1)

Q1: What outcomes could EvoAudit access while designing the revisions? Were the final AGE test and transfer results unavailable until those revisions were fixed? (Weakness 2 Paragraph 1 and 3)

A: Thank you for highlighting the need to clarify the information available during redesign. In our paper, we choose A-Evolve as the improved method for EvoAudit, and EvoAudit had access only to the original A-Evolve search records: Train trajectories, Validation scores, candidate modifications, and admission records. Neither Test outcomes nor external transfer outcomes were provided to EvoAudit when it proposed the operator revisions.

The four revisions to select, analyze, mutate, and gate were fixed before the final AGE test results became available. The online AGE search also uses only Train and Validation evidence. Thus, the final AGE test outcomes did not inform these frozen revisions. We will clarify the distinction between the retrospective analyses reported in the paper and the evidence actually available to EvoAudit during redesign.

EvoAudit organizes our diagnostic approach into a workflow for auditing and revising an existing evolution method. In this study, we apply that workflow to A-Evolve, and AGE is the resulting instantiation used to evaluate the proposed revisions.

We have clarfied the corresponding contents in the manuscript.

Q2: How stable are the AGE gains across independent 20-generation runs at the same budget? If those runs are not available, can the authors restrict the conclusion to the evaluated lineages and separate execution uncertainty from search uncertainty? (Weakness 1)

A: Thanks for your suggestion. We have completed three independent AGE runs with search seeds 0, 42, and 3407 under the same 20-generation budget. The table reports mean ± sample standard deviation across the three runs for A-Evolve and AGE.

BenchmarkHarnessVariantAvg@8 (%)Pass@8 (%)
τ³-BankingMiniSWEA-Evolve26.5 ± 0.656.4 ± 0.6
AGE33.9 ± 0.365.6 ± 0.6
τ³-BankingCodexA-Evolve27.9 ± 0.857.4 ± 2.6
AGE37.8 ± 0.366.3 ± 1.2
Terminal-Bench 2MiniSWEA-Evolve68.4 ± 0.486.9 ± 0.6
AGE71.9 ± 0.489.1 ± 1.3
Terminal-Bench 2CodexA-Evolve67.9 ± 0.787.3 ± 1.7
AGE73.2 ± 0.791.4 ± 1.3

As shown in the results, all 12 AGE runs exceed the corresponding A-Evolve mean on both metrics, supporting consistency of gains across the evaluated runs. AGE also shows lower or comparable variability across runs.

Q3: What explicitly defined notion of usefulness is not captured by the benchmark score? Which findings demonstrate a limitation specific to harness evolution, rather than the familiar fact that averages, costs, preservation and transfer are different objectives? (Weakness 2 Paragraph 2)

A: Thank you for this distinction. We agree that average benchmark success is a valid measure of its stated outcome. By useful improvement, we mean higher success (Section 3.1) on the intended deployment workload (Section 4), within a cost budget (Section 3.3) and an acceptable regression tolerance (Sections 3.2 and 5.4). For an evolved agent AA relative to the Seed A0A_0, this criterion is

C_{\mathcal D}(A)\le B,\qquad R_{\mathcal P}(A,A_0)\le\epsilon.$$ Here, $S_{\mathcal D}$ and $C_{\mathcal D}$ denote average success and mean deployment cost under a fixed execution protocol. Regression on designated previously supported tasks $\mathcal P$ is $$R_{\mathcal P}(A,A_0) = \frac{1}{|\mathcal P|} \sum_{t\in\mathcal P} \max\{0,\ p_t(A_0)-p_t(A)\},$$ where $p_t(A)$ is task success probability. Thus, gains elsewhere do not cancel regressions. For transfer, $\mathcal D$ is the target workload. Because $B$ and $\epsilon$ depend on deployment requirements, we report the underlying outcomes separately. Our contribution beyond metric disagreement is controlled evidence about modification interactions and search retention. Six of eight Champions outperform either isolated Component, while two favor a subset. In one case, both Components individually underperform the Seed, but their combination improves upon it. This demonstrates a limitation of judging modifications in isolation. With candidate pools and evaluation budgets fixed, the diagnostic constrained gate also reduces retention of the candidate with the highest reporting score from 15 of 16 pools to 6 of 16. These findings show how interactions between harness modifications affect agent performance and how admission rules affect candidate retention. Our contribution is a controlled empirical analysis of these mechanisms in the evaluated settings, providing evidence to guide the evaluation and redesign of harness evolution. We have revised section 2.2.2 and the corresponding contents of the manuscript.
AuthorsOpenReview ↗

Response to Reviewer 46YA (2)

Q4: Which of the four AGE revisions contributes most to the observed improvement? Can the existing diagnostic and evaluation records separate the benefit of a particular revision from that of changing the bundle, and show the sensitivity of preservation decisions to execution noise? (Weakness 2 Paragraph 3)

A: Thank you for your question. We added two complementary ablation studies. “+” applies the indicated AGE revisions to A-Evolve, while “AGE w/o” restores the corresponding original operators and retains the remaining AGE revisions. We evaluate select and gate jointly because the revised gate admits equal-score candidates with lower comparable cost that the original gate rejects, allowing select to choose among them based on Seed-relative regressions. Thus, only their combination enables this preservation criterion to influence subsequent evolution. We also analyze task preservation and cost changes using these ablation runs.

Benchmarkτ³-BankingTerminal-Bench 2
HarnessMiniSWECodexMiniSWECodex
VariantAvg@8Pass@8Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8
A-Evolve26.355.728.157.768.086.568.087.6
+ select and gate26.957.728.958.868.387.668.788.8
+ analyze30.460.832.961.969.588.869.989.9
+ mutate29.859.832.361.970.688.870.989.9
AGE w/o select and gate33.864.937.666.071.588.873.291.0
AGE w/o analyze30.761.933.562.971.187.671.689.9
AGE w/o mutate31.361.934.563.970.187.670.888.8
AGE34.066.038.167.071.889.973.592.1

Both ablation designs identify analyze as the largest contributor to Avg@8 on general tasks and mutate on coding tasks. The additions assess standalone effects of each tested revision unit, while the reversions assess contributions conditional on the other revisions.

To clarify the joint effects of select and gate, we further examined task coverage changes and deployment and evolution costs. All deltas below compare + select and gate with A-Evolve.

SettingΔGainedΔLostΔDeployment (%)ΔEvolution (%)
τ³-banking / MiniSWE+1−1+3.3−1.2
τ³-banking / Codex0−1−5.1+1.4
Terminal-Bench 2 / MiniSWE+2+1−4.7−0.2
Terminal-Bench 2 / Codex0−1−6.1−2.9

Adding select and gate jointly reduces lost tasks in three of four settings and reduces deployment cost by 4.7% to 6.1% in three settings. These results show the individual contributions of analyze and mutate and the combined contribution of select and gate.

Our split-half analysis in appendix C.3 shows execution sensitivity in some aggregate trends on τ³-Banking for the original methods. We count a Seed-covered task as lost only if all eight evaluation attempts fail. Under this criterion, joint select and gate revisions reduce losses in three of four settings and improve net coverage in all four.

We have revised lines 470–477 and 507–518 of the manuscript.

Q5: A compact explanation of which experiments use whole-benchmark reporting, held-out test tasks, decision panels and external transfer would help readers follow the evidence. These distinctions matter to the claims, not just to presentation.

A: Thanks for your advice. We have clarified the evaluation scope in the table captions. Full benchmark results describe overall outcomes, Test results assess gains beyond search feedback, separate decision and reporting panels distinguish admission from evaluation, and external benchmarks assess transfer.

Reviewer_PZZ1OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether agent evolution improves reliability, efficiency, and generalization beyond benchmark scores. The authors evaluate four methods with two harnesses on general and coding tasks, and analyze how agent selection, diagnosis, mutation, and admission affect the results. They find that higher scores can come with task regressions, increased cost, and limited transfer. Based on these findings, they introduce EvoAudit and the resulting AGE method, which improves reliability and task coverage in the four evaluated settings.

Critical strengths
  1. The topic of the paper is important, and the experimental analysis covers several aspects.
  2. The results provide some interesting findings about the limitations of agent evolution. The authors also use these findings to improve the evolution procedure, which gives the analysis practical value.
Critical weaknesses
  1. The experiments use only one evolution run for each combination of method and harness. Evaluating the resulting agent eight times can measure variation in its execution, but does not show whether the evolution algorithm consistently produces good agents. I would like to see multiple independent evolution runs to better assess the stability of the findings.

  2. The criterion for identifying the best parent in Section 5.1 is not fully justified. The authors use the improvement of the child over its parent, but a lower-scoring parent may show a larger improvement while still producing a worse child. It would be helpful to compare the conclusions using the children's absolute performance. In addition, three candidates per parent may be insufficient to reliably estimate its improvement potential.

  3. The contribution of each operator in AGE is unclear. The gains could mainly come from improved diagnosis and mutation prompts rather than the revised selection and admission rules. Individual and leave-one-out ablations would help establish which changes drive the improvement and whether their combination provides additional benefits.

  4. The overall readability needs improvement. For example, Table 4 reports Components A and B without explaining their contents in the main text, while the actual AGE operator changes are mainly described in Appendix F.2. I understand the page limit, but the main paper should still provide enough detail to follow the experiments and understand the method without repeatedly consulting the appendix.

Questions for authors

Please see the Weaknesses part.

3 comments
AuthorsOpenReview ↗

Response to Reviewer PZZ1 (1)

Q1: The experiments use only one evolution run for each combination of method and harness. Evaluating the resulting agent eight times can measure variation in its execution, but does not show whether the evolution algorithm consistently produces good agents. I would like to see multiple independent evolution runs to better assess the stability of the findings.

A: Thank you for your suggestion. We have added two independent evolution runs for every method, benchmark, and harness combination. The table reports mean ± standard deviation across the three independent evolution runs.

BenchmarkHarnessVariantAvg@8 (%)Pass@8 (%)
τ³-BankingMiniSWEAHE32.5 ± 0.565.6 ± 1.6
HyperAgents31.7 ± 2.558.1 ± 4.6
A-Evolve26.5 ± 0.656.4 ± 0.6
GEPA28.1 ± 0.458.4 ± 2.6
CodexAHE36.1 ± 0.362.5 ± 1.6
HyperAgents36.8 ± 0.366.7 ± 0.6
A-Evolve27.9 ± 0.857.4 ± 2.6
GEPA29.8 ± 0.454.6 ± 1.0
Terminal-Bench 2MiniSWEAHE70.0 ± 0.887.3 ± 1.7
HyperAgents67.6 ± 2.583.1 ± 3.0
A-Evolve68.4 ± 0.486.9 ± 0.6
GEPA68.7 ± 1.689.1 ± 1.7
CodexAHE71.9 ± 0.590.6 ± 1.7
HyperAgents72.8 ± 1.386.9 ± 2.8
A-Evolve67.9 ± 0.787.3 ± 1.7
GEPA68.4 ± 0.486.9 ± 0.6

The three-run averages broadly agree with the original results. Relative to the Seed, all 16 Avg@8 comparisons and 15 of the 16 Pass@8 comparisons retain their original trends, with one Pass@8 comparison changing from a tie to a small average gain. Four settings still show improvement in one metric and decline in the other, supporting our finding that reliability and coverage gains can diverge.

Moreover, we find that evolution stability differs across methods. HyperAgents has the largest standard deviations on both metrics in most settings. This may reflect its ability to modify evolution operators, which expands the search space. AHE’s sequential updates avoid random parent sampling. A-Evolve’s recurring patterns may encourage similar revisions across runs, while GEPA’s Pareto selection may reduce sensitivity to aggregate-score fluctuations.

We have revised line 944–978 of the manuscript.

Q2: The criterion for identifying the best parent in Section 5.1 is not fully justified. The authors use the improvement of the child over its parent, but a lower-scoring parent may show a larger improvement while still producing a worse child. It would be helpful to compare the conclusions using the children's absolute performance. In addition, three candidates per parent may be insufficient to reliably estimate its improvement potential.

A: Thank you for this suggestion. We add a comparison using children’s absolute performance, defining the best parent as the one whose children achieve the highest mean Test Avg@8. We also expand the analysis from three to ten candidates per parent under the same generation and evaluation protocol. The table reports best-parent selection rates under both criteria.

BenchmarkHarnessParent-relative improvement (%)Absolute child performance (%)Random (%)
τ³-BankingMiniSWE17.525.225.0
τ³-BankingCodex7.98.825.0
Terminal-Bench 2MiniSWE59.59.025.0
Terminal-Bench 2Codex55.717.625.0

With ten candidates, selection remains below random on general tasks and above random on coding tasks under the improvement criterion. Under absolute performance, it is below random in three settings and near random in one. We think that coding tasks provide concrete execution feedback that supports targeted repairs, allowing selected parents to achieve larger gains without producing the strongest children, while general tasks require coordinated retrieval, policy reasoning, and user interaction, limiting the usefulness of current performance for predicting either outcome.

We consider these two criteria are both meaningful and should be used according to the evaluation objective. Parent-relative improvement is appropriate for studying which parents benefit most from continued optimization under a common process and budget. Absolute child performance is appropriate for identifying starting points that produce the highest-performing children on average.

We have revised line 330–347 and line 1328–1373 of the manuscript.

AuthorsOpenReview ↗

Response to Reviewer PZZ1 (2)

Q3: The contribution of each operator in AGE is unclear. The gains could mainly come from improved diagnosis and mutation prompts rather than the revised selection and admission rules. Individual and leave-one-out ablations would help establish which changes drive the improvement and whether their combination provides additional benefits.

A: Thank you for your question. We added two complementary ablation studies. “+” applies the indicated AGE revisions to A-Evolve, while “AGE w/o” restores the corresponding original operators and retains the remaining AGE revisions. We evaluate select and gate jointly because the revised gate admits equal-score candidates with lower comparable cost that the original gate rejects, allowing select to choose among them based on Seed-relative regressions. Thus, only their combination enables this preservation criterion to influence subsequent evolution. We also analyze task preservation and cost changes using these ablation runs.

Benchmarkτ³-BankingTerminal-Bench 2
HarnessMiniSWECodexMiniSWECodex
VariantAvg@8Pass@8Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8
A-Evolve26.355.728.157.768.086.568.087.6
+ select and gate26.957.728.958.868.387.668.788.8
+ analyze30.460.832.961.969.588.869.989.9
+ mutate29.859.832.361.970.688.870.989.9
AGE w/o select and gate33.864.937.666.071.588.873.291.0
AGE w/o analyze30.761.933.562.971.187.671.689.9
AGE w/o mutate31.361.934.563.970.187.670.888.8
AGE34.066.038.167.071.889.973.592.1

Both ablation designs identify analyze as the largest contributor to Avg@8 on general tasks and mutate on coding tasks. The additions assess standalone effects of each tested revision unit, while the reversions assess contributions conditional on the other revisions.

To clarify the joint effects of select and gate, we further examined task coverage changes and deployment and evolution costs. All deltas below compare + select and gate with A-Evolve.

SettingΔGainedΔLostΔDeployment (%)ΔEvolution (%)
τ³-banking / MiniSWE+1−1+3.3−1.2
τ³-banking / Codex0−1−5.1+1.4
Terminal-Bench 2 / MiniSWE+2+1−4.7−0.2
Terminal-Bench 2 / Codex0−1−6.1−2.9

Adding select and gate jointly reduces lost tasks in three of four settings and reduces deployment cost by 4.7% to 6.1% in three settings. These results show the individual contributions of analyze and mutate and the combined contribution of select and gate.

We have revised lines 470–477 and 507–518 of the manuscript.

Q4: The overall readability needs improvement. For example, Table 4 reports Components A and B without explaining their contents in the main text, while the actual AGE operator changes are mainly described in Appendix F.2. I understand the page limit, but the main paper should still provide enough detail to follow the experiments and understand the method without repeatedly consulting the appendix.

A: Thank you for your suggestion. We have added concise explanations of Components A and B across benchmarks and harnesses, including tool-interface improvements and workflow guidance on general tasks, and task-solving guidance and execution support on coding tasks. We also clarify that each component is evaluated on the matching Seed with its required dependencies. The revisions are highlighted in blue in lines 355–361 for Components A and B and lines 431–441 for AGE’s operator revisions.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer PZZ1,

Thank you again for your thoughtful feedback. We have revised the manuscript and provided additional results on independent evolution runs, parent selection, and operator ablations, along with clarifications to improve readability.

We would appreciate hearing whether our responses and revisions address your concerns. If any questions remain, we would be happy to provide further clarification and improve the work accordingly.

If you find that the revisions address your concerns, we would be grateful if you could reconsider your rating in light of these changes.

Thank you for your time and consideration. We look forward to your feedback.

Reviewer_HzTpOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the limitations of existing agent harness evolution methods beyond their final benchmark scores. The authors evaluate four evolution methods across two agent harnesses and two benchmarks, analyzing task-level regressions, deployment cost, generalization, and the effects of different evolution operators. Based on these findings, the paper proposes AGE, which revises the selection, diagnosis, mutation, and admission procedures to improve agent evolution.

Critical strengths

The paper provides a detailed analysis of existing agent evolution methods. In particular, the finding that evolved agents can gain new capabilities while losing previously solved tasks is interesting and highlights a limitation of relying on aggregate benchmark scores.

The experiments cover multiple evolution methods, harnesses, and task domains. The analysis of reliability, coverage, deployment cost, and cross-benchmark transfer provides useful insights into the behavior of evolved agents.

Critical weaknesses

The novelty of the proposed AGE method is relatively limited. Most of its components are reasonable engineering improvements to existing evolution operators, such as more specific diagnosis, preserving dependencies during mutation, and considering cost during admission. It is unclear how much methodological novelty these changes provide.

The paper does not sufficiently isolate the contribution of each AGE component. Since selection, diagnosis, mutation, and admission are modified together, it is difficult to determine which changes are responsible for the improvements. An ablation study of these four components would strengthen the empirical conclusions.

Questions for authors

Could the authors provide an ablation of the four operator revisions in AGE? Which components contribute most to the performance improvements?

2 comments
AuthorsOpenReview ↗

Response to Reviewer HzTp

Q1: The novelty of the proposed AGE method is relatively limited. Most of its components are reasonable engineering improvements to existing evolution operators, such as more specific diagnosis, preserving dependencies during mutation, and considering cost during admission. It is unclear how much methodological novelty these changes provide.

A: Thank you for raising this point. We agree that AGE’s individual operator revisions build on existing evolution techniques. Our main contribution is a systematic analysis of agent harness evolution beyond aggregate benchmark scores rather than an evolution algorithm. Therefore, AGE serves as an empirical case study to assess the practical utility of our analysis. Its revisions to the four evolution operators address the identified limitations, testing whether our findings can guide effective improvements to agent evolution.

Q2: The paper does not sufficiently isolate the contribution of each AGE component. Since selection, diagnosis, mutation, and admission are modified together, it is difficult to determine which changes are responsible for the improvements. An ablation study of these four components would strengthen the empirical conclusions. Could the authors provide an ablation of the four operator revisions in AGE? Which components contribute most to the performance improvements?

A: Thank you for your question. We added two complementary ablation studies. “+” applies the indicated AGE revisions to A-Evolve, while “AGE w/o” restores the corresponding original operators and retains the remaining AGE revisions. We evaluate select and gate jointly because the revised gate admits equal-score candidates with lower comparable cost that the original gate rejects, allowing select to choose among them based on Seed-relative regressions. Thus, only their combination enables this preservation criterion to influence subsequent evolution. We also analyze task preservation and cost changes using these ablation runs.

Benchmarkτ³-BankingTerminal-Bench 2
HarnessMiniSWECodexMiniSWECodex
VariantAvg@8Pass@8Avg@8Pass@8Avg@8Pass@8Avg@8Pass@8
A-Evolve26.355.728.157.768.086.568.087.6
+ select and gate26.957.728.958.868.387.668.788.8
+ analyze30.460.832.961.969.588.869.989.9
+ mutate29.859.832.361.970.688.870.989.9
AGE w/o select and gate33.864.937.666.071.588.873.291.0
AGE w/o analyze30.761.933.562.971.187.671.689.9
AGE w/o mutate31.361.934.563.970.187.670.888.8
AGE34.066.038.167.071.889.973.592.1

Both ablation designs identify analyze as the largest contributor to Avg@8 on general tasks and mutate on coding tasks. The additions assess standalone effects of each tested revision unit, while the reversions assess contributions conditional on the other revisions.

To clarify the joint effects of select and gate, we further examined task coverage changes and deployment and evolution costs. All deltas below compare + select and gate with A-Evolve.

SettingΔGainedΔLostΔDeployment (%)ΔEvolution (%)
τ³-banking / MiniSWE+1−1+3.3−1.2
τ³-banking / Codex0−1−5.1+1.4
Terminal-Bench 2 / MiniSWE+2+1−4.7−0.2
Terminal-Bench 2 / Codex0−1−6.1−2.9

Adding select and gate jointly reduces lost tasks in three of four settings and reduces deployment cost by 4.7% to 6.1% in three settings. These results show the individual contributions of analyze and mutate and the combined contribution of select and gate.

We have revised lines 470–477 and 507–518 of the manuscript.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer HzTp,

Thank you again for your insightful feedback and valuable comments on our paper. We have clarified AGE’s role in validating our analysis and added ablations of its four operators to assess their contributions.

We would appreciate your feedback on whether these updates address your concerns. If any questions remain, we would be happy to clarify them. If the updates have addressed your concerns, we would appreciate it if you could update the rating score accordingly.

Thank you for your time and consideration. Looking forward to your feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers and Area Chair,

We thank all the reviewers for their valuable comments and constructive feedback. We have carefully revised the manuscript and conducted additional experiments in response to their feedback, with the revised parts highlighted in blue. For your convenience, we summarize the main updates below.

  1. Clarifying the contribution and AGE’s redesign protocol. (Reviewers HzTp and 46YA) Our main contribution is a systematic empirical analysis of agent harness evolution, including how interactions between harness modifications and admission decisions shape the resulting agents. We have explicitly defined useful improvement as higher success on the intended workload within cost and regression constraints. EvoAudit applies our diagnostic approach to A-Evolve, with AGE as the resulting case study. During redesign, EvoAudit used only A-Evolve’s Train and Validation search records. The operator revisions were fixed before the final AGE Test results became available.

  2. Assessing stability across independent evolution runs. (Reviewers PZZ1 and 46YA) We now report results from three independent evolution runs per evaluated method, benchmark and harness combination, distinguishing search variability from repeated execution of a fixed agent. The three-run averages broadly preserve the original performance trends, including the divergence between reliability and coverage gains.

  3. Identifying the contributions of AGE’s operator revisions. (Reviewers HzTp, PZZ1, and 46YA) We added complementary addition and reversion ablations, evaluating analyze and mutate individually and select and gate jointly because of their coupled admission and selection mechanism. Both designs identify analyze as the largest contributor to Avg@8 on τ³-Banking and mutate on Terminal-Bench 2. The joint select and gate revision reduces lost tasks and cost in most settings and improves net task coverage. These experiments distinguish the contributions of AGE’s operator revisions.

  4. Strengthening the diagnostic analysis and presentation. (Reviewers PZZ1 and 46YA) We expanded the parent-selection analysis from three to ten candidates per parent and compared parent-relative improvement with absolute child performance, clarifying how the conclusions depend on the evaluation objective. We also added explanations of the components and AGE operator revisions, and clarified the roles of whole benchmark reporting, held-out Test tasks, decision and reporting panels, and external transfer evaluations.

We believe these updates adequately address all the reviewers' concerns and improve the quality of our manuscript. We sincerely appreciate your time and consideration of our revised manuscript.

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.