- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether improving an agent’s benchmark score actually gives us a better agent. The authors compare four harness-evolution methods across two harnesses and two task domains, with the model weights held fixed. They examine average success, gained and lost tasks, cost, and transfer, then use those observations to revise A-Evolve into AGE. The main finding is that improvement often comes with regressions, and that gains do not transfer uniformly. AGE improves reliability and coverage over A-Evolve in all four evaluated settings.
The study separates average success across repeated attempts from coverage of tasks solved at least once, and supplements both with task-level turnover. This is a useful distinction: a champion can become more consistent on a smaller set of tasks, or cover new tasks while becoming less consistent elsewhere. The frozen-pool selection and admission experiments examine decisions within the search procedure, while the component interventions examine whether useful changes combine additively. I understand the paper’s contribution as an empirical diagnosis of these archived evolution processes, followed by a constructive redesign, rather than a general demonstration of open-ended recursive improvement.
Critical strengths
-
I find the distinction between gaining new tasks and preserving old ones useful. A higher average score can conceal the loss of behavior that already worked. The task-level comparisons and trace examples make this problem concrete, and the cost and transfer results help explain why it matters in practice.
The paper deserves credit for preserving the task identities and examining which outcomes changed. A gain in the aggregate is much easier to interpret when the reader can see the tasks gained, tasks lost and accompanying resource changes. The trace audit also offers plausible behavioral explanations, including state preservation, grounding and completion, rather than treating every success-count change as a new capability. This is a promising direction for evaluating agent updates, especially where regressions on a small set of important workflows would matter.
-
The paper goes beyond pointing out problems. It uses the component experiments to motivate changes and then evaluates the revised method. The gains across both harnesses and domains give some support to the proposed approach, even though the experiments do not tell us which revision is responsible.
The mechanism experiments make the paper more interesting than a catalogue of benchmark gains. The subset comparisons test whether changes remain useful in combination, and the admission replay distinguishes retaining a good candidate from retaining a large set of candidates. I appreciate that the authors do not interpret the size-normalized retention measure in isolation. The improvement of AGE over its reference recipe across both harnesses and domains is encouraging evidence for further work on these decision-specific interventions.
Critical weaknesses
-
My main concern is the lack of repeated evolution runs. Appendix C.1 says each configuration contributes one canonical lineage, and AGE uses search seed 0 in Appendix F.2. Running the selected agent eight times tells us how variable its execution is. It does not tell us how variable the outcome of the evolutionary search is. I therefore find the method-level conclusions stronger than this design supports.
This distinction also affects the interpretation of the claimed preservation benefits. Repeated execution of a fixed champion can estimate its performance, but the champion itself is the product of a stochastic search and selection process. A favorable search path may produce both the chosen artifact and the apparent benefit of a particular operator. I would therefore separate uncertainty conditional on the selected artifact from uncertainty over the procedure that produces it. Even a modest number of independent lineages, or a clear limitation of the conclusions when those are unavailable, would make the evidence easier to evaluate.
-
I would also like a clearer explanation of what was held out during the redesign. The mechanism analyses use test results, whereas the online AGE search excludes them. But if the test results informed the choice of the AGE operators, those tasks have already influenced the method at another stage. That would make the final comparison useful development evidence, but not an independent test of the redesign.
There is a second distinction I would like the paper to make more explicitly. A valid benchmark score can measure average success correctly without guaranteeing lower cost, preservation of every previously solved task or performance on a different distribution. Those are different objectives. ABC recommends matching the measured outcome to the intended capability and explaining its interpretation. The observation that aggregate gains coexist with regressions does not, by itself, show that the benchmark is invalid or that its score lacks meaning. The authors should define the additional notion of usefulness they require and identify what their experiments establish beyond ordinary aggregation and optimization tradeoffs.
The proposed revisions are best assessed against that definition. For example, cost-aware tie-breaking addresses one particular decision, while avoiding regressions addresses another; neither necessarily produces the best agent for every deployment objective. Likewise, a parent with the highest current score need not have the largest remaining improvement potential, but the important contribution is evidence about which selection rule helps under specified budgets. I would be more convinced by a clear explanation of how the revised decisions are justified independently of the same evidence used to discover their weaknesses.
Overall Assessment
Presentation: Generally understandable, but some of the definitions and experimental stages take effort to disentangle.
A compact explanation of which experiments use whole-benchmark reporting, held-out test tasks, decision panels and external transfer would help readers follow the evidence. These distinctions matter to the claims, not just to presentation.
Informativeness: The task-preservation and transfer analyses provide a useful account of what the headline scores miss.
Soundness: The single-lineage design and the separation of development from evaluation need clarification.
I think this is a reasonable paper to accept for its analysis of how harness evolution succeeds and fails. My reservation is that the current experiment supports conclusions about these runs more readily than conclusions about the methods in general. A clear account of the redesign holdouts would help. If the final evaluation informed the redesign itself, or the gains proved unstable across search runs, I would lower my recommendation.
The authors have put substantial effort into examining the process rather than simply reporting a final winner, and I encourage that direction. My support depends on presenting the result at the level its experimental design supports. The strongest version of the paper would explain a concrete deployment/evaluation objective, show why a particular search decision is inadequate for that objective, and validate the proposed correction on evidence not used to design it. That would make the contribution more precise than a general warning about benchmark scores.
Questions for authors
-
What outcomes could EvoAudit access while designing the revisions? Were the final AGE test and transfer results unavailable until those revisions were fixed?
-
How stable are the AGE gains across independent 20-generation runs at the same budget? If those runs are not available, can the authors restrict the conclusion to the evaluated lineages and separate execution uncertainty from search uncertainty?
-
What explicitly defined notion of usefulness is not captured by the benchmark score? Which findings demonstrate a limitation specific to harness evolution, rather than the familiar fact that averages, costs, preservation and transfer are different objectives?
-
Which of the four AGE revisions contributes most to the observed improvement? Can the existing diagnostic and evaluation records separate the benefit of a particular revision from that of changing the bundle, and show the sensitivity of preservation decisions to execution noise?
2 comments
Response to Reviewer 46YA (2)
Q4: Which of the four AGE revisions contributes most to the observed improvement? Can the existing diagnostic and evaluation records separate the benefit of a particular revision from that of changing the bundle, and show the sensitivity of preservation decisions to execution noise? (Weakness 2 Paragraph 3)
A: Thank you for your question. We added two complementary ablation studies. “+” applies the indicated AGE revisions to A-Evolve, while “AGE w/o” restores the corresponding original operators and retains the remaining AGE revisions. We evaluate select and gate jointly because the revised gate admits equal-score candidates with lower comparable cost that the original gate rejects, allowing select to choose among them based on Seed-relative regressions. Thus, only their combination enables this preservation criterion to influence subsequent evolution. We also analyze task preservation and cost changes using these ablation runs.
| Benchmark | τ³-Banking | Terminal-Bench 2 | ||||||
|---|---|---|---|---|---|---|---|---|
| Harness | MiniSWE | Codex | MiniSWE | Codex | ||||
| Variant | Avg@8 | Pass@8 | Avg@8 | Pass@8 | Avg@8 | Pass@8 | Avg@8 | Pass@8 |
| A-Evolve | 26.3 | 55.7 | 28.1 | 57.7 | 68.0 | 86.5 | 68.0 | 87.6 |
| + select and gate | 26.9 | 57.7 | 28.9 | 58.8 | 68.3 | 87.6 | 68.7 | 88.8 |
| + analyze | 30.4 | 60.8 | 32.9 | 61.9 | 69.5 | 88.8 | 69.9 | 89.9 |
| + mutate | 29.8 | 59.8 | 32.3 | 61.9 | 70.6 | 88.8 | 70.9 | 89.9 |
| AGE w/o select and gate | 33.8 | 64.9 | 37.6 | 66.0 | 71.5 | 88.8 | 73.2 | 91.0 |
| AGE w/o analyze | 30.7 | 61.9 | 33.5 | 62.9 | 71.1 | 87.6 | 71.6 | 89.9 |
| AGE w/o mutate | 31.3 | 61.9 | 34.5 | 63.9 | 70.1 | 87.6 | 70.8 | 88.8 |
| AGE | 34.0 | 66.0 | 38.1 | 67.0 | 71.8 | 89.9 | 73.5 | 92.1 |
Both ablation designs identify analyze as the largest contributor to Avg@8 on general tasks and mutate on coding tasks. The additions assess standalone effects of each tested revision unit, while the reversions assess contributions conditional on the other revisions.
To clarify the joint effects of select and gate, we further examined task coverage changes and deployment and evolution costs. All deltas below compare + select and gate with A-Evolve.
| Setting | ΔGained | ΔLost | ΔDeployment (%) | ΔEvolution (%) |
|---|---|---|---|---|
| τ³-banking / MiniSWE | +1 | −1 | +3.3 | −1.2 |
| τ³-banking / Codex | 0 | −1 | −5.1 | +1.4 |
| Terminal-Bench 2 / MiniSWE | +2 | +1 | −4.7 | −0.2 |
| Terminal-Bench 2 / Codex | 0 | −1 | −6.1 | −2.9 |
Adding select and gate jointly reduces lost tasks in three of four settings and reduces deployment cost by 4.7% to 6.1% in three settings. These results show the individual contributions of analyze and mutate and the combined contribution of select and gate.
Our split-half analysis in appendix C.3 shows execution sensitivity in some aggregate trends on τ³-Banking for the original methods. We count a Seed-covered task as lost only if all eight evaluation attempts fail. Under this criterion, joint select and gate revisions reduce losses in three of four settings and improve net coverage in all four.
We have revised lines 470–477 and 507–518 of the manuscript.
Q5: A compact explanation of which experiments use whole-benchmark reporting, held-out test tasks, decision panels and external transfer would help readers follow the evidence. These distinctions matter to the claims, not just to presentation.
A: Thanks for your advice. We have clarified the evaluation scope in the table captions. Full benchmark results describe overall outcomes, Test results assess gains beyond search feedback, separate decision and reporting panels distinguish admission from evaluation, and external benchmarks assess transfer.
Response to Reviewer 46YA (1)
A: Thank you for highlighting the need to clarify the information available during redesign. In our paper, we choose A-Evolve as the improved method for EvoAudit, and EvoAudit had access only to the original A-Evolve search records: Train trajectories, Validation scores, candidate modifications, and admission records. Neither Test outcomes nor external transfer outcomes were provided to EvoAudit when it proposed the operator revisions.
The four revisions to select, analyze, mutate, and gate were fixed before the final AGE test results became available. The online AGE search also uses only Train and Validation evidence. Thus, the final AGE test outcomes did not inform these frozen revisions. We will clarify the distinction between the retrospective analyses reported in the paper and the evidence actually available to EvoAudit during redesign.
EvoAudit organizes our diagnostic approach into a workflow for auditing and revising an existing evolution method. In this study, we apply that workflow to A-Evolve, and AGE is the resulting instantiation used to evaluate the proposed revisions.
We have clarfied the corresponding contents in the manuscript.
A: Thanks for your suggestion. We have completed three independent AGE runs with search seeds 0, 42, and 3407 under the same 20-generation budget. The table reports mean ± sample standard deviation across the three runs for A-Evolve and AGE.
As shown in the results, all 12 AGE runs exceed the corresponding A-Evolve mean on both metrics, supporting consistency of gains across the evaluated runs. AGE also shows lower or comparable variability across runs.
A: Thank you for this distinction. We agree that average benchmark success is a valid measure of its stated outcome. By useful improvement, we mean higher success (Section 3.1) on the intended deployment workload (Section 4), within a cost budget (Section 3.3) and an acceptable regression tolerance (Sections 3.2 and 5.4). For an evolved agent A relative to the Seed A0, this criterion is
C_{\mathcal D}(A)\le B,\qquad R_{\mathcal P}(A,A_0)\le\epsilon.$$ Here, $S_{\mathcal D}$ and $C_{\mathcal D}$ denote average success and mean deployment cost under a fixed execution protocol. Regression on designated previously supported tasks $\mathcal P$ is $$R_{\mathcal P}(A,A_0) = \frac{1}{|\mathcal P|} \sum_{t\in\mathcal P} \max\{0,\ p_t(A_0)-p_t(A)\},$$ where $p_t(A)$ is task success probability. Thus, gains elsewhere do not cancel regressions. For transfer, $\mathcal D$ is the target workload. Because $B$ and $\epsilon$ depend on deployment requirements, we report the underlying outcomes separately. Our contribution beyond metric disagreement is controlled evidence about modification interactions and search retention. Six of eight Champions outperform either isolated Component, while two favor a subset. In one case, both Components individually underperform the Seed, but their combination improves upon it. This demonstrates a limitation of judging modifications in isolation. With candidate pools and evaluation budgets fixed, the diagnostic constrained gate also reduces retention of the candidate with the highest reporting score from 15 of 16 pools to 6 of 16. These findings show how interactions between harness modifications affect agent performance and how admission rules affect candidate retention. Our contribution is a controlled empirical analysis of these mechanisms in the evaluated settings, providing evidence to guide the evaluation and redesign of harness evolution. We have revised section 2.2.2 and the corresponding contents of the manuscript.