- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes VERSE, a framework that jointly evolves an LLM agent’s harness and its optimizer while keeping model weights fixed. It uses execution-based verification, failure analysis, and regression tracking to guide improvements to both components. Experiments show that VERSE improves four harness optimizers on held-out SWE-rebench and out-of-distribution tasks.
Critical strengths
The paper addresses an important limitation of existing harness evolution methods, where the optimizer itself remains fixed. The motivation for combining optimizer self-evolution with execution-based verification is clear and practically relevant. The paper is well organized and easy to follow.
Critical weaknesses
-
The main components, including execution-based verification, failure attribution, regression tracking, and optimizer self-improvement, have been explored in prior work. The contribution appears to lie primarily in combining these mechanisms into a unified framework. Although the paper discusses related approaches, more direct comparisons or stronger evidence of the unique benefits of their integration would help establish its novelty.
-
The independent contribution of optimizer self-evolution is not consistently demonstrated. In Table 2, adding self-evolution to VERSE improves validation-selected test accuracy in only three of eight comparisons relative to VERSE without self-evolution.
Questions for authors
In Table 2, adding self-evolution lowers validation-selected test accuracy relative to VERSE without self-evolution in five of the eight comparisons. Under what conditions is optimizer self-evolution actually beneficial (e.g., which hosts, executors or failure types)? Is there statistically reliable evidence for this, such as results over multiple independent evolution runs per configuration?
3 comments
Response to Reviewer DXmN (2/2)
Weakness 2 and Question: The independent contribution of self-evolution is not consistently shown.
Self-evolution makes the improvement hold on every baseline: with it, VERSE is above its baseline in all 16 comparisons of Table 2 (General Response, Part 1, point 2).
The full VERSE method is the harness optimizer with self-evolution; VERSE without it is an ablation that tests whether the verification tools alone are enough. In Table 2, the tools alone improve on the baseline in some comparisons but not in all. With self-evolution, the optimizer learns how to use the tools: it also tests an edit on tasks that the current harness already solves, and it revises a draft more often when a test fails (Section 4.4).
With the closed-weight models of Appendix F, VERSE with self-evolution also matches or exceeds its baseline in all 8 comparisons (Table G2). There, the tools alone already do so, and adding self-evolution scores higher in 6 of the 8, by 0.4 to 2.6 points.
"Under what conditions is self-evolution beneficial?" When the optimizer can verify its edits by execution. We compare self-evolution with and without verification on two benchmarks: SWE-rebench (Table 1) and Terminal-Bench (Appendix J.1, Table 18). On both, self-evolution alone does not improve on Meta-Harness at the selected round, and self-evolution with verification is the best of the four configurations. With the verification tools, it then keeps VERSE above the baseline on all four baselines and on both test sets (Table 2).
"Is there evidence over multiple independent evolution runs?" Yes: in Appendix F, we ran the whole evolution three times (General Response, Part 1, Table G1). VERSE has the highest mean on both benchmarks. On HarnessX, VERSE with self-evolution has three runs and VERSE without it has two (General Response, Part 1, Table G3). VERSE with self-evolution has the higher mean on both benchmarks (32.82% against 32.10% on SWE-rebench, and 40.57% against 36.89% on Terminal-Bench), and each of the three runs with self-evolution scores at least as high as both runs without it. On τ³-bench and AppWorld, with three runs per configuration, VERSE with self-evolution is also higher than without it on both test sets: by 1.4 and 1.1 points on τ³-bench (in distribution and OOD), and by 1.6 and 0.6 on AppWorld (Section "Additional Experiments").
We appreciate your valuable feedback and hope that our answers address your concerns! Please let us know if you have any further questions or suggestions.
Follow-up on Our Response
Dear Reviewer DXmN,
Thank you again for your time and feedback! In addition to our response, we have added Section "Additional Experiments" to the General Response: results on two more benchmarks, τ³-bench and AppWorld, and an ablation study of self-evolution.
As the author discussion closes on Nov 18, we would be grateful for any feedback, and we would be happy to answer further questions.
Sincerely, The Authors
Response to Reviewer DXmN (1/2)
Dear Reviewer DXmN,
Thank you for the careful reading and constructive feedback. We answer each point below. Weakness 2 and your question ask the same thing, so we answer them together. The General Response lists the changes in the revised PDF.
Weakness 1: The components have been explored in prior work; the contribution is their combination.
VERSE focuses on a problem that deserves more attention in harness evolution: how to evolve a harness that generalizes to unseen tasks (General Response, Parts 2 and 3). Its design follows from this goal: an optimizer that tests its edits while it writes them, and that revises its own harness between rounds.
(1) The goal: a harness that generalizes to unseen tasks. Most harness evolution work optimizes a harness for a given benchmark: the harness is evolved on the tasks of the benchmark and evaluated on the same tasks, sometimes with an additional check on other tasks after the search. Higher accuracy on the search tasks does not show that the harness has learned lessons that carry over to other tasks (General Response, Parts 2 and 3). In our setting, generalization is the goal of the optimization: the optimizer learns on training tasks, and the harness is selected and tested on tasks it has never seen (Section 2.2). There, out of distribution, the four baselines stay within two points of the initial harness, while VERSE gains 2.8 to 10.3 points (Table R1).
Table R1: Each baseline alone and with VERSE (with self-evolution), on the two test sets (Table 2). Avg@3 test accuracy (%) of the harness selected on validation. Bold: higher than the baseline alone.
(2) The design follows from this goal. For the harness to generalize, the optimizer has to learn lessons that are not tied to particular tasks. Self-evolution leads the optimizer to such lessons. Its own harness does not change how the executor solves a task, only how the optimizer finds the cause of a failure and tests an edit. So what the optimizer writes there is about how to improve a harness, not about particular tasks. For example, after an edit fixed three tasks but broke two, an optimizer wrote these two lessons for itself (Appendix L.3):
We also found that self-evolution needs verification (Observation 1). In Table 1, the self-evolving optimizer without verification never produced a harness that beat the initial one on validation, and with verification it gives the best harness of the table; the same comparison on Terminal-Bench shows the same pattern (Table 18). This is why VERSE gives the self-evolving optimizer verification tools.
(3) "The main components have been explored in prior work." The closest methods do not do them as VERSE does.
Verification. In earlier harness optimizers, an edit is executed only after it is written, on tasks fixed by the method: after the round has been evaluated (Meta-Harness, AHE), or to decide whether a finished edit is kept (AutoSaddler, HarnessEvolve). In VERSE, the optimizer runs tests itself while it is still writing the edit, and it decides what to test (Section 3.1).
Self-improvement. Recursive self-improvement methods such as STOP, Gödel Agent, and DGM have a single agent that rewrites itself; none of them improves the harness of another agent. The four baselines we compare against improve the harness of an executor with an optimizer that stays fixed. VERSE does both: its optimizer improves the executor's harness and also its own harness.
(4) "More direct comparisons or stronger evidence of the unique benefits of their integration." The paper has both.
Direct comparisons. We compare with four published harness optimizers under one protocol, in distribution and out of distribution. VERSE improves each of them on both test sets (Table R1).
Benefit of the integration. Table 1 tests the two parts separately and together on the same baseline. With self-evolution alone, the optimizer never produced a harness that beat the initial one on validation. With simple verification alone, it reaches 39.81%, close to Meta-Harness (39.20%). With both, it reaches 41.05%, the best of the table. Neither part gives this gain alone.