- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The author investigates the challenge of coordinating multiple evolvers within an Agent Harness framework during the self-improvement process.
Three approaches—Debate, Pipeline, and Master-Worker—are compared and analyzed using various metrics, including Best/Final performance, as well as the exploration gains and performance degradation observed throughout the evolutionary process.
Furthermore, the author introduces two additional methods, PEG and EOL, to explore whether they can be combined to drive further improvements in final performance.
Critical strengths
The paper draws a clear distinction between "finding a better version during the evolutionary process" and "whether the version ultimately retained is actually better"—a perspective that holds value for the practical deployment of self-improving agents. The authors report results based on five different seeds, incorporating experimental controls such as role decomposition, multiple proposers, and various aggregation methods; they also highlight instances where modifications led to the degradation of other capabilities. These findings provide a useful empirical foundation for investigating stability issues within the evolutionary process.
Critical weaknesses
Weakness 1: Lack of analysis on model–harness interaction; the scope of applicability of the core design claims remains unclear.
I do not see a clear logical chain that connects the failure cases and failure analysis presented in the paper to the two proposed improvements—PEG and EoI—and further substantiates the authors’ core design conclusions. To me, these sections feel disjointed: the authors observe certain failure phenomena and propose interventions that improve empirical outcomes, yet it is never explained what specific problems identified in the failure analysis these interventions resolve, nor why those improved results lend credence to the stated design principles.
Concretely, the effectiveness of a harness is tightly coupled with the underlying foundation model. The paper’s existing failure analysis already touches upon code editing, tool invocation, and error handling—issues that are manifestly tied to model–harness interaction. As such, it is difficult to disentangle whether performance degradation stems from the harness structure itself, the design of tool interfaces, the evolver’s capacity to modify code, or the policy model’s inability to operate effectively within the revised scaffolding. The drop may not reflect an intrinsic deficit of the model at all; rather, a new harness may surface or amplify latent behavioral quirks of the model. Different attributions here would point toward divergent design principles and would materially reshape how a general-purpose harness evolution framework ought to be built—yet the paper currently does not adequately disambiguate among them.
Holding the model fixed does help compare different protocols under the current configuration. However, I would argue that this setup does not resolve the generalization concern introduced by model–harness coupling. The authors would benefit from targeted behavioral and failure analysis, or from cross-model ablations, to demonstrate which gains arise from the coordination protocol’s design versus which are merely artifacts of adaptation tuned to a particular model. Without such evidence, what we observe may simply be a set of modifications that happen to align better with this model, rather than proof of any broadly transferable advantage of the protocol itself.
Therefore, what I expect the authors to establish is: to what extent the claimed design benefits can be cleanly separated from adaptation effects specific to a given model. Absent this distinction, the contribution reads closer to an empirical ranking over a specific model–harness pairing. I do not currently see how the presented results can robustly ground design conclusions at the level of harness evolution or recursive self-improvement (RSI), nor can I infer what concrete design principles other agent systems should adopt on the basis of this work.
Weakness 2: Inconsistencies and undocumented details in the evaluation protocol and evolution pipeline undermine confidence in the validity of key experimental results.
-
Internal inconsistency in the evaluated task suite.
§4.1 distinguishes 160 validation tasks from 448 final evaluation tasks. However, the tasks enumerated in Table 8, combined with the undisclosed APEX subset, sum to 448, while the table footnote claims that Macro is computed over all 608 tasks. The surrounding text references 32 Claw tasks, whereas the table labels this count as 64. It remains ambiguous whether the reported scores are derived from a consistent task set, and how these figures map onto the results in the main tables. -
Unclear whether key metrics adhere to the original benchmark protocol.
Upon inspecting the public documentation and implementation of Evo-Bench, we note that Claw-Eval adopts Pass³: for each task, three independent executions are launched, and success is counted only if all three pass. MEVO does not explicitly state whether it follows this same protocol, raising two issues that directly affect result interpretation:
• (A) The General score of 0.7372 in Table 9 is discussed in relation to 32 tasks, yet 0.7372 × 32 does not yield an integer number of successful tasks. It is unclear whether this figure represents per-task success rate, a continuous score, or a further averaged quantity.• (B) The EoI variant in Table 7 draws answers from three harnesses before selecting a final output. If Pass³ is preserved, a faithful implementation would require three full independent ensemble runs (i.e., nine rollouts per task); this differs materially from running each harness once and then picking an answer. The authors need to specify the exact evaluation procedure actually used; otherwise, it is impossible to determine whether EoI and the other baselines are compared under identical metrics. https://github.com/RUCAIBox/Evo-Bench/blob/889e4fc8b197f426b444dbf8de217ea15b596fd2/evobench/evaluation/runner.py#L490-L492
-
The experimental pipeline through which PEG attains net gains cannot be unambiguously reconstructed from the current write-up.
In Figure 3, the starting points (Round 1) of PEG across the five methods all coincide with the Final scores in Table 1—i.e., the outcomes after three rounds of evolution without PEG. This leaves it unclear whether PEG is initialized from the original scaffold or continues evolution atop an already-evolved version, and whether the compute budget reported in Table 2 subsumes this prior evolution stage. Moreover, the held-out fitness score used by PEG to accept candidate harnesses and the scores reported in Figure 3 may or may not originate from the same data split; validation-set holdout does not guarantee monotonic improvement on the independent test set. Since PEG constitutes a central component responsible for the paper’s claimed net improvements, clarifying these procedural details is essential for assessing both its true efficacy and the fairness of the stated comparisons.
Questions for authors
Questions for Authors
-
What evidence can disentangle the gains attributable to the coordination protocol from those arising merely from adaptation to a specific model–harness pairing? The reported failure cases span code modification, tool interface usage, and the model’s tool-use behavior. Could the authors provide targeted analysis or controlled ablations clarifying how these failures directly motivated the designs of PEG and EoI, and how the observed improvements substantiate the central claim that “independent review stabilizes evolution”?
-
Could the authors explicitly specify the task sets and scoring protocols underlying each reported result, and reconcile the inconsistencies noted above? The key points concern the 448-vs.-608 task discrepancy in Table 8, as well as how the General score in Table 9 is computed. The original Evo-Bench Claw-Eval adopts Pass³ (three independent executions per task, counted as successful only if all three pass). Does MEVO follow the same protocol? And how is EoI in Table 7 reconciled with this three-run evaluation scheme?
-
At exactly which scaffold version does the PEG experiment begin, how are candidate harnesses selected, and how is the evolution budget accounted for? Please clarify why the Round-1 starting points in Figure 3 align with the no-PEG three-round Final scores in Table 1, and confirm whether this prior evolution stage is included in the budget reported in Table 2. Additionally, please state which data split the held-out fitness used for candidate acceptance draws from, and which split the scores plotted in the figures correspond to.
9 comments
Response to Reviewer RP9Y (part 2/4)
W1 / Q1 (second half): How the failure cases motivate PEG and EoI
I do not see a clear logical chain that connects the failure cases and failure analysis presented in the paper to the two proposed improvements—PEG and EoI
The failure cases motivate interventions at three points in the loop: before an edit is promoted, between evolution iterations and at deployment. The table below connects the cases in Appendices K and L to the corresponding intervention and experimental evidence.
| What goes wrong | Example in the paper | Intervention | Evidence |
|---|---|---|---|
| A promoted edit breaks behaviour outside its target | Single, seed 1: a rewrite of dispatch_tool removes the injected-tool path (App. K); five scaffolded configurations fail to import in seed 3 (App. L) | Import guard; separate-context review before promotion | Pipeline Final 0.44 with verifier, 0.38 with self-check, 0.35 without; harmful promotions 2/15 vs 6/15 (App. N) |
| A candidate that passed local review is worse than the retained parent | Debate, seed 1: 0.5699 at iteration 2, then 0.1691 after a refactor that passed the import check | PEG: accept only if validation fitness is no lower than the parent's | Final after +5 iterations, PEG vs ungated: Pipeline 0.55 vs 0.39, Debate 0.61 vs 0.33; the difference is positive in all five seeds for both (App. R) |
| Strong earlier iterates are discarded by last-iterate deployment | Debate: Best 0.59, Final 0.30 | EoI: a judge selects among iterates at deployment | Debate with EoI 0.63; Pipeline with EoI 0.55 (last iterate 0.44) (App. J) |
These comparisons address distinct questions. EoI evaluates deployment quality at three times the inference compute of last-iterate deployment, so its gain is not a causal decomposition. PEG's paired comparison measures the benefit of validation-based preservation; the verifier's contribution is evaluated separately by the within-Pipeline controls.
The conclusion and Section 4.3 distinguish these contributions: separate-context review lowers harmful promotions within Pipeline, validation gating preserves gains across iterations, and candidate quality (Debate's Best) and retention (Pipeline's gap) require separate evaluation. This makes the stabilization claim specific to the interventions and settings tested, since a stronger intermediate candidate does not guarantee a stronger deployed harness.
On model–harness coupling: when a regression arises from the policy mishandling a changed tool interface, review and gating still act on the harness side. The policy-only transfer test in part 1 assesses how method rankings change with the policy while harnesses stay fixed.
Response to Reviewer RP9Y (part 3/4)
W2 / Q2: Task sets, scoring, and Pass³
Internal inconsistency in the evaluated task suite ... Unclear whether key metrics adhere to the original benchmark protocol.
Thank you for highlighting the task-accounting discrepancies. Tables 8 and 9 and the surrounding text retained entries from an earlier subset-based analysis; we have reconciled them with the final task suite. Section 4.1 and Appendix O now provide the consistent accounting below:
| Source | Validation | Evaluation | Main-result score |
|---|---|---|---|
| Search: BrowseComp-EN | 32 | 128 | pass rate |
| Search: HLE | 32 | 128 | pass rate |
| Office: GDPval | 32 | 64 | rubric mean |
| Office: APEX | 32 | 64 | pass rate |
| General: Claw-Eval | 32 | 64 | pass rate |
| Total | 160 | 448 |
The reported macro is 0.4·Search + 0.4·Office + 0.2·General on the 448 evaluation tasks only. The 160 validation tasks select checkpoints and accept PEG candidates and never enter a reported macro; 608 is the benchmark size, 160 + 448. Table 8 now includes APEX, and Claw-Eval is 32 validation and 64 evaluation tasks.
General score (0.7372). This value is not a success count over 32 or 64 tasks.This is an error caused by my carelessness. We replaced it with the value recomputed from task-level outcomes, 47/64 = 0.7344, for the pre-guard Single example in Table 9. With Office 0.6557 and Search 0.3750, the macro is 0.5592.
Pass³. The main tables use one execution per task, not Claw-Eval's native Pass³, and the paper now says so. To check that this does not drive the conclusions, we fixed the Best and Final checkpoints of the methods listed in Table 13 and re-scored General with Pass³ (three independent executions, all must pass). Search and Office are unchanged (Table 13, five seeds).
| Method | Best | Final | Gap |
|---|---|---|---|
| Seed harness | 0.5177 | 0.5177 | 0.0000 |
| Single | 0.4544 | 0.2900 | 0.1644 |
| RolePrompt | 0.5170 | 0.3548 | 0.1622 |
| MultiProp-Judge | 0.5149 | 0.3269 | 0.1880 |
| Debate | 0.5604 | 0.2912 | 0.2692 |
| Pipeline | 0.4783 | 0.4256 | 0.0526 |
| MasterWorker | 0.4719 | 0.3915 | 0.0804 |
Among the evolved methods listed in Table 13, Debate retains the highest Best, while Pipeline retains the highest Final and smallest gap. This is a fixed-checkpoint check, not a re-run of evolution under Pass³, and Concat, Vote and PEG endpoints are not included.
EoI. Under Pass³, EoI needs three independent complete ensembles of three harness rollouts each, so nine General rollouts per task, and every selected output must pass. Our main EoI numbers run each harness once and let a judge without labels or rubrics pick one output, which is one ensemble trial, not Pass³. We now report both (Table 14):
| Method | Single-trial macro | Pass³ macro |
|---|---|---|
| Single + EoI | 0.4700 | 0.4556 |
| RolePrompt + EoI | 0.5300 | 0.5088 |
| MultiProp-Judge + EoI | 0.5400 | 0.5288 |
| Debate + EoI | 0.6300 | 0.6056 |
| Pipeline + EoI | 0.5500 | 0.5363 |
| MasterWorker + EoI | 0.5300 | 0.5200 |
Drops are 0.010–0.024, Debate stays highest, and no pair of methods that differed swaps order.
Response to Reviewer RP9Y (part 1/4)
Thank you for recognizing the deployment value of distinguishing the best harness found from the one ultimately retained, and for highlighting the five-seed design, controls and documented regressions. We also appreciate your detailed examination of the evaluation protocol against the benchmark code. The revision makes the protocol explicit and adds evidence on the questions you raised: corrected task accounting and the Table 9 entry, with Pass³ results for General and EoI (part 3); PEG initialization, acceptance and budget accounting (part 4); and a second model configuration plus frozen-harness transfer (below). Part 2 connects the failure cases to PEG and EoI. Changes are marked in red in the revised PDF.
W1 / Q1 (first half): Separating protocol effects from model-specific adaptation
Fixing the model supports a controlled comparison of protocols. We extend that comparison with two complementary tests of sensitivity and transfer.
Re-evolution under a second configuration (Appendix P, three seeds): policy Qwen3.7-Plus, evolver Kimi-K3.1-Pro, adjudicator DeepSeek-V4-Pro, rubric scorer GLM-5.3.
Debate has the largest Best and Pipeline the smallest gap in all three seeds.
Frozen-harness transfer (Table 17): the five main-run Final harnesses of each method, with only the policy changed to Qwen3.7-Plus. The original scorer, rubrics and tasks are unchanged, with no re-evolution or reselection.
The ranking largely persists (Spearman 0.87 over the nine method-level means, including the seed harness), and Pipeline remains the highest among evolved harnesses.
The re-evolution and frozen-harness results provide evidence that the observed pattern extends beyond the original model pairing. Their scope remains a comparison across the two tested configurations. The revised Limitations makes this attribution boundary explicit.