acceptodds
Under review as a conference paper at ICLR 2027

MEvo: Multi-Evolver Coordination for Self-Improving Agent Harnesses

Abstract

Self-improving agent harnesses in the DGM lineage all delegate each self-modification to a single LLM proposer—the single-editor assumption. We introduce MEvo (Multi-Evolver Coordination), to our knowledge the first harness-evolution configuration whose final write authority rests with an explicit protocol over k independent evolvers. We instantiate three protocols: MEvo-Debate (competing proposals under judge adjudication), MEvo-Pipeline (staged diagnose–modify–verify), and MEvo-MasterWorker (persistent cross-iteration authority). Across 5 seeds on the full 608-task EvoBench benchmark, Debate attains the highest Best macro, above the single-evolver baseline, while Pipeline holds the deployed score within 0.05 of its highest iterate—several-fold tighter than every other tested configuration. No protocol's final iterate surpasses the seed harness under the 3-iteration budget; the same incentive-disjoint discipline, lifted to the outer loop as a Preserve-and-Extend Gate, pushes Pipeline above seed at 5 iterations, and an Ensemble-of-Iterates recipe further improves inference-time quality. These results establish multi-evolver coordination with incentive-disjoint review as the pre-condition for converting transient exploration into net improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, self-evolution, multi-agent coordination, agent harness, automated optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RP9YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author investigates the challenge of coordinating multiple evolvers within an Agent Harness framework during the self-improvement process.

Three approaches—Debate, Pipeline, and Master-Worker—are compared and analyzed using various metrics, including Best/Final performance, as well as the exploration gains and performance degradation observed throughout the evolutionary process.

Furthermore, the author introduces two additional methods, PEG and EOL, to explore whether they can be combined to drive further improvements in final performance.

Critical strengths

The paper draws a clear distinction between "finding a better version during the evolutionary process" and "whether the version ultimately retained is actually better"—a perspective that holds value for the practical deployment of self-improving agents. The authors report results based on five different seeds, incorporating experimental controls such as role decomposition, multiple proposers, and various aggregation methods; they also highlight instances where modifications led to the degradation of other capabilities. These findings provide a useful empirical foundation for investigating stability issues within the evolutionary process.

Critical weaknesses

Weakness 1: Lack of analysis on model–harness interaction; the scope of applicability of the core design claims remains unclear.

I do not see a clear logical chain that connects the failure cases and failure analysis presented in the paper to the two proposed improvements—PEG and EoI—and further substantiates the authors’ core design conclusions. To me, these sections feel disjointed: the authors observe certain failure phenomena and propose interventions that improve empirical outcomes, yet it is never explained what specific problems identified in the failure analysis these interventions resolve, nor why those improved results lend credence to the stated design principles.

Concretely, the effectiveness of a harness is tightly coupled with the underlying foundation model. The paper’s existing failure analysis already touches upon code editing, tool invocation, and error handling—issues that are manifestly tied to model–harness interaction. As such, it is difficult to disentangle whether performance degradation stems from the harness structure itself, the design of tool interfaces, the evolver’s capacity to modify code, or the policy model’s inability to operate effectively within the revised scaffolding. The drop may not reflect an intrinsic deficit of the model at all; rather, a new harness may surface or amplify latent behavioral quirks of the model. Different attributions here would point toward divergent design principles and would materially reshape how a general-purpose harness evolution framework ought to be built—yet the paper currently does not adequately disambiguate among them.

Holding the model fixed does help compare different protocols under the current configuration. However, I would argue that this setup does not resolve the generalization concern introduced by model–harness coupling. The authors would benefit from targeted behavioral and failure analysis, or from cross-model ablations, to demonstrate which gains arise from the coordination protocol’s design versus which are merely artifacts of adaptation tuned to a particular model. Without such evidence, what we observe may simply be a set of modifications that happen to align better with this model, rather than proof of any broadly transferable advantage of the protocol itself.

Therefore, what I expect the authors to establish is: to what extent the claimed design benefits can be cleanly separated from adaptation effects specific to a given model. Absent this distinction, the contribution reads closer to an empirical ranking over a specific model–harness pairing. I do not currently see how the presented results can robustly ground design conclusions at the level of harness evolution or recursive self-improvement (RSI), nor can I infer what concrete design principles other agent systems should adopt on the basis of this work.

Weakness 2: Inconsistencies and undocumented details in the evaluation protocol and evolution pipeline undermine confidence in the validity of key experimental results.

  1. Internal inconsistency in the evaluated task suite.
    §4.1 distinguishes 160 validation tasks from 448 final evaluation tasks. However, the tasks enumerated in Table 8, combined with the undisclosed APEX subset, sum to 448, while the table footnote claims that Macro is computed over all 608 tasks. The surrounding text references 32 Claw tasks, whereas the table labels this count as 64. It remains ambiguous whether the reported scores are derived from a consistent task set, and how these figures map onto the results in the main tables.

  2. Unclear whether key metrics adhere to the original benchmark protocol.
    Upon inspecting the public documentation and implementation of Evo-Bench, we note that Claw-Eval adopts Pass³: for each task, three independent executions are launched, and success is counted only if all three pass. MEVO does not explicitly state whether it follows this same protocol, raising two issues that directly affect result interpretation:
    • (A) The General score of 0.7372 in Table 9 is discussed in relation to 32 tasks, yet 0.7372 × 32 does not yield an integer number of successful tasks. It is unclear whether this figure represents per-task success rate, a continuous score, or a further averaged quantity.

    • (B) The EoI variant in Table 7 draws answers from three harnesses before selecting a final output. If Pass³ is preserved, a faithful implementation would require three full independent ensemble runs (i.e., nine rollouts per task); this differs materially from running each harness once and then picking an answer. The authors need to specify the exact evaluation procedure actually used; otherwise, it is impossible to determine whether EoI and the other baselines are compared under identical metrics.​ https://github.com/RUCAIBox/Evo-Bench/blob/889e4fc8b197f426b444dbf8de217ea15b596fd2/evobench/evaluation/runner.py#L490-L492

  3. The experimental pipeline through which PEG attains net gains cannot be unambiguously reconstructed from the current write-up.
    In Figure 3, the starting points (Round 1) of PEG across the five methods all coincide with the Final scores in Table 1—i.e., the outcomes after three rounds of evolution without PEG. This leaves it unclear whether PEG is initialized from the original scaffold or continues evolution atop an already-evolved version, and whether the compute budget reported in Table 2 subsumes this prior evolution stage. Moreover, the held-out fitness score used by PEG to accept candidate harnesses and the scores reported in Figure 3 may or may not originate from the same data split; validation-set holdout does not guarantee monotonic improvement on the independent test set. Since PEG constitutes a central component responsible for the paper’s claimed net improvements, clarifying these procedural details is essential for assessing both its true efficacy and the fairness of the stated comparisons.

Questions for authors

Questions for Authors

  1. What evidence can disentangle the gains attributable to the coordination protocol from those arising merely from adaptation to a specific model–harness pairing? The reported failure cases span code modification, tool interface usage, and the model’s tool-use behavior. Could the authors provide targeted analysis or controlled ablations clarifying how these failures directly motivated the designs of PEG and EoI, and how the observed improvements substantiate the central claim that “independent review stabilizes evolution”?

  2. Could the authors explicitly specify the task sets and scoring protocols underlying each reported result, and reconcile the inconsistencies noted above? The key points concern the 448-vs.-608 task discrepancy in Table 8, as well as how the General score in Table 9 is computed. The original Evo-Bench Claw-Eval adopts Pass³ (three independent executions per task, counted as successful only if all three pass). Does MEVO follow the same protocol? And how is EoI in Table 7 reconciled with this three-run evaluation scheme?

  3. At exactly which scaffold version does the PEG experiment begin, how are candidate harnesses selected, and how is the evolution budget accounted for? Please clarify why the Round-1 starting points in Figure 3 align with the no-PEG three-round Final scores in Table 1, and confirm whether this prior evolution stage is included in the budget reported in Table 2. Additionally, please state which data split the held-out fitness used for candidate acceptance draws from, and which split the scores plotted in the figures correspond to.

9 comments
AuthorsOpenReview ↗

Response to Reviewer RP9Y (part 1/4)

Thank you for recognizing the deployment value of distinguishing the best harness found from the one ultimately retained, and for highlighting the five-seed design, controls and documented regressions. We also appreciate your detailed examination of the evaluation protocol against the benchmark code. The revision makes the protocol explicit and adds evidence on the questions you raised: corrected task accounting and the Table 9 entry, with Pass³ results for General and EoI (part 3); PEG initialization, acceptance and budget accounting (part 4); and a second model configuration plus frozen-harness transfer (below). Part 2 connects the failure cases to PEG and EoI. Changes are marked in red in the revised PDF.

W1 / Q1 (first half): Separating protocol effects from model-specific adaptation

... what evidence can disentangle the gains attributable to the coordination protocol from those arising merely from adaptation to a specific model–harness pairing?

Fixing the model supports a controlled comparison of protocols. We extend that comparison with two complementary tests of sensitivity and transfer.

Re-evolution under a second configuration (Appendix P, three seeds): policy Qwen3.7-Plus, evolver Kimi-K3.1-Pro, adjudicator DeepSeek-V4-Pro, rubric scorer GLM-5.3.

MethodBestFinalGap
Seed harness0.480.480.00
Single0.440.290.15
Debate0.540.310.23
Pipeline0.460.410.05
MasterWorker0.450.370.08

Debate has the largest Best and Pipeline the smallest gap in all three seeds.

Frozen-harness transfer (Table 17): the five main-run Final harnesses of each method, with only the policy changed to Qwen3.7-Plus. The original scorer, rubrics and tasks are unchanged, with no re-evolution or reselection.

MethodOriginal FinalTransferred
Seed harness0.530.48
Single0.300.28
RolePrompt0.380.33
MultiProp-Concat0.330.35
MultiProp-Vote0.360.31
MultiProp-Judge0.340.36
Debate0.300.29
Pipeline0.440.41
MasterWorker0.400.37

The ranking largely persists (Spearman 0.87 over the nine method-level means, including the seed harness), and Pipeline remains the highest among evolved harnesses.

The re-evolution and frozen-harness results provide evidence that the observed pattern extends beyond the original model pairing. Their scope remains a comparison across the two tested configurations. The revised Limitations makes this attribution boundary explicit.

AuthorsOpenReview ↗

Response to Reviewer RP9Y (part 2/4)

W1 / Q1 (second half): How the failure cases motivate PEG and EoI

I do not see a clear logical chain that connects the failure cases and failure analysis presented in the paper to the two proposed improvements—PEG and EoI

The failure cases motivate interventions at three points in the loop: before an edit is promoted, between evolution iterations and at deployment. The table below connects the cases in Appendices K and L to the corresponding intervention and experimental evidence.

What goes wrongExample in the paperInterventionEvidence
A promoted edit breaks behaviour outside its targetSingle, seed 1: a rewrite of dispatch_tool removes the injected-tool path (App. K); five scaffolded configurations fail to import in seed 3 (App. L)Import guard; separate-context review before promotionPipeline Final 0.44 with verifier, 0.38 with self-check, 0.35 without; harmful promotions 2/15 vs 6/15 (App. N)
A candidate that passed local review is worse than the retained parentDebate, seed 1: 0.5699 at iteration 2, then 0.1691 after a refactor that passed the import checkPEG: accept only if validation fitness is no lower than the parent'sFinal after +5 iterations, PEG vs ungated: Pipeline 0.55 vs 0.39, Debate 0.61 vs 0.33; the difference is positive in all five seeds for both (App. R)
Strong earlier iterates are discarded by last-iterate deploymentDebate: Best 0.59, Final 0.30EoI: a judge selects among iterates at deploymentDebate with EoI 0.63; Pipeline with EoI 0.55 (last iterate 0.44) (App. J)

These comparisons address distinct questions. EoI evaluates deployment quality at three times the inference compute of last-iterate deployment, so its gain is not a causal decomposition. PEG's paired comparison measures the benefit of validation-based preservation; the verifier's contribution is evaluated separately by the within-Pipeline controls.

The conclusion and Section 4.3 distinguish these contributions: separate-context review lowers harmful promotions within Pipeline, validation gating preserves gains across iterations, and candidate quality (Debate's Best) and retention (Pipeline's gap) require separate evaluation. This makes the stabilization claim specific to the interventions and settings tested, since a stronger intermediate candidate does not guarantee a stronger deployed harness.

On model–harness coupling: when a regression arises from the policy mishandling a changed tool interface, review and gating still act on the harness side. The policy-only transfer test in part 1 assesses how method rankings change with the policy while harnesses stay fixed.

AuthorsOpenReview ↗

Response to Reviewer RP9Y (part 3/4)

W2 / Q2: Task sets, scoring, and Pass³

Internal inconsistency in the evaluated task suite ... Unclear whether key metrics adhere to the original benchmark protocol.

Thank you for highlighting the task-accounting discrepancies. Tables 8 and 9 and the surrounding text retained entries from an earlier subset-based analysis; we have reconciled them with the final task suite. Section 4.1 and Appendix O now provide the consistent accounting below:

SourceValidationEvaluationMain-result score
Search: BrowseComp-EN32128pass rate
Search: HLE32128pass rate
Office: GDPval3264rubric mean
Office: APEX3264pass rate
General: Claw-Eval3264pass rate
Total160448

The reported macro is 0.4·Search + 0.4·Office + 0.2·General on the 448 evaluation tasks only. The 160 validation tasks select checkpoints and accept PEG candidates and never enter a reported macro; 608 is the benchmark size, 160 + 448. Table 8 now includes APEX, and Claw-Eval is 32 validation and 64 evaluation tasks.

General score (0.7372). This value is not a success count over 32 or 64 tasks.This is an error caused by my carelessness. We replaced it with the value recomputed from task-level outcomes, 47/64 = 0.7344, for the pre-guard Single example in Table 9. With Office 0.6557 and Search 0.3750, the macro is 0.5592.

Pass³. The main tables use one execution per task, not Claw-Eval's native Pass³, and the paper now says so. To check that this does not drive the conclusions, we fixed the Best and Final checkpoints of the methods listed in Table 13 and re-scored General with Pass³ (three independent executions, all must pass). Search and Office are unchanged (Table 13, five seeds).

MethodBestFinalGap
Seed harness0.51770.51770.0000
Single0.45440.29000.1644
RolePrompt0.51700.35480.1622
MultiProp-Judge0.51490.32690.1880
Debate0.56040.29120.2692
Pipeline0.47830.42560.0526
MasterWorker0.47190.39150.0804

Among the evolved methods listed in Table 13, Debate retains the highest Best, while Pipeline retains the highest Final and smallest gap. This is a fixed-checkpoint check, not a re-run of evolution under Pass³, and Concat, Vote and PEG endpoints are not included.

EoI. Under Pass³, EoI needs three independent complete ensembles of three harness rollouts each, so nine General rollouts per task, and every selected output must pass. Our main EoI numbers run each harness once and let a judge without labels or rubrics pick one output, which is one ensemble trial, not Pass³. We now report both (Table 14):

MethodSingle-trial macroPass³ macro
Single + EoI0.47000.4556
RolePrompt + EoI0.53000.5088
MultiProp-Judge + EoI0.54000.5288
Debate + EoI0.63000.6056
Pipeline + EoI0.55000.5363
MasterWorker + EoI0.53000.5200

Drops are 0.010–0.024, Debate stays highest, and no pair of methods that differed swaps order.

Reviewer_kXjbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MEvo studies how multiple LLM evolvers can coordinate to modify an agent harness. It compares Debate, Pipeline, and MasterWorker protocols on Evo-Bench. Debate achieves the highest Best score, while Pipeline has the smallest Best–Final gap. Version-preservation and ensemble mechanisms further improve deployed performance.

Critical strengths

The paper addresses a practical problem in harness evolution: modifications that improve one capability can break others. Reporting both Best and Final scores makes these regressions visible.

Critical weaknesses
  1. The additional value of Pipeline is unclear. Debate already includes critique and revision. Pipeline reduces the Best–Final gap, but both its Best and Final scores remain below the initial harness in the main experiment. With PEG, Debate also achieves a higher final score. The paper should isolate what Pipeline’s verifier adds beyond critique and revision.

  2. A parallel multi-agent search baseline is missing. Systems such as CORAL allow agents to explore and evaluate different candidates while sharing knowledge. MEvo’s Debate selects one candidate for benchmark evaluation after deliberation. The authors should compare with CORAL or a similar parallel-search baseline under matched models, token budgets, and evaluation budgets.

  3. There is no comparison with Meta-Harness. Meta-Harness is directly relevant because it searches over harness code using prior candidates, execution traces, and scores. The author-implemented Single baseline does not establish how MEvo compares with this approach. A comparison under the same models, editable harness, and budget is needed.

  4. The model configuration needs better justification and ablation. The experiments use DeepSeek-V4-Flash as the policy, DeepSeek-V4-Pro as the evolver, and Qwen3.7-Plus for scoring and Debate adjudication. The paper does not show whether the protocol rankings hold with other model combinations or when proposal selection and final scoring use separate models.

Questions for authors
  1. How does MEvo compare with Meta-Harness under the same models, editable harness, and total budget?

  2. How does MEvo compare with CORAL-style parallel exploration under matched token and evaluation budgets?

  3. What does Pipeline’s independent verifier add beyond Debate when both use the same regression checks and promotion rule?

  4. How sensitive are the protocol rankings to the choice of policy, evolver, and judge models?

4 comments
AuthorsOpenReview ↗

Response to Reviewer kXjb (part 1/2)

Thank you for recognizing the practical importance of regressions during harness evolution and the value of reporting both Best and Final. The revision adds parallel-search and Meta-Harness-inspired controls, verifier ablations and a second model configuration (Appendices N, P, Q; changes marked in red). The external-inspired controls reach a Best close to Debate's with gaps of 0.21–0.25; the component controls quantify the verifier's contribution within Pipeline; and the Best/gap pattern persists under the second model configuration. Below we distinguish what each comparison establishes.

W1 / Q3: What does Pipeline's verifier add beyond Debate?

The additional value of Pipeline is unclear. Debate already includes critique and revision. Pipeline reduces the Best–Final gap, but both its Best and Final scores remain below the initial harness in the main experiment. With PEG, Debate also achieves a higher final score. The paper should isolate what Pipeline’s verifier adds beyond critique and revision.

What does Pipeline’s independent verifier add beyond Debate when both use the same regression checks and promotion rule?

The comparison highlights complementary roles: Debate finds stronger candidates (highest Best in 5/5 seeds), while Pipeline retains more of what it finds when every edit is promoted (gap 0.05 vs 0.29, smaller in 5/5 seeds). Pipeline's Best/Final (0.49/0.44) remain below the initial harness (0.53), and with PEG Debate reaches a higher Final (0.61 vs 0.55 after +5 iterations). Pipeline's contribution therefore concerns retention under ungated deployment.

On the verifier itself, the cleanest evidence is the within-Pipeline control (five seeds, Appendix N):

Pipeline variantBestFinalGap
Separate-context verifier0.490.440.05
No verifier0.500.350.15
Same-context self-check0.490.380.11

Best remains similar across these variants, while separate-context verification raises Final by 0.09 relative to no verifier and by 0.06 relative to a same-context self-check. Harmful promotions (macro at least 0.02 below the parent) are 2/15 with the verifier and 6/15 without.

For "the same regression checks and promotion rule," the bare comparison already uses the same import check and always-promote rule for both protocols; neither uses a regression suite. Pipeline's verifier reviews the edit within an iteration. Under PEG, both share the same validation gate, and Debate has the higher Final. The within-Pipeline ablations isolate the effect of verification in that protocol. Appendix E identifies a Debate variant with a dedicated verifier as an untested follow-up to separate review from the remaining differences in instructions, context and number of patch authors.

W2 / Q2: CORAL-style parallel exploration

A parallel multi-agent search baseline is missing. Systems such as CORAL allow agents to explore and evaluate different candidates while sharing knowledge. MEvo’s Debate selects one candidate for benchmark evaluation after deliberation. The authors should compare with CORAL or a similar parallel-search baseline under matched models, token budgets, and evaluation budgets.

How does MEvo compare with CORAL-style parallel exploration under matched token and evaluation budgets?

We implemented a CORAL-style control. Three agents work asynchronously in isolated copies of the harness, share a persistent notes directory (attempts, notes, skills), and submit candidates to the same validation evaluator. The highest-validation candidate of each iteration is promoted. It uses the main configuration's models, the same editable harness and import guard, and the same per-run token and validation-evaluation caps as Debate. The caps cover the whole run and are shared across the three agents. We match caps, not realized consumption.

SeedBestFinalGap
10.550.290.26
20.600.370.23
30.560.300.26
Mean0.570.320.25

Parallel exploration with shared notes reaches strong candidates (Best 0.57; Debate's five-seed mean is 0.59), with a gap of 0.25 under the common always-promote output rule. This control evaluates parallel search and shared knowledge within our deployment setting. The reported gap characterizes the adaptation.

AuthorsOpenReview ↗

Response to Reviewer kXjb (part 2/2)

W3 / Q1: Meta-Harness

There is no comparison with Meta-Harness. Meta-Harness is directly relevant because it searches over harness code using prior candidates, execution traces, and scores. The author-implemented Single baseline does not establish how MEvo compares with this approach. A comparison under the same models, editable harness, and budget is needed.

How does MEvo compare with Meta-Harness under the same models, editable harness, and total budget?

We added a Meta-Harness-inspired control (Appendix Q). One proposer has filesystem access to every prior candidate's source code, scores and execution traces and proposes the next harness. It uses the same models, editable harness, guard and run-level caps as the CORAL-style control.

SeedBestFinalGap
10.520.310.21
20.580.380.20
30.550.330.22
Mean0.550.340.21

Putting everything side by side:

MethodSeedsBestFinalGap
CORAL-style30.570.320.25
Meta-Harness-inspired30.550.340.21
MEvo-Debate50.590.300.29
MEvo-Pipeline50.490.440.05

The search-heavy methods reach a Best of 0.55–0.59 with gaps of 0.21–0.29, while Pipeline combines a lower Best with a smaller gap. These controls show why candidate generation and deployed-version selection deserve separate evaluation.

This control evaluates search with access to prior code, scores and traces under our common always-promote output rule. Native Meta-Harness instead returns a Pareto frontier of an archive in Algorithm 1. A native-system comparison with its original output selection and measured token accounting remains a separate follow-up, as stated in Appendix Q and Related Work.

W4 / Q4: Sensitivity to policy, evolver and judge models

The model configuration needs better justification and ablation. The experiments use DeepSeek-V4-Flash as the policy, DeepSeek-V4-Pro as the evolver, and Qwen3.7-Plus for scoring and Debate adjudication. The paper does not show whether the protocol rankings hold with other model combinations or when proposal selection and final scoring use separate models.

How sensitive are the protocol rankings to the choice of policy, evolver, and judge models?

Config. B re-runs evolution for three seeds with Qwen3.7-Plus as policy, Kimi-K3.1-Pro as evolver, DeepSeek-V4-Pro for Debate adjudication and GLM-5.3 for rubric scoring, so selection and scoring use separate models.

Method (Best / Final / Gap)Config. A, 5 seedsConfig. B, 3 seeds
Seed harness0.53 / 0.53 / 0.000.48 / 0.48 / 0.00
Single0.47 / 0.30 / 0.170.44 / 0.29 / 0.15
Debate0.59 / 0.30 / 0.290.54 / 0.31 / 0.23
Pipeline0.49 / 0.44 / 0.050.46 / 0.41 / 0.05
MasterWorker0.49 / 0.40 / 0.090.45 / 0.37 / 0.08

Debate has the largest Best and Pipeline the smallest gap in all three Config. B seeds. In a separate transfer test, we froze each method's five main-run Final harnesses, changed only the policy and re-scored with the original scorer. The method ranking mostly holds (Spearman 0.87 over nine entries including the seed harness; Appendix P, Table 17).

We thank Reviewer kXjb again for the constructive suggestions on baseline comparisons and model sensitivity, which have helped us clarify the contribution and practical value of our findings.

Dear Reviewer kXjb, thank you again for your constructive review. We wanted to follow up on our responses to see whether they address your questions, in particular the within-Pipeline verifier ablation, the CORAL-style and Meta-Harness-inspired controls, and the second model configuration. If any point remains unclear or you would like additional analysis, we would be glad to try our best to provide it before the discussion period ends.

Dear Reviewer kXjb,

In case it is helpful for the upcoming discussion, here is a short summary of how the revision addresses your four questions (details in our two-part response before and the revised PDF, new material in red):

  • Verifier vs. Debate (W1/Q3): within Pipeline, a separate-context verifier gives Final 0.44, versus 0.38 with a same-context self-check and 0.35 with no verifier; harmful promotions drop from 6/15 to 2/15 (Appendix N).
  • CORAL-style control (W2/Q2): Best 0.57, Final 0.32, gap 0.25 under matched caps, three seeds (Appendix Q).
  • Meta-Harness-inspired control (W3/Q1): Best 0.55, Final 0.34, gap 0.21, same setting (Appendix Q).
  • Other model configurations (W4/Q4): Debate has the largest Best and Pipeline the smallest gap in all three seeds of a second configuration, and the ranking largely persists in a frozen-harness transfer test, Spearman 0.87 (Appendix P).

Thank you again for your time and constructive review.

Best regards, The Authors

Reviewer_ouioOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper's motivation is that the single editor takes ownership of each modification decision, so it may be biased. The authors study the multi-evolver coordination and trying to answer which coordination gives the smallest Best-Final gap. To do this, the authors design MasterWorker, Debate, and Pipeline, different types of coordination, and run 5 seed eval on 608 tasks (160 tasks validation and 448 tasks evaluation). The conclusion is that Debate gives highest Best and Pipeline gives smallest Best-Final Gap.

Critical strengths
  1. The paper raises an interesting question that multiple agent decide together when editing hardness maybe better than a single one. I believe isolating and studying this single aspect in harness evolution is important. Overall, I agree with the motivation.
  2. The paper tests a setting in harness self-modification, Debate, where multiple independent proposer critic each other at the harness-edit step and compare. This setting gives the best on all 5 runs.
Critical weaknesses
  1. Some settings are not new, such as, Pipeline vs AEGIS from HarnessX, and most coordination setting is somehow appeared elsewhere, for example AutoGen, MetaGPT. Therefore, the expectation is that the coordination settings on harness self-modification would tell solid and interesting findings. However, I find that the finding is not solidly proved. For example, Pipeline has the smallest Best-Final Gap is even reversed on 2 runs (Appendix K5 K6). Also, for MasterWorker setting, only 1 runs shows it recover from its own bad iteration, but the paper claims so. The experiments are not enough to show solid claims or even "how frequent will this claim holds". The analysis between runs is more like an afterward explanation and much of the effect was shown in whether an destructive rewrite happens rather than the effect of different multi-evolver setting.
Questions for authors
  1. Can you clarify why HarnessX's multi agent vs single agent ablation shows no effect?
  2. Can you run Pipeline without the verifier, and MasterWorker with a fresh Master each iteration?
  3. Can you give concrete numbers on how often a bad edit go through with/without an independent reviewer? How often Pipeline's verifier reject edits and how much the rejected edits would hurt?
  4. Can you explain why a different set of model (policy, evolver, and judge models) is not tested? The scope only count as case study if only 1 set is used, and I believe another set is very important to be include in the current version.
5 comments
AuthorsOpenReview ↗

Response to Reviewer ouio (part 1/2)

Thank you for recognizing the importance of isolating coordination at the harness-edit step and for noting Debate's highest Best in all five runs. To address your questions about how often the findings hold and what drives them, we provide a per-seed breakdown, component controls, candidate-level statistics and a second model configuration. The detailed additions appear in Appendices M–R; the main text refines the claims, summarizes key control results and points to these analyses. Changes are marked in red in the revised PDF.

W1.1: Novelty relative to HarnessX/AEGIS, AutoGen and MetaGPT

Some settings are not new, such as Pipeline vs AEGIS from HarnessX, and most coordination setting is somehow appeared elsewhere, for example AutoGen, MetaGPT.

Thank you for highlighting these connections. MEvo builds on established coordination patterns to study how proposal, review and write authority affect executable harness evolution. AutoGen and MetaGPT coordinate agents around a fixed system at inference time, and AEGIS stages evidence processing with a single patch author. MEvo examines what happens when the edited object is executable harness code and every edit is promoted without human review. Pipeline has one patch author and a separate-context verifier, Debate has three independent patch authors and a judge, and MasterWorker has a persistent Master that reconciles worker proposals across iterations. Our contribution is a controlled comparison of these decision structures using the Best–Final gap, supported by the controls in Q2–Q4 below (structural comparison in Appendix D). We have removed the "first" wording from the abstract and contributions to make this scope explicit.

W1.2: How often do the findings hold across seeds?

Pipeline has the smallest Best-Final Gap is even reversed on 2 runs (Appendix K5 K6). Also, for MasterWorker setting, only 1 runs shows it recover from its own bad iteration, but the paper claims so.

To make the frequency of the findings explicit, we report the per-seed Best–Final gaps below (from the two-decimal endpoints in Tables 10–11):

SeedSingleDebatePipelineMasterWorker
10.180.400.000.00
20.000.200.080.15
30.270.310.050.09
40.180.260.070.10
50.240.260.060.09
Mean0.170.290.050.09

Debate has the highest Best of all eight bare methods in 5/5 seeds. Pipeline's gap is smaller than Debate's in 5/5 seeds. Against all bare methods, it is strictly smallest in seeds 3–5, tied with MasterWorker at 0.00 in seed 1, and larger than Single's in seed 2. In seed 2, Single's zero gap reflects its best iterate also being its last; its Final (0.43) remains below that seed's initial harness (0.57). Appendix M now reports these observed frequencies.

The trajectory 0.4624 → 0.4557 → 0.5012 illustrates recovery in seed 1, which the introduction and Appendix K.4 now label explicitly as a case study. The fresh-Master control in Q2 separately evaluates the average contribution of Master persistence.

W1.3: Destructive rewrites versus an effect of coordination

The analysis between runs is more like an afterward explanation and much of the effect was shown in whether an destructive rewrite happens...

To test whether coordination affects the passage of destructive edits, we added controls that change one component at a time and count harmful promotions directly (Q2, Q3), complementing the trajectory case studies. Removing the verifier changes Best only slightly (0.50 vs 0.49), lowers Final from 0.44 to 0.35, and raises harmful promotions from 2/15 to 6/15. These endpoint and candidate-level results support a contribution of separate-context review to retention within Pipeline. We have removed the incentive-based wording and confined the interpretation to these observed outcomes.

AuthorsOpenReview ↗

Response to Reviewer ouio (part 2/2)

Q1: Why does HarnessX's multi-agent vs single-agent ablation show no effect?

Can you clarify why HarnessX's multi agent vs single agent ablation shows no effect?

The experimental settings differ in how candidate versions are retained. In HarnessX (Section 6.4, Table 6), AEGIS and the single-agent evolver reach 87.4% and 86.4% accuracy at 107.8M and 123.1M tokens, and both keep variant isolation, so a bad variant is discarded before it can become the deployed parent. Our main comparison uses an always-promote rule: every edit is promoted, and the last iterate is deployed. With a validation gate (PEG), protocol differences remain (Final after +5 iterations: Single 0.48, Pipeline 0.55, Debate 0.61). Output preservation therefore helps contextualize the comparison.

Q2: Pipeline without the verifier; MasterWorker with a fresh Master

Can you run Pipeline without the verifier, and MasterWorker with a fresh Master each iteration?

We completed both requested controls across five seeds (Appendix N, Table 12), together with a same-context self-check variant.

SettingBestFinalGap
Pipeline (separate-context verifier)0.490.440.05
Pipeline, no verifier0.500.350.15
Pipeline, same-context self-check0.490.380.11
MasterWorker (persistent Master)0.490.400.09
MasterWorker, fresh Master each iteration0.490.350.14

Best remains similar across these controls. The separate-context verifier raises Final by 0.09 relative to removing it, and the persistent Master raises Final by 0.05 relative to a fresh Master. A self-check inside the author's own context recovers part of the verifier's benefit. All variants share the same nominal token and step caps.

Q3: How often do bad edits get through, and how much would rejected edits hurt?

Can you give concrete numbers on how often a bad edit go through with/without an independent reviewer? How often Pipeline's verifier reject edits and how much the rejected edits would hurt?

Appendix N now reports candidate-level counts. A promoted edit is harmful if its evaluation macro is at least 0.02 below its parent. Each condition covers 15 iterations (5 seeds × 3).

MeasureResult
Harmful promotions, no verifier6/15
Harmful promotions, with verifier2/15
First patch returned for revision7/15
Returned patches whose replay lowers macro by ≥0.025/7
Mean macro change over the 7 replays−0.08
Worst replay−0.29

Replays evaluate each originally returned patch against its own parent. 7/15 counts revision requests, not final rejections. The verifier lowers harmful promotions from 6/15 to 2/15, and most patches it sends back would have hurt, but two harmful edits still passed.

Q4: Why not another policy, evolver and judge set?

Can you explain why a different set of model (policy, evolver, and judge models) is not tested? The scope only count as case study if only 1 set is used, and I believe another set is very important to be include in the current version.

We added a second model configuration to test sensitivity to the policy, evolver and judge choices directly. Config. B re-runs evolution for three seeds with Qwen3.7-Plus as policy, Kimi-K3.1-Pro as evolver, DeepSeek-V4-Pro for adjudication and GLM-5.3 for rubric scoring, so adjudicator and scorer are now different models.

Method (Best / Final / Gap)Config. A, 5 seedsConfig. B, 3 seeds
Seed harness0.53 / 0.53 / 0.000.48 / 0.48 / 0.00
Single0.47 / 0.30 / 0.170.44 / 0.29 / 0.15
Debate0.59 / 0.30 / 0.290.54 / 0.31 / 0.23
Pipeline0.49 / 0.44 / 0.050.46 / 0.41 / 0.05
MasterWorker0.49 / 0.40 / 0.090.45 / 0.37 / 0.08

In all three Config. B seeds Debate has the largest Best and Pipeline the smallest gap. We also froze each method's five main-run Final harnesses, swapped only the policy to Qwen3.7-Plus and re-scored with the original scorer (Appendix P, Table 17). The method ranking largely persists (Spearman 0.87 over nine entries including the seed harness; Pipeline 0.41, Debate 0.29, Single 0.28). Config. B changes all roles at once and uses three seeds, providing evidence across the tested configurations .

We thank Reviewer ouio again for the thoughtful questions and constructive feedback, which have helped us strengthen the empirical analysis of coordination and retention.

Dear Reviewer ouio, thank you again for your thoughtful review. We posted our responses a few days ago and wanted to check whether they address your concerns, in particular the per-seed breakdown of the Best–Final gaps, the verifier and fresh-Master controls with the candidate-level counts, and the second model configuration. If anything is still unclear or you would like us to run or clarify something further, please let us know and we will gladly follow up before the discussion period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.