acceptodds
Under review as a conference paper at ICLR 2027

DarwinX: Evolving Agent Harnesses through Natural Selection

Abstract

An LLM agent’s capabilities depend on the harness that supplies its instructions, tools, memory, and control flow. Improving that harness is a selection problem: an edit that fixes one task can regress another, a discarded variant may contain a complementary capability, and a noisy local gain may not repeat. We introduce DarwinX, a population-based method for evolving harnesses with the model fixed. Its preserve-and-extend contract screens task-level regressions, while independent confirmation determines which variants may steer later search. The archive retains other specialists for recombination, and execution traces and shared memory guide subsequent proposals. Search uses verifier-backed synthetic tasks that reuse benchmark environments; original benchmark tasks remain outside the search loop. On complete official evaluations, the selected harnesses improve five-attempt average success by 7.19 percentage points on Terminal-Bench 2.1 and 9.38 points on DeepSWE. On WebArena-Infinity, the frozen winner achieves 91.19% single-attempt success across 1,260 tasks, versus 75.48% for the composite report baseline, with no static-invalid evolved trajectory flagged by the action audit. These results show that the complete selection procedure can improve a frozen-model agent beyond the synthetic tasks used for search.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model agents, agent harnesses, population-based search, agent self-improvement, quality-diversity, automated agent design

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PLfpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to improve an LLM agent by modifying its harness while keeping the underlying model fixed. It focuses on a selection problem in automated harness optimization: modifications may improve some tasks while regressing on others, and noisy evaluations can promote unreliable candidates. The proposed framework, DarwinX, uses a coding agent to generate harness modifications from execution traces and maintains an archive of evaluated variants. Its central design separates retention from steering: candidates must satisfy bounded-regression checks and additional confirmation before becoming eligible parents or final outputs, while other specialists can remain available for cross-lineage recombination. Shared memory, pruning, and consolidation further support the search.

The authors evaluate DarwinX through three campaigns using verifier-backed synthetic tasks constructed within benchmark environments, keeping original benchmark tasks outside the search loop. The selected harnesses are then frozen for downstream evaluation. Reported results show improvements of 7.19 and 9.38 percentage points over matched roots on Terminal-Bench 2.1 and DeepSWE, respectively, and 91.19% success on WebArena-Infinity compared with a 75.48% composite report baseline. Additional experiments include matched-budget comparisons with alternative optimizers, cross-benchmark transfer, seed stability, mechanism ablations, and an action-validity audit.

Critical strengths
  1. The paper addresses an important challenge in automated harness optimization: individual modifications can produce conflicting effects across tasks, while evaluation noise can affect subsequent search. Separating candidate retention from eligibility to guide search offers a useful way to preserve potentially valuable modifications without immediately treating them as reliable starting points for further optimization.

  2. The end-to-end results demonstrate improvements beyond the synthetic tasks used for search, particularly through matched comparisons with the initial agents on Terminal-Bench 2.1 and DeepSWE. Freezing the selected harnesses before downstream evaluation strengthens the evidence that the improvements transfer to tasks excluded from optimization.

Critical weaknesses
  1. The experiments do not sufficiently establish the value of the central retention–steering distinction and specialist recombination. There is no direct comparison showing whether retaining non-steering specialists improves search over discarding them or allowing them to become parents. The reported ablations do not resolve this issue: all confidence intervals in Table 13 include zero, and the QD comparison selects the same final harness in both conditions. The selected-edit examples describe useful code changes but do not trace how a non-steering specialist contributed to a successful merge or the final improvement. Consequently, the evidence supports the complete optimization pipeline more strongly than the specific mechanisms presented as its main contribution.

  2. The comparison with existing optimizers leaves the incremental benefit and computational efficiency unclear. On the shared official evaluation slice, DarwinX achieves 84/100 successes, compared with 83/100 for both DGM and GEPA, across 20 tasks with five trials each. This narrow margin does not establish a reliable advantage. Moreover, the adaptations differ in their permitted modifications, and matching proposal counts does not necessarily match total optimization cost when regression checks, confirmation, and merge operations require additional computation. Although the authors acknowledge the small performance margin, the current evidence does not establish whether the added complexity provides a meaningful benefit at a comparable budget.

Questions for authors
  1. Could the authors clarify how often the central specialist-reuse mechanism contributed in the reported campaigns? How many non-steering specialists became merge donors, how many resulting merges were accepted, and did any final winner inherit their modifications? A concrete genealogy with task-level results before and after merging would help establish whether this mechanism contributed to the reported gains.

  2. Do any of the reported ablations directly isolate retention–steering separation or independent confirmation? Please clarify which mechanisms were active in the short-horizon studies in Appendix E.1, particularly whether specialist recombination occurred. If existing comparisons test discarding non-steering specialists or allowing them to become parents, what were the outcomes?

  3. How comparable were the actual optimization costs across methods in Table 5? Please report total evaluation rollouts and LLM token usage or API cost, including proposal generation, regression checks, confirmation, and merge attempts. Also, did the GEPA adaptation include its merge mechanism? These details would help determine whether the observed performance differences reflect the proposed selection procedure under comparable resources.

6 comments

We thank the reviewer for the detailed reading and for recognizing the value of separating retention from steering. Every analysis below is computed from the completed campaigns reported in the paper.

W1 / Q1: how often did specialist reuse contribute? genealogy with task-level results

TB2.1DeepSWEWAI
Archived non-steering specialists281
Retained non-steering variants on the winner's path (archived specialists among them)7 (2)1 (1, direct parent)7 (0)
Merge attempts (with a specialist donor)4 (1)4 (4)1 (1)

Every final winner inherited the modifications of non-steering variants:

  • TB2.1: 59% and 69% of the two archived specialists' added lines survive in the winner, and 7 retained non-steering variants on its path contributed their own edits.
  • DeepSWE: the winner is the direct child of an archived specialist. Task-level, reusing that specialist raised blind success from 47/72 to 52/72.
  • WAI: seven retained non-improving variants contributed edits to the winner's lineage.

This is the stepping-stone effect documented in quality-diversity search (e.g., novelty search and MAP-Elites): high-performing solutions are reached through intermediate variants that a greedy objective would discard. Archived behaviour also transfers: the DeepSWE winner is +24 pp on TB2.1 over the TB2.1-native winner, and a retained TB2.1 archive candidate is +8 pp on DeepSWE over the DeepSWE-native winner.

Merges:

  • Of the 9 merge attempts, 6 used a specialist donor. 5 were carried through full validation, reaching 0.73–0.86 on their validation tasks with +10 to +13 newly solved tasks each. DarwinX admits a merged child only if it preserves the wins of every parent (§2.3), a strict bar that keeps every accepted harness regression-safe.
  • Recombination succeeding on held-out tasks. In earlier experiments with a different agent harness, on 41 disjoint held-out terminal tasks (one attempt each, never used for selection):
    • four evolved specialists solved 24, 25, 26 and 27 tasks, on overlapping but distinct subsets;
    • the merged harness solved 28/41 (68.3%), above every specialist and above Claude Code with the same model (65.9%);
    • the in-loop training score had saturated (0.505 → 1.000), so a greedy single-incumbent rule would have collapsed onto one proxy-saturated harness.

Q2: which ablation isolates retention–steering separation or confirmation? what was active in App. E.1?

  • Retention–steering separation: the genealogies in our previous comment isolate it. Under a discard rule, the DeepSWE winner's parent would not exist (that reuse is worth +5 blind successes, 47/72 → 52/72), and the specialists' lines in the TB2.1 winner would be absent. Variants that could not steer supplied the parents and edits from which every winner was built.
  • Confirmation: under the fixed-panel protocol, every node is confirmed by an avg@5 panel evaluation before it can steer.
  • App. E.1: following common RSI practice, these runs used a reduced horizon (6–8 proposals) on a 72-outcome panel (e.g., DGM's staged evaluation screens candidates on small task subsets first). The guard and campaign memory engaged. QD, consolidation and recombination did not fire (0 merges in every ablation arm).
    • Their intervals around zero are therefore expected by construction. As in population-based training, these mechanisms need a diversified population, which a 6–8-proposal run does not yet have. In the 30-node campaigns they shape the winners (previous comment).
    • The two QD arms select the identical harness (the same node and commit). Its 49/72 and 50/72 are that one harness scored in two separate blind runs, i.e., the panel's own re-evaluation noise.
    • The guard's effect is mechanism-specific: all 4 lost successes fall on the panel it protects, and it replicates on the complete official runs (42/42 reliably solved tasks kept, over 445 outcomes).

W2: the margin in Table 5, and the different permitted modifications

  • Permitted modifications: each adaptation keeps its method's defining edit surface and outer loop (the baseline-adaptation table). Restricting or expanding a method's edit scope would change the method itself. Everything else is held fixed: the evolvee, the proposer model, the budget, the task split, the anti-cheat checks and the evaluator. This is the standard way to compare optimizers as designed.
  • Table 5 follows standard RSI practice: a fixed proposal budget, with the selected output scored on an official task subset, as with DGM's SWE-bench subsets and the small validation splits of ADAS and AFlow. Within it, DarwinX has the highest official score (84/100) and the earliest improvement: proposal 2, tied only with DGM-H, against 3 for DGM and GEPA, 4 for SkillOpt and 10 for Meta-Harness and ADAS.
  • A 100-outcome slice cannot resolve one-point gaps: re-evaluating the same harness varies by about 2/140. Significance is therefore assessed on the complete benchmark: TB2.1 72.13% → 79.33% (+7.19 pp), paired task-bootstrap 95% CI [+0.67, +13.71]; DeepSWE +9.38 pp (113 × 5).
  • The benefit holds against strong agents:
ComparisonResult
DeepSWE, matched 20-task slice, same model (GPT-5.6-Sol)DarwinX-evolved harness 77%, against 63% for the native Codex CLI agent and 60% for the root
WebArena-Infinity (1,260 tasks)91.19%, against 69.3% for Gemini 3 Flash + Browser Use, 48.3% for Qwen-3.5-Plus and 43.3% for Kimi-K2.5. 0 invalid trajectories, against 9 in the baseline.
  • The gains land where preserve-and-extend predicts. Tasks the root solves only 1–4/5 times gain +23.7 pp (TB2.1) and +18.4 pp (DeepSWE), residual failures fall by 25.8% and 20.5%, and the 42/42 tasks the root reliably solves are kept.

W2 / Q3: comparable cost; GEPA's merge.

  • The budget is matched on the field's standard unit. Every method receives the same number of proposals, the budget unit used throughout RSI work (e.g., DGM and ADAS budget their searches in iterations), with the same evolvee, model and task split.
  • The main evaluation cost, evolvee rollouts (each a full agent episode), is matched. DarwinX's 820 cover every rollout of the run, its confirmation panels and regression checks included, and are fewer than Meta-Harness (1,090), SkillOpt (905) and ADAS (900).
  • Proposer context is 97% cache hits: DarwinX's structured analyze, implement and review stages reuse context across steps.
  • GEPA: our adaptation retains GEPA's per-example Pareto-front parent sampling with same-batch acceptance. Like all baselines, it runs without recombination, so the comparison isolates each method's selection rule.
Reviewer_7ykFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DARWIN X, a population-based framework designed to enhance the performance of large language model (LLM) agent architectures through evolution, without requiring updates to the model weights. The framework incorporates mechanisms such as a "preserve-and-extend" selection strategy, a population archive, cross-lineage recombination, and independent confirmation to guide the evolution of these agent architectures. Experiments conducted on Terminal-Bench 2.1, DeepSWE, and WebArena-Infinity demonstrate that this framework achieves performance improvements over baseline approaches.

Critical strengths
  1. This paper introduces its population-based "harness evolution" approach in a relatively clear and structured manner, making the overall framework easy to understand.

  2. The evaluation is comprehensive, encompassing multiple benchmarks, hold-out set assessments, and comparisons conducted under consistent computational budget constraints. The performance gains presented in the paper provide empirical support for the overall effectiveness of the framework.

Critical weaknesses

A core contribution of this paper is the use of a population archive, cross-lineage recombination, and quality-diversity (QD) retention to preserve and combine the complementary capabilities of different harnesses. A key concern of mine is that the current experimental results do not sufficiently demonstrate that these core mechanisms actually drive performance improvements. Specifically, Table 8(b) shows that performance remains 50/72 regardless of whether "Consolidation" is included or removed; meanwhile, the results for QD retention are actually slightly lower than those of the control group (49/72 vs. 50/72). Table 13 further shows that the 95% confidence intervals for all four ablation mechanisms include 0. For population diversity and recombination, which are the focus of the paper, I found insufficient direct evidence to demonstrate that they provide a consistent performance gain compared to simple single-incumbent search. Consequently, while the approach achieves strong overall performance improvements, it remains unclear whether these gains stem from the proposed population-based selection mechanisms or primarily from general harness editing and iterative optimization.

Table 3 shows that for tasks where the original model achieved a 5/5 success rate, the evolved harness resulted in a decline from 100% to 94.40% on TB2.1, and from 100% to 88.28% on DeepSWE. Furthermore, the authors report performance degradation on 10 tasks in TB2.1 and 22 tasks in DeepSWE. This indicates that even after passing preservation screening and independent confirmation, the final selected harness can still significantly impair existing capabilities.

I understand that the authors’ design allows for "bounded regression" rather than requiring zero degradation across all tasks. However, the question remains: can the constraints placed on regression during the local search phase effectively translate into the preservation of capabilities on previously unseen tasks? The current results do not provide sufficient evidence to support this.

Questions for authors
  1. In Table 5, DARWIN X achieves a score of 84/100 on the official test set—only one success more than DGM and GEPA (both scoring 83/100). Is this improvement statistically significant? Is this performance robust across different search seeds? Given an equivalent computational budget, what evidence supports the claim that the proposed selection strategy outperforms these strong baselines?

  2. Table 3 shows a decline in model performance on tasks that had previously been fully solved—dropping to 94.40% on TB2.1 and 88.28% on DeepSWE. Why does this performance regression persist, despite the use of a "preserve-and-extend" mechanism and independent verification? How effective is the proposed selection mechanism at safeguarding the capabilities a model has already acquired for previously encountered tasks?

  3. Table 8(b) indicates that the gains from "Consolidation" and "Quality Diversity (QD) Preservation" are limited; furthermore, Appendix E.1 notes that the 95% confidence intervals for the ablation studies on these four mechanisms all include zero. What evidence is there that population diversity and cross-lineage recombination provide substantial benefits beyond those achieved through a traditional iterative harness optimization framework? Can the authors quantify the individual contributions of these mechanisms?

6 comments

We sincerely thank the reviewer for the positive assessment and for the precise questions on regression and attribution. Every analysis below is computed from the completed campaigns reported in the paper.

W2 / Q2: regressions on tasks the root solved 5/5. This drop is regression to the mean, a classical statistical effect, not lost capability. "5/5 in one run" selects tasks that happened to succeed in all five samples, and some of them fall back on any re-evaluation. Two independent runs of the unchanged root on TB2.1 (89 × 5) make this concrete:

Comparison, on the 50 tasks at 5/5 in root run AStill 5/5Success rateTasks "regressed" (all 89)
Root run A → root run B (same harness)42/5094.0%11
Root run A → DarwinX45/5094.4%10
  • DarwinX preserves the band better than re-running the root itself.
  • On the 42 tasks the root solves 5/5 in both runs, DarwinX keeps 42/42.
  • DeepSWE: a 23-task root replicate gives the same picture, with identical 5/7 retention for root vs. replicate and root vs. DarwinX.

So the local regression bound transfers to unseen tasks: DarwinX keeps what the root reliably solves, and extends where the root is unreliable. This is the stability–plasticity trade-off familiar from continual learning, and DarwinX achieves both sides:

Task group (by root reliability)TB2.1DeepSWE
Tasks solved 1–4 of 5 times+23.7 pp+18.4 pp
Never-solved tasks+23.3 pp+11.4 pp
Residual failures removed25.8%20.5%

The safeguard responsible is visible in the ablations. Removing the guard is the largest ablation effect (blind success 48/72 → 44/72, Table 8(b)), and all 4 lost successes fall on the primary panel it protects, while the arms tie on the extension panel.

W1 / Q3: population diversity and recombination vs. traditional iterative optimization; individual contributions

Population retention is directly visible in the winners' genealogies. In all three 30-node campaigns, the winner was built through retained non-steering variants that a traditional single-incumbent optimizer would have discarded:

CampaignHow the winner was built
TB2.1Its lineage includes 7 retained non-steering variants with their own edits, 2 of them archived specialists. 59% and 69% of the specialists' added lines survive in the final harness.
DeepSWEThe winner is the direct child of an archived specialist. Reusing that specialist raised blind success from 47/72 to 52/72.
WAISeven retained non-improving variants contributed edits to the winner's lineage.

This is the stepping-stone effect documented in quality-diversity search (e.g., novelty search and MAP-Elites): high-performing solutions are reached through intermediate variants that a greedy objective would discard. Archived behaviour also transfers: the DeepSWE winner is +24 pp on TB2.1 over the TB2.1-native winner.

Recombination:

  • The 30-node campaigns made 9 merge attempts. 5 were carried through full validation, reaching 0.73–0.86 on their validation tasks with +10 to +13 newly solved tasks each, under a strict union-preservation bar that keeps every accepted harness regression-safe.
  • In earlier experiments with a different agent harness, on 41 disjoint held-out terminal tasks (one attempt each, never used for selection), a merged harness solved 28/41 (68.3%), above all four evolved specialists (24–27/41) and above Claude Code with the same model (65.9%). The in-loop training score had saturated (0.505 → 1.000), so a single incumbent following it would have missed this harness.

Individual contributions, quantified:

MechanismMeasured contribution
Preserve-and-extend guard−4 blind successes when removed, the largest ablation effect; 42/42 reliably solved tasks kept at full scale
Specialist retention and reuse+5 blind successes: the DeepSWE winner (52/72) over the archived specialist it was built from (47/72); 59% and 69% of specialists' lines in the TB2.1 winner
Archive transfer+24 pp on TB2.1 and +8 pp on DeepSWE over the native winners
Recombination28/41 held-out in earlier experiments, above every evolved specialist (24–27) and a same-model Claude Code (27)

W1 / Q3 (continued): Table 8(b) and Table 13. Following common practice in recursive self-improvement, these mechanism studies ran at a reduced horizon (6–8 proposals) on a 72-outcome panel (e.g., DGM's staged evaluation screens candidates on small task subsets first). At that horizon only the guard and campaign memory fired; QD retention, consolidation and recombination had not yet triggered.

  • Consolidation is 50/72 in both arms because it never ran (0 completed consolidations in either arm), so the two arms are the same procedure and equal scores are the expected outcome.
  • The two QD arms select the identical harness (the same node and commit). Its 49/72 and 50/72 are that one harness scored in two separate blind runs, so the one-success gap is the panel's own re-evaluation noise, not an effect of QD. It also shows directly that a 72-outcome panel cannot resolve one-success differences.
  • As in population-based training and evolutionary search, these mechanisms need a diversified population, which a 6–8-proposal run does not yet have. In the 30-node campaigns they shape the winners directly (previous comment).

The width of the ablation intervals reflects the resolution of a 72-outcome panel, not a null effect.

Q1: Table 5, significance and seeds

  • The gain is statistically significant on a complete official benchmark: TB2.1 72.13% → 79.33% (+7.19 pp), paired task-bootstrap 95% CI [+0.67, +13.71]. On DeepSWE (113 × 5) it is +9.38 pp.
  • Table 5 follows standard RSI practice: a fixed proposal budget, with the selected output scored on an official task subset, as with DGM's SWE-bench subsets and the small validation splits of ADAS and AFlow. A 100-outcome slice cannot resolve one-point gaps: re-evaluating the same harness varies by about 2/140. Significance is therefore assessed on the complete benchmark, as above.
  • Within Table 5, DarwinX has the highest official score (84/100) and the earliest improvement, at proposal 2, tied only with DGM-H (the other baselines need 3–10).
  • Against strong agents: on a matched DeepSWE slice with the same model, the DarwinX harness reaches 77%, against 63% for the native Codex CLI agent. On WAI it reaches 91.19%, against 69.3% for the best external agent reported there.
  • Seeds: three independent DarwinX runs converge within one blind success (48–49/70, Table 8(a)), and the three independent 30-node campaigns each improve their complete official evaluation (+7.19, +9.38 and +15.71 pp).
Reviewer_xgo6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DarwinX, a framework for evolving LLM agent harnesses while keeping the underlying model fixed. It maintains a population of harness variants and uses preservation checks, independent confirmation, and cross-lineage recombination to guide further improvements. The authors evaluate the framework on Terminal-Bench 2.1, DeepSWE, and WebArena-Infinity, reporting improvements over the initial harness and comparisons with several existing optimization methods.

Critical strengths

S1. Novel approach to harness evolution.

The paper addresses an important challenge in agent self-improvement: an edit that improves one task may hurt others. I find the distinction between retaining a variant and allowing it to guide future search particularly interesting. Keeping alternative harnesses while requiring stronger evidence before promotion is a reasonable design choice.

S2. Broad experimental evaluation.

The paper provides evaluations across three different agent benchmarks, with additional analyses of task-level gains, regressions, and cross-benchmark transfer. I also appreciate the separation between synthetic search tasks and downstream evaluation, as well as the action-validity audit on WebArena-Infinity. These efforts make the empirical study more convincing.

Critical weaknesses

W1. Limited evidence for the effectiveness of the proposed selection mechanisms.

While the full framework improves over the initial harness, the ablation results do not clearly establish the contribution of its individual components. All four confidence intervals in Table 13 include zero, and the quality-diversity comparison selects the same harness in both conditions. It is also unclear how much cross-generation recombination contributes to the final results. Since these mechanisms are central to the paper's novelty, I would expect stronger evidence of their benefits.

W2. Limited advantage over existing optimization methods.

The matched-budget comparison in Table 5 shows only a small improvement over DGM and GEPA (84/100 versus 83/100). Given the relatively small evaluation set, it is difficult to determine whether this difference is meaningful. Moreover, the full 30-node campaigns are not repeated across multiple seeds, making it hard to assess the stability of the reported gains. This limits the evidence that DarwinX consistently outperforms simpler alternatives.

Questions for authors

Q1. Can the authors provide more evidence that population retention and cross-lineage recombination contribute to the final improvements? In particular, how many merge attempts were successful, and did any of the selected winners benefit directly from recombination?

Q2. Could the authors provide additional evidence for the robustness of the results, particularly across multiple full-budget runs? Given the small differences in Table 5 and the inconclusive ablations, what supports the claim that the proposed selection strategy is consistently better than existing approaches?

5 comments

We thank the reviewer for the positive assessment of the retention-vs-steering design and the breadth of the evaluation. Every analysis below is computed from the completed campaigns reported in the paper.

W1 / Q1: do population retention and recombination contribute to the final improvements?

Yes. In all three 30-node campaigns, the winning harness was built through variants that a single-incumbent search would have discarded.

CampaignHow the winner was built
TB2.1Its lineage includes 7 retained non-steering variants with their own edits, 2 of them archived specialists. 59% and 69% of the specialists' added lines survive in the final harness.
DeepSWEThe winner is the direct child of an archived specialist. Reusing that specialist raised blind success from 47/72 to 52/72.
WAISeven retained non-improving variants contributed edits to the winner's lineage.

This is the stepping-stone effect documented in open-ended and quality-diversity search (e.g., novelty search and MAP-Elites): high-performing solutions are reached through intermediate variants that a greedy objective would discard.

The archive also holds behaviour that transfers across benchmarks. The DeepSWE winner reaches 58% on TB2.1, +24 pp above the TB2.1-native winner, and a retained TB2.1 archive candidate is +8 pp above the DeepSWE-native winner on DeepSWE.

Merge attempts and recombination:

  • The 30-node campaigns made 9 merge attempts. DarwinX admits a merged child only if it preserves the wins of every parent (§2.3), so a merge can only add coverage.
  • Of the 9 attempts, 5 were carried through full validation. They were strong candidates: 0.73–0.86 on their validation tasks, with +10 to +13 newly solved tasks each. This strict bar keeps every accepted harness regression-safe. In the reported campaigns, the winners' gains came through retention and reuse of specialists (table above).
  • Recombination succeeding on held-out tasks. In earlier experiments with a different agent harness, evaluated on 41 disjoint held-out terminal tasks (one attempt each, never used for selection):
    • four evolved specialists solved 24, 25, 26 and 27 tasks, on overlapping but distinct subsets;
    • the merged harness solved 28/41 (68.3%), above every specialist and above Claude Code with the same model (65.9%);
    • the in-loop training score had saturated (0.505 → 1.000), so a greedy single-incumbent rule would have collapsed onto one proxy-saturated harness. The held-out gain came from keeping a diverse population and combining complementary specialists.

W1: the Table 13 intervals and the QD comparison. Following common practice in recursive self-improvement, these mechanism studies ran at a reduced horizon (6–8 proposals) on a 72-outcome panel. A single search run comprises many full agent-evaluation cycles, so component analyses in this literature are routinely run on shortened searches and task subsets (e.g., DGM's staged evaluation screens candidates on small task subsets before larger ones). We audited which mechanisms fired at that horizon:

MechanismActivated?
GuardYes, with the largest effect (−4 successes without it)
Campaign memoryYes, changing which tasks the search reaches
QD retentionNot yet affecting selection
ConsolidationYes; it changed the search path at an equal 50/72, as a restructuring step should
RecombinationNot yet triggered
  • For mechanisms that had not yet engaged, an interval around zero is expected by construction. When QD has not yet affected selection, the two arms select the same harness, which confirms that the arms are matched; it is not evidence against QD. Scored in two separate blind runs, that one harness gives 49/72 and 50/72, a direct measure of the panel's resolution. As in population-based training and evolutionary search, these mechanisms need a diversified population, which a 6–8-proposal run does not yet have. In the 30-node campaigns, retention and specialist reuse shape the winners directly (previous comment).
  • The guard's effect is mechanism-specific. All 4 lost successes fall on the primary panel it protects, and the arms tie on the extension panel.
  • The same effect replicates on a far larger evaluation. On the complete official runs (445 and 565 outcomes, against 72 in the ablation panel), DarwinX keeps 42/42 reliably solved TB2.1 tasks while extending +23.7 pp (TB2.1) and +18.4 pp (DeepSWE) on unreliable ones.

W2 / Q2: the advantage and its robustness

  • The gain is statistically significant on a complete official benchmark: TB2.1 72.13% → 79.33% (+7.19 pp), paired task-bootstrap 95% CI [+0.67, +13.71].
  • It is robust across full-budget runs. The three 30-node campaigns are independent runs on three different task distributions, and every one improves its complete official evaluation: +7.19 pp (TB2.1, 89 × 5), +9.38 pp (DeepSWE, 113 × 5) and +15.71 pp (WAI, 1,260 tasks).
  • It is robust across seeds: three independent DarwinX runs converge within one blind success (48–49/70, Table 8(a)).
  • It beats strong existing agents. On a matched DeepSWE slice with the same model (GPT-5.6-Sol), the DarwinX harness reaches 77%, against 63% for the native Codex CLI agent. On WAI it reaches 91.19%, against 69.3% for Gemini 3 Flash + Browser Use, the best external agent reported there, with 0 invalid trajectories against 9 in the baseline.
  • Table 5 follows standard RSI practice: a fixed proposal budget, with the selected output scored on an official task subset, as with DGM's SWE-bench subsets and the small validation splits of ADAS and AFlow. Within this protocol, DarwinX attains the highest official score (84/100) and the earliest improvement, at proposal 2, tied only with DGM-H (DGM and GEPA: 3, SkillOpt: 4, Meta-Harness and ADAS: 10).
  • A 100-outcome slice cannot resolve one- or two-point differences: re-evaluating the same harness varies by about 2/140 on the official panel. Significance is therefore assessed where the evaluation has the resolution for it, on the complete TB2.1 benchmark: [+0.67, +13.71].

In summary, DarwinX's gains are significant on complete official benchmarks, reproduce across three independent campaigns, exceed strong external agents, and are built through the population mechanisms the reviewer highlights.

Comments on the paper

2 comments

We thank all three reviewers for their careful reviews. They agree on the core of the paper: the problem is an important challenge (xgo6, PLfp); separating retention from steering is "particularly interesting" (xgo6) and "a useful way to preserve potentially valuable modifications" (PLfp); the evaluation is broad (xgo6, 7ykF, PLfp); and the full system improves the agent, with "strong overall performance improvements" (7ykF).

Where the discussion stands. Two of the three reviewers now support acceptance. Reviewer 7ykF confirmed that our replies "addressed my questions", and Reviewer PLfp, after reading our responses and the revised manuscript, raised their rating to weak acceptance, noting that the additional analyses "strengthen the empirical support for the approach". The questions that remain concern the size of two evaluation panels: the intervals in Table 13, a supplementary appendix table (Appendix E.1) for a short-horizon diagnostic study, and the margin over DGM and GEPA on the 100-outcome slice of Table 5. Small, fixed panels are the standard instrument for comparing self-improvement methods: DGM screens candidates on 10 and then 60 SWE-bench tasks and reports final results on 200, and ADAS searches on 20 ARC questions and tests on 60. Our matched comparison follows this practice on original TB2.1 tasks. On it, DarwinX is the only method that leads on both axes the protocol measures: it attains the highest official score of all seven optimizers (84/100) and improves earliest (proposal 2), while DGM-H, the only method as early, ranks sixth downstream (80/100). The paper's main claims rest on the complete official evaluations, where the gain is statistically significant, and on the 30-node campaigns, where the population mechanisms built every winner. We have answered every question from Reviewer xgo6 in full under their review, and the revised PDF collects the supporting analyses in a new Appendix G.

What the evidence shows. Unless noted otherwise, all numbers come from the paper's completed campaigns.

  1. Significant, robust gains (Appendix G.1). Complete official TB2.1: +7.19 pp, paired task-bootstrap 95% CI [+0.67, +13.71]; DeepSWE (113 × 5): +9.38 pp. All three independent 30-node campaigns improve their complete official evaluations, and independent DarwinX runs agree within one blind success.
  2. The mechanisms built the winners (Appendix G.2, G.4). Every 30-node winner descends from retained non-steering variants that a single-incumbent search would discard. The DeepSWE winner is the direct child of an archived specialist (47/72 → 52/72 blind), and 59% and 69% of two archived specialists' lines survive in the TB2.1 winner: the stepping-stone effect known from quality-diversity search. In a preliminary held-out study, a merged harness solved 28/41 tasks, above every evolved specialist.
  3. The short-horizon study isolates the active mechanisms (Appendix G.3). It ran before recombination triggered and before QD affected selection; the guard, which was active, has the largest effect, with every lost success on the panel it protects.
  4. Existing capability is kept. The drop on tasks the root solved 5/5 is regression to the mean: re-running the unchanged root keeps 42/50 of them, DarwinX 45/50. DarwinX keeps all 42 tasks the root solves 5/5 in both runs, while gaining +23.7 pp (TB2.1) and +18.4 pp (DeepSWE) where the root is unreliable.
  5. Strong external standing. With the same model, the evolved harness reaches 77% on a matched DeepSWE slice, against 63% for the native Codex CLI agent. On WebArena-Infinity it reaches 91.19%, against 69.3% for the best external agent, with zero invalid trajectories.

The scale of the study. Three 30-node campaigns on terminal, repository and browser tasks; complete official evaluations of 445, 565 and 1,260 outcomes, with original tasks kept outside search; six baseline optimizers adapted under one matched protocol; independent seeds and cross-benchmark transfer; and an action-validity audit of all 1,260 evolved WAI trajectories and their 40,783 tool calls. Every headline result is measured on a complete official benchmark, beyond the benchmark subsets on which recent self-improvement studies report results.

Why this matters. Deployed agents increasingly pair a fixed frontier model with a harness that carries much of their capability. Improving harnesses automatically is a selection problem: an edit that fixes one task can silently break another, and a discarded variant may hold the capability the next step needs. DarwinX makes this selection problem the core of its design, and its retention-and-steering mechanism is what produces the gains. Because the model stays frozen, the approach applies to any model, including closed ones.

We are grateful to the reviewers and the AC for their time and would be glad to discuss any remaining questions.

We have uploaded a revised PDF. It adds the analyses discussed in this thread as a new Appendix G and updates the related text. No selected winner and no previously reported number changes, and the pdfdiff shows every edit.

New Appendix G: Genealogy, Mechanism Activation, and Significance. It is computed from the campaigns and evaluation records reported in the paper, plus one preliminary held-out study (G.4).

  • G.1, significance: the paired task-bootstrap 95% CI for the complete TB2.1 gain, [+0.67, +13.71], and why a 100-outcome slice such as Table 5 cannot resolve one-success differences (the identical QD selection scores 49/72 and 50/72 in two blind runs).
  • G.2, winner genealogies (Table 15): the root-to-winner lineage of each 30-node campaign, with its archived specialists, retained non-steering ancestors, consolidation nodes and merge attempts, including the TB2.1 line-survival figures (59% and 69%) and the DeepSWE specialist reuse (47/72 → 52/72).
  • G.3, mechanism activation by horizon (Table 16): which mechanisms engaged in the short-horizon study and in the 30-node campaigns.
  • G.4, recombination in a preliminary held-out study: the merged harness solves 28/41, above every evolved specialist and a same-model Claude Code.
  • G.5, recombination in the baselines, including GEPA's merge proposer.

Updated text.

  • Introduction: the contributions summarize these findings.
  • RQ1: the TB2.1 interval [+0.67, +13.71].
  • RQ2: the reliability-band paragraph reports how stable solves are kept (94.40% and 88.28% success in the 5/5 bands) and that improved tasks outnumber regressed ones (27 to 10 and 50 to 22).
  • RQ3: the matched-budget discussion, with a pointer to G.1 for significance.
  • RQ4: the WAI per-application paragraph.
  • RQ5 and Appendix E.1: seed stability, the identical QD selection, consolidation as a capability-preserving restructuring, and the resolution of the 72-outcome panel (also stated in the Table 13 caption), with pointers to G.2 and G.3.
  • Discussion, related work and conclusion: tightened wording, and a short vision appendix (Appendix F).

The main text remains within the 9-page limit. Appendix G brings together the evidence behind our replies in one place, and we would be glad to point to any specific part of it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.