- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to improve an LLM agent by modifying its harness while keeping the underlying model fixed. It focuses on a selection problem in automated harness optimization: modifications may improve some tasks while regressing on others, and noisy evaluations can promote unreliable candidates. The proposed framework, DarwinX, uses a coding agent to generate harness modifications from execution traces and maintains an archive of evaluated variants. Its central design separates retention from steering: candidates must satisfy bounded-regression checks and additional confirmation before becoming eligible parents or final outputs, while other specialists can remain available for cross-lineage recombination. Shared memory, pruning, and consolidation further support the search.
The authors evaluate DarwinX through three campaigns using verifier-backed synthetic tasks constructed within benchmark environments, keeping original benchmark tasks outside the search loop. The selected harnesses are then frozen for downstream evaluation. Reported results show improvements of 7.19 and 9.38 percentage points over matched roots on Terminal-Bench 2.1 and DeepSWE, respectively, and 91.19% success on WebArena-Infinity compared with a 75.48% composite report baseline. Additional experiments include matched-budget comparisons with alternative optimizers, cross-benchmark transfer, seed stability, mechanism ablations, and an action-validity audit.
Critical strengths
-
The paper addresses an important challenge in automated harness optimization: individual modifications can produce conflicting effects across tasks, while evaluation noise can affect subsequent search. Separating candidate retention from eligibility to guide search offers a useful way to preserve potentially valuable modifications without immediately treating them as reliable starting points for further optimization.
-
The end-to-end results demonstrate improvements beyond the synthetic tasks used for search, particularly through matched comparisons with the initial agents on Terminal-Bench 2.1 and DeepSWE. Freezing the selected harnesses before downstream evaluation strengthens the evidence that the improvements transfer to tasks excluded from optimization.
Critical weaknesses
-
The experiments do not sufficiently establish the value of the central retention–steering distinction and specialist recombination. There is no direct comparison showing whether retaining non-steering specialists improves search over discarding them or allowing them to become parents. The reported ablations do not resolve this issue: all confidence intervals in Table 13 include zero, and the QD comparison selects the same final harness in both conditions. The selected-edit examples describe useful code changes but do not trace how a non-steering specialist contributed to a successful merge or the final improvement. Consequently, the evidence supports the complete optimization pipeline more strongly than the specific mechanisms presented as its main contribution.
-
The comparison with existing optimizers leaves the incremental benefit and computational efficiency unclear. On the shared official evaluation slice, DarwinX achieves 84/100 successes, compared with 83/100 for both DGM and GEPA, across 20 tasks with five trials each. This narrow margin does not establish a reliable advantage. Moreover, the adaptations differ in their permitted modifications, and matching proposal counts does not necessarily match total optimization cost when regression checks, confirmation, and merge operations require additional computation. Although the authors acknowledge the small performance margin, the current evidence does not establish whether the added complexity provides a meaningful benefit at a comparable budget.
Questions for authors
-
Could the authors clarify how often the central specialist-reuse mechanism contributed in the reported campaigns? How many non-steering specialists became merge donors, how many resulting merges were accepted, and did any final winner inherit their modifications? A concrete genealogy with task-level results before and after merging would help establish whether this mechanism contributed to the reported gains.
-
Do any of the reported ablations directly isolate retention–steering separation or independent confirmation? Please clarify which mechanisms were active in the short-horizon studies in Appendix E.1, particularly whether specialist recombination occurred. If existing comparisons test discarding non-steering specialists or allowing them to become parents, what were the outcomes?
-
How comparable were the actual optimization costs across methods in Table 5? Please report total evaluation rollouts and LLM token usage or API cost, including proposal generation, regression checks, confirmation, and merge attempts. Also, did the GEPA adaptation include its merge mechanism? These details would help determine whether the observed performance differences reflect the proposed selection procedure under comparable resources.
6 comments
Q2: which ablation isolates retention–steering separation or confirmation? what was active in App. E.1?
- Retention–steering separation: the genealogies in our previous comment isolate it. Under a discard rule, the DeepSWE winner's parent would not exist (that reuse is worth +5 blind successes, 47/72 → 52/72), and the specialists' lines in the TB2.1 winner would be absent. Variants that could not steer supplied the parents and edits from which every winner was built.
- Confirmation: under the fixed-panel protocol, every node is confirmed by an avg@5 panel evaluation before it can steer.
- App. E.1: following common RSI practice, these runs used a reduced horizon (6–8 proposals) on a 72-outcome panel
(e.g., DGM's staged evaluation screens candidates on small task subsets first). The guard and campaign memory
engaged. QD, consolidation and recombination did not fire (0 merges in every ablation arm).
- Their intervals around zero are therefore expected by construction. As in population-based training, these mechanisms need a diversified population, which a 6–8-proposal run does not yet have. In the 30-node campaigns they shape the winners (previous comment).
- The two QD arms select the identical harness (the same node and commit). Its 49/72 and 50/72 are that one harness scored in two separate blind runs, i.e., the panel's own re-evaluation noise.
- The guard's effect is mechanism-specific: all 4 lost successes fall on the panel it protects, and it replicates on the complete official runs (42/42 reliably solved tasks kept, over 445 outcomes).
W2: the margin in Table 5, and the different permitted modifications
- Permitted modifications: each adaptation keeps its method's defining edit surface and outer loop (the baseline-adaptation table). Restricting or expanding a method's edit scope would change the method itself. Everything else is held fixed: the evolvee, the proposer model, the budget, the task split, the anti-cheat checks and the evaluator. This is the standard way to compare optimizers as designed.
- Table 5 follows standard RSI practice: a fixed proposal budget, with the selected output scored on an official task subset, as with DGM's SWE-bench subsets and the small validation splits of ADAS and AFlow. Within it, DarwinX has the highest official score (84/100) and the earliest improvement: proposal 2, tied only with DGM-H, against 3 for DGM and GEPA, 4 for SkillOpt and 10 for Meta-Harness and ADAS.
- A 100-outcome slice cannot resolve one-point gaps: re-evaluating the same harness varies by about 2/140. Significance is therefore assessed on the complete benchmark: TB2.1 72.13% → 79.33% (+7.19 pp), paired task-bootstrap 95% CI [+0.67, +13.71]; DeepSWE +9.38 pp (113 × 5).
- The benefit holds against strong agents:
| Comparison | Result |
|---|---|
| DeepSWE, matched 20-task slice, same model (GPT-5.6-Sol) | DarwinX-evolved harness 77%, against 63% for the native Codex CLI agent and 60% for the root |
| WebArena-Infinity (1,260 tasks) | 91.19%, against 69.3% for Gemini 3 Flash + Browser Use, 48.3% for Qwen-3.5-Plus and 43.3% for Kimi-K2.5. 0 invalid trajectories, against 9 in the baseline. |
- The gains land where preserve-and-extend predicts. Tasks the root solves only 1–4/5 times gain +23.7 pp (TB2.1) and +18.4 pp (DeepSWE), residual failures fall by 25.8% and 20.5%, and the 42/42 tasks the root reliably solves are kept.
W2 / Q3: comparable cost; GEPA's merge.
- The budget is matched on the field's standard unit. Every method receives the same number of proposals, the budget unit used throughout RSI work (e.g., DGM and ADAS budget their searches in iterations), with the same evolvee, model and task split.
- The main evaluation cost, evolvee rollouts (each a full agent episode), is matched. DarwinX's 820 cover every rollout of the run, its confirmation panels and regression checks included, and are fewer than Meta-Harness (1,090), SkillOpt (905) and ADAS (900).
- Proposer context is 97% cache hits: DarwinX's structured analyze, implement and review stages reuse context across steps.
- GEPA: our adaptation retains GEPA's per-example Pareto-front parent sampling with same-batch acceptance. Like all baselines, it runs without recombination, so the comparison isolates each method's selection rule.
We thank the reviewer for the detailed reading and for recognizing the value of separating retention from steering. Every analysis below is computed from the completed campaigns reported in the paper.
W1 / Q1: how often did specialist reuse contribute? genealogy with task-level results
Every final winner inherited the modifications of non-steering variants:
This is the stepping-stone effect documented in quality-diversity search (e.g., novelty search and MAP-Elites): high-performing solutions are reached through intermediate variants that a greedy objective would discard. Archived behaviour also transfers: the DeepSWE winner is +24 pp on TB2.1 over the TB2.1-native winner, and a retained TB2.1 archive candidate is +8 pp on DeepSWE over the DeepSWE-native winner.
Merges: