acceptodds
Under review as a conference paper at ICLR 2027

MILO: Automated Harness Discovery via Orchestrated Multi-Agent Evolution

19 views

Abstract

Modern agentic systems typically consist of an artificial intelligence (AI) model and a harness, a software layer that manages its control flow and interactions with the environment. Agent performance on long-horizon tasks is often strongly influenced by its harness design. Yet building effective harnesses requires significant human effort due to the combinatorial design space, and this effort must be repeated as models are updated. To automate harness design, existing methods provide limited exploration of this search space. Most optimize only parts of the harness, such as prompts or skills, while others struggle to escape local optima due to their fixed search strategies and exploitative bias in LLM-driven search. In this paper, we present MILO (Meta-evolutionary Island Orchestration), a framework for automated harness discovery that co-evolves the harness and its own search strategy. Three components drive the search: (i) a hierarchical lineage memory of island-based trees that uses rejected mutations as negative evidence to prune unpromising paths and steer toward promising lineages; (ii) per-island mutator agents that combine global search history with feedback on parent weaknesses to rewrite entire harnesses; and (iii) an orchestrator agent that adapts the search by grafting and speciating lineages, reassigning mutators, and revising the curriculum. Together, they make MILO a meta-evolutionary harness-discovery framework that self-adapts its memory, mutators, and curriculum to balance exploration and exploitation. Across three long-horizon benchmarks, Terminal-Bench 2.1, PaperBench, and DeepSWE, MILO-discovered harnesses outperform eight state-of-the-art harnesses and six search methods with frontier (Opus 4.8) and open-weight (gpt-oss-120b) backbones. With Opus 4.8, it improves resolution rate over its initial harness by %, % and %, respectively, versus the best prior search gains % (GEPA), % (Meta-Harness) and % (no improvement observed). On Terminal-Bench 2.1, it reaches %, above the official leaderboard's top entry (%), while consuming 26% fewer tokens than its initial harness. On EinsteinArena's open problems, MILO-evolved harnesses tighten the best-known upper bounds for Erdős minimum-overlap () and the first and third autocorrelation inequalities (, ).

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
automated harness discovery, evolutionary search, LLM agents, long-horizon software engineering, scientific discovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aSWYOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MILO for automated harness discovery. It combines island-based lineage memory, evidence-driven mutator agents, and an orchestrator that adapts the search strategy. Experiments on 3 benchmarks show improvements over existing harnesses and search methods. The paper also reports transfer results on Frontier-Bench and improvements on three mathematical optimization problems.

Critical strengths
  1. MILO automates the entire network optimization process, not just prompt tuning. Its approach, which leverages failure history and adaptive multi-agent search, shows great promise.
  2. The study achieved outstanding results. MILO delivered significant improvements across three benchmarks and two backbone models. Comparisons with existing search methods, cost analysis, and transfer experiments further solidify the contributions of this research.
Critical weaknesses
  1. The reserved validation set was repeatedly used to monitor progress and guide the coordination process (Algo 1), and to select the final test framework. This introduces a risk of overfitting. The complete benchmark results reported in the paper are not entirely independent of the search process. Also, the paper lacks a comparison of results from different independent search runs and a comparison of total search costs.
  2. The implementation details and component contributions are unclear. The "accuracy-first" dominance rule in section 4.2 is not equivalent to the hypervolume criterion in Eq7. The course adaptation mechanism claimed in the main text is not reflected in the disclosed coordinator action patterns (Appendix C.2). The variator initialization settings in Appendix A.1 also contradict the content in section 4.1.2. Finally, the ablation experiments neither isolated the benefits of retaining eliminated candidate solutions nor compared the coordinator with simpler intervention rules.
Questions for authors
  1. Can the authors report results on an unmodified test set, across independent runs, and including the total search cost?
  2. Which admission rule was implemented: the accuracy-priority method or the hypervolume-contribution method? In what sense are these criteria equivalent?
  3. Was curriculum adaptation used in the experiments? Why does the reported mutator initialization method differ from the description in the main text?
  4. How does the coordinator perform compared to simple heuristic interventions?
7 comments
AuthorsOpenReview ↗

Authors' response to Reviewer aSWY (Part 1/3)

We sincerely thank the reviewer for the thoughtful review. We are especially encouraged that the reviewer recognizes the promise of MILO's approach beyond prompt tuning and finds its results outstanding. We have carefully addressed each of the reviewer's concerns with new experiments and corresponding revisions to the paper. The feedback also helped us clarify aspects of both the evaluation and the method description.

Below, we respond to each point under the reviewer's Weaknesses (W) and Questions (Q), with corresponding revisions highlighted in blue in the revised PDF. We very much hope these revisions address the reviewer's concerns and further strengthen the paper.


[R3-W1 + R3-Q1] The reserved validation set was repeatedly used to monitor progress … and to select the final … The complete benchmark results … are not entirely independent of the search process. … Can the authors report results on an unmodified test set, across independent runs, and including the total search cost?

We thank the reviewer for this crucial point. For our experiments, we indeed partition each benchmark into three: B_search (used by the mutators to propose better harnesses), B_val (used to monitor progress, guide the orchestration, and select the final harness), and B_test (a separate set that remains completely inaccessible to the search). In the initial submission, we reported results on the full benchmark. Following the reviewer's suggestion, the revised paper now reports results only on B_test and across independent searches, and we also report the total search cost.

  • (a) Main results only on a test set (B_test) that remains inaccessible during search. In Section 5.1, we describe how each benchmark (TB2.1, PaperBench, and DeepSWE) is split into B_search, B_val, and B_test in a 30:10:60 ratio. The partitioning is group-disjoint, ensuring that closely related tasks never end up in multiple partitions, and it is stratified by difficulty (Appendix A.2). In Section 5.1, we also state that B_test is held out and used only to score the final output harness of each search, for MILO and all SoTA search methods alike. Algorithm 1 (line 19) adds that the whole search uses only B_search and B_val, while B_test stays hidden. So any overfitting to B_val cannot inflate the reported results.

    Tables 1(a–b), 2(a), 3, and 4 now report results only on B_test. As we now state in the abstract, on held-out B_test tasks, MILO-discovered harnesses rank first in all six settings (3 benchmarks × 2 backbone LLMs) against eight SoTA harnesses and six SoTA evolutionary search frameworks. With Opus 4.8 as the backbone, MILO's RR@k gain over Best-of-3 is +12.8%, +36.2%, and +8.9% on TB2.1, PaperBench, and DeepSWE, vs. +6.0%, +27.8%, and +3.0% for the best prior search. On TB2.1/Opus, MILO's task-clustered interval does not overlap with that of the strongest prior search, GEPA (86.0 ± 2.5 vs. 79.2 ± 2.7%).

    Transfer to unseen benchmarks (Table 4). We also report the reusability of all MILO-evolved harnesses: in Table 4, we evaluate each benchmark's final harness, without further search, on the B_test of every other benchmark and on the never-searched Frontier-Bench. RR@k (%), with the gain over the seed:

    Evolved on (B_search, B_val) ↓ / run on (B_test) →TB2.1PaperBenchDeepSWEFrontier-Bench
    Best-of-3 seed73.219.458.310.0
    TB2.186.0 (+12.8%, in-domain)33.3 (+13.9%)63.2 (+4.9%)13.8 (+3.8%)
    PaperBench75.8 (+2.6%)55.6 (+36.2%, in-domain)60.8 (+2.5%)11.4 (+1.4%)
    DeepSWE81.1 (+7.9%)36.1 (+16.7%)67.2 (+8.9%, in-domain)14.8 (+4.8%)

    As we report in Section 5.5:

    • Every transferred harness beats the seed, and almost all also beat SoTA harnesses such as Mini-SWE-Agent.
    • Two transferred harnesses even beat all target-tuned prior searches. DeepSWE → TB2.1 reaches 81.1% RR@5, above every prior search tuned on TB2.1 (at most 79.2%), and TB2.1 → DeepSWE reaches 63.2% RR@3 (vs. at most 61.3%). So MILO-evolved harnesses are reusable artifacts, not tied to the benchmark they were searched on.
    • Transfer holds even on the most out-of-distribution benchmark. TB2.1 and DeepSWE are software engineering tasks, whereas PaperBench is paper replication. Yet TB2.1 → PaperBench and DeepSWE → PaperBench still gain +13.9% and +16.7% over the seed.

[continued in next comment box 2/3...]

AuthorsOpenReview ↗

Authors' response to Reviewer aSWY (Part 2/3)

[...continued from previous comment box 1/3]

  • (b) Independent runs ([R3-W1 + R3-Q1], continued). In Table 3, we now run MILO and each of its six ablation controls as three independent searches from the same seed harnesses (B10–B12) on TB2.1/Opus. Full MILO reaches 87.0 ± 1.2% RR@5, a gain of +13.7% over Best-of-3. Its spread is also the smallest of all seven search configurations (A–G). These error bars include both the variation across whole searches and the final-evaluation noise. In Section 5.5, we also report two MILO searches on TB2.1/gpt-oss that start from different initial harnesses. From the expert-designed seeds used throughout the paper (B10–B12), MILO reaches 36.2% RR@5 (Table 2). From the simplest human-written seeds (B1–B3, Table 7), it reaches 34.7%, only slightly below. This is still above every prior search started from the expert-designed seeds (at most 32.1% RR@5, EvoX, Table 2a).

  • (c) Total search cost. Tables 1–2 now report, for every search in all six settings, the average USD cost per candidate explored (mutation and all rollouts on B_search and B_val) and the number of candidates explored per hour. Their product over the 72-hour search gives the total spend, which we plot against B_test accuracy in Figure 4. In Section 5.5, we compare the costs directly:

    1. Lowest cost per candidate. With Opus 4.8 as the backbone, MILO achieves the lowest search cost per candidate on TB2.1 (USD 53 vs. 63–72), PaperBench (USD 79 vs. 83–119), and DeepSWE (USD 360 vs. 427–489). Mutator calls are under 5% of spend on all benchmarks. Most of it comes from evaluating candidates, so a cheaper harness makes the search cheaper. Among our baselines, only Meta-Harness optimizes token cost, and none optimizes latency. MILO's fitness includes both, so it increasingly explores cheaper candidates (Table 2c) and ends up with the lowest cost per candidate.
    2. Higher accuracy at lower spend. (i) On TB2.1/Opus, OpenEvolve and ShinkaEvolve, which also evolve three islands, explore 140–166 candidates for USD 9.5k–10.8k, yet reach at most 76.2% RR@5 on B_test. MILO reaches 86.0% after only 121 candidates for USD 6.4k (Figure 4a,b). (ii) On PaperBench/Opus, MILO reaches the highest accuracy (55.6% RR@3 on B_test vs. 47.2% by Meta-Harness) at the lowest spend, while OpenEvolve and ShinkaEvolve explore 2.4× the candidates at 2.6× the spend, yet reach at most 19.4% (Table 1, Figure 4a,b). (iii) On DeepSWE/gpt-oss, MILO reaches 9.7% after only USD 65, nearly 3× the best of any other search (3.4% by OpenEvolve) over its entire run (USD 313 spend; Figure 4c).
    3. A one-time cost. The total spend is paid once, whereas the discovered harness saves inference cost on every task it runs. On TB2.1/Opus, MILO's harness gains +12.8% RR@5 over Best-of-3 with 0.84× its tokens per task (Table 1b) and transfers to unseen benchmarks without further search (Table 4).

Thus, MILO finds better harnesses at a lower search cost.


[R3-W2 + R3-Q2] The "accuracy-first" dominance rule in section 4.2 is not equivalent to the hypervolume criterion in Eq7. … Which admission rule was implemented: the accuracy-priority method or the hypervolume-contribution method? In what sense are these criteria equivalent?

We apologize that the criterion was not clear in the original version, partly because some details had to move to the Appendix. The revised main paper now clarifies both the dominance rule and the child-acceptance criterion. MILO implements both as one rule, applied in a fixed order (Eq. 7, Section 4.2): accuracy decides across ε-buckets, and hypervolume gain decides within the top bucket. We bucketize accuracy at tolerance ε = 0.02 (Appendix A.1), and the top bucket is the highest one occupied by the island's admitted harnesses. A child h is then handled as follows:

  • Above the top bucket: admitted, at any cost.
  • Below the top bucket: rejected, at any cost.
  • Within the top bucket: admitted only if it adds hypervolume, i.e., no admitted member has F ≥ F(h). Here, F is a vector that holds accuracy and the economy of each cost.

In what sense the two criteria are equivalent. The dominance relation in Section 4.2 uses the same bucket form: h dominates h′ if it lies in a higher ε-bucket of accuracy, or in the same one with F(h) ≥ F(h′). A child is therefore admitted exactly when no admitted harness dominates it. Within the top bucket, this holds exactly when its hypervolume gain is positive, and Eq. 7 writes the test in both forms. A duplicate fitness vector is dominated, which matches its zero hypervolume gain. Two accuracies closer than ε can fall in adjacent buckets. We accept this in exchange for transitivity, so that a higher accuracy bucket wins at any cost. As we note in Section 4.2, this transitive ε-box dominance lets us test only the island's Pareto front instead of its whole admitted set.

[continued in next comment box 3/3...]

AuthorsOpenReview ↗

Authors' response to Reviewer aSWY (Part 3/3)

[...continued from previous comment box 2/3]

[R3-W2 + R3-Q3] The course adaptation mechanism claimed in the main text is not reflected in the disclosed coordinator action patterns (Appendix C.2). … Was curriculum adaptation used in the experiments? Why does the reported mutator initialization method differ from the description in the main text?

Curriculum. Yes. CURRICULUM is one of the orchestrator's four actions (Eq. 6, Section 4.1.3), and all reported MILO results use it. In Section 4.1.3, we describe how CURRICULUM replaces the search set, B_search ← B′, favoring high-regret tasks. Over the admitted harnesses, a task's regret is the difference between the best attainable and the population's mean accuracy. The orchestration prompt in Appendix C.2 shows this move explicitly. Its stall taxonomy maps a whole-population plateau to curriculum, its move (iii) is CURRICULUM with the same regret as in Section 4.1.3, and its JSON schema has a curriculum field. Its rules require every curriculum task to be in the search split, so B_val and B_test never enter the curriculum.

Mutator initialization. We would like to clarify that, before round r = 1, m^(1) assigns each island a mutator drawn uniformly at random from G (Section 4.1.2), which the orchestrator may reassign in later rounds. The orchestrator itself is a Claude Code agent. We now state this explicitly in Appendix A.1.


[R3-W2 + R3-Q4] … the ablation experiments neither isolated the benefits of retaining eliminated candidate solutions nor compared the coordinator with simpler intervention rules. … How does the coordinator perform compared to simple heuristic interventions?

Following the reviewer's insightful suggestion, we now add both controls to Table 3 and run each configuration as three independent searches.

Retained rejections. The ablation (Table 3) now separates the lineage structure from the retained rejections:

  • (B) flat archive, agentic mutator: 77.0 ± 2.3%
  • (C) + lineage memory, rejections discarded: 77.5 ± 2.4%
  • (D) + rejections retained: 81.8 ± 2.0%

As we report in Section 5.5, lineage memory alone adds only +0.5% (B → C), but retaining rejections adds a further +4.3% (C → D) by pruning weak designs. As we explain in Section 1, failures in the memory prune later mutations and push exploration to bolder structural changes.

A simple heuristic coordinator. In configuration (F) of the ablation (Table 3), we replace MILO's orchestrator with a simple fixed rule: periodic best-harness migration across islands on a fixed schedule, as in OpenEvolve.

  • (E) + three islands, no coordination: 83.1 ± 1.6%
  • (F) three islands, fixed-rule coordinator: 80.6 ± 2.4%
  • (G) three islands, MILO's orchestrator: 87.0 ± 1.2%

MILO's orchestrator outperforms the fixed rule by +6.4% (F → G) and no coordination by +3.9% (E → G). The fixed rule itself lowers RR@5 (−2.5% vs. E). From its lineages, we attribute this to lost diversity: the islands quickly converge to local optima (Section 5.5) and eventually cannot escape them. (F) differs from MILO in both the timing (on a schedule rather than at stalls) and the type of interventions (migration only). So Table 3 shows that MILO's stall-triggered orchestrator outperforms both no coordination and a standard migration rule.


As requested by the reviewer, we have added results on a separate test set, across independent runs, and with the total search cost. MILO remains first in all six settings on B_test. We have also clarified the admission rule, the use of curriculum, and the mutator initialization, and added the two requested ablation controls. We very much hope that these results resolve the reviewer's main concerns about held-out evaluation, search variance, search cost, and the method details. We greatly appreciate the reviewer's encouraging assessment of MILO's approach and results, and believe that these revisions and additional experiments have substantially strengthened the paper.

If the reviewer feels that these revisions merit it, we would be very grateful for any appropriate adjustment to the evaluation score. We would also be happy to clarify any remaining questions and thank the reviewer again for their time and thoughtful engagement with our work.

Reviewer_QqN5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes MILO, a framework for automated harness discovery that jointly evolves agent harnesses and their search strategies. It combines hierarchical lineage memory, island-based mutator agents, and an adaptive orchestrator to balance exploration and exploitation. Experiments on Terminal-Bench 2.1, PaperBench, and DeepSWE show that MILO outperforms existing harnesses and search methods.

Critical strengths

The paper evaluates MILO on three diverse long-horizon benchmarks using both closed-source and open-weight models. It compares against eight state-of-the-art harnesses and six evolutionary search methods. All search baselines use the same initial harnesses, evaluator, ensuring a fair and controlled comparison.

Critical weaknesses
  1. Single search run per method. All results come from one search run per method, so the error bars reflect only evaluation noise of the final harness, not the variance of the search itself, which is typically large for LLM-driven evolution. The ablation (Table 1d) is also single-run, and some steps fall within the CIs. Repeating each search with multiple seeds and reporting mean ± std would make the conclusions more convincing.

  2. Computational resources are not aligned. Only wall-clock time (72h) is matched. The number of evaluated candidates, token usage and cost are not reported. MILO runs three parallel islands with frontier coding-agent mutators, whereas most baselines use single LLM calls, so part of the gain may come from extra compute rather than the search design.

Questions for authors

The search process uses (B_{\text{search}}), while both progress monitoring and final harness selection rely on (B_{\text{val}}). However, Table 1 reports results evaluated on the “full benchmark,” which appears to include tasks used during search and selection. Could the authors clarify whether this is the case?

The text in several key figures and tables, notably Figure 1 and Tables 1 and 2 (including the embedded cost plots and ablation), is too small to read without heavy zooming, which makes the main results hard to follow.

6 comments
AuthorsOpenReview ↗

Authors' response to Reviewer QqN5 (Part 1/3)

We sincerely thank the reviewer for the constructive feedback. We are especially encouraged that the reviewer recognizes the breadth of our evaluation (three long-horizon benchmarks, closed-source and open-weight models, eight state-of-the-art harnesses, and six SoTA evolutionary search frameworks) and finds our comparison fair and controlled, as all search baselines use the same initial harnesses and evaluator. We have carefully addressed each of the reviewer's concerns with new experiments and corresponding revisions to the paper. The feedback on search variance, compute, held-out evaluation, and readability also helped us make our evidence more convincing and the main results easier to follow.

Below, we respond to each point under the reviewer's Weaknesses (W) and Questions (Q), with corresponding revisions highlighted in blue in the revised PDF. We truly hope these revisions address the reviewer's concerns and further strengthen the paper.


[R2-W1] Single search run per method … the error bars reflect only evaluation noise of the final harness, not the variance of the search itself … The ablation (Table 1d) is also single-run, and some steps fall within the CIs. Repeating each search with multiple seeds and reporting mean ± std would make the conclusions more convincing.

We agree that search-level variance matters, and, as the reviewer suggested, we now repeat MILO and each of its ablation configurations three times and report mean ± std. The ablation formerly in Table 1(d) is now Table 3, which reports the mean ± std on B_test over three runs. In Section 5.5, we run each configuration in Table 3 as three independent searches from the same seeds (B10–B12). These error bars include both the variation across whole searches and the final-evaluation noise.

MILO's gains are reproducible across independent searches. Over three independent 72-hour searches, full MILO reaches 87.0 ± 1.2% RR@5, a gain of +13.7% over Best-of-3. Its std is close to the evaluation noise alone, so MILO's results vary little from search to search, given the same initial seeds. MILO's mean is also 7.8% above the strongest prior search on TB2.1/Opus (79.2%, GEPA, Table 1).

Each component's contribution (Table 3). We have also made the ablation more fine-grained. It now adds MILO's components one at a time, with three new controls: lineage memory without retained rejections (C), islands without coordination (E), and a fixed-rule coordinator (F). Each configuration is run as three independent searches (RR@5, mean ± std):

  • Best-of-3 (no search): 73.3 ± 1.1%
  • (A) flat archive, LLM mutator: 73.8 ± 1.3%
  • (B) flat archive, agentic mutator: 77.0 ± 2.3%
  • (C) + lineage memory, rejections discarded: 77.5 ± 2.4%
  • (D) + rejections retained: 81.8 ± 2.0%
  • (E) + three islands, no coordination: 83.1 ± 1.6%
  • (F) three islands, fixed-rule coordinator: 80.6 ± 2.4%
  • (G) three islands, MILO's orchestrator: 87.0 ± 1.2%

In Table 3, replacing the LLM mutator with an agentic mutator that can access the search history adds +3.2% (A → B). Retaining rejected candidates in the lineage memory adds +4.3% by pruning weak designs (C → D). MILO's orchestrator adds +3.9% over uncoordinated islands (E → G) and +6.4% over the fixed-rule coordinator (F → G). Each of these four gains is larger than the std of both configurations it compares. As the reviewer noted for the original ablation, two steps remain small: the lineage structure alone (+0.5%, B → C) and uncoordinated islands (+1.3%, D → E). Both steps add structure that the larger gains then use: retained rejections live in the lineage memory (D), and the orchestrator coordinates the islands (G).

Different initial seeds. We also report two MILO searches on TB2.1/gpt-oss that start from different initial harnesses (Section 5.5). From the expert-designed seeds used throughout the paper (B10–B12), MILO reaches 36.2% RR@5 (Table 2). From the simplest human-written seeds (B1–B3, Table 7), it reaches 34.7%, only slightly below. This is still above every prior search started from the expert-designed seeds (at most 32.1% RR@5, EvoX, Table 2a).

[continued in next comment box 2/3...]

AuthorsOpenReview ↗

Authors' response to Reviewer QqN5 (Part 2/3)

[...continued from previous comment box 1/3]

[R2-W2] Computational resources are not aligned. Only wall-clock time (72h) is matched. The number of evaluated candidates, token usage and cost are not reported. MILO runs three parallel islands with frontier coding-agent mutators … so part of the gain may come from extra compute rather than the search design.

We deeply thank the reviewer for raising this point. It gave us the opportunity to dive deeper and compare every search framework not only by the accuracy and token consumption of its final harness, but also by its search cost in USD and the number of candidates it explores. To our knowledge, this is one of the first comparisons of LLM-driven search methods along all these dimensions. In short, most of MILO's gain come from its search design. Of its +13.7% gain over Best-of-3 on TB2.1/Opus (Table 3), the agentic mutator and the islands together add only +4.5%, while the search design adds +9.2%, mainly through retained rejections (+4.3%) and the orchestrator (+3.9%).

We match wall-clock time because it is the budget that matters in practice: a practitioner typically wants a harness within at most 72 hours (3 days), which is already long, and a longer search also costs more. R1 (kvNy) likewise called this "a reasonable end-to-end engineering comparison". However, we agree that equal wall-clock time does not mean equal compute. In Section 5.5, we now state that differences in mutator type (LLM or agent) and concurrency (islands) can imply unequal compute.

We therefore now report the cost of every search. We add two columns to Tables 1–2 for every search in all six settings (3 benchmarks × 2 backbone LLMs): the average USD cost per candidate explored, and the number of candidates explored per hour. The cost includes mutation and rollouts on B_search and B_val. Their product over 72 hours gives the total spend, which we plot in Figure 4. We use USD as the common unit, as the searches mix models with different token prices.

The new cost data show that MILO's advantage comes from a more effective search:

  1. Lowest cost per candidate. With Opus 4.8 as the backbone, MILO achieves the lowest search cost per candidate on TB2.1 (USD 53 vs. 63–72), PaperBench (USD 79 vs. 83–119), and DeepSWE (USD 360 vs. 427–489). Mutator calls are under 5% of spend on all benchmarks, so agentic mutators add little to the cost. Most of it comes from evaluating candidates, so a cheaper harness makes the search cheaper. Yet almost all evolutionary search methods optimize accuracy alone and drift to costly harnesses (Section 1). Among our baselines, only Meta-Harness optimizes token cost, and none optimizes latency. MILO's fitness includes both, so it increasingly explores cheaper candidates (Table 2c) and ends up with the lowest cost per candidate.
  2. Highest accuracy at equal exploration and at lower spend. (i) With gpt-oss-120b as the backbone, we plot each search's best B_val pass rate on TB2.1 against candidates explored (Table 2c). MILO achieves the best final pass rate at the same exploration budget. (ii) On DeepSWE/gpt-oss, we plot the B_test accuracy of each search's output harness against spend (Figure 4c). MILO reaches far higher accuracy at a much lower spend: 9.7% after only USD 65, nearly 3× the best of any other search (3.4% by OpenEvolve) over its entire run (USD 313 spend). (iii) On PaperBench/Opus, MILO reaches the highest accuracy (55.6% RR@3 on B_test vs. 47.2% by Meta-Harness) at the lowest spend, while OpenEvolve and ShinkaEvolve explore 2.4× the candidates at 2.6× the spend, yet reach at most 19.4% (Table 1, Figure 4a,b).
  3. With less exploration and lower spend than the other island-based searches, MILO finds a better harness. OpenEvolve and ShinkaEvolve also evolve three islands, as MILO does. The other baselines keep their native single population, because lifting them to multiple islands is a non-trivial design space and would not be faithful to their frameworks (Appendix A.3). On TB2.1/Opus, OpenEvolve and ShinkaEvolve explore 140–166 candidates for USD 9.5k–10.8k, yet reach at most 76.2% RR@5 on B_test. MILO reaches 86.0% after only 121 candidates for USD 6.4k (Figure 4a,b). Keeping the islands fixed, MILO's orchestrator alone lifts RR@5 to 87.0% on B_test, vs. 83.1% without coordination and 80.6% with a fixed-rule coordinator (Table 3; see our answer to W1). So MILO's gain comes from how it searches rather than from its islands.
  4. Most of MILO's gain comes from its search strategy rather than a stronger mutator. Meta-Harness also uses an agentic mutator (Claude Code with Opus 4.8), which is used by MILO (Appendices A.1, A.3). Yet MILO outperforms it in all six settings (Tables 1–2).

Thus, MILO's gain comes from its search design, not from extra compute.

[continued in next comment box 3/3...]

AuthorsOpenReview ↗

Authors' response to Reviewer QqN5 (Part 3/3)

[...continued from previous comment box 2/3]

[R2-Q1] … Table 1 reports results evaluated on the "full benchmark," which appears to include tasks used during search and selection. Could the authors clarify whether this is the case?

We thank the reviewer for raising this crucial point, and we are happy to clarify. For our experiments, we indeed partition each benchmark into three: B_search (used by the mutators to propose better harnesses), B_val (used for harness selection and progress monitoring), and B_test (a separate set that remains completely inaccessible to the search). In the initial submission, Tables 1–2 reported results on the full benchmark, which includes the search and validation tasks. Following the reviewer's comment, the revised paper reports every harness only on B_test.

  • (a) The held-out split. In Section 5.1, we describe how each benchmark (TB2.1, PaperBench, and DeepSWE) is split into B_search, B_val, and B_test in a 30:10:60 ratio. The partitioning is group-disjoint, ensuring that closely related tasks never end up in multiple partitions, and it is stratified by difficulty (Appendix A.2). For example, all DeepSWE tasks from the same source repository form one group. A human expert verified all groups. In Section 5.1, we also state that B_test is held out and used only to score the final output harness of each search. Algorithm 1 (line 19) adds that the whole search uses only B_search and B_val, while B_test stays hidden.

  • (b) MILO's advantage holds on B_test. Tables 1(a–b), 2(a), 3, and 4 now report results only on B_test. As we now state in the abstract, on held-out B_test tasks, MILO-discovered harnesses rank first in all six settings. With Opus 4.8 as the backbone, MILO's RR@k gain over Best-of-3 is +12.8%, +36.2%, and +8.9% on TB2.1, PaperBench, and DeepSWE, vs. +6.0%, +27.8%, and +3.0% for the best prior search.

  • (c) MILO's harnesses transfer to unseen benchmarks (Table 4). We also report the reusability of all MILO-evolved harnesses: in Table 4, we evaluate each benchmark's final harness, without further search, on the B_test of every other benchmark and on the never-searched Frontier-Bench. RR@k (%), with the gain over the seed:

    Evolved on (B_search, B_val) ↓ / run on (B_test) →TB2.1PaperBenchDeepSWEFrontier-Bench
    Best-of-3 seed73.219.458.310.0
    TB2.186.0 (+12.8%, in-domain)33.3 (+13.9%)63.2 (+4.9%)13.8 (+3.8%)
    PaperBench75.8 (+2.6%)55.6 (+36.2%, in-domain)60.8 (+2.5%)11.4 (+1.4%)
    DeepSWE81.1 (+7.9%)36.1 (+16.7%)67.2 (+8.9%, in-domain)14.8 (+4.8%)

    As we report in Section 5.5:

    • Every transferred harness beats the seed, and almost all also beat SoTA harnesses such as Mini-SWE-Agent.
    • Two transferred harnesses even beat all target-tuned prior searches. DeepSWE → TB2.1 reaches 81.1% RR@5 (vs. at most 79.2%), and TB2.1 → DeepSWE reaches 63.2% RR@3 (vs. at most 61.3%).
    • Transfer holds even on the most out-of-distribution benchmark. TB2.1 and DeepSWE are software engineering tasks, whereas PaperBench is paper replication. Yet TB2.1 → PaperBench and DeepSWE → PaperBench still gain +13.9% and +16.7% over the seed.

[R2-Q2] The text in several key figures and tables, notably Figure 1 and Tables 1 and 2 (including the embedded cost plots and ablation), is too small to read without heavy zooming, which makes the main results hard to follow.

We thank the reviewer for pointing this out, and we apologize for the readability issues. They arose partly because we chose to show the complete evidence side by side in the main paper: all six settings, the inference costs, and the ablation. Following the reviewer's feedback, the revised paper now gives the key analyses more room:

  1. The ablation has moved out of Table 1 into its own table with a larger font (Table 3).
  2. The search-cost analysis now has its own figure (Figure 4), and we add new search-cost columns to Tables 1–2.
  3. The inference-cost plots in Table 1(b) and the evolution plots in Table 2(b–c) are enlarged, and Table 2(c) now spans the full width.

As requested by the reviewer, we have added repeated searches, the search cost of every method, results on tasks unseen during search, and more readable figures. MILO remains first in all six settings on the held-out B_test. We very much hope that these results resolve the reviewer's main concerns about search variance and compute. We greatly appreciate the reviewer's recognition of our broad and fair evaluation, and believe that these revisions and additional experiments have made the paper clearer and its evidence stronger.

If the reviewer feels that these revisions merit it, we would be very grateful for any appropriate adjustment to the evaluation score. We would also be happy to clarify any remaining questions and thank the reviewer again for the time and care invested in our paper.

Reviewer_kvNyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MILO evolves complete agent harnesses using multiple islands, lineage memory retaining rejected candidates, agentic mutators, and an orchestrator that changes mutator assignments, exchanges lineages, creates islands, and revises the task curriculum when progress stalls. The paper reports improvements over expert harnesses and automated search baselines on Terminal-Bench 2.1, PaperBench-CodeDev, and DeepSWE with two backbones, plus transfer to Frontier-Bench. An extension evolves harnesses that generate optimizer programs and reports improved constructions for three mathematical problems.

Critical strengths
  1. The system tackles a significant practical problem at an appropriate level. Evolving control flow and verification mechanisms is a meaningful extension beyond prompt optimization. Explicit failure history and intervention on stalled searches are well motivated.

  2. The empirical scope is broad. Tables 1–2 compare multiple expert and automatic harnesses across three demanding benchmarks and two backbones. The frontier-backbone results include 86.1 versus 74.1 Terminal-Bench resolution rate and 55.0 versus 26.7 PaperBench resolution rate relative to Best-of-3. These are meaningful reported gains, not merely tiny leaderboard changes.

Critical weaknesses
  1. The validation split is an optimization signal, not an untouched test. Algorithm 1, lines 12–18, evaluates admitted children on B_val, resets stall counters on validation gains, and uses those counters to trigger orchestration. Line 22 selects the final harness using B_val. These are legitimate validation uses, but they make the claim that this monitoring guards against overfitting incomplete: validation outcomes affect the future search trajectory and final selection. Tables 1–2 then evaluate on the full benchmark, which includes search and validation tasks. Repeating rollouts on the same tasks does not remove adaptive selection bias. Table 3 provides real external evidence, but only for a subset of the overall comparison, and its +3.8-point resolution gain over Best-of-3 needs a paired analysis before being treated as decisive.

  2. Equal wall-clock limits do not establish equal search resources. Section 5.3 and Appendix A allocate 72 hours per method, but MILO runs concurrent islands and can access six mutator pairings. Several baselines retain different native model ensembles and mutation styles. This is a reasonable end-to-end engineering comparison, but total proposer tokens, evaluated candidate-task attempts, API spending, concurrency, and validation evaluations should be reported to establish whether the advantage comes from orchestration rather than additional throughput or more capable mutation. The inference savings do not include the cost of discovering the harness.

  3. Uncertainty largely concerns deployment, not discovery. Task-clustered intervals and multiple final rollouts are useful, but do not measure variation across independently repeated 72-hour searches. The additive ablation's 79.1 to 79.3 increment is not convincing evidence for lineage memory without search-level repetitions. A fixed-rule orchestrator given comparable intervention opportunities is also an important control for the claim that adaptive strategy changes themselves explain the improvement.

Questions for authors
  1. Which reported task sets were never used for search feedback, stall decisions, or final harness selection? Can the principal comparison be reported on such a set, with a paired analysis of the existing Frontier-Bench results?

  2. Does the implementation use accuracy-primary admission, geometric hypervolume gain, or both? How are duplicates, epsilon ties, and comparison against a parent versus an entire island handled?

  3. What are the total search tokens/cost, candidate-task evaluations, validation evaluations, and parallelism for each method and ablation? How many independent search runs were performed?

7 comments
AuthorsOpenReview ↗

Authors' response to Reviewer kvNy (Part 1/4)

We sincerely thank the reviewer for the thoughtful and detailed review. We are especially encouraged that the reviewer recognizes MILO as tackling a significant practical problem, appreciates its evolution of control flow and verification as a meaningful step beyond prompt optimization, and finds its gains meaningful rather than marginal. We have carefully addressed each of the reviewer's concerns with new experiments and corresponding revisions to the paper. The feedback on held-out evaluation, search resources, and search variance also helped us make our evidence considerably more rigorous.

Below, we respond to each point under the reviewer's Weaknesses (W) and Questions (Q), with corresponding revisions highlighted in blue in the revised PDF. We truly hope these revisions address the reviewer's concerns and further strengthen the paper.


[R1-W1 + R1-Q1] The validation split is an optimization signal, not an untouched test … Which reported task sets were never used for search feedback, stall decisions, or final harness selection? Can the principal comparison be reported on such a set …?

We thank the reviewer for this crucial point, and we agree that the validation split is not an untouched test. For our experiments, we indeed partition each benchmark into three: B_search (used by the mutators to propose better harnesses), B_val (used for harness selection and progress monitoring), and B_test (a separate set that remains completely inaccessible to the search). In the initial submission, we reported results on the full benchmark. Following the reviewer's suggestion, the revised paper reports the principal comparison only on B_test.

  • (a) Evaluation on task sets that no search uses: B_test and Frontier-Bench. In Section 5.1, we describe how each benchmark (TB2.1, PaperBench, and DeepSWE) is split into B_search, B_val, and B_test in a 30:10:60 ratio. The partitioning is group-disjoint, ensuring that closely related tasks never end up in multiple partitions, and it is stratified by difficulty (Appendix A.2). In Section 5.1, we also state that B_test is held out and used only to score the final output harness of each search. Algorithm 1 (line 19) adds that the whole search uses only B_search and B_val, while B_test stays hidden. We also evaluate on Frontier-Bench, which no search ever uses (see (c) below).

  • (b) The principal comparison is now on B_test. Tables 1(a–b), 2(a), 3, and 4 now report results only on B_test. As we now state in the abstract, on held-out B_test tasks, MILO-discovered harnesses rank first in all six settings (3 benchmarks × 2 backbone LLMs) against eight SoTA harnesses and six SoTA evolutionary search frameworks. With Opus 4.8 as the backbone, MILO's RR@k gain over Best-of-3 is +12.8%, +36.2%, and +8.9% on TB2.1, PaperBench, and DeepSWE, vs. +6.0%, +27.8%, and +3.0% for the best prior search. On TB2.1/Opus, MILO's task-clustered interval does not overlap with that of the strongest prior search, GEPA (86.0 ± 2.5 vs. 79.2 ± 2.7%).

  • (c) Frontier-Bench and transfer to unseen benchmarks (Table 4). To broaden this evidence on unseen tasks, we now report the reusability of all MILO-evolved harnesses: in Table 4, we evaluate each benchmark's final harness, without further search, on the B_test of every other benchmark and on the never-searched Frontier-Bench. As we describe in Section 5.5, its own B_test is hidden during search, and the other benchmarks' B_test sets are out of distribution. RR@k (%), with the gain over the seed:

    Evolved on (B_search, B_val) ↓ / run on (B_test) →TB2.1PaperBenchDeepSWEFrontier-Bench
    Best-of-3 seed73.219.458.310.0
    TB2.186.0 (+12.8%, in-domain)33.3 (+13.9%)63.2 (+4.9%)13.8 (+3.8%)
    PaperBench75.8 (+2.6%)55.6 (+36.2%, in-domain)60.8 (+2.5%)11.4 (+1.4%)
    DeepSWE81.1 (+7.9%)36.1 (+16.7%)67.2 (+8.9%, in-domain)14.8 (+4.8%)

    As we report in Section 5.5:

    • Every transferred harness beats the seed, and almost all also beat SoTA harnesses such as Mini-SWE-Agent.
    • Two transferred harnesses even beat all target-tuned prior searches. DeepSWE → TB2.1 reaches 81.1% RR@5, above every prior search tuned on TB2.1 (at most 79.2%), and TB2.1 → DeepSWE reaches 63.2% RR@3 (vs. at most 61.3%). So MILO-evolved harnesses are reusable artifacts, not tied to the benchmark they were searched on.
    • Transfer holds even on the most out-of-distribution benchmark. TB2.1 and DeepSWE are software engineering tasks, whereas PaperBench is paper replication. Yet TB2.1 → PaperBench and DeepSWE → PaperBench still gain +13.9% and +16.7% over the seed.

[continued in next comment box 2/4...]

AuthorsOpenReview ↗

Authors' response to Reviewer kvNy (Part 2/4)

[...continued from previous comment box 1/4]

[R1-W2 + R1-Q3] Equal wall-clock limits do not establish equal search resources … whether the advantage comes from orchestration rather than additional throughput or more capable mutation. … What are the total search tokens/cost, candidate-task evaluations, validation evaluations, and parallelism for each method and ablation?

We thank the reviewer for raising this important point, and we completely agree that equal wall-clock time does not mean equal compute. Because differences in mutator type (LLM or agent) and concurrency (islands) can imply unequal compute, we now report the cost of every search in Section 5.5. With as many islands as OpenEvolve and ShinkaEvolve, MILO outperforms both at a lower search cost on all three benchmarks with Opus 4.8.

  • (a) What we now report. We add two columns to Tables 1–2 for every search in all six settings: the average USD cost per candidate explored, and the number of candidates explored per hour. Their product over 72 hours gives the total spend, which we plot in Figure 4. We use USD as the common unit, as the searches mix models with different token prices.

    The cost per candidate includes mutation and all rollouts, i.e., all candidate-task attempts and validation evaluations. Each MILO candidate runs on every B_search task (Appendix A.1), and only admitted harnesses are re-scored on B_val (Section 4.2). For a fair comparison, every search runs on the same machine, starts from the same three seeds, and is scored by one shared scorer (Appendices A.1, A.3). OpenEvolve and ShinkaEvolve also evolve three islands, as MILO does. The other baselines keep their native single population, because lifting them to multiple islands is a non-trivial design space and would not be faithful to their frameworks (Appendix A.3).

    Finally, the total spend is the one-time cost of discovering a harness, whereas the harness saves inference cost on every task it runs. On TB2.1/Opus, MILO's USD 6.4k search (Figure 4a) yields a harness that gains +12.8% RR@5 over Best-of-3 with 0.84× its tokens per task (Table 1b) and transfers to unseen benchmarks without further search (Table 4). OpenEvolve and ShinkaEvolve spend USD 9.5k–10.8k for at most +3.0%.

  • (b) What the cost data show. MILO's advantage comes from a more effective search rather than additional throughput or more capable mutation:

    1. Lowest cost per candidate. With Opus 4.8 as the backbone, MILO achieves the lowest search cost per candidate on TB2.1 (USD 53 vs. 63–72), PaperBench (USD 79 vs. 83–119), and DeepSWE (USD 360 vs. 427–489). Mutator calls are under 5% of spend on all benchmarks, so agentic mutators add little to the cost. Most of it comes from evaluating candidates, so a cheaper harness makes the search cheaper. Among our baselines, only Meta-Harness optimizes token cost, and none optimizes latency. MILO's fitness includes both, so it increasingly explores cheaper candidates (Table 2c) and ends up with the lowest cost per candidate.
    2. Highest accuracy at equal exploration and at lower spend. (i) With gpt-oss-120b as the backbone, we plot each search's best B_val pass rate on TB2.1 against candidates explored (Table 2c). MILO achieves the best final pass rate at the same exploration budget. (ii) On DeepSWE/gpt-oss, we plot the B_test accuracy of each search's output harness against spend (Figure 4c). MILO reaches far higher accuracy at a much lower spend: 9.7% after only USD 65, nearly 3× the best of any other search (3.4% by OpenEvolve) over its entire run (USD 313 spend). (iii) On PaperBench/Opus, MILO reaches the highest accuracy (55.6% RR@3 on B_test vs. 47.2% by Meta-Harness) at the lowest spend, while OpenEvolve and ShinkaEvolve explore 2.4× the candidates at 2.6× the spend, yet reach at most 19.4% (Table 1, Figure 4a,b).
    3. With less exploration and lower spend than the other island-based searches, MILO finds a better harness. On TB2.1/Opus, OpenEvolve and ShinkaEvolve explore 140–166 candidates at a higher spend, yet reach at most 76.2% RR@5 on B_test. MILO reaches 86.0% after only 121 candidates (Figure 4a,b). Keeping the islands fixed, MILO's orchestrator alone lifts RR@5 to 87.0% on B_test, vs. 83.1% without coordination and 80.6% with a fixed-rule coordinator (Table 3, mean over three independent searches; please see our answer to W3).
    4. Most of MILO's gain comes from its search strategy rather than more capable mutation. Meta-Harness also uses an agentic mutator (Claude Code with Opus 4.8), which is used by MILO (Appendices A.1, A.3). Yet MILO outperforms it in all six settings (Tables 1–2). In Table 3, replacing the LLM mutator with an agentic one adds +3.2% (A → B), while MILO's lineage memory, islands, and orchestrator add a further +10.0% (B → G).

Thus, MILO's gain comes from how it searches, not from extra throughput.

[continued in next comment box 3/4...]

AuthorsOpenReview ↗

Authors' response to Reviewer kvNy (Part 3/4)

[...continued from previous comment box 2/4]

[R1-W3 + R1-Q3] Uncertainty largely concerns deployment, not discovery … The additive ablation's 79.1 to 79.3 increment is not convincing evidence for lineage memory without search-level repetitions. A fixed-rule orchestrator … is also an important control … How many independent search runs were performed?

We thank the reviewer for these valuable suggestions, which motivated substantial new experiments in our revision. We agree that the task-clustered confidence intervals in Tables 1–2 capture deployment uncertainty rather than discovery uncertainty. To address the latter, we now report in Section 5.5 the mean RR@5 across three independent searches for each configuration in Table 3, initialized from the same seeds (B10–B12). We also substantially expand the ablation to examine lineage memory and to compare MILO's orchestrator with a fixed-rule coordinator.

  • (a) Same seeds, repeated searches. Across these three independent searches from the same seeds, full MILO reaches 87.0 ± 1.2% RR@5, a gain of +13.7% over Best-of-3 and 7.8% above the strongest prior search (79.2%, GEPA, Table 1). These error bars include both search variation and final-evaluation noise. The latter alone is 1.1% for a fixed harness (Best-of-3 re-evaluated three times), and MILO's std of 1.2% is close to it, suggesting limited variation across independent searches.

  • (b) A more fine-grained ablation (Table 3). Following the reviewer's suggestion, the ablation now adds MILO's components one at a time and includes three new controls: lineage memory without retained rejections (C), islands without coordination (E), and a fixed-rule coordinator (F). RR@5, mean ± std over three searches:

    • Best-of-3 (no search): 73.3 ± 1.1%
    • (A) flat archive, LLM mutator: 73.8 ± 1.3%
    • (B) flat archive, agentic mutator: 77.0 ± 2.3%
    • (C) + lineage memory, rejections discarded: 77.5 ± 2.4%
    • (D) + rejections retained: 81.8 ± 2.0%
    • (E) + three islands, no coordination: 83.1 ± 1.6%
    • (F) three islands, fixed-rule coordinator: 80.6 ± 2.4%
    • (G) three islands, MILO's orchestrator: 87.0 ± 1.2%

    As we report in Section 5.5, an agentic mutator with search-history access adds +3.2% (A → B). Lineage memory alone adds only +0.5% (B → C), but retaining rejections adds a further +4.3% (C → D) by pruning weak designs. The repeated ablation thus shows that the memory's gain comes from retaining rejected candidates rather than from the lineage structure alone. Three uncoordinated islands (best one picked on B_val) then add +1.3% (D → E).

  • (c) Fixed-rule coordinator (F). To test whether coordination alone explains the gain, configuration (F) keeps the three islands of (E) and replaces MILO's orchestrator with the OpenEvolve-style rule: periodic best-harness migration on a fixed schedule. As we note in Section 2, such a hand-set strategy cannot react to stalls. Interestingly, it lowers RR@5 by 2.5% (E → F). From its lineages, we attribute this to lost diversity, as islands converge to local optima. MILO's orchestrator instead adds +3.9% over uncoordinated islands (E → G) and +6.4% over the fixed rule (F → G), reaching 87.0%. (F) differs from MILO in both the timing (on a schedule rather than at stalls) and the type of interventions (migration only). So Table 3 shows that MILO's stall-triggered orchestrator outperforms both no coordination and a standard migration rule.

  • (d) Different seed initializations. We also report two MILO searches on TB2.1/gpt-oss that start from different initial harnesses (Section 5.5). From the expert-designed seeds used throughout the paper (B10–B12), MILO reaches 36.2% RR@5 (Table 2). From the simplest human-written seeds (B1–B3, Table 7), it reaches 34.7%, only slightly below. This is still above every prior search started from the expert-designed seeds (at most 32.1% RR@5, EvoX, Table 2a).

  • (e) Number of independent searches. In Table 3, each configuration is run as three independent searches from the same seeds (B10–B12). We also report a MILO search from a different initialization (B1–B3), as described in (d).

We thank the reviewer again for suggesting these additional experiments. The repeated searches provide evidence that MILO's gains are consistent across independent searches, with comparable performance under a different initialization. The expanded ablation clarifies the contributions of retaining rejected candidates and of MILO's orchestration strategy. We believe these additions substantially strengthen the empirical evidence for MILO and address the reviewer's concerns.

[continued in next comment box 4/4...]

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response by Authors (Part 1/2)

We sincerely thank all reviewers for their thoughtful and constructive feedback, which has helped us improve our work considerably. Below, R1, R2, and R3 refer to Reviewers kvNy, QqN5, and aSWY. The reviewers found that MILO tackles "a significant practical problem at an appropriate level" (R1) and automates harness design as a whole, "not just prompt tuning" (R3). Its evaluation spans three long-horizon benchmarks, two backbones, eight state-of-the-art harnesses, and six SoTA evolutionary search frameworks with shared seeds and evaluator, "ensuring a fair and controlled comparison" (R2). The reviewers also called the gains "meaningful … not merely tiny leaderboard changes" (R1) and "outstanding" (R3). We have addressed every comment and concern of all three reviewers with new experiments and corresponding revisions to the paper, summarized below.

The reviewers shared four main requests: (i) results on tasks the search never used, (ii) the search cost of every method, (iii) repeated searches and controls for individual components, and (iv) a clearer admission rule, mutator initialization, and figures. On held-out B_test tasks, MILO-discovered harnesses still rank first in all six settings (3 benchmarks × 2 backbone LLMs) against eight SoTA harnesses and six SoTA evolutionary search frameworks. We now provide all of the following in the revised PDF, where every change is highlighted in blue:

  1. Results on a held-out test split that no search can access (Section 5.1, Appendix A.2). Terminal-Bench 2.1 (TB2.1), PaperBench, and DeepSWE are split into B_search, B_val, and B_test in a 30:10:60 ratio, group-disjoint and stratified by difficulty. MILO and all six SoTA evolutionary search frameworks use B_search for evolution and B_val for harness selection and progress monitoring. B_test is held out and used only to score the final output harness of each search (Section 5.1, Algorithm 1). In the initial submission, we reported results on the full benchmark. Tables 1(a–b), 2(a), 3, and 4 now report results only on B_test.

  2. Search cost for every method (Tables 1–2, Figure 4, Section 5.5). For every search in all six settings, Tables 1–2 now report the average USD cost per candidate explored and the number of candidates explored per hour. Their product over the 72-hour search gives the total spend, which we plot against B_test accuracy in Figure 4.

  3. Repeated searches and new controls (Table 3, Section 5.5). MILO and its six ablation controls are each run as three independent searches from the same seeds on TB2.1/Opus (mean ± std on B_test). The new controls isolate retained rejections, uncoordinated islands, and a fixed-rule coordinator. We also run MILO from a different, simpler set of seeds on TB2.1/gpt-oss.

  4. Transfer to unseen benchmarks (Table 4). Each evolved harness is run, without further search, on the B_test of the other benchmarks and on the never-searched Frontier-Bench.

  5. Method clarifications (Sections 4.1–4.2, Appendices A.1 and C.2). Eq. 7 and the dominance relation above it now state the admission rule explicitly: ε-bucketed accuracy first, then hypervolume gain within the top bucket. Appendix A.1 now states how duplicates are handled and describes the mutator initialization consistently with Section 4.1.2. Section 4.1.3 and Appendix C.2 detail the orchestrator's curriculum move.


Headline results on the held-out B_test (Tables 1–2):

Setting (metric, %)Best-of-3 seedBest prior searchMILO
TB2.1 / Opus 4.8 (RR@5)73.279.2 (GEPA)86.0
PaperBench / Opus 4.8 (RR@3)19.447.2 (Meta-Harness)55.6
DeepSWE / Opus 4.8 (RR@3)58.361.3 (GEPA, opt. anything)67.2
TB2.1 / gpt-oss-120b (PR@5)45.051.8 (ShinkaEvolve)56.5
PaperBench / gpt-oss-120b (PR@3)13.717.8 (Meta-Harness)20.5
DeepSWE / gpt-oss-120b (PR@3)0.91.5 (OpenEvolve)19.0

[continued in next comment box 2/2...]

AuthorsOpenReview ↗

Overall Response by Authors (Part 2/2)

[...continued from previous comment box 1/2]

Key findings from the new experiments:

  1. MILO's gains hold on held-out tasks (Tables 1–2). With Opus 4.8 as the backbone, MILO's RR@k gain over Best-of-3 is +12.8% on TB2.1 (vs. +6.0% for the best prior search), +36.2% on PaperBench (vs. +27.8%), and +8.9% on DeepSWE (vs. +3.0%). With gpt-oss-120b as the backbone, its PR@k gain is +11.5% (vs. +6.8%), +6.8% (vs. +4.1%), and +18.1% (vs. +0.6%).

  2. Lowest search cost per candidate (Tables 1–2, Section 5.5). With Opus 4.8 as the backbone, MILO achieves the lowest search cost per candidate on TB2.1 (USD 53 vs. 63–72), PaperBench (USD 79 vs. 83–119), and DeepSWE (USD 360 vs. 427–489). Mutator calls are under 5% of spend on all benchmarks, so agentic mutators add little to the cost. Most of it comes from evaluating candidates, so a cheaper harness makes the search cheaper. Among our baselines, only Meta-Harness optimizes token cost, and none optimizes latency. MILO's fitness includes both, so it increasingly explores cheaper candidates (Table 2c) and ends up with the lowest cost per candidate.

  3. Better harnesses with less exploration and lower spend (Figure 4). On TB2.1/Opus, OpenEvolve and ShinkaEvolve, which also evolve three islands, explore 140–166 candidates for USD 9.5k–10.8k, yet reach at most 76.2% RR@5 on B_test. MILO reaches 86.0% after only 121 candidates for USD 6.4k. On PaperBench/Opus, MILO reaches the highest accuracy at the lowest spend, while OpenEvolve and ShinkaEvolve explore 2.4× the candidates at 2.6× the spend, yet reach at most 19.4% RR@3. On DeepSWE/gpt-oss, MILO reaches 9.7% after only USD 65, nearly 3× the best of any other search (3.4% by OpenEvolve) over its entire run (USD 313 spend).

  4. Robust across searches and seeds (Table 3, Section 5.5). Over three independent searches on TB2.1/Opus, MILO reaches 87.0 ± 1.2% RR@5, with the smallest spread of all seven search configurations. The clearest component gains come from retained rejections and orchestration: retaining rejected candidates adds +4.3%, and MILO's orchestrator adds +3.9% over the same three islands without coordination and +6.4% over a fixed-rule coordinator. From the simplest human-written seeds (B1–B3), MILO still reaches 34.7% RR@5 on TB2.1/gpt-oss, above every prior search started from the expert-designed seeds (at most 32.1%).

  5. Transfer to unseen benchmarks (Table 4). Every transferred harness beats the seed, and two even beat all prior searches tuned on the target itself (81.1 vs. at most 79.2% RR@5 on TB2.1, and 63.2 vs. at most 61.3% RR@3 on DeepSWE). Transfer holds even from software engineering to paper replication: the TB2.1- and DeepSWE-evolved harnesses gain +13.9% and +16.7% over the seed on PaperBench.


Every request and where we addressed it. The table below covers every weakness (W) and question (Q) raised by the reviewers, and each reviewer's thread gives the detailed, point-by-point response.

RequestRaised inWhere addressed
Results on tasks never used for search, stall decisions, or selectionR1-W1, R1-Q1, R2-Q1, R3-W1, R3-Q1Section 5.1, Appendix A.2, Algorithm 1, Tables 1–2
Search cost, candidates, and compute fairnessR1-W2, R1-Q3, R2-W2, R3-W1, R3-Q1Tables 1–2, Figure 4, Section 5.5
Repeated independent searchesR1-W3, R1-Q3, R2-W1, R3-W1, R3-Q1Table 3, Section 5.5
Isolate the value of retaining rejected candidatesR1-W3, R3-W2Table 3, (C) vs. (D)
Compare the orchestrator with a simple fixed ruleR1-W3, R3-W2, R3-Q4Table 3, (E)–(G)
Accuracy-first vs. hypervolume admissionR1-Q2, R3-W2, R3-Q2Section 4.2, Eq. 7
Mutator initializationR3-W2, R3-Q3Section 4.1.2, Appendix A.1
Use of curriculum adaptationR3-W2, R3-Q3Section 4.1.3, Appendix C.2
External evidence beyond the searched tasksR1-W1, R1-Q1Table 4
Readability of Figure 1 and Tables 1–2R2-Q2Tables 1(b), 2(b–c), 3, Figure 4

We sincerely thank the reviewers again for their time and insightful feedback, which has made the paper considerably stronger. We truly hope that these revisions address the reviewers' concerns, and we would be happy to clarify any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.