- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes MILO for automated harness discovery. It combines island-based lineage memory, evidence-driven mutator agents, and an orchestrator that adapts the search strategy. Experiments on 3 benchmarks show improvements over existing harnesses and search methods. The paper also reports transfer results on Frontier-Bench and improvements on three mathematical optimization problems.
Critical strengths
- MILO automates the entire network optimization process, not just prompt tuning. Its approach, which leverages failure history and adaptive multi-agent search, shows great promise.
- The study achieved outstanding results. MILO delivered significant improvements across three benchmarks and two backbone models. Comparisons with existing search methods, cost analysis, and transfer experiments further solidify the contributions of this research.
Critical weaknesses
- The reserved validation set was repeatedly used to monitor progress and guide the coordination process (Algo 1), and to select the final test framework. This introduces a risk of overfitting. The complete benchmark results reported in the paper are not entirely independent of the search process. Also, the paper lacks a comparison of results from different independent search runs and a comparison of total search costs.
- The implementation details and component contributions are unclear. The "accuracy-first" dominance rule in section 4.2 is not equivalent to the hypervolume criterion in Eq7. The course adaptation mechanism claimed in the main text is not reflected in the disclosed coordinator action patterns (Appendix C.2). The variator initialization settings in Appendix A.1 also contradict the content in section 4.1.2. Finally, the ablation experiments neither isolated the benefits of retaining eliminated candidate solutions nor compared the coordinator with simpler intervention rules.
Questions for authors
- Can the authors report results on an unmodified test set, across independent runs, and including the total search cost?
- Which admission rule was implemented: the accuracy-priority method or the hypervolume-contribution method? In what sense are these criteria equivalent?
- Was curriculum adaptation used in the experiments? Why does the reported mutator initialization method differ from the description in the main text?
- How does the coordinator perform compared to simple heuristic interventions?
7 comments
Authors' response to Reviewer aSWY (Part 2/3)
[...continued from previous comment box 1/3]
-
(b) Independent runs ([R3-W1 + R3-Q1], continued). In Table 3, we now run MILO and each of its six ablation controls as three independent searches from the same seed harnesses (B10–B12) on TB2.1/Opus. Full MILO reaches 87.0 ± 1.2% RR@5, a gain of +13.7% over Best-of-3. Its spread is also the smallest of all seven search configurations (A–G). These error bars include both the variation across whole searches and the final-evaluation noise. In Section 5.5, we also report two MILO searches on TB2.1/gpt-oss that start from different initial harnesses. From the expert-designed seeds used throughout the paper (B10–B12), MILO reaches 36.2% RR@5 (Table 2). From the simplest human-written seeds (B1–B3, Table 7), it reaches 34.7%, only slightly below. This is still above every prior search started from the expert-designed seeds (at most 32.1% RR@5, EvoX, Table 2a).
-
(c) Total search cost. Tables 1–2 now report, for every search in all six settings, the average USD cost per candidate explored (mutation and all rollouts on B_search and B_val) and the number of candidates explored per hour. Their product over the 72-hour search gives the total spend, which we plot against B_test accuracy in Figure 4. In Section 5.5, we compare the costs directly:
- Lowest cost per candidate. With Opus 4.8 as the backbone, MILO achieves the lowest search cost per candidate on TB2.1 (USD 53 vs. 63–72), PaperBench (USD 79 vs. 83–119), and DeepSWE (USD 360 vs. 427–489). Mutator calls are under 5% of spend on all benchmarks. Most of it comes from evaluating candidates, so a cheaper harness makes the search cheaper. Among our baselines, only Meta-Harness optimizes token cost, and none optimizes latency. MILO's fitness includes both, so it increasingly explores cheaper candidates (Table 2c) and ends up with the lowest cost per candidate.
- Higher accuracy at lower spend. (i) On TB2.1/Opus, OpenEvolve and ShinkaEvolve, which also evolve three islands, explore 140–166 candidates for USD 9.5k–10.8k, yet reach at most 76.2% RR@5 on B_test. MILO reaches 86.0% after only 121 candidates for USD 6.4k (Figure 4a,b). (ii) On PaperBench/Opus, MILO reaches the highest accuracy (55.6% RR@3 on B_test vs. 47.2% by Meta-Harness) at the lowest spend, while OpenEvolve and ShinkaEvolve explore 2.4× the candidates at 2.6× the spend, yet reach at most 19.4% (Table 1, Figure 4a,b). (iii) On DeepSWE/gpt-oss, MILO reaches 9.7% after only USD 65, nearly 3× the best of any other search (3.4% by OpenEvolve) over its entire run (USD 313 spend; Figure 4c).
- A one-time cost. The total spend is paid once, whereas the discovered harness saves inference cost on every task it runs. On TB2.1/Opus, MILO's harness gains +12.8% RR@5 over Best-of-3 with 0.84× its tokens per task (Table 1b) and transfers to unseen benchmarks without further search (Table 4).
Thus, MILO finds better harnesses at a lower search cost.
[R3-W2 + R3-Q2] The "accuracy-first" dominance rule in section 4.2 is not equivalent to the hypervolume criterion in Eq7. … Which admission rule was implemented: the accuracy-priority method or the hypervolume-contribution method? In what sense are these criteria equivalent?
We apologize that the criterion was not clear in the original version, partly because some details had to move to the Appendix. The revised main paper now clarifies both the dominance rule and the child-acceptance criterion. MILO implements both as one rule, applied in a fixed order (Eq. 7, Section 4.2): accuracy decides across ε-buckets, and hypervolume gain decides within the top bucket. We bucketize accuracy at tolerance ε = 0.02 (Appendix A.1), and the top bucket is the highest one occupied by the island's admitted harnesses. A child h is then handled as follows:
- Above the top bucket: admitted, at any cost.
- Below the top bucket: rejected, at any cost.
- Within the top bucket: admitted only if it adds hypervolume, i.e., no admitted member has F ≥ F(h). Here, F is a vector that holds accuracy and the economy of each cost.
In what sense the two criteria are equivalent. The dominance relation in Section 4.2 uses the same bucket form: h dominates h′ if it lies in a higher ε-bucket of accuracy, or in the same one with F(h) ≥ F(h′). A child is therefore admitted exactly when no admitted harness dominates it. Within the top bucket, this holds exactly when its hypervolume gain is positive, and Eq. 7 writes the test in both forms. A duplicate fitness vector is dominated, which matches its zero hypervolume gain. Two accuracies closer than ε can fall in adjacent buckets. We accept this in exchange for transitivity, so that a higher accuracy bucket wins at any cost. As we note in Section 4.2, this transitive ε-box dominance lets us test only the island's Pareto front instead of its whole admitted set.
[continued in next comment box 3/3...]
Authors' response to Reviewer aSWY (Part 3/3)
[...continued from previous comment box 2/3]
[R3-W2 + R3-Q3] The course adaptation mechanism claimed in the main text is not reflected in the disclosed coordinator action patterns (Appendix C.2). … Was curriculum adaptation used in the experiments? Why does the reported mutator initialization method differ from the description in the main text?
Curriculum. Yes. CURRICULUM is one of the orchestrator's four actions (Eq. 6, Section 4.1.3), and all reported MILO results use it. In Section 4.1.3, we describe how CURRICULUM replaces the search set, B_search ← B′, favoring high-regret tasks. Over the admitted harnesses, a task's regret is the difference between the best attainable and the population's mean accuracy. The orchestration prompt in Appendix C.2 shows this move explicitly. Its stall taxonomy maps a whole-population plateau to curriculum, its move (iii) is CURRICULUM with the same regret as in Section 4.1.3, and its JSON schema has a curriculum field. Its rules require every curriculum task to be in the search split, so B_val and B_test never enter the curriculum.
Mutator initialization. We would like to clarify that, before round r = 1, m^(1) assigns each island a mutator drawn uniformly at random from G (Section 4.1.2), which the orchestrator may reassign in later rounds. The orchestrator itself is a Claude Code agent. We now state this explicitly in Appendix A.1.
[R3-W2 + R3-Q4] … the ablation experiments neither isolated the benefits of retaining eliminated candidate solutions nor compared the coordinator with simpler intervention rules. … How does the coordinator perform compared to simple heuristic interventions?
Following the reviewer's insightful suggestion, we now add both controls to Table 3 and run each configuration as three independent searches.
Retained rejections. The ablation (Table 3) now separates the lineage structure from the retained rejections:
- (B) flat archive, agentic mutator: 77.0 ± 2.3%
- (C) + lineage memory, rejections discarded: 77.5 ± 2.4%
- (D) + rejections retained: 81.8 ± 2.0%
As we report in Section 5.5, lineage memory alone adds only +0.5% (B → C), but retaining rejections adds a further +4.3% (C → D) by pruning weak designs. As we explain in Section 1, failures in the memory prune later mutations and push exploration to bolder structural changes.
A simple heuristic coordinator. In configuration (F) of the ablation (Table 3), we replace MILO's orchestrator with a simple fixed rule: periodic best-harness migration across islands on a fixed schedule, as in OpenEvolve.
- (E) + three islands, no coordination: 83.1 ± 1.6%
- (F) three islands, fixed-rule coordinator: 80.6 ± 2.4%
- (G) three islands, MILO's orchestrator: 87.0 ± 1.2%
MILO's orchestrator outperforms the fixed rule by +6.4% (F → G) and no coordination by +3.9% (E → G). The fixed rule itself lowers RR@5 (−2.5% vs. E). From its lineages, we attribute this to lost diversity: the islands quickly converge to local optima (Section 5.5) and eventually cannot escape them. (F) differs from MILO in both the timing (on a schedule rather than at stalls) and the type of interventions (migration only). So Table 3 shows that MILO's stall-triggered orchestrator outperforms both no coordination and a standard migration rule.
As requested by the reviewer, we have added results on a separate test set, across independent runs, and with the total search cost. MILO remains first in all six settings on B_test. We have also clarified the admission rule, the use of curriculum, and the mutator initialization, and added the two requested ablation controls. We very much hope that these results resolve the reviewer's main concerns about held-out evaluation, search variance, search cost, and the method details. We greatly appreciate the reviewer's encouraging assessment of MILO's approach and results, and believe that these revisions and additional experiments have substantially strengthened the paper.
If the reviewer feels that these revisions merit it, we would be very grateful for any appropriate adjustment to the evaluation score. We would also be happy to clarify any remaining questions and thank the reviewer again for their time and thoughtful engagement with our work.
Authors' response to Reviewer aSWY (Part 1/3)
We sincerely thank the reviewer for the thoughtful review. We are especially encouraged that the reviewer recognizes the promise of MILO's approach beyond prompt tuning and finds its results outstanding. We have carefully addressed each of the reviewer's concerns with new experiments and corresponding revisions to the paper. The feedback also helped us clarify aspects of both the evaluation and the method description.
Below, we respond to each point under the reviewer's Weaknesses (W) and Questions (Q), with corresponding revisions highlighted in blue in the revised PDF. We very much hope these revisions address the reviewer's concerns and further strengthen the paper.
[R3-W1 + R3-Q1] The reserved validation set was repeatedly used to monitor progress … and to select the final … The complete benchmark results … are not entirely independent of the search process. … Can the authors report results on an unmodified test set, across independent runs, and including the total search cost?
We thank the reviewer for this crucial point. For our experiments, we indeed partition each benchmark into three: B_search (used by the mutators to propose better harnesses), B_val (used to monitor progress, guide the orchestration, and select the final harness), and B_test (a separate set that remains completely inaccessible to the search). In the initial submission, we reported results on the full benchmark. Following the reviewer's suggestion, the revised paper now reports results only on B_test and across independent searches, and we also report the total search cost.
(a) Main results only on a test set (B_test) that remains inaccessible during search. In Section 5.1, we describe how each benchmark (TB2.1, PaperBench, and DeepSWE) is split into B_search, B_val, and B_test in a 30:10:60 ratio. The partitioning is group-disjoint, ensuring that closely related tasks never end up in multiple partitions, and it is stratified by difficulty (Appendix A.2). In Section 5.1, we also state that B_test is held out and used only to score the final output harness of each search, for MILO and all SoTA search methods alike. Algorithm 1 (line 19) adds that the whole search uses only B_search and B_val, while B_test stays hidden. So any overfitting to B_val cannot inflate the reported results.
Tables 1(a–b), 2(a), 3, and 4 now report results only on B_test. As we now state in the abstract, on held-out B_test tasks, MILO-discovered harnesses rank first in all six settings (3 benchmarks × 2 backbone LLMs) against eight SoTA harnesses and six SoTA evolutionary search frameworks. With Opus 4.8 as the backbone, MILO's RR@k gain over Best-of-3 is +12.8%, +36.2%, and +8.9% on TB2.1, PaperBench, and DeepSWE, vs. +6.0%, +27.8%, and +3.0% for the best prior search. On TB2.1/Opus, MILO's task-clustered interval does not overlap with that of the strongest prior search, GEPA (86.0 ± 2.5 vs. 79.2 ± 2.7%).
Transfer to unseen benchmarks (Table 4). We also report the reusability of all MILO-evolved harnesses: in Table 4, we evaluate each benchmark's final harness, without further search, on the B_test of every other benchmark and on the never-searched Frontier-Bench. RR@k (%), with the gain over the seed:
As we report in Section 5.5:
[continued in next comment box 2/3...]