acceptodds
Under review as a conference paper at ICLR 2027

Harness-MAP: Budget-Efficient Harness Search via Structure-Token Factorization

Abstract

Automating harness evolution via naive meta-agent search is prohibitively expensive, requiring hundreds of meta-agent calls and minutes-to-hours per evaluation. How can we efficiently search for a diverse set of high-quality harnesses under a tight evaluation budget? To address this, we introduce Harness-MAP to structure the search space, factorizing graph-level program structures from token-level implementations. Bilevel search over this structure-token representation space illuminates an archive of diverse, high-quality harnesses. The map also enables a resolution-adaptive search strategy that navigates this space efficiently, sub-splitting partitions with high fitness variance while retracting ineffective refinements. Across three interactive spatial navigation benchmarks with the frozen Qwen3-VL-30B-A3B-Instruct backbone model, our search approach rapidly achieves performance leaps under minimal evaluation budgets: on SpatialWorld-3dgames (39 evals), on BALROG-MiniHack (41 evals), and on SimWorld-Robotics (6 evals). Harness-MAP outperforms prior harness evolution methods across both search and held-out tasks, establishing factorized representations as an effective Quality-Diversity paradigm for budget-aware agent harness search.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-Evolving Agents, Agent Harness, Quality-Diversity Search

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7PhLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes to do the efficient search for a diverse set of Harness with a tight budget and the method Harness-MAP demonstrates good performance on different tasks. Harness-MAP tries to use a descriptor to make each candidate harness on a map and keep the best harness in each cell, which also proposes a method for doing the search. Harness-MAP demonstrates great performances compared with baselines.

Critical strengths
  1. The search loop shows great performance on benchmarks.
  2. This paper provides detailed information for each task, which is helpful for people to continue to work on this method.
Critical weaknesses
  1. It looks like that the gains are not from the claimed main contributions. For example, the adaptive resolution shows no help for the unsaturated benchmark.
  2. The role of VLM is overstated.
Questions for authors
  1. It is better if you report the held-out results with k0 fixed in advance and over >= 3 independent search runnings.
  2. If there is no archive, how is the performance of Harness-MAP?
  3. Which meta-agent produced each Table 1 entry? Do all baselines get the same meta-agent?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 7PhL (PART 1/6)

Thank you for your constructive review and for recognizing the novelty of our map-based harness search loop, its strong performance, and our detailed implementation documentation. Per your suggestions, we expanded on the details of our component ablations, multi-seed held-out evaluations, meta-agent consistency, and the role of VLM calls. All corresponding revisions are highlighted in blue in the updated manuscript.

[W1] Attribution of performance gains to the proposed structured representation map and adaptive-resolution search designs.

We respectfully disagree with this assessment because controlled ablations and experiments on 3 resolution modes confirm that our gains come directly from both core designs across saturated and unsaturated benchmarks.

[A1-1] Structured Representation Map (φS,φT\varphi_S, \varphi_T)

Controlled ablations confirm that the factorized structure-token space and QD archive are essential for search accuracy and efficiency (Tab. 3[original-result]). Replacing OpenEvolve's surface-text descriptor with our structure-token CVT space (OpenEvolve_st-cvt) raises SR on SpatialWorld-3DGames (k0=8k_0 = 8) from 21.9% to 71.4% under the same search loop. Swapping our descriptor for OpenEvolve's 2D descriptor (Harness-MAP_cd) lowers performance from 100.0% to 96.2%, more than doubles Rubik evaluations (14 to 37), and also causes coverage failure on Maze3D. Removing Weisfeiler-Lehman features (Harness-MAP_w/o WL) drops SR to 97.6% and triples Block3D evaluations (5 to 16), while removing the QD archive (Harness-MAP_w/o archivein Tab. R1[new-result]) drops SR to 84.0% and Block3D from 100.0% to 35.0%.

Tab. R1. Component and descriptor ablation results on SpatialWorld-3DGames (k0=8k_0 = 8, reporting SR% / N* / T*).

VariantMazeMaze3DBlock3DRubikSnakeWtd. Avg. (SR%)
Harness-MAP (Full)100.0 / 12 / 2100.0 / 45 / 7100.0 / 5 / 1100.0 / 14 / 2100.0 / 2 / 0100.0
Harness-MAP w/o WL100.0 / 12 / 290.0 / 28 / 16100.0 / 16 / 4100.0 / 27 / 4100.0 / 7 / 197.6
Harness-MAP_cd100.0 / 14 / 484.0 / 23 / 4100.0 / 18 / 3100.0 / 37 / 7100.0 / 8 / 196.2
Harness-MAP w/o archive100.0 / 7 / 185.0 / 30 / 535.0 / 30 / 5100.0 / 39 / 6100.0 / 30 / 584.0
OpenEvolve60.0 / 37 / 365.0 / 50 / 4920.0 / 50 / 4915.0 / 50 / 4910.0 / 50 / 4921.9
OpenEvolve_st-cvt100.0 / 6 / 5100.0 / 6 / 548.0 / 50 / 4915.0 / 50 / 49100.0 / 26 / 2571.4

Dynamic Weisfeiler-Lehman representations capture substantial architectural variance on unsaturated tasks without suffering from high-dimensional sparsity (Tab. 14 [new-result] and Tab. R2[new-result]). While φT\varphi_T stays fixed at 786 dimensions, the WL structural block φS\varphi_S expands dynamically, e.g., from 118 to 211 keys on SimWorld-Robotics-Hard, where φS\varphi_S accounts for 10% to 40% of total descriptor variance and becomes the largest block at k0=4k_0 = 4 (40%), separating distinct program architectures that token features alone would merge.

Tab. R2. Share of descriptor variance per block on SimWorld-Robotics-Hard.

k0k_0Structure φS\varphi_Sastnumdoc
210%43%36%11%
440%21%17%22%
819%43%25%12%
AuthorsOpenReview ↗

Response to Reviewer 7PhL (PART 2/6)

[A1-2] Resolution-Adaptive Search

Adaptive resolution eliminates sensitivity to initial grid granularity k0k_0 on saturated benchmarks (Fig. 9[new-figure], Tab. 10[original-result]). As shown in Fig. 9, adaptive resolution (K_adaptive) is far more robust across different k0k_0 than static (K_fixed) and scheduled (K_hier) modes on SpatialWorld-3DGames.

Adaptive partitioning enables steadier and more robust optimization across unsaturated benchmarks. Across unsaturated SimWorld-Robotics-Hard (Tab. R3[new-result]) and BALROG-MiniHack (Tab. 12[orignial-result], Tab. R4[new-result]), K_adaptive consistently ranks 1st or 2nd best across varying initial resolutions, proving largely insensitive to the choice of k0k_0. In contrast, static K_fixed and scheduled K_hier exhibit brittle performance drops under suboptimal granularities, forcing practitioners to run multiple expensive trials to tune k0k_0 and wasting substantial rollout compute. Plus, partition retention/reversion rates across all benchmarks (§A.7, Tab. 13[new-result]) futher confirm this self-correcting robustness, where 66.2% (45/68) of attempted sub-splits beat their pre-split parent baselines and are retained.

Tab. R3. Resolution ablation on unsaturated SimWorld-Robotics-Hard (10/2 search/val split, SubSR, N=16N = 16).

k0k_0ModeBest@4Best@8Best@12Best@16
2adaptive0.500‾\underline{0.500}0.575‾\underline{0.575}0.6250.650‾\underline{0.650}
2fixed0.6250.6250.6250.700
2hier0.4750.4750.475‾\underline{0.475}0.550
4adaptive0.525‾\underline{0.525}0.525‾\underline{0.525}0.525‾\underline{0.525}0.575
4fixed0.5500.5500.5500.550‾\underline{0.550}
4hier0.4500.4500.4500.475
8adaptive0.4250.450‾\underline{0.450}0.5250.525‾\underline{0.525}
8fixed0.350‾\underline{0.350}0.5250.5250.525‾\underline{0.525}
8hier0.4250.4250.500‾\underline{0.500}0.625

Tab. R4. Resolution ablation on unsaturated BALROG-MiniHack (SR%, N=50N = 50).

k0k_0ModeBest@10Best@20Best@30Best@40Best@50
2adaptive67.575.075.075.075.0
2fixed30.0‾\underline{30.0}62.5‾\underline{62.5}67.5‾\underline{67.5}67.5‾\underline{67.5}72.5‾\underline{72.5}
2hier12.535.062.567.5‾\underline{67.5}67.5
4adaptive62.5‾\underline{62.5}72.577.5‾\underline{77.5}80.0‾\underline{80.0}85.0
4fixed72.587.587.587.592.5
4hier57.585.0‾\underline{85.0}87.587.590.0‾\underline{90.0}
8adaptive32.557.580.0‾\underline{80.0}80.0‾\underline{80.0}82.5
8fixed62.5‾\underline{62.5}82.5‾\underline{82.5}87.587.587.5‾\underline{87.5}
8hier77.585.087.587.592.5
16adaptive37.537.560.0‾\underline{60.0}87.590.0
16fixed32.560.0‾\underline{60.0}60.0‾\underline{60.0}62.562.5‾\underline{62.5}
16hier35.0‾\underline{35.0}80.080.080.0‾\underline{80.0}90.0
32adaptive55.065.065.072.5‾\underline{72.5}72.5‾\underline{72.5}
32fixed35.0‾\underline{35.0}42.565.075.075.0
32hier30.050.0‾\underline{50.0}62.5‾\underline{62.5}65.072.5‾\underline{72.5}

Dynamic cell refinement sustains high archive coverage and yields superior QD-scores across intermediate resolutions. In Tab. 2[original-result], K_adaptive achieves higher Quality-Diversity scores at intermediate resolutions k0∈{4,8,16}k_0 \in \{4, 8, 16\} and preserves superior archive coverage, while baselines collapse at finer resolutions.

In sum: Ablations show the structure-token map drives the gains, lifting OpenEvolve from 21.9% to 71.4% while removing any of its components lowers Harness-MAP from 100.0% to 84.0–97.6% (Tab. 3). Adaptive resolution mitigates k0k_0 sensitivity on saturated benchmarks and stays robust with effective sub-partitioning on unsaturated ones (Fig. 9, Tab. R3–R4).

AuthorsOpenReview ↗

Response to Reviewer 7PhL (PART 3/6)

This comment "The role of VLM is overstated" is every short. We do not see precisely what this weakness is referring to. Could you please elaborate or rephrase this point? At the meantime, we address the interpretation regarding why some top-performing harnesses discovered by Harness-MAP can operate without VLM calls.

[W2] Clarification of the role of non-VLM and VLM harnesses.

[A2-1] Harness-MAP formulates harness evolution as executable program search where mandating a VLM call at every step could be an optional rule for the meta-agent to make VLM-centric proposals, even though we didn't do it.

Program search discovers whichever orchestration of symbolic code and VLM calls best solves the task. Technically, harnesses optimize control flow around a frozen backbone, but deterministic solvers handle the transitions cleanly, and skipping redundant VLM calls improves reliability and efficiency, which is totally reasonable.

[A2-2] Across three benchmarks, Harness-MAP discovers a spectrum of zero-VLM, hybrid, and VLM-centric harnesses depending on environment structure. We have found the following emergent patterns:

  1. Zero-VLM symbolic solvers on structured grid tasks. In BALROG-MiniHack, incremental modular edits produce deterministic code solvers such as BFS planners.
  2. Conditional VLM workflows on hybrid reasoning tasks. Harness-MAP also discovers selective-querying harnesses that invoke the frozen VLM only when deterministic rules cannot decide.
  3. Per-step VLM workflows on perceptual grounding tasks. When visual grounding or situational routing is required, discovered harnesses invoke the VLM on every step.

Because different environments vary in how much perceptual grounding versus combinatorial search they require, discovering both zero-VLM solvers and VLM-driven workflows reflects adaptive task specialization.

In sum: Our structure-token program search naturally discovers both zero-VLM solvers and VLM-centric workflows tailored to environment demands. Please let us know if your comment targeted another point.

Reviewer_2WbTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Harness-MAP searches for agent harnesses under a limited budget of candidate evaluations. It combines a structural descriptor based on Weisfeiler–Lehman refinement with a token-level behavior descriptor to organize a CVT-MAP-Elites archive. The best candidate in each occupied cell is retained, and an LLM meta-agent proposes edits using execution feedback and component-aware directives. During search, the occupied cell with the largest fitness range is selected for subdivision, and unproductive splits are reverted. Experiments use the frozen Qwen3-VL-30B-A3B-Instruct backbone on SpatialWorld-3DGames, BALROG-MiniHack, and SimWorld-Robotics. The paper reports improved search efficiency and examines descriptor choices, resolution strategies, and performance on held-out tasks.

Critical strengths
  1. The structure-token descriptor is used both to organize the archive and to guide edits. The archive can refine individual cells and revert subdivisions that do not improve upon the parent elite. These changes reuse cached descriptors and fitness values, avoiding additional task rollouts for repartitioning.
  2. Harness-MAP finds competitive harnesses early in the search. Table 7 reports 71.0% on SpatialWorld-3DGames after 10 candidate evaluations with k₀ = 4, exceeding Meta-Harness's 68.6% after 50. On BALROG-MiniHack, Harness-MAP reaches 75.0% after 20 evaluations with k₀ = 2, matching Meta-Harness's final score.
Critical weaknesses
  1. Table 3 shows an advantage for the full descriptor, but does not isolate the contribution of the WL structural features. Harness-MAP_cd replaces the entire representation with OpenEvolve's two-dimensional descriptor of code complexity and diversity. This also changes the implementation features, so the result does not establish whether the structural descriptor improves search beyond the AST, numeric, and documentation features.
  2. Table 5 does not specify which resolution strategies and selected harnesses were used for the held-out evaluation. Section 7 reports 90.0% search success for adaptive Harness-MAP at k₀ = 16, while Table 5 lists 62.5% at that initial resolution. These may be different configurations, but the held-out results for the 90.0% harness cannot be identified. The results also vary by split. Harness-MAP's best reported Holdout A score is 75.0%, compared with Meta-Harness's 67.5%, while its best Holdout B score is 65.0%, compared with 72.5%. The abstract should qualify its claim of better held-out performance.
Questions for authors
  1. How does Harness-MAP perform without the WL structural descriptor, while retaining the AST, numeric, and documentation features under the same search settings?
  2. Which resolution strategies and selected harnesses were evaluated in Table 5? What are the Holdout A and Holdout B results for the adaptive BALROG-MiniHack harness that reaches 90.0% search success at k₀ = 16?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 2WbT (PART 1/2)

Thank you for your constructive review and for recognizing the novelty of our structure-token archive with zero-rollout adaptive partitioning and its early search efficiency. Per your suggestions, we have added the Weisfeiler-Lehman descriptor ablation, and clarified the evaluation protocol in Tab. 5 with updated held-out results. All corresponding revisions are highlighted in blue in the updated manuscript.

[W1, Q1] Ablations on the Weisfeiler-Lehman (WL) structural descriptor.

[A1-1] Removing the WL structural descriptor (φS\varphi_S) degrades task success rate and sample efficiency on complex tasks. To isolate φS\varphi_S under identical search settings (k0=8k_0=8, N=50N=50 evaluations on SpatialWorld-3DGames), we added Harness-MAPw/o WL\text{Harness-MAP}_{\text{w/o WL}} (Tab. 3[updated-result], §A.3). This variant with Harness-MAP search loop retains the full token descriptor while removing the graph refinement block φS(s)\varphi_S(s). Result-wise, weighted average success rate drops from 100.0% to 97.6%, and evaluation cost that fist reaches their best scores rises from 5 to 16 on Block3D, 14 to 27 on Rubik, and 2 to 7 on Snake (Tab. R1[new-result]).

Mechanistically, two-round WL refinement encodes role-aware multi-hop call graphs (such as distinguishing whether Perception routes directly to Controller or through a Planner). Without φS\varphi_S, harnesses with distinct control-flow topologies but similar token distributions collapse into identical Voronoi cells and reduce structural exploration diversity.

Furthermore, our contribution of the joint structure-token representation is also validated by comparing it against alternative descriptors Harness-MAPcd\text{Harness-MAP}_{\text{cd}}, and by equipping a prior MAP-Elites method with our descriptor space (OpenEvolvest-cvt\text{OpenEvolve}_{\text{st-cvt}}) (Tab. 3).

Tab. R1. Descriptor ablation results (SR/N/T) with SR first reaching at N/T.

Harness-MAP Loop with different descriptorsMazeMaze3DBlock3DRubikSnakeWeighted Avg. SR
Harness-MAP (Full)100.0 / 12 / 2100.0 / 45 / 7100.0 / 5 / 1100.0 / 14 / 2100.0 / 2 / 0 ‡\ddagger100.0
Harness-MAPw/o WL\text{Harness-MAP}_{\text{w/o WL}}100.0 / 12 / 290.0 / 28 / 4100.0 / 16 / 2100.0 / 27 / 4100.0 / 7 / 197.6
Harness-MAPcd\text{Harness-MAP}_{\text{cd}}100.0 / 14 / 484.0 / 23 / 4$\dagger$100.0 / 18 / 3100.0 / 37 / 7100.0 / 8 / 196.2

†\dagger CovErr indicates that OpenEvolve's descriptor could not fill 50% of k0k_0 on Maze3D, so this cell ran at a 25% target. ‡\ddagger Saturated during initialization before the first search step.

[A1-2] Dynamic WL vocabularies scale with codebase complexity while occupying low-dimensional subspaces. Across all three benchmarks, candidates map into the joint structure-token space (§3.2, §A.8). While φT\varphi_T stays fixed at 786 dimensions, it expands dynamically as search discovers novel inter-module wiring patterns on SpatialWorld-3DGames and SimWorld-Robotics (Tab. R2, §A.8, Tab. 14[new-result]). The representation populates smooth low-dimensional subspaces, enabling Centroidal Voronoi Tessellation to partition the space effectively without high-dimensional sparsity issues.

Tab. R2. Representation dimensionality across benchmarks (§A.8, Tab. 14).

BenchmarkNominal Dim.Structural Block (φS\varphi_S)Token Block (φT\varphi_T)Dominant Feature
SpatialWorld-3DGames844 to 2,36658 to 1,580 dynamic WL keys (scales with k0k_0)786 dims (15AST+3num+768doc15_{\text{AST}} + 3_{\text{num}} + 768_{\text{doc}})Balanced: doc 29 to 76%; structure φS\varphi_S 0 to 36%
SimWorld-Robotics904 to 997118 to 211 dynamic WL keys (scales with k0k_0)786 dims (15AST+3num+768doc15_{\text{AST}} + 3_{\text{num}} + 768_{\text{doc}})Balanced: structure φS\varphi_S 10 to 40%; ast largest at k0=2/8k_0 = 2/8 (43%); doc 11 to 22%

On SimWorld-Robotics-Hard, the structure block carries 10 to 40% of the descriptor variance among evolved harnesses and is the largest block at k0=4k_0=4 (Tab. R3[new-result]), so φS\varphi_S separates candidates that the token block alone would place together.

Tab. R3. Proportion of descriptor variance per block on SimWorld-Robotics-Hard.

k0k_0Structure φS\varphi_SastnumdocLargest block
210%43%36%11%ast
440%21%17%22%structure φS\varphi_S
819%43%25%12%ast

In sum, the Harness-MAPw/o WL\text{Harness-MAP}_{\text{w/o WL}} ablation (Tab. R1) and cross-benchmark dimensionality analysis (Tab. R2) show that WL structural features provide essential topological discrimination beyond token-level features, improving both task success rate and sample efficiency.

AuthorsOpenReview ↗

Response to Reviewer 2WbT (PART 2/2)

[W2, Q2] Clarifying Tab. 5 configurations, resolution strategies, and held-out performance.

[A2-1]: Tab. 5 evaluates the best search-split harness from each adaptive-resolution search. For every method and initial resolution k0k_0, we evaluate the single best-performing search-split candidate on the held-out splits. All Harness-MAP entries in the Tab. 5[updated-result] correspond directly to the adaptive runs in Tab. 1 under the unified (P,C,m)(P, C, m) schedules in Tab. 8[new-result] (Tab. R3a).

Table R3a. Search configurations of Harness-MAP across settings, where P=log⁡2k0P = \log_2 k_0 parents per generation, CC children per parent, coverage fraction mm, and evaluation budget NN.

Settingk0k_0PPCCmmNN
SpatialWorld-3DGames2 / 4 / 8 / 16 / 321 / 2 / 3 / 4 / 52 / 2 / 1 / 1 / 10.550
BALROG-MiniHack2 / 4 / 8 / 16 / 321 / 2 / 3 / 4 / 52 / 2 / 1 / 1 / 10.550
SimWorld-Robotics-Easy2 / 4 / 81 / 2 / 32 / 2 / 20.516
SimWorld-Robotics-Hard2 / 4 / 81 / 2 / 32 / 1 / 10.516

[A2-2]: Corrected BALROG-MiniHack held-out results under consistent (P,C)(P, C) configurations. The earlier MiniHack held-out results in Tab. 5[previous-result] were from a very preliminary experiment with different batch PP settings where P=1P=1. Our final results in Tab.1[original-result] are under the consistent adaptive configuration P=log⁡2k0P = \log_2 k_0, C=2C=2, or C=1C=1 for larger resolutions to save more chances to evolve. The improvement from the batched evolution also proved the importance of keeping a widely diverse archive per sequential step.

Table R3b. Updated BALROG-MiniHack generalization performance (from revised Tab. 5) under KadaptiveK_{\text{adaptive}}.

MethodSearch (SR%)Holdout A (SR%)Holdout B (SR%)
Qwen3-VL-30B (Direct)10.0 ± 4.710.0 ± 4.70.0 ± 0.0
Meta-Harness75.0 ± 6.867.5 ± 7.472.5 ± 7.1
GEPA30.0 ± 7.220.0 ± 6.330.0 ± 7.2
DGM72.5 ± 7.165.0 ± 7.555.0 ± 7.9
ShinkaEvolve20.0 ± 6.310.0 ± 4.717.5 ± 7.1
OpenEvolve27.5 ± 7.115.0 ± 5.617.5 ± 6.0
Harness-MAP (k0=2k_0=2)75.0 ± 6.875.0 ± 6.865.0 ± 7.5
Harness-MAP (k0=4k_0=4)85.0 ± 5.657.5 ± 7.885.0 ± 5.6
Harness-MAP (k0=8k_0=8)82.5 ± 6.075.0 ± 6.860.0 ± 7.7
Harness-MAP (k0=16k_0=16)90.0 ± 4.772.5 ± 7.147.5 ± 7.9
Harness-MAP (k0=32k_0=32)72.5 ± 7.160.0 ± 7.752.5 ± 7.9

In sum: Tab. 5 evaluates the top search-split elite from each adaptive run under the unified configurations in Tab. 8 (Tab. R3a).

Thank you again for your valuable feedback. We hope this clarification resolves your concern.

Thank you for the clarifications. I will keep my original score.

Reviewer_P1E2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Harness-MAP represents agent harnesses using graph structure and implementation descriptors, partitions this space through CVT, and uses a MAP-Elites-style archive with LLM diagnosis and directed mutation. An adaptive procedure splits high-fitness-variance cells and reverts unproductive subdivisions. The paper reports large search-set improvements on SpatialWorld-3DGames, BALROG-MiniHack, and SimWorld navigation under a limited number of candidate evaluations.

Critical strengths
  1. A structured search representation is a sensible contribution. Combining component connectivity, control-flow features, numeric statistics, and documentation embeddings gives the archive a more informative coordinate system than raw code length. Table 3's descriptor transplant into OpenEvolve produces a substantial reported gain, supporting practical value in the representation.

  2. The search-set gains are large. Table 1 reports 100% on SpatialWorld and 90% on MiniHack against 68.6% and 75% for Meta-Harness. The qualitative analysis explains that search can discover algorithmic modules and recombine them, providing a plausible mechanism for improvement.

Critical weaknesses
  1. The adaptive split procedure is incomplete as stated. Algorithm 2 first partitions cached members and then immediately reverts if the maximum child-cell fitness is no larger than the parent elite. If this comparison uses the same cached candidates, the best child cannot exceed the already best parent, so every such split will revert. If the implementation allows new evaluations before checking improvement, it needs an explicit grace period, budget allocation, and delayed test. The geometric split itself can be free of rollout cost, but demonstrating that the split is productive requires subsequent search. This distinction is central to the proposed mechanism.

  2. The budget unit and initialization formula are inconsistent. Section 4 describes N as the total task-level executions, while Tables 6–7 and Appendix A.5 treat N as harness evaluations, each potentially involving multiple episodes. Equation 9 counts initialization as exactly ceil(m k0). However, Algorithm 1 evaluates valid proposals until that many cells are occupied: evaluated candidates landing in an already occupied cell also consume budget unless filtered before rollout. Invalid proposals and LLM validity checks consume resources as well. The paper needs actual counters for candidate evaluations, task episodes, all initialization attempts, and proposer/verifier costs. Table 6 also runs baselines serially while the proposed method batches candidates, so sequential-step superiority alone does not establish wall-clock or dollar efficiency.

Questions for authors
  1. What is the actual number of task episodes and all evaluated initialization proposals, including collisions? How much proposer/verifier compute and tuning compute did each method consume?

  2. When is a split evaluated for retention relative to new candidate generation? Please reconcile the implementation with Algorithm 2's immediate reversion condition.

3 comments
AuthorsOpenReview ↗

Response to Reviewer P1E2 (PART 1/3)

Thank you for your constructive review and for recognizing the novelty of our structured search representation and its strong search-set gains. Per your suggestions, we have clarified the deferred split-retention mechanism, unified the evaluation budget definitions, and added full empirical accounting across all search loops. All corresponding revisions are highlighted in blue in the updated manuscript.

[W1, Q2] Timing of split retention evaluation before new candidate generation and how the budget allocation works.

[A1-1] Harness-MAP strictly defers retention testing across a grace period of τ=1\tau = 1 breeding turn so that newly bred candidates can populate and compete inside child sub-cells before split productivity is judged. We have refined §3.4, Alg. 2, and §A.5 to make this deferred mechanism explicit in both the text and pseudocode:

Sub-splitting defers retention testing across a grace period of τ=1\tau = 1 breeding turn with zero extra rollout budget. Technically,

New splits are not judged immediately on cached members at creation. When a high-variance cell c∗c^* is partitioned into ksplit=2k_{\text{split}} = 2 sub-cells (up to maximum depth dmax⁡=2d_{\max} = 2), it is registered as an open split with turn counter tc=0t_c = 0 and pre-split elite baseline (Alg. 2, Line 5). Because retention testing is gated on tc≥τt_c \ge \tau (we chose τ=1\tau = 1) (Alg. 2, Line 7), evaluation will therefore wait until subsequent breeding occurs.

Child sub-cells require zero extra rollout budget for testing. Child sub-cells enter the standard archive pool and compete for the regular PP parent-selection slots per generation under the exact same selection rule as all other cells.

Retention is tested after breeding, and unproductive splits merge back with memory. Selecting any sub-cell within an open split as a parent increases tc←tc+1t_c \leftarrow t_c + 1. Once tc≥τt_c \ge \tau, ADAPT_RESOLUTION retains the split if any leaf sub-cell beats the pre-split baseline. Otherwise, the sub-cells merge back without losing any members and store the parent's sorted member score tuple (Eq. 12), preventing redundant re-splits until a new candidate alters the cell's score distribution.

[A1-2] To reconcile Alg. 1(b) and Alg. 2, every search generation tt executes in the following order:

  1. Select parents. Sample PP occupied cells, and each selection inside an open split's region increases its counter tc←tc+1t_c \leftarrow t_c + 1.
  2. Evaluate children. Each selected parent breeds CC children, which undergo task rollouts (the sole source of evaluation cost).
  3. Assign to Voronoi cells. Evaluated candidates map to their nearest centroid via φ(H)\varphi(H), populating newly created sub-cells.
  4. Sub-split a high-variance cell. At most one eligible cell with the widest fitness range is partitioned via zero-cost local CVT and initialized with tc=0t_c = 0.
  5. Judge mature splits. Only open splits with tc≥τt_c \ge \tau are tested for retention or reversion.

[A1-3] Across all benchmarks, 66.2% (45/68) of attempted sub-splits succeed and are retained (§A.7, Tab. 13[new-result]). To verify that splits do not revert by construction, we report the full partition trajectories across all benchmarks. Tab. 13 shows that 45/68 (66.2%) attempted sub-splits surpassed their pre-split parent baselines and were retained, which further demonstrates the effectiveness of dynamic-resolution search.

In sum: Geometric splitting incurs zero rollout cost at tc=0t_c = 0, retention testing is deferred until tc≥τt_c \ge \tau (τ=1\tau = 1) after standard breeding and evaluation, and Tab. 13 confirms high empirical retention rates across benchmarks.

AuthorsOpenReview ↗

Response to Reviewer P1E2 (PART 2/3)

[W2, Q1] Budget-aware evaluation protocol, counts of task episodes, and proposer compute.

[A2-1] Number of task-level executions equals candidate-harness evaluations (NN). One budget unit (NN) counts candidate-harness evaluations on the fixed search set. We clarified in §A.4 so that NN unambiguously denotes full candidate-harness evaluations instead of individual episodes. One evaluation runs 20 or 25 episodes on each game of SpatialWorld-3DGames, 40 episodes on BALROG-MiniHack (N=50N=50), and 50 episodes on SimWorld (N=16N=16).

[A2-2] Initialization proposals do not waste evaluations because candidates are assigned to partitions using offline structure-token representations (φ\varphi). Rather than evaluating proposals until ⌈mk0⌉\lceil m k_0 \rceil cells fill, φ\varphi maps each proposed starter from its offline structure-token features first and discards any collision (landing in an already occupied cell) at zero rollout cost. Only newly covered cells undergo rollouts (M=⌈mk0⌉M = \lceil m k_0 \rceil, baseline harnesses included), reopening a cell only on invalid evaluation. We report the proposal counters during initialization in Tab. R1[new-result] across all benchmarks, where zero initialization evaluations failed validity.

Table R1. Initialization counters for Harness-MAP adaptive (m=0.5m = 0.5). 3DGames rows report means over 5 games and other rows report single runs.

Settingk0k_0Target ⌈mk0⌉\lceil m k_0 \rceilInit proposalsCollisions (discarded pre-rollout)Init evaluations(M=⌈mk0⌉M= \lceil m k_0 \rceil)
3DGames, Codex2 / 4 / 8 / 16 / 321 / 2 / 4 / 8 / 160 / 1.2 / 6.6 / 35.0 / 54.60 / 0.2 / 3.6 / 28.0 / 39.61 / 2 / 4 / 8 / 16
3DGames, Claude2 / 4 / 8 / 16 / 321 / 2 / 4 / 8 / 160 / 1.0 / 3.0 / 9.0 / 18.40 / 0 / 0 / 2.0 / 3.01 / 2 / 4 / 8 / 16
MiniHack2 / 4 / 8 / 16 / 321 / 2 / 4 / 8 / 160 / 9 / 19 / 35 / 670 / 8 / 16 / 28 / 521 / 2 / 4 / 8 / 16
SimWorld-Easy2 / 4 / 81 / 2 / 40 / 2 / 90 / 1 / 61 / 2 / 4
SimWorld-Hard2 / 4 / 81 / 2 / 40 / 1 / 40 / 0 / 11 / 2 / 4
AuthorsOpenReview ↗

Response to Reviewer P1E2 (PART 3/3)

[A2-3] All runs executed sequentially, and Tab. R2a–R2b[new-result] report exact episodes, meta-agent calls, and wall-clock hours.

In our actual implementation, we evaluate candidates one at a time according to a serialized queue. It is equivalent to seralizing siblings sharing the same depth in a Tree structure into a sequential order. Therefore our reported sequential-step efficiency faithfully represent the true wall-clock or dollar efficiency.

Sequential steps TT measure parallel proposal depth, and the batch size affects archive informativeness per generation. It is the harness evaluation budget NN spent that directly decides the wall-clock search time, Because task rollouts dominate runtime in complex embodied environments (e.g., 10.9 h/eval on 50 episodes of SimWorld-Robotics-Hard), finding an optimal harness with fewer evaluations directly reduces search hours. Every method evaluated one candidate at a time via a single worker and Tab. 7-8 [new-setup] reflects parallel depth. We run all the experiments using Claude Max subscription plans and Azure APIs.

Harness-MAP invokes meta-agent calls during proposal and diagnosis stages, with zero per-task hyperparameter tuning. Harness-MAP calls the proposer once per child and the diagnostician once per parent slot per round, while baselines fold failure analysis into a single proposer call. See more detailed configurations of each method in the updated §A.4.

Table R2a. BALROG-MiniHack. Budget N=50N = 50.

MethodNNTask episodesProposer callsDiagnostician callsHours
Harness-MAP (k0=2k_0=2)502,000496965.5
Harness-MAP (k0=4k_0=4)502,000578110.0
Harness-MAP (k0=8k_0=8)502,0006511117.7
Harness-MAP (k0=16k_0=16)502,0007711914.0
Harness-MAP (k0=32k_0=32)502,00010113520.9
Meta-Harness502,00049068.5
GEPA503,17049045.5
DGM501,26028034.9
ShinkaEvolve501,96849071.0
OpenEvolve501,85349093.1

Table R2b. SimWorld-Robotics. Budget N=16N = 16.

SettingMethodNNTask episodesProposer callsDiagnostician callsHours
EasyHarness-MAP (k0=2k_0=2)16800208113.5
EasyHarness-MAP (k0=4k_0=4)16800239118.4
EasyHarness-MAP (k0=8k_0=8)168003215116.1
EasyMeta-Harness16773150107.1
EasyGEPA1676515063.2
EasyDGM1670914070.5
EasyShinkaEvolve1669415045.7
EasyOpenEvolve1680015043.3
HardHarness-MAP (k0=2k_0=2)16800158193.8
HardHarness-MAP (k0=4k_0=4)168001515207.2
HardHarness-MAP (k0=8k_0=8)168001616132.9
HardMeta-Harness16800150185.6
HardGEPA16422110102.0
HardDGM16800150183.7
HardShinkaEvolve16800150185.5
HardOpenEvolve16800150177.9

In sum: Number of task-level executions equals candidate-harness evaluations (NN), offline structure-token partitioning prevents initialization collisions from wasting evaluations (M=⌈mk0⌉M = \lceil m k_0 \rceil). Tab R1–R2b and §A.4 report exact task episode, LLM-call, and runtime counts across all methods.

Thank you again for your valuable feedback, which has helped us improve our manuscript. We hope this clarification resolves your concern.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviews and Rebuttal

We thank all reviewers and the Area Chairs for reviewing our paper.


We first highlight key strengths of our work acknowledged by reviewers:

  1. A novel structure-token representation for harness search. The structure-token descriptor gives harness search an explicit coordinate system that both organizes the archive and guides targeted edits, with value beyond our own loop. ([P1E2], [2WbT])

  2. Budget-efficient search with zero-rollout repartitioning: Dynamically subdividing high-variance cells and reverting unproductive splits via cached descriptors concentrates compute without extra rollouts. ([2WbT])

  3. Substantial gains with early sample efficiency. Large search-set improvements stem from discovering and recombining algorithmic modules at early stage. The detailed per-task documentation supports follow-up work. ([P1E2], [2WbT], [7PhL])


We then summarize the key results and additional evidence presented in our revised manuscript (highlighted in blue) and rebuttal:

  1. The structure-token representation drives the gains. Transplanting our descriptor into OpenEvolve more than triples its success, while removing the WL structural block lowers success and up to triples search cost. Additionally, the WL vocabulary grows with new topologies while populations stay low-dimensional, avoiding high-dimensional sparsity. (Tabs. 3, 14; §A.3, §A.8; [2WbT W1, Q1], [7PhL W1])

  2. The Quality-Diversity archive provides essential stepping stones. Without it, search collapses to greedy hill climbing and stalls on hard tasks, as non-best archive elites lead to solvers that greedy search never finds. (Tab. 3; §5; [7PhL Q2])

  3. Task-adaptive VLM usage emerges automatically. Harness-MAP discovers a spectrum of zero-VLM, hybrid, and VLM-centric harnesses depending on environment structure, without predefined rules. (Figs. 2; §5, §A.9; [7PhL W2])

  4. Adaptive resolution is self-adjusting at zero rollout cost with granularity robustness. Splits are judged after a breeding grace period. Empirically, 66.2% of splits succeed, eliminating sensitivity to initial cell count k0k_0 across both saturated and unsaturated benchmarks. (Alg. 2; §A.7; [P1E2 W1, Q2], [7PhL W1])

  5. Large, robust gains under transparent, budget-matched evaluation. All methods share the same meta-agent and comparable episode budgets, with no rollouts spent on initialization collisions. Across three independent search runs with k0k_0 fixed in advance, Harness-MAP significantly improves over the backbone on every split. (§A.4; rebuttal Tabs. R1–R2; [P1E2 W2, Q1], [7PhL Q1, Q3], [2WbT W2, Q2])


In light of the above, we believe Harness-MAP makes a significant contribution toward budget-efficient agent harness search. Our extensive empirical evidence shows that a factorized structure-token map, with resolution-adaptive Quality-Diversity search, discovers diverse and high-performing harnesses under tight evaluation budgets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.