- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes to do the efficient search for a diverse set of Harness with a tight budget and the method Harness-MAP demonstrates good performance on different tasks. Harness-MAP tries to use a descriptor to make each candidate harness on a map and keep the best harness in each cell, which also proposes a method for doing the search. Harness-MAP demonstrates great performances compared with baselines.
Critical strengths
- The search loop shows great performance on benchmarks.
- This paper provides detailed information for each task, which is helpful for people to continue to work on this method.
Critical weaknesses
- It looks like that the gains are not from the claimed main contributions. For example, the adaptive resolution shows no help for the unsaturated benchmark.
- The role of VLM is overstated.
Questions for authors
- It is better if you report the held-out results with k0 fixed in advance and over >= 3 independent search runnings.
- If there is no archive, how is the performance of Harness-MAP?
- Which meta-agent produced each Table 1 entry? Do all baselines get the same meta-agent?
6 comments
Response to Reviewer 7PhL (PART 2/6)
[A1-2] Resolution-Adaptive Search
Adaptive resolution eliminates sensitivity to initial grid granularity on saturated benchmarks (Fig. 9[new-figure], Tab. 10[original-result]). As shown in Fig. 9, adaptive resolution (K_adaptive) is far more robust across different than static (K_fixed) and scheduled (K_hier) modes on SpatialWorld-3DGames.
Adaptive partitioning enables steadier and more robust optimization across unsaturated benchmarks.
Across unsaturated SimWorld-Robotics-Hard (Tab. R3[new-result]) and BALROG-MiniHack (Tab. 12[orignial-result], Tab. R4[new-result]), K_adaptive consistently ranks 1st or 2nd best across varying initial resolutions, proving largely insensitive to the choice of . In contrast, static K_fixed and scheduled K_hier exhibit brittle performance drops under suboptimal granularities, forcing practitioners to run multiple expensive trials to tune and wasting substantial rollout compute. Plus, partition retention/reversion rates across all benchmarks (§A.7, Tab. 13[new-result]) futher confirm this self-correcting robustness, where 66.2% (45/68) of attempted sub-splits beat their pre-split parent baselines and are retained.
Tab. R3. Resolution ablation on unsaturated SimWorld-Robotics-Hard (10/2 search/val split, SubSR, ).
| Mode | Best@4 | Best@8 | Best@12 | Best@16 | |
|---|---|---|---|---|---|
| 2 | adaptive | 0.625 | |||
| 2 | fixed | 0.625 | 0.625 | 0.625 | 0.700 |
| 2 | hier | 0.475 | 0.475 | 0.550 | |
| 4 | adaptive | 0.575 | |||
| 4 | fixed | 0.550 | 0.550 | 0.550 | |
| 4 | hier | 0.450 | 0.450 | 0.450 | 0.475 |
| 8 | adaptive | 0.425 | 0.525 | ||
| 8 | fixed | 0.525 | 0.525 | ||
| 8 | hier | 0.425 | 0.425 | 0.625 |
Tab. R4. Resolution ablation on unsaturated BALROG-MiniHack (SR%, ).
| Mode | Best@10 | Best@20 | Best@30 | Best@40 | Best@50 | |
|---|---|---|---|---|---|---|
| 2 | adaptive | 67.5 | 75.0 | 75.0 | 75.0 | 75.0 |
| 2 | fixed | |||||
| 2 | hier | 12.5 | 35.0 | 62.5 | 67.5 | |
| 4 | adaptive | 72.5 | 85.0 | |||
| 4 | fixed | 72.5 | 87.5 | 87.5 | 87.5 | 92.5 |
| 4 | hier | 57.5 | 87.5 | 87.5 | ||
| 8 | adaptive | 32.5 | 57.5 | 82.5 | ||
| 8 | fixed | 87.5 | 87.5 | |||
| 8 | hier | 77.5 | 85.0 | 87.5 | 87.5 | 92.5 |
| 16 | adaptive | 37.5 | 37.5 | 87.5 | 90.0 | |
| 16 | fixed | 32.5 | 62.5 | |||
| 16 | hier | 80.0 | 80.0 | 90.0 | ||
| 32 | adaptive | 55.0 | 65.0 | 65.0 | ||
| 32 | fixed | 42.5 | 65.0 | 75.0 | 75.0 | |
| 32 | hier | 30.0 | 65.0 |
Dynamic cell refinement sustains high archive coverage and yields superior QD-scores across intermediate resolutions. In Tab. 2[original-result], K_adaptive achieves higher Quality-Diversity scores at intermediate resolutions and preserves superior archive coverage, while baselines collapse at finer resolutions.
In sum: Ablations show the structure-token map drives the gains, lifting OpenEvolve from 21.9% to 71.4% while removing any of its components lowers Harness-MAP from 100.0% to 84.0–97.6% (Tab. 3). Adaptive resolution mitigates sensitivity on saturated benchmarks and stays robust with effective sub-partitioning on unsaturated ones (Fig. 9, Tab. R3–R4).
Response to Reviewer 7PhL (PART 3/6)
This comment "The role of VLM is overstated" is every short. We do not see precisely what this weakness is referring to. Could you please elaborate or rephrase this point? At the meantime, we address the interpretation regarding why some top-performing harnesses discovered by Harness-MAP can operate without VLM calls.
[W2] Clarification of the role of non-VLM and VLM harnesses.
[A2-1] Harness-MAP formulates harness evolution as executable program search where mandating a VLM call at every step could be an optional rule for the meta-agent to make VLM-centric proposals, even though we didn't do it.
Program search discovers whichever orchestration of symbolic code and VLM calls best solves the task. Technically, harnesses optimize control flow around a frozen backbone, but deterministic solvers handle the transitions cleanly, and skipping redundant VLM calls improves reliability and efficiency, which is totally reasonable.
[A2-2] Across three benchmarks, Harness-MAP discovers a spectrum of zero-VLM, hybrid, and VLM-centric harnesses depending on environment structure. We have found the following emergent patterns:
- Zero-VLM symbolic solvers on structured grid tasks. In BALROG-MiniHack, incremental modular edits produce deterministic code solvers such as BFS planners.
- Conditional VLM workflows on hybrid reasoning tasks. Harness-MAP also discovers selective-querying harnesses that invoke the frozen VLM only when deterministic rules cannot decide.
- Per-step VLM workflows on perceptual grounding tasks. When visual grounding or situational routing is required, discovered harnesses invoke the VLM on every step.
Because different environments vary in how much perceptual grounding versus combinatorial search they require, discovering both zero-VLM solvers and VLM-driven workflows reflects adaptive task specialization.
In sum: Our structure-token program search naturally discovers both zero-VLM solvers and VLM-centric workflows tailored to environment demands. Please let us know if your comment targeted another point.
Response to Reviewer 7PhL (PART 1/6)
Thank you for your constructive review and for recognizing the novelty of our map-based harness search loop, its strong performance, and our detailed implementation documentation. Per your suggestions, we expanded on the details of our component ablations, multi-seed held-out evaluations, meta-agent consistency, and the role of VLM calls. All corresponding revisions are highlighted in blue in the updated manuscript.
[W1] Attribution of performance gains to the proposed structured representation map and adaptive-resolution search designs.
We respectfully disagree with this assessment because controlled ablations and experiments on 3 resolution modes confirm that our gains come directly from both core designs across saturated and unsaturated benchmarks.
[A1-1] Structured Representation Map (φS,φT)
Controlled ablations confirm that the factorized structure-token space and QD archive are essential for search accuracy and efficiency (Tab. 3[original-result]). Replacing OpenEvolve's surface-text descriptor with our structure-token CVT space (
OpenEvolve_st-cvt) raises SR on SpatialWorld-3DGames (k0=8) from 21.9% to 71.4% under the same search loop. Swapping our descriptor for OpenEvolve's 2D descriptor (Harness-MAP_cd) lowers performance from 100.0% to 96.2%, more than doubles Rubik evaluations (14 to 37), and also causes coverage failure on Maze3D. Removing Weisfeiler-Lehman features (Harness-MAP_w/o WL) drops SR to 97.6% and triples Block3D evaluations (5 to 16), while removing the QD archive (Harness-MAP_w/o archivein Tab. R1[new-result]) drops SR to 84.0% and Block3D from 100.0% to 35.0%.Tab. R1. Component and descriptor ablation results on SpatialWorld-3DGames (k0=8, reporting SR% / N* / T*).
Harness-MAP w/o archiveDynamic Weisfeiler-Lehman representations capture substantial architectural variance on unsaturated tasks without suffering from high-dimensional sparsity (Tab. 14 [new-result] and Tab. R2[new-result]). While φT stays fixed at 786 dimensions, the WL structural block φS expands dynamically, e.g., from 118 to 211 keys on SimWorld-Robotics-Hard, where φS accounts for 10% to 40% of total descriptor variance and becomes the largest block at k0=4 (40%), separating distinct program architectures that token features alone would merge.
Tab. R2. Share of descriptor variance per block on SimWorld-Robotics-Hard.
astnumdoc