- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a "harness evolution" framework consisting of two stages. Starting with "Web Thinker" as the initial harness—while keeping the underlying model API constant—the authors independently search for an improved prompt and an improved harness (controller) code.
Leveraging an external language model (LM), they perform an iterative process—akin to meta-learning—using a very small development set. Finally, they combine the selected prompt and controller (i.e., the harness) and evaluate the resulting agent on specific subsets of the HLE and GAIA benchmarks.
The authors argue that modifying these two components allows for the optimization of agent performance with fewer samples and lower costs. Both components prove effective—yielding even better results when combined—and demonstrate a degree of generalizability.
Critical strengths
None
Critical weaknesses
- There is a significant contradiction between the statement in the main text regarding the optimization data’s independence and the information provided in the supplementary material.
In lines 334–336, the authors explicitly state that the 30 optimization problems used across the two stages do not overlap with the GAIA or HLE datasets, and that no problems from these benchmarks were utilized during the evolutionary process. However, the submitted Stage II code directly loads GAIA data. More importantly, a set of 20 problems found in four candidate evaluation logs—including both the problem prompts and the ground-truth answers—matches entries in the final 103-problem GAIA benchmark exactly. Furthermore, these evaluation logs can be used to reproduce the 52.4% figure reported in the main table.
This indicates that the candidate optimization problems in the submission overlap with the final GAIA benchmark, which directly contradicts the claim in the main text regarding data isolation. Consequently, the reported GAIA results cannot be considered a strictly independent generalization evaluation.
- The main table lacks sufficient detail regarding the evaluation subsets and the sources of the baselines, making the comparability of the results unclear.
While the authors state generally that they follow the evaluation setup used in WebThinker, the main table fails to clearly distinguish between the full benchmarks and the specific subsets actually employed. It also does not specify, on an item-by-item basis, whether the baselines utilized the same problem sets and scoring protocols. For instance, the HLE results in the supplementary material are calculated using a denominator of 500 problems, rather than the full HLE dataset. While using a subset is not inherently problematic, the composition of the subset, the selection criteria, and whether all direct comparisons are based on the identical subset must be explicitly stated.
Questions for authors
Please show the evolving details(FULL TRAJS ON ALL EPISODES), so reviewers can check whether there is tuning on the test set.