- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces HARNESSFLOW, a framework that uses a coding agent to optimize complete LLM-agent harnesses while keeping the executor model and task environment fixed. The method organizes candidate harnesses and their refinement relationships into a tree. A selection policy combines validation scores and expansion counts to determine which harness to refine next, while the coding agent uses archived code, execution traces, and a component library to generate new candidates. The paper evaluates the framework on six benchmarks and compares the task performance and execution costs of the resulting harnesses
Critical strengths
S1. The paper addresses a practically meaningful problem of automated harness optimization. It allows coordinated modifications to context construction, state management, control flow, and tool interfaces, supporting changes that account for interactions between these mechanisms.
S2. The experiments cover interactive tasks, code generation and repair, and web interaction. The reported results show average performance improvements over both the initial harnesses and Meta-Harness, with particularly substantial gains on TAU2-bench and SWE-bench Verified.
S3. The paper is clearly presented. The selection policy, algorithm description, and harness-evolution examples help readers understand how candidates are generated and how the resulting implementations change during optimization.
Critical weaknesses
W1. AFlow already combines code-based workflow refinement with uniform–softmax parent selection. However, this paper lacks a controlled comparison with this closely related selection strategy using the same full-harness search space, coding agent, experience archive, and component library.
W2. The five repeated runs described in Section 4.1 and Appendix C.4 evaluate candidate or selected harnesses, while independent repetitions of the complete optimization process are not clearly documented. Therefore, the reported standard deviations do not establish the stability of HARNESSFLOW with respect to stochastic parent selection and code generation.
W3. Appendix C.1 reports 100 validation tasks and 200 test tasks for BigCodeBench-Hard, whereas the official Hard dataset contains only 148 tasks. Although the paper mentions sampling from the broader BigCodeBench release using published difficulty criteria, it does not provide the exact filtering rules or task IDs. This makes it difficult to verify the construction of the evaluation set and its comparability with the standard BigCodeBench-Hard benchmark.
W4. The optimization procedure involves repeated end-to-end candidate evaluations and code refinement using GPT-5.5 at the highest reasoning effort. However, the cost analysis only reports deployment execution costs and does not quantify candidate-evaluation costs, coding-agent costs, or optimization time. Consequently, the practical computational efficiency of the complete framework remains unclear, despite its promising performance under a limited candidate budget.
Questions for authors
Q1. Did Meta-Harness receive identical component-library access and comparable access to archived code and execution feedback? Could you provide matched comparisons with , deterministic best-parent selection, and uniform-parent selection, specifying whether uniform sampling operates within the top-K shortlist or over all evaluated harnesses?
Q2. How many independent complete optimization runs were performed for each experimental setting? Do the performance improvements and selection-policy ablation results persist across different search seeds?
Q3. How were the 300 BigCodeBench-Hard tasks constructed? Please provide the exact dataset revision, filtering rules, and task IDs for both the validation and test splits, and explain how this subset relates to the official 148-task Hard dataset.
Q4. Figure 4 introduces retrieval of repository commit subjects and source diffs. Was accessible Git history restricted to each task’s base commit and its ancestors, with gold fixes, solution branches, and later commits inaccessible?
Q5. What were the total proposal costs, candidate-evaluation costs, and optimization times for HARNESSFLOW and Meta-Harness? Were comparable resource budgets imposed on coding-agent execution and debugging?