acceptodds
Withdrawn from ICLR 2027

HARNESSFLOW: MCTS-GUIDED HARNESS OPTIMIZATION FOR LLM AGENTS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The performance of large language model (LLM) agents depends on both the underlying model and the harness that governs interaction with tools and the environment. Yet harness design remains largely manual, and automating it requires navigating an open-ended search space through costly end-to-end evaluations. We introduce **HarnessFlow**, a framework for optimizing complete executable harnesses under a limited candidate-evaluation budget, with the model and task environment held fixed. HarnessFlow organizes candidates and their evaluation histories in a refinement tree, using a Monte Carlo Tree Search (MCTS)-based policy to balance exploitation and exploration. A coding agent refines selected harnesses using archived code, execution feedback, and a library of reusable harness mechanisms. With 15 refinement attempts per benchmark, HarnessFlow achieves the highest mean test scores among the compared methods on six benchmarks spanning agentic tasks, coding, and web interaction. The optimized harnesses achieve an average score of 69.48%, outperforming the initial harnesses and Meta-Harness by 10.89 and 3.95 percentage points, respectively, while incurring lower average per-task execution costs than hand-engineered baselines. These results suggest that structured harness search offers a promising route toward automating the design of capable and efficient LLM agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Harness Optimization, LLM Agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_q88SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HARNESSFLOW, a framework that uses a coding agent to optimize complete LLM-agent harnesses while keeping the executor model and task environment fixed. The method organizes candidate harnesses and their refinement relationships into a tree. A selection policy combines validation scores and expansion counts to determine which harness to refine next, while the coding agent uses archived code, execution traces, and a component library to generate new candidates. The paper evaluates the framework on six benchmarks and compares the task performance and execution costs of the resulting harnesses

Critical strengths

S1. The paper addresses a practically meaningful problem of automated harness optimization. It allows coordinated modifications to context construction, state management, control flow, and tool interfaces, supporting changes that account for interactions between these mechanisms.

S2. The experiments cover interactive tasks, code generation and repair, and web interaction. The reported results show average performance improvements over both the initial harnesses and Meta-Harness, with particularly substantial gains on TAU2-bench and SWE-bench Verified.

S3. The paper is clearly presented. The selection policy, algorithm description, and harness-evolution examples help readers understand how candidates are generated and how the resulting implementations change during optimization.

Critical weaknesses

W1. AFlow already combines code-based workflow refinement with uniform–softmax parent selection. However, this paper lacks a controlled comparison with this closely related selection strategy using the same full-harness search space, coding agent, experience archive, and component library.

W2. The five repeated runs described in Section 4.1 and Appendix C.4 evaluate candidate or selected harnesses, while independent repetitions of the complete optimization process are not clearly documented. Therefore, the reported standard deviations do not establish the stability of HARNESSFLOW with respect to stochastic parent selection and code generation.

W3. Appendix C.1 reports 100 validation tasks and 200 test tasks for BigCodeBench-Hard, whereas the official Hard dataset contains only 148 tasks. Although the paper mentions sampling from the broader BigCodeBench release using published difficulty criteria, it does not provide the exact filtering rules or task IDs. This makes it difficult to verify the construction of the evaluation set and its comparability with the standard BigCodeBench-Hard benchmark.

W4. The optimization procedure involves repeated end-to-end candidate evaluations and code refinement using GPT-5.5 at the highest reasoning effort. However, the cost analysis only reports deployment execution costs and does not quantify candidate-evaluation costs, coding-agent costs, or optimization time. Consequently, the practical computational efficiency of the complete framework remains unclear, despite its promising performance under a limited candidate budget.

Questions for authors

Q1. Did Meta-Harness receive identical component-library access and comparable access to archived code and execution feedback? Could you provide matched comparisons with α=0\alpha=0, deterministic best-parent selection, and uniform-parent selection, specifying whether uniform sampling operates within the top-K shortlist or over all evaluated harnesses?

Q2. How many independent complete optimization runs were performed for each experimental setting? Do the performance improvements and selection-policy ablation results persist across different search seeds?

Q3. How were the 300 BigCodeBench-Hard tasks constructed? Please provide the exact dataset revision, filtering rules, and task IDs for both the validation and test splits, and explain how this subset relates to the official 148-task Hard dataset.

Q4. Figure 4 introduces retrieval of repository commit subjects and source diffs. Was accessible Git history restricted to each task’s base commit and its ancestors, with gold fixes, solution branches, and later commits inaccessible?

Q5. What were the total proposal costs, candidate-evaluation costs, and optimization times for HARNESSFLOW and Meta-Harness? Were comparable resource budgets imposed on coding-agent execution and debugging?

Reviewer_L29WOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes an optimization method for agent harnesses. The proposed method, Harnessflow, uses a search over a branching tree and a parent selection policy to trade off exploitation and exploration in the harness design space. It also relies on a library of reusable harness components at each revision. Harnessflow is evaluated on several agentic benchmarks and outperforms other common harnesses such as codex and claude code but also its closest competitor Meta-harness.

Critical strengths
  • the paper is well written and easy to follow
  • good number of benchmarks in the evaluation
  • the ablation on the library and stronger backbone model is informative
  • inference time cost is reported
  • useful figures describing the search process.
Critical weaknesses

Main concerns:

  • While mean and variance across benchmark evaluations are reported, what we really need to compare different harness optimization methods are mean and variance across multiple search runs.
  • How was the library created? Was it designed with the benchmarks in mind? The ablation helps, and it seems to take away a large portion of the gain on tau2 bench. Would it be possible to run meta-harness with the library as well?
  • As far as I know, meta-harness has access to all prior candidates when designing the next harness. In that sense it has access to a persistent archive. Making the core differences the parent sampling and the library. Please correct me if this is inaccurate.
  • The cost of performing the actual search / harness optimization seems to be missing? Could the authors please report this for harness flow and meta-harness, for all benchmarks ideally, tokens and time per search step or something like that would be very useful.
  • I am surprised by the poor performance of meta-harness (almost no gains and very early plateauing). Was the official code and prompt used? Was there a specific reason the authors chose to run on none of the benchmarks present in the meta-harness paper? Would it be possible to run e.g. terminal bench 2 for a clear comparison?
  • Is there leakage of the solution on swe-bench verified? I am asking because 0->2 yields a large improvement and relies on searching the git history of the repo.
  • In Figure 3 a we can see that the gains over meta-harness appear very early on before the parent selection policy can really have an effect (also Figure 6). This raises the question whether the highlighted parent selection policy is actually useful. An ablation would be very useful to isolate the effects of the paper's contributions.
  • How were hyperparameters chosen? The ablation on epsilon shows high parameter sensitivity and reports test scores. Did these inform the hyperparameter choice? Hyperparameter tuning is a limitation that should be acknowledged.
  • Would it be possible for the no library ablation to be run on the set of all six benchmarks?
  • Would it be possible to add fully random parent selection and fully greedy selection?
  • All models used are openai models, also the user simulator and grader, this limits generalisation across families
  • My understanding of Table 4 is that the search itself was rerun with the stronger model? It would be interesting to see whether the harness found for gpt 5.4 mini generalises to stronger / other models.

Other:

  • It would be good to state the models used for the search process in the main text.
  • I think the intro overstates manual harness design and this could be softened.
  • Tree search over code is an established approach e.g. AIDE or AI Scientist v2 (also alpha evolve, open evolve, shinka evolve), I think this should be stated more clearly.
  • Darwin Godel Machines by Zhang et al.,2025 should be cited. It would also be useful to add a discussion comparing the selection policy in DGM to that of harness flow, maybe even an empirical ablation.

If the above points are addressed and ablations / additional results support the core claims, especially those critical for assessing the real impact of the claimed contributions (the parent selection policy, search cost, randomness over searches, library impact, meta-harness weakness, swe bench leakage to name a few) I would be happy to increase my score.

Questions for authors

See above

Reviewer_TeQXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HarnessFlow, which treats a complete agent harness (context construction, state management, tool interfaces, control flow) as an executable codebase and optimizes it with a coding agent (Codex with GPT-5.5 at the highest reasoning effort). Candidates are kept in a refinement tree; the parent to refine is sampled from a mixture of a score-based softmax over the top-K nodes and an exploration term based on the number of direct children (ε = 0.35). A 21-module Harness Component Library is offered to the coding agent as optional reference code. With a budget of 15 candidates per benchmark, HarnessFlow reports an average test score of 69.48% across TAU2-bench, AppWorld, WebShop, BrowseComp-Plus, SWE-bench Verified, and BigCodeBench-Hard, 10.89 points above the initial harness and 3.95 points above Meta-Harness, with lower per-task execution cost than Claude Code, Codex, OpenHands, and Hermes.

Critical strengths
  1. Realistic optimization target and benchmarks. Optimizing the full harness, rather than only prompts or workflow wiring, and evaluating on interactive benchmarks (TAU2-bench, AppWorld, SWE-bench Verified) is closer to real agent deployment than AFlow, which the paper notes was evaluated on single-turn question answering, mathematics, and code generation.
Critical weaknesses
  1. Limited algorithmic novelty; the "MCTS" is a parent sampler. Selection samples a parent from (1−ε)·softmax(score/τ) + ε·(preference for nodes with fewer children) over the top-K nodes. There are no rollouts and no value backup—the paper states that each harness's directly measured score is preserved—and "experience backpropagation" amounts to recording the outcome in the archive. As described in the paper's own Related Work, AFlow already combines MCTS with code-represented workflow refinement, and Meta-Harness already lets a proposer inspect archived code, scores, and traces to modify a complete harness. Removing the component library lowers the three-benchmark average only from 65.03 to 63.39 (Table 2), mostly on TAU2-bench, so the remaining contribution is essentially the parent-selection rule.
  2. The efficiency claim omits search cost. Cost-efficiency is one of the paper's three headline claims, yet Figure 3b and the accompanying text explicitly exclude one-time harness-search costs. Each candidate requires GPT-5.5 (highest effort) code generation plus R = 5 full executions on the validation split (e.g., 100 SWE-bench Verified tasks × 5), so the search cost could well exceed the deployment-time savings. In addition, the final harness is the one with the highest validation score among the root plus 15 candidates on validation splits of only 92–105 tasks; the mean best-so-far validation score (74.2) is about 5 points above the mean test score (69.48), which raises a selection-bias concern that is not discussed.
Questions for authors
  1. (Clarification) What is the total search cost per benchmark (coding-agent tokens, validation executions, dollars, wall-clock)?
  2. (Clarification) What is the validation–test gap per benchmark for the selected harness?
  3. (Suggestion) How benchmark-specific are the discovered harnesses—e.g., how does the harness found on SWE-bench Verified perform on BigCodeBench-Hard?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.