acceptodds
Under review as a conference paper at ICLR 2027

Joint Model-Harness Optimization: Cost-Aware Resource Allocation for LLM Agents

Abstract

An LLM agent's quality and cost depend jointly on its model and harness, yet execution-time gains alone do not establish whether harness optimization is cost-effective. We study model–harness design as resource allocation, accounting separately for the cost of searching for a harness and the cost of executing it. We introduce HARP-Static, which combines component addition, trace-guided graph rewrites, and backward compression while tuning node-level resources and model assignments. Across six benchmarks, with harness search on five, we find both opportunities and limits to reallocating computation. Two searched architectures reduce to a minimal reasoning chain, and development-set analyses identify verification changes that lower cost without changing observed scores; larger savings from cheaper node-level models incur quality loss. On HotpotQA, Haiku-4.5 with a searched harness outperforms Opus-4.5 with its default harness by 10.4 percentage points at 55.3% of its execution cost. However, the only finite break-even point estimate in the matched-quality analysis requires approximately 1.72 million served tasks, and the underlying saving is not statistically established. Generalization poses a further constraint: on BFCL, the compressed harness loses 15.0 percentage points to the official harness, with post-hoc analysis locating the entire deficit in multi-turn tasks requiring repeated tool interaction. These results show that evaluating harness optimization requires distinguishing local execution gains from deployment-level savings and testing whether compression preserves the interactions a task requires.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, harness optimization, cost-aware optimization, resource allocation, compute reallocation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wvdxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a cost-aware framework for jointly optimizing LLM model selection, agent harness architecture, and computational resource allocation. HARP-Static searches over harness components, execution topology, node configurations, and model assignments, while HARP-Adaptive adjusts resource allocation at runtime. Experiments across six benchmarks show that optimized harnesses can sometimes outperform stronger models at substantially lower execution cost, particularly on HotpotQA.

Critical strengths
  • The paper demonstrates substantial improvements on HotpotQA, with HARP-Haiku outperforming Opus with its default harness. The investigation of which computational components can be reduced is also informative.

  • The authors explicitly examine amortization costs, transferability, and failure cases. The BFCL analysis highlights how harness compression can inadvertently eliminate necessary multi-turn interactions.

Critical weaknesses
  • Although the paper discusses Meta-Harness, AutoHarness, AFlow, and related approaches, experiments primarily compare against default or human-designed harnesses. Without comparisons to competitive optimization baselines under matched search budgets, it is difficult to establish the effectiveness of HARP's specific optimization strategy.

  • The paper's central motivation is cost-aware optimization, yet its only identified matched-quality break-even estimate requires approximately 1.72 million tasks, with statistically insignificant savings.

  • Cross-task transfer incurs substantial performance degradation (−16.1 pp on average), while BFCL exhibits a 15.0 pp loss against the official harness. HARP-Adaptive also fails to demonstrate measurable improvements over HARP-Static, weakening the contribution of the proposed joint offline–online optimization framework.

Questions for authors
  1. How does HARP compare against existing automated harness optimization methods, such as Meta-Harness or AFlow, under identical model and search budgets? Which components of HARP are responsible for its improvements?

  2. Given the substantial optimization cost and the lack of statistically significant matched-quality savings, under what deployment scenarios does HARP provide a reliable economic advantage?

  3. Can the authors investigate whether larger, more representative development sets or robustness-aware search objectives prevent the substantial BFCL degradation and improve cross-task transfer?

Reviewer_oETiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how LLM agents allocate costs between model selection and framework design, separating one-time search costs from execution costs. The authors propose two methods: HARP-Static and HARP-Adaptive. HARP-Static searches for framework components, structure, node configurations, and model assignments for each node at a single cost; HARP-Adaptive is a runtime controller trained based on search trajectories. On the HotpotQA dataset, Haiku 4.5 using the searched framework outperforms Opus 4.5 using the default framework at a lower cost, but this method does not help on the SimpleQA dataset and actually degrades performance on the BFCL dataset. The searched framework can be transferred between different models but not across tasks, and the search cost is rarely recovered in break-even analysis.

Critical strengths

The question is useful. Separating the one-time search cost from the execution cost, and asking whether harness optimization pays off at deployment, is missing from most prior harness search work.

Critical weaknesses

I do not think this is a ready manuscript that meets the ICLR standard. I found problems in several parts of the paper, and overall it does not look complete to me.

  • The numbers are not consistent. For the HotpotQA comparison against Opus, the main text and Table A2 report +12.5 pp at 0.04× the cost, but the abstract and Table A5 report +10.4 pp at 0.55×. I also could not reconcile the SimpleQA results between these two tables.
  • Some planned experiments are missing. τ-bench is listed as a benchmark, but no τ-bench results are reported, and there are no results under the degraded or failure conditions described in the introduction. For the break-even analysis, which I think is the main contribution, the matched-quality analysis gives only one finite estimate of about 1.72 million tasks, and the saving behind it is not significant.
  • Most of the theory restates known results, such as Lagrangian scalarization and greedy submodular cover. The authors say they will report the constants behind each guarantee, but I could not find them in the paper, so I do not see how the theory connects to the experiments.
  • The title is confusing. ``Joint model–harness optimization'' made me expect that the model is trained together with the harness, but no model weights are updated. The model side is only model selection, and the search itself is run with Haiku fixed.

I think separating search cost from execution cost is a good question, and I hope the authors can submit a more complete version.

Questions for authors

See the weakness section above

Reviewer_h41gOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether an LLM agent is better served by a stronger model or a better harness, and frames this as cost-aware resource allocation.

It separates the one-time cost of harness search from the recurring cost of execution. It introduces HARP-Static which searches jointly over components, graph topology, node configurations and per-node model assignments. It also introduces HARP-Adaptive which reuses the search traces to learn a state-dependent allocation policy.

Harness search is run on five of six benchmarks while τ\tau-bench has a model axis only.

The main finding is that a searched harness can substitute for model capability on procedural tasks such as HotpotQA, but not on knowledge bound ones. Online adaptation bring no measurable gain, and the search cost is hard to amortize.

Critical strengths

A well-posed, practical problem. Most harness-optimization work reports only final performance. The paper treats model and harness as competing uses of one budget and separates search cost from deployment cost. The break-even analysis asks whether an improvement pays for itself which is quite an interesting question.

Informative experiments, including negative results. The paper provides useful insights into when harness optimization can substitute for model capability specially the distinction between procedural and knowledge-intensive tasks. I also like the transfer experiments, which suggest that optimized harness structures can generalize across models while remaining task-specific. The authors are transparent about the limitations of their approach and report negative results rather than only successful runs.

Critical weaknesses

1. The practical benefit of the full method is unclear. The adaptive part of HARP does not seem to help: in Section 5.2 it ties the static harness at every cost-matched point. On the cost side, the abstract reports that the only break-even estimate needs about 1.71.7 million served tasks, and even this saving is not statistically significant. I also could not find a direct comparison with existing harness-optimization methods such as Meta-Harness or AutoHarness, which are discussed in the related work. Without it, it is hard to tell how much HARP itself contributes.

2. Some promised evaluations seem to be missing. The introduction and Section 3.4 say the method is tested under degraded and failure conditions, and that these are where adaptivity should pay off. I could not find these results. Similarly, Section 3.4 says the paper reports measured values for the constants behind the theory (γ^\hat{\gamma}, η^\hat{\eta}, ξ^\hat{\xi}), but I did not see them. Without these, it is hard to judge whether the assumptions of the theoretical analysis hold in practice.

3. Limited generalization of the optimized resource allocations. The BFCL results show that reducing the tool-call budget preserves accuracy on all 70 development tasks but causes the optimized harness to fail all 25 held-out multi-turn tasks, compared with 18 successes for the official harness. This suggests that the optimization procedure can remove capabilities that are rarely needed during development but essential for other tasks. Given the relatively small development sets, it is unclear how reliably HARP can identify which resources are safe to remove.

Questions for authors

1 - How does HARP compare with Meta-Harness or AutoHarness at matched search budgets?

2 - Where are the degraded and failure results, and the Section 3.4 diagnostics?

3 - How frequent are the excluded runs for each method, and do the results hold if they are counted as failures? What did the "harness superseded" exclusions remove?

4 - Which HotpotQA cost figures are correct, and is the searched harness cheaper or more expensive than Haiku’s default?

5 - Could more conservative allocation, such as floors on tool-call budgets, prevent failures like the one on BFCL?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.