- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a cost-aware framework for jointly optimizing LLM model selection, agent harness architecture, and computational resource allocation. HARP-Static searches over harness components, execution topology, node configurations, and model assignments, while HARP-Adaptive adjusts resource allocation at runtime. Experiments across six benchmarks show that optimized harnesses can sometimes outperform stronger models at substantially lower execution cost, particularly on HotpotQA.
Critical strengths
-
The paper demonstrates substantial improvements on HotpotQA, with HARP-Haiku outperforming Opus with its default harness. The investigation of which computational components can be reduced is also informative.
-
The authors explicitly examine amortization costs, transferability, and failure cases. The BFCL analysis highlights how harness compression can inadvertently eliminate necessary multi-turn interactions.
Critical weaknesses
-
Although the paper discusses Meta-Harness, AutoHarness, AFlow, and related approaches, experiments primarily compare against default or human-designed harnesses. Without comparisons to competitive optimization baselines under matched search budgets, it is difficult to establish the effectiveness of HARP's specific optimization strategy.
-
The paper's central motivation is cost-aware optimization, yet its only identified matched-quality break-even estimate requires approximately 1.72 million tasks, with statistically insignificant savings.
-
Cross-task transfer incurs substantial performance degradation (−16.1 pp on average), while BFCL exhibits a 15.0 pp loss against the official harness. HARP-Adaptive also fails to demonstrate measurable improvements over HARP-Static, weakening the contribution of the proposed joint offline–online optimization framework.
Questions for authors
-
How does HARP compare against existing automated harness optimization methods, such as Meta-Harness or AFlow, under identical model and search budgets? Which components of HARP are responsible for its improvements?
-
Given the substantial optimization cost and the lack of statistically significant matched-quality savings, under what deployment scenarios does HARP provide a reliable economic advantage?
-
Can the authors investigate whether larger, more representative development sets or robustness-aware search objectives prevent the substantial BFCL degradation and improve cross-task transfer?