acceptodds
Under review as a conference paper at ICLR 2027

HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance–Cost Trade-offs

Abstract

Harness development plays a crucial role in building capable AI agents. Recent approaches improve harnesses through recursive self-improvement (RSI), but their performance gains often come with higher execution costs. We introduce HarnessPro, a label-free method for evolving harnesses to improve task performance while reducing execution cost. In this setting, candidate harnesses are evaluated using LLM-generated tests without access to ground-truth solutions or official tests. Specialized agents diagnose execution trajectories, plan updates with different performance-cost priorities, and implement them as candidate harnesses. Rather than returning a single harness, HarnessPro uses normalized weighted squared-distance selection to provide cost-focused (lite), balanced (medium), and performance-focused (heavy) harnesses for different deployment budgets. Selected harnesses enter the next iteration, where execution feedback guides further updates. Our experiments cover Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, with GPT-5.5 and Codex as well as GLM-5.2 and Claude Code. Across these benchmarks with GPT-5.5 and Codex, the heavy mode of HarnessPro averages a 2.0 percentage point gain in pass rate and a 26.1% reduction in per-task cost relative to the strongest prior label-free baseline. This advantage also holds for GLM-5.2 and Claude Code. Code is available at [this link](https://anonymous.4open.science/r/HarnessPro-ICLR-CDB2).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Recursive self-improvement, Agent harnesses, Multi-agent systems, Cost-efficient agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wTTWOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper improves RHO, a label-free method that evolves the agent harness from past trajectories. HarnessPro keeps the RHO framework but adds cost to the objective, so that the evolved harness uses tokens more efficiently and lowers the total cost. In each round, an analyzer diagnoses repeated rollouts on a small set of training tasks, a planner writes three update plans with different priorities between cost and performance, and an editor turns each plan into a new harness. Instead of asking an LLM judge to compare trajectories, the method scores the candidates on a validation set with tests that a test synthesizer writes once at the start. It then removes candidates that perform worse than the initial harness or that another candidate beats in both pass rate and cost, and it keeps a lite, a medium and a heavy harness (the three modes) for the next round. On Terminal-Bench 2, SWE-Bench Pro and ProgramBench, with GPT-5.5 and Codex as well as GLM-5.2 and Claude Code, the heavy mode reaches a pass rate similar to or higher than RHO at a clearly lower cost.

Critical strengths

Overall, the method makes sense. Cost is a real concern when agents are deployed, and offering several harnesses for different budgets is useful in practice. The evaluation uses three established external benchmarks and two pairs of models and agents, and each experiment is repeated three times (twice on ProgramBench). The appendix also provides the evolution prompts and the evolved harness files, which makes it easy to see what the method actually produces. Finally, replacing the LLM judge with generated tests is a reasonable choice, and the ablation shows that removing the test synthesizer lowers the pass rate by 3.4 to 10.0 points.

Critical weaknesses
  1. The cost of the evolution process is not reported. Since the main claim is lower cost, the authors should report the total cost of evolving the harness (rollouts, analyzer, planner, editor, validation and test synthesis) and how many deployed tasks are needed to recover it.

  2. The increment over RHO is relatively small. The data splits, the task selection and the label-free setting all follow RHO. Against RHO+Iteration, which runs the same number of rounds with a cost-aware instruction, the heavy mode gains only 0.7 points at 3.7% higher cost on SWE-Bench Pro and 1.3 points at 5.8% lower cost on ProgramBench, and the baselines are run only once. I suggest using RHO+Iteration as the main comparison.

  3. The latency claim is inconsistent with Table 2. HarnessPro is mostly faster than RHO, but it is often slower than the vanilla harness. For example, with GPT-5.5 on ProgramBench all three modes take 745 to 985 seconds against 669, so this claim should be narrowed.

  4. The ablations are incomplete. They are single runs on two benchmarks with one model, and the paper does not explain what replaces the planner in the "w/o planner" setting. More importantly, no baseline adds reasoning-effort routing, tool-call budget hooks and tool-output compression together to RHO, so it is unclear whether the gains come from the evolution framework or from these general techniques.

  5. Several writing details need fixing. The abstract says "Code is available at this link" without a link, the selection hyperparameters are in Appendix A.2 rather than in Section 5.1 as the reproducibility statement says, and Figure 4 misspells Sleep-time Compute.

Questions for authors

Please see weakness

Reviewer_XReoOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HarnessPro, a label-free framework for recursively improving agent harnesses while jointly considering task success and execution cost. It uses specialized agents to diagnose execution trajectories, plan and implement harness revisions, and synthesize evaluation tests without relying on official test labels. Candidate harnesses are evaluated under a performance–cost objective, and Pareto-based selection produces lite, medium, and heavy variants for different deployment budgets.

The authors evaluate HarnessPro on Terminal-Bench 2, SWE-Bench Pro, and ProgramBench with two model–agent configurations. On GPT-5.5/Codex, the heavy configuration reportedly improves average task success by 2.0 percentage points and reduces per-task execution cost by 26.1% relative to RHO. The paper also examines individual components and compares LLM-generated tests with direct LLM judging as feedback for harness evolution.

Critical strengths
  1. The paper addresses a practical limitation of approaches that optimize task success without explicitly accounting for inference expenditure. Maintaining alternative Pareto-selected harnesses is useful for deployment settings with different cost constraints. The contribution is significant even if some individual harness modifications resemble established engineering techniques, because the joint optimization objective is relevant to real-world agent operation.

  2. The evaluation covers three benchmarks and two model–agent configurations. The reported improvements combine task success and per-task cost rather than relying on a single metric. The component ablations also help characterize the contribution of planning, synthesized tests, dynamic routing, and budget-related mechanisms. Importantly, the paper does provide an initial comparison of synthesized tests against direct LLM judging, so the proposed feedback mechanism is not entirely unvalidated.

Critical weaknesses
  1. The comparison with direct LLM judging is useful, but it is relatively small and primarily measures failure detection. Recursive optimization also requires candidate-level feedback that reliably distinguishes improvements from regressions across diverse tasks. The available evidence does not fully establish how stable this ranking is over successive evolution rounds. This matters because systematic feedback errors could be amplified when candidate selection repeatedly depends on the same proxy.

  2. The reported reductions concern the execution cost of the selected harnesses. Recursive optimization itself requires candidate generation, testing, and selection, which can be expensive. Without an explicit accounting of total evolution expenditure and the deployment volume needed to amortize it, the strength of the practical cost-efficiency claim remains somewhat uncertain. This does not negate the observed per-task savings, but it limits the deployment-level interpretation.

Questions for authors
  1. Beyond failure-detection accuracy, do the synthesized tests reliably rank candidate harnesses in agreement with independent official evaluations? Any existing candidate-level correlation or disagreement analysis would help establish their suitability for recursive selection.

  2. What is the total cost of one HarnessPro evolution run, including candidate generation and testing? Can the authors estimate the deployment volume at which the observed per-task savings offset the optimization expenditure?

  3. How stable are the performance–cost improvements across independent optimization runs? In particular, how often does the Pareto-selection procedure retain candidates that improve the proxy evaluation but regress on held-out tasks?

Reviewer_muwjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HARNESSPRO, a novel, label-free framework for recursive self-improvement (RSI) of AI agent harnesses. While prior RSI methods often improve task performance at the expense of significantly higher execution costs (token usage and latency), HARNESSPRO explicitly optimizes for the performance-cost Pareto frontier. It employs a multi-agent framework consisting of a Trajectory Analyzer to diagnose failures and overhead, an Evolution Planner to propose targeted modifications, and a Harness Editor to implement these changes. Crucially, instead of relying on optimistic LLM-as-a-judge evaluations, it utilizes a Test Synthesizer to generate fixed, LLM-derived test cases for label-free candidate evaluation.

Critical strengths
  • Targeting the performance-cost trade-off in agent harnesses addresses a critical, real-world deployment bottleneck. Moving beyond pure performance improvement to Pareto optimization is a significant and timely contribution.
  • The introduction of the Test Synthesizer is a major strength. By generating fixed test cases from initial trajectories rather than relying on direct LLM judgment, the framework significantly improves failure detection recall.
  • The clear division of labor among the Analyzer, Planner, and Editor, combined with Git-based version control for edits, ensures that harness evolution is controlled and generalizable.
Critical weaknesses
  • The paper heavily emphasizes the reduction in execution cost per task. However, it does not thoroughly quantify or discuss the computational and financial cost of running the multi-agent evolution process itself, including multiple rollouts, analysis, and planning per epoch. A further discussion on the break-even point would be valuable.
  • The framework relies on the Test Synthesizer to create evaluation criteria without ground truth. For highly complex, ambiguous, or novel tasks, the synthesized tests might be incomplete or brittle, potentially guiding the harness to overfit to these synthetic tests. The authors are encouraged to provide more evidences that the proposed method can address this problem.
Questions for authors

NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.