- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper improves RHO, a label-free method that evolves the agent harness from past trajectories. HarnessPro keeps the RHO framework but adds cost to the objective, so that the evolved harness uses tokens more efficiently and lowers the total cost. In each round, an analyzer diagnoses repeated rollouts on a small set of training tasks, a planner writes three update plans with different priorities between cost and performance, and an editor turns each plan into a new harness. Instead of asking an LLM judge to compare trajectories, the method scores the candidates on a validation set with tests that a test synthesizer writes once at the start. It then removes candidates that perform worse than the initial harness or that another candidate beats in both pass rate and cost, and it keeps a lite, a medium and a heavy harness (the three modes) for the next round. On Terminal-Bench 2, SWE-Bench Pro and ProgramBench, with GPT-5.5 and Codex as well as GLM-5.2 and Claude Code, the heavy mode reaches a pass rate similar to or higher than RHO at a clearly lower cost.
Critical strengths
Overall, the method makes sense. Cost is a real concern when agents are deployed, and offering several harnesses for different budgets is useful in practice. The evaluation uses three established external benchmarks and two pairs of models and agents, and each experiment is repeated three times (twice on ProgramBench). The appendix also provides the evolution prompts and the evolved harness files, which makes it easy to see what the method actually produces. Finally, replacing the LLM judge with generated tests is a reasonable choice, and the ablation shows that removing the test synthesizer lowers the pass rate by 3.4 to 10.0 points.
Critical weaknesses
-
The cost of the evolution process is not reported. Since the main claim is lower cost, the authors should report the total cost of evolving the harness (rollouts, analyzer, planner, editor, validation and test synthesis) and how many deployed tasks are needed to recover it.
-
The increment over RHO is relatively small. The data splits, the task selection and the label-free setting all follow RHO. Against RHO+Iteration, which runs the same number of rounds with a cost-aware instruction, the heavy mode gains only 0.7 points at 3.7% higher cost on SWE-Bench Pro and 1.3 points at 5.8% lower cost on ProgramBench, and the baselines are run only once. I suggest using RHO+Iteration as the main comparison.
-
The latency claim is inconsistent with Table 2. HarnessPro is mostly faster than RHO, but it is often slower than the vanilla harness. For example, with GPT-5.5 on ProgramBench all three modes take 745 to 985 seconds against 669, so this claim should be narrowed.
-
The ablations are incomplete. They are single runs on two benchmarks with one model, and the paper does not explain what replaces the planner in the "w/o planner" setting. More importantly, no baseline adds reasoning-effort routing, tool-call budget hooks and tool-output compression together to RHO, so it is unclear whether the gains come from the evolution framework or from these general techniques.
-
Several writing details need fixing. The abstract says "Code is available at this link" without a link, the selection hyperparameters are in Appendix A.2 rather than in Section 5.1 as the reproducibility statement says, and Figure 4 misspells Sleep-time Compute.
Questions for authors
Please see weakness