- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates if the improvements of the agentic harness optimization methods could be distinguished from run-to-run noise. It evaluates three harness optimization methods on three benchmarks with 8 combinations in total. Each optimized harness is compared with its seed harness on held-out tasks with 8 runs. The result shows, only 2 out of 8 optimizations show distinguishable improvements, 1 reliable regression, and 5 undistinguishable from run-to-run noise. It also includes a case study of the specific modifications of the optimization of Meta-Harness on Terminal-Bench. Result shows the changes are mostly relevant to environmental repair addressing the crash of shell, not directly relevant to the problem solving process. The paper further divides the gain into environmen part and problem solving part, it shows only Meta-Harness on FrontierCS achieving reliable gain in problem solving.
Critical strengths
- The problem is important for harness optimization research. This paper provides compares the gain with the run-to-run noise with an estimate of confidence interval.
- Most of the experimental settings are reasonable. Comparison within the same benchmark and optimization method makes sense for harness optimization.
- The case analysis checked the exact changes the optimization methods have made and clearly showed the changes focus on environmental repair.
- The division of optimization gain between environment and problem solving further attributes the major improvements of harness optimization come from environment repair.
The paper studies an important topic that is relevant to the true value of the research of harness optimization methods. It provides estimation of confidence interval and criterion of distinguishing run-to-run noise with true performance gain. The result shows most improvements very likely comes from the fixing of crash of environment instead of truly problem solving.
Critical weaknesses
- The evidence is not strong enough to support the claim and lacks of causal analysis.
- Each search only runs once per benchmark. For LLM-based search, it could introduce uncertainty to the selected harness itself. Multiple runs of search seems to support the later experiments.
- The subgroup division between environment and problem is unvalidated. The labels are given by Fable 5, but its validity is not mentioned in the paper.
- For different benchmarks, the number of held-out tasks is imbalanced: 19 for TB, 23 for DS and 38 for FCS, which also influences the calculation of the interval. To achieve comparable and more general conclusion, the balance of held-out tasks seems to be necessary.
- The splitting of the runs into environment part and problem part by its failure mode can't isolate the cause of the improvements. E.g. a slow and long trace may be more likely to cause shell crash, if an optimization reduces the length of trace, it also reduces the crash rate.
- Each method uses different seed harness. It would be better if they could search under the same seed to eliminate the bias introduced by the seed itself.
Questions for authors
see weaknesses