acceptodds
Desk rejected at ICLR 2027

Can Harness Optimization Improve Agents? Generalizable Gains Versus Evaluation Noise

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Harness optimization methods revise an agent’s prompts, tools, memory, and run- time control without updating model weights. We ask which of their improve- ments can be distinguished from evaluation noise, and where the reliable ones come from. We evaluate Meta-Harness, Agentic Harness Engineering, and Ret- rospective Harness Optimization on Terminal-Bench 2.1, DeepSWE, and Fron- tierCS, eight method–benchmark evaluations in all. In each, the selected harness and its seed harness receive repeated independent runs on every held-out task un- der the same model, budgets, and environment. The benchmark-level effect is an equally weighted mean over tasks, and its confidence interval reflects run-to-run variation. Two of the eight selected harnesses improve reliably on their seed har- nesses, five effects cannot be distinguished from evaluation noise, and one is a reliable regression. In an additional Meta-Harness search on Terminal-Bench 2.1, most of the selected harness’s held-out improvement comes from tasks on which the seed harness’s shell tends to crash. The selected harness restarts the shell after a crash; on the other tasks, there is no clear evidence of improvement. Repairing such an environment failure can raise held-out performance without showing that the agent solves problems better. We analyze change targets and observed failure causes to distinguish evidence of environment repair from evidence of improved problem solving.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Harness Optimization, Agent Evaluation, Statistical Reliability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_o7scOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates if the improvements of the agentic harness optimization methods could be distinguished from run-to-run noise. It evaluates three harness optimization methods on three benchmarks with 8 combinations in total. Each optimized harness is compared with its seed harness on held-out tasks with 8 runs. The result shows, only 2 out of 8 optimizations show distinguishable improvements, 1 reliable regression, and 5 undistinguishable from run-to-run noise. It also includes a case study of the specific modifications of the optimization of Meta-Harness on Terminal-Bench. Result shows the changes are mostly relevant to environmental repair addressing the crash of shell, not directly relevant to the problem solving process. The paper further divides the gain into environmen part and problem solving part, it shows only Meta-Harness on FrontierCS achieving reliable gain in problem solving.

Critical strengths
  • The problem is important for harness optimization research. This paper provides compares the gain with the run-to-run noise with an estimate of confidence interval.
  • Most of the experimental settings are reasonable. Comparison within the same benchmark and optimization method makes sense for harness optimization.
  • The case analysis checked the exact changes the optimization methods have made and clearly showed the changes focus on environmental repair.
  • The division of optimization gain between environment and problem solving further attributes the major improvements of harness optimization come from environment repair.

The paper studies an important topic that is relevant to the true value of the research of harness optimization methods. It provides estimation of confidence interval and criterion of distinguishing run-to-run noise with true performance gain. The result shows most improvements very likely comes from the fixing of crash of environment instead of truly problem solving.

Critical weaknesses
  • The evidence is not strong enough to support the claim and lacks of causal analysis.
  • Each search only runs once per benchmark. For LLM-based search, it could introduce uncertainty to the selected harness itself. Multiple runs of search seems to support the later experiments.
  • The subgroup division between environment and problem is unvalidated. The labels are given by Fable 5, but its validity is not mentioned in the paper.
  • For different benchmarks, the number of held-out tasks is imbalanced: 19 for TB, 23 for DS and 38 for FCS, which also influences the calculation of the interval. To achieve comparable and more general conclusion, the balance of held-out tasks seems to be necessary.
  • The splitting of the runs into environment part and problem part by its failure mode can't isolate the cause of the improvements. E.g. a slow and long trace may be more likely to cause shell crash, if an optimization reduces the length of trace, it also reduces the crash rate.
  • Each method uses different seed harness. It would be better if they could search under the same seed to eliminate the bias introduced by the seed itself.
Questions for authors

see weaknesses

Reviewer_Try7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines whether improvements from automatic harness optimization are distinguishable from evaluation noise and what explains the improvements that remain reliable. It evaluates Meta-Harness, Agentic Harness Engineering, and Retrospective Harness Optimization across Terminal-Bench 2.1, DeepSWE, and FrontierCS, yielding eight method–benchmark configurations.

Critical strengths
  1. The evaluation protocol contains several thoughtful controls.
Critical weaknesses

The main concern is that the paper includes an explicitly provisional evaluation in its headline results without adequately documenting its completion status. Table 1 and Figure 1 state that “MH/DS results are provisional,” while Appendix A states that “The provisional evaluations described in Section 3.1 await completion of this protocol.” However, the manuscript does not specify which protocol requirements remain unmet, how many scored runs have actually been completed for each task and harness, or how the outstanding work affects the reported estimate and confidence interval.

This omission is particularly consequential because the paper’s central contribution is to assess statistical reliability through repeated evaluation. The headline conclusion of two reliable improvements, five unresolved effects, and one reliable regression includes MH/DeepSWE, yet readers cannot verify that this comparison satisfies the stated requirement of eight scored runs per task and harness. Labeling a result as provisional acknowledges the limitation but does not provide the information needed to assess its validity.

The authors should complete the MH/DeepSWE evaluation and report its actual sample sizes and any departures from the protocol. Alternatively, they should exclude this configuration from the completed evaluation and revise the aggregate claims accordingly. Until this is resolved, the headline eight-comparison summary is not fully supported by a documented, completed experimental protocol.

Questions for authors

see weakness

Reviewer_RRNSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether the improvements produced by harness optimization methods are reliable or just evaluation noise, through a statistical view. It evaluates eight method–benchmark settings, using gpt-5.6-terra with high reasoning effort. The results show two clear improvements, one clear regression, and five effects that are not statistically significant.

Critical strengths
  • the problem of “whether the gains of harness evolution are reliable or just random noise” is interesting and important to explore
  • I like the way it addresses the question through a statistical view, analyzing if the effects are statistically significant
  • The evaluation is comprehensive, covering eight method–benchmark settings.
  • The case study is interesting, and I agree with the authors that examining traces is necessary to understand where improvements come from.
Critical weaknesses
  • My primary concern is that only the evaluation is repeated, and the harness evolution algorithms themselves are run only once per setting, so the results cannot show whether a method reliably produces improvements across optimization runs.

  • The paper separates “environment repair” from evidence of improved problem solving. However, fixing failures in the agent’s interaction with its environment is precisely where a harness can help. The authors should justify why this distinction matters.

  • The writing needs substantial improvement: -- Several sentences are unclear; for example, it is not clear what “The selected harness and its seed harness receive repeated independent runs” means. -- The structure is hard to follow. -- The introduction is too long. Some details can be skipped.

Overall, I think the contribution is limited.

Questions for authors

The main text lacks important experimental details. For example, Section 3.1 states: “We evaluate Meta-Harness (MH), AHE, and Retro Harness (RH), which optimize different components of an agent harness.” Which components does each method optimize, and how do they differ?

It is unclear why the noise range is extrapolated to J = 64 and J = 256, or whether such extrapolation is meaningful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.