- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces GP‑Harness, an end‑to‑end framework for optimising LLM agent harnesses under limited expensive evaluation budgets. It couples evolutionary candidate generation with a Gaussian‑process surrogate and expected‑improvement acquisition to select promising harness candidates for full benchmark evaluation. The system compares graph‑structured harness representations against dimension‑matched text representations, and provides theoretical analysis quantifying the performance gain achievable by surrogate‑guided candidate selection. Empirical validation is carried out on two agent‑optimisation benchmarks, with comparisons against several existing agent‑harness optimisation baselines, alongside internal ablations over representation types and acquisition functions.
Critical strengths
- It addresses a practically important real‑world pain point within LLM‑agent research: exhaustive evaluation of every generated agent harness is computationally prohibitive. The paper builds a complete, runnable optimisation pipeline and adopts a rigorous experimental protocol with fixed seeds, held‑out test partitions and explicit confirmation rules to mitigate over‑optimism on in‑sample metrics. This careful experimental setup increases trust in its reported empirical outcomes.
- The manuscript provides formal theoretical analysis that characterises conditions under which surrogate‑driven candidate selection outperforms parent‑ranked evolutionary selection. This theoretical framing offers interpretable guidance for understanding when surrogate‑guided evaluation allocation can deliver practical benefits for harness search.
Critical weaknesses
- Since Gaussian‑process surrogates and expected‑improvement acquisition are well‑studied combinatorial optimisation tools, the core novelty hinges on harness representation and candidate‑selection design. However, controlled comparisons against alternative surrogate models are absent under shared candidate generation and evaluation budget. Without these results it is impossible to isolate whether observed gains stem from the Gaussian‑process or other pipeline components, which undermines the paper’s central contribution claim.
- The graph versus text representation comparison suffers from an information‑asymmetry confound due to text‑encoder truncation, so observed performance gaps cannot be reliably attributed to graph topology. Furthermore, theoretical thresholds are derived for Pearson correlation but empirical diagnostics use non‑equivalent Spearman rank correlation, and correlations are only computed over selected candidates rather than full candidate pools. In addition, evidence for GP uncertainty calibration is poor and the small margin over the strong HarnessBank baseline lacks proven statistical significance. Collectively these issues weaken both the interpretability of ablation results and the credibility of the main empirical conclusions.
Questions for authors
- Under identical candidate generation and evaluation budget settings, can you provide results for alternative surrogate models such as random forests, to identify whether the performance gains are specifically attributable to Gaussian‑process modelling?
- In your graph‑text representation comparison, given the text encoder truncates long harness descriptions while graph encoding preserves workflow‑state information, how do you rule out differing input‑information volume as the cause of the observed performance gap, rather than graph‑structural features themselves?
- Your theoretical bounds in Proposition 1 and Corollary 1 rely on Pearson correlation, yet you report Spearman rank correlation in Table 3, and compute these values only on acquisition‑selected candidates. Can you clarify whether your theoretical thresholds still apply to your reported empirical metrics, given the metric mismatch and restricted candidate sample?
- Your GP shows low correlation between predictive standard deviation and actual prediction error. Is the improved performance of EI over posterior‑mean selection driven by calibrated uncertainty estimates, or purely by general exploratory behaviour of the acquisition function?
- Can you provide paired confidence intervals for the performance difference between GP‑Harness and HarnessBank, and comment on whether the observed performance margin is statistically significant under your experimental setup?
Response to Reviewer 2XTb
Thank you for the review. We reply to each question.
Q1. Is the GP necessary? Alternative surrogates. Our theoretical claim concerns candidate-level evaluation allocation, not superiority of the GP family: in Proposition 1 and its dependent-candidate extension the gain depends on score-fitness association, candidate variability and dependence, so other predictors with good scores can also benefit. The GP's specific role is to supply the predictive distribution that EI uses. We added a result that a fitted score whose error is at most ε loses at most 2ε relative to the conditional mean, and a sensitivity bound for Gaussian EI in terms of errors in the predictive mean and standard deviation (Appendix B.5 and B.7). Empirically, offline on identical histories and the same 16 features (New result 7), Bayesian ridge regression and a random forest rank candidates nearly as well as the GP (0.42 and 0.41 versus 0.48 on Terminal-Bench 2.0; 0.39 and 0.38 versus 0.44 on SWE-bench Lite), within about one standard error; the GP's advantage is in predictive density (NLPD about 0.3 better than the random forest). We therefore do not claim that the GP is necessary for ranking. These are offline shared-history comparisons; we did not run online searches with the alternatives.
Q2. Information asymmetry in the graph-text comparison. Your concern is well founded. The text encoder retains a median of 32% and 29% of each serialised harness, against full retention of per-state descriptions on the graph route (New result 6). On identical histories, removing the truncation raises the text route's within-pool rank correlation from 0.33 to 0.37 and from 0.30 to 0.35; removing the graph route's structural features lowers it by only 0.03-0.04, and structure alone is weakest (New result 7). We no longer attribute the gap to topology, and Sec. 4.2 has been revised accordingly.
Q3. Do the thresholds apply to the reported metrics? The two statistics differ, but under the bivariate Gaussian model of Proposition 1 they satisfy the exact identity r=2sin(πρs/6) (proof in Appendix B.6), which maps 0.48 and 0.44 to about 0.50 and 0.46. These are plug-in conversions, not estimates of the latent within-frontier correlation. The larger issue is the sample: Table 3 uses noisy labels on acquisition-selected candidates pooled across rounds, so it cannot certify a within-frontier threshold, and independent noise attenuates Pearson correlation by σf/σf2+σϵ2 without fixing the direction of the selection effect. We therefore do not use those values as a lower bound and have removed our claim in Sec. 3.5 and Appendix B.1 of the submitted version that they are conservative estimates. The shared-frontier experiment measures ranking on unselected candidates directly (New result 1): within-pool r = 0.43 and 0.41 for the graph GP, comparable to the independent-candidate reference 0.41; with correlated candidates the relevant condition also involves the covariance term qM(R).
Q4. Calibration versus exploration. A low correlation between predicted standard deviation and absolute error does not by itself show poor calibration: we give an exact counterexample in which a perfectly calibrated Gaussian predictor has Pearson and Spearman correlations between scale and error arbitrarily close to zero (Appendix B.8). Table 3 also compares latent-function uncertainty with residuals that include execution noise. Conversely, NLPD alone does not isolate calibrated uncertainty. EI's role is nevertheless well defined: it is the model-expected positive improvement and increases strictly with predictive standard deviation at fixed mean. Empirically, EI and posterior-mean selection are indistinguishable within a round on identical pools (+0.006 ± 0.011, +0.004 ± 0.009; New result 2), and the online VALID gaps (6.2 and 5.5 points) have intervals including zero on each benchmark (New result 3). We therefore do not claim a benefit from calibrated uncertainty, nor that the online difference measures exploration; Sec. 4.2 of the revision states that any benefit of EI over posterior-mean selection is not individually established at six seeds and that the present evidence does not identify how the uncertainty term contributes to search.
Q5. Paired intervals against HarnessBank. We found that the submitted Table 8 standard errors treated the two methods as independent rather than using seed-level paired differences. New result 4 gives the corrected comparison against the founder. For the baseline rows, including HarnessBank, the revised Table 8 reports means only, because their seed-level records are still being re-verified against the raw run logs. The paired intervals cannot exclude zero for HarnessBank under any of the standard errors we have computed, so we do not claim significance. Sec. 4.1 now states explicitly that the margin over HarnessBank is not established.