acceptodds
Under review as a conference paper at ICLR 2027

GP-Harness: Sequential Bayesian–Evolutionary Optimisation of LLM Harnesses

Abstract

Large language model agents rely on harnesses that organise planning, tool use, verification, and failure recovery around a fixed model. Automatic harness search can generate more candidates than can be evaluated on long-horizon tasks, making the choice of which candidates to evaluate a central challenge. We introduce , which combines evolutionary candidate generation with Bayesian optimisation to guide this choice. Starting from eight adapted open-source harnesses, programmatic mutation and recombination generate candidate pools. A Gaussian-process surrogate over graph features uses previous evaluations to predict candidate fitness and uncertainty, and expected improvement selects one candidate for evaluation in each round. Within each benchmark, all harnesses use a shared executor, with the task model, tool interfaces, evaluator, and per-task execution limits held fixed. With 20 new-candidate evaluations per run, achieves 64.6% mean success on a held-out 24-task Terminal-Bench 2.0 subset and a 43.3% mean resolved rate on the official 300-instance SWE-bench Lite test set across six search seeds, including runs that fall back to an initial harness. These results improve on the initial harness selected by its performance on the search tasks by 10.4 and 8.0 percentage points, respectively, and on budget-matched surrogate-free evolution by 4.9 and 4.6 points. Online ablations favour graph features over dimension-matched text features and uncertainty-aware acquisition over posterior-mean selection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model agents, harness optimisation, Bayesian optimisation, evolutionary search, Gaussian processes, tool use

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2XTbOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces GP‑Harness, an end‑to‑end framework for optimising LLM agent harnesses under limited expensive evaluation budgets. It couples evolutionary candidate generation with a Gaussian‑process surrogate and expected‑improvement acquisition to select promising harness candidates for full benchmark evaluation. The system compares graph‑structured harness representations against dimension‑matched text representations, and provides theoretical analysis quantifying the performance gain achievable by surrogate‑guided candidate selection. Empirical validation is carried out on two agent‑optimisation benchmarks, with comparisons against several existing agent‑harness optimisation baselines, alongside internal ablations over representation types and acquisition functions.

Critical strengths
  1. It addresses a practically important real‑world pain point within LLM‑agent research: exhaustive evaluation of every generated agent harness is computationally prohibitive. The paper builds a complete, runnable optimisation pipeline and adopts a rigorous experimental protocol with fixed seeds, held‑out test partitions and explicit confirmation rules to mitigate over‑optimism on in‑sample metrics. This careful experimental setup increases trust in its reported empirical outcomes.
  2. The manuscript provides formal theoretical analysis that characterises conditions under which surrogate‑driven candidate selection outperforms parent‑ranked evolutionary selection. This theoretical framing offers interpretable guidance for understanding when surrogate‑guided evaluation allocation can deliver practical benefits for harness search.
Critical weaknesses
  1. Since Gaussian‑process surrogates and expected‑improvement acquisition are well‑studied combinatorial optimisation tools, the core novelty hinges on harness representation and candidate‑selection design. However, controlled comparisons against alternative surrogate models are absent under shared candidate generation and evaluation budget. Without these results it is impossible to isolate whether observed gains stem from the Gaussian‑process or other pipeline components, which undermines the paper’s central contribution claim.
  2. The graph versus text representation comparison suffers from an information‑asymmetry confound due to text‑encoder truncation, so observed performance gaps cannot be reliably attributed to graph topology. Furthermore, theoretical thresholds are derived for Pearson correlation but empirical diagnostics use non‑equivalent Spearman rank correlation, and correlations are only computed over selected candidates rather than full candidate pools. In addition, evidence for GP uncertainty calibration is poor and the small margin over the strong HarnessBank baseline lacks proven statistical significance. Collectively these issues weaken both the interpretability of ablation results and the credibility of the main empirical conclusions.
Questions for authors
  1. Under identical candidate generation and evaluation budget settings, can you provide results for alternative surrogate models such as random forests, to identify whether the performance gains are specifically attributable to Gaussian‑process modelling?
  2. In your graph‑text representation comparison, given the text encoder truncates long harness descriptions while graph encoding preserves workflow‑state information, how do you rule out differing input‑information volume as the cause of the observed performance gap, rather than graph‑structural features themselves?
  3. Your theoretical bounds in Proposition 1 and Corollary 1 rely on Pearson correlation, yet you report Spearman rank correlation in Table 3, and compute these values only on acquisition‑selected candidates. Can you clarify whether your theoretical thresholds still apply to your reported empirical metrics, given the metric mismatch and restricted candidate sample?
  4. Your GP shows low correlation between predictive standard deviation and actual prediction error. Is the improved performance of EI over posterior‑mean selection driven by calibrated uncertainty estimates, or purely by general exploratory behaviour of the acquisition function?
  5. Can you provide paired confidence intervals for the performance difference between GP‑Harness and HarnessBank, and comment on whether the observed performance margin is statistically significant under your experimental setup?
1 comment

Response to Reviewer 2XTb

Thank you for the review. We reply to each question.

Q1. Is the GP necessary? Alternative surrogates. Our theoretical claim concerns candidate-level evaluation allocation, not superiority of the GP family: in Proposition 1 and its dependent-candidate extension the gain depends on score-fitness association, candidate variability and dependence, so other predictors with good scores can also benefit. The GP's specific role is to supply the predictive distribution that EI uses. We added a result that a fitted score whose error is at most ε loses at most 2ε relative to the conditional mean, and a sensitivity bound for Gaussian EI in terms of errors in the predictive mean and standard deviation (Appendix B.5 and B.7). Empirically, offline on identical histories and the same 16 features (New result 7), Bayesian ridge regression and a random forest rank candidates nearly as well as the GP (0.42 and 0.41 versus 0.48 on Terminal-Bench 2.0; 0.39 and 0.38 versus 0.44 on SWE-bench Lite), within about one standard error; the GP's advantage is in predictive density (NLPD about 0.3 better than the random forest). We therefore do not claim that the GP is necessary for ranking. These are offline shared-history comparisons; we did not run online searches with the alternatives.

Q2. Information asymmetry in the graph-text comparison. Your concern is well founded. The text encoder retains a median of 32% and 29% of each serialised harness, against full retention of per-state descriptions on the graph route (New result 6). On identical histories, removing the truncation raises the text route's within-pool rank correlation from 0.33 to 0.37 and from 0.30 to 0.35; removing the graph route's structural features lowers it by only 0.03-0.04, and structure alone is weakest (New result 7). We no longer attribute the gap to topology, and Sec. 4.2 has been revised accordingly.

Q3. Do the thresholds apply to the reported metrics? The two statistics differ, but under the bivariate Gaussian model of Proposition 1 they satisfy the exact identity r=2sin⁡(πρs/6)r = 2\sin(\pi\rho_s/6) (proof in Appendix B.6), which maps 0.48 and 0.44 to about 0.50 and 0.46. These are plug-in conversions, not estimates of the latent within-frontier correlation. The larger issue is the sample: Table 3 uses noisy labels on acquisition-selected candidates pooled across rounds, so it cannot certify a within-frontier threshold, and independent noise attenuates Pearson correlation by σf/σf2+σϵ2\sigma_f/\sqrt{\sigma_f^2+\sigma_\epsilon^2} without fixing the direction of the selection effect. We therefore do not use those values as a lower bound and have removed our claim in Sec. 3.5 and Appendix B.1 of the submitted version that they are conservative estimates. The shared-frontier experiment measures ranking on unselected candidates directly (New result 1): within-pool r = 0.43 and 0.41 for the graph GP, comparable to the independent-candidate reference 0.41; with correlated candidates the relevant condition also involves the covariance term qM(R)q_M(R).

Q4. Calibration versus exploration. A low correlation between predicted standard deviation and absolute error does not by itself show poor calibration: we give an exact counterexample in which a perfectly calibrated Gaussian predictor has Pearson and Spearman correlations between scale and error arbitrarily close to zero (Appendix B.8). Table 3 also compares latent-function uncertainty with residuals that include execution noise. Conversely, NLPD alone does not isolate calibrated uncertainty. EI's role is nevertheless well defined: it is the model-expected positive improvement and increases strictly with predictive standard deviation at fixed mean. Empirically, EI and posterior-mean selection are indistinguishable within a round on identical pools (+0.006 ± 0.011, +0.004 ± 0.009; New result 2), and the online VALID gaps (6.2 and 5.5 points) have intervals including zero on each benchmark (New result 3). We therefore do not claim a benefit from calibrated uncertainty, nor that the online difference measures exploration; Sec. 4.2 of the revision states that any benefit of EI over posterior-mean selection is not individually established at six seeds and that the present evidence does not identify how the uncertainty term contributes to search.

Q5. Paired intervals against HarnessBank. We found that the submitted Table 8 standard errors treated the two methods as independent rather than using seed-level paired differences. New result 4 gives the corrected comparison against the founder. For the baseline rows, including HarnessBank, the revised Table 8 reports means only, because their seed-level records are still being re-verified against the raw run logs. The paired intervals cannot exclude zero for HarnessBank under any of the standard errors we have computed, so we do not claim significance. Sec. 4.1 now states explicitly that the margin over HarnessBank is not established.

Reviewer_nBUiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

GP-Harness studies which candidates to evaluate during automated harness search for LLM agents. It mutates and recombines the current best harnesses, then uses a Gaussian process over frozen graph features of their declared workflows and expected improvement to select one candidate per round for full evaluation. The final candidate must pass a validation check against the founders. On Terminal-Bench 2.0 and SWE-bench Lite, it reports higher mean held-out scores than the founders, surrogate-free evolution, and four adapted harness optimisers.

Critical strengths
  • The paper addresses a practical bottleneck. Internal controls differ only in candidate selection, making the comparison clear.
  • The protocol separates search, validation, and test data, freezes representations, and includes founder fallbacks in test means. Limitations are stated openly.
Critical weaknesses
  • The main empirical claim is not statistically supported. Table 1 is the central evidence that GP-Harness beats prior methods, but its gap over the strongest baseline, HarnessBank, is smaller than the reported standard errors, and the authors themselves note that these results do not establish the margin (Sec. 4.1). With so few seeds for the main comparison and the ablations (Sec. 4.3), it is hard to tell whether the gains are real.
  • Novelty over graph-based Bayesian optimisation needs clarification. NAS-BOWL also mutates the best architectures, scores candidates with a Weisfeiler-Lehman GP, and selects by expected improvement. The paper does not discuss this overlap.
  • The graph-versus-text comparison does not isolate structure: text inputs are truncated, while graph features cover every state and include text embeddings (Secs. 3.2, 4.2). Since the declared workflow is not enforced at runtime (Sec. 1), the method appears closer to structured prompt optimisation for a fixed agent. Only one task model is tested.
  • Total cost is missing. Budgets match only full candidate evaluations, excluding parent comparisons, screening, and baseline proposer calls (Sec. 4.3). Mutation and recombination operators also lack detail (Sec. 3.3).

Reference:

Questions for authors
  1. How does GP-Harness differ from NAS-BOWL and related graph-based Bayesian optimisation methods?
  2. Would more seeds or repeated test runs establish gains over surrogate-free evolution and HarnessBank?
  3. What is each method's total cost, including proposer calls, parent comparisons, and screening? 4.. What exactly do mutation and recombination change? Could you show an evolved harness?
  4. How often does the executor follow the declared workflow when transitions are not enforced?
2 comments

Response to Reviewer nBUi

Thank you for the review and for pointing us to NAS-BOWL. We reply to each question.

Q1. Relation to NAS-BOWL and graph-based Bayesian optimisation. NAS-BOWL (Ru et al., 2021) is a relevant antecedent: it combines a Weisfeiler-Lehman graph-kernel GP, candidate generation including mutations of strong observed architectures, and acquisition-based evaluation. We do not claim GP regression, EI or evolutionary proposals as new; our contribution is the formulation and implementation of sequential evaluation allocation for LLM harness search. The searched objects and representation differ: harness nodes carry state-level instructions, tool and skill requirements, recovery logic and declared transitions interpreted by a shared LLM executor, so we combine semantic and structural descriptors, freeze their transformation before search, and fit a Matérn GP in that feature space rather than a WL kernel on architecture graphs. The search also separates the data used for surrogate learning from the population admitted for breeding. On identical search histories, a GP with a WL graph kernel ranks candidates less well than our feature-based GP (prequential Spearman 0.36 versus 0.48 on Terminal-Bench 2.0, 0.34 versus 0.44 on SWE-bench Lite), close to structure-only features, consistent with harness nodes carrying free-text content that a label-based kernel does not see (New result 7). Sec. 2 of the revised version discusses NAS-BOWL.

Q2. More seeds or repeated test runs. We agree that six seeds and a 24-task Terminal-Bench test set limit what can be established, and Sec. 4.3 states both limits. New result 4 gives the corrected paired comparison against the founder; for the baseline rows the revised Table 8 reports means only, because their seed-level records are still being re-verified. Sec. 4.1 states explicitly that the margin over HarnessBank is not established. The shared-frontier experiment (New results 1-2) adds evidence that does not depend on independent search trajectories.

Q3. Total cost. New result 5. GP-Harness makes no proposer-model calls and uses the fewest task-model tokens and the least wall-clock time on both benchmarks; its executions are within 2% of surrogate-free evolution, while the methods that screen proposals use 13-29% more. The one-time founder-calibration pool is shared by all methods and reported separately. Sec. 4.3 summarises it, and Appendix A.7 gives the full table.

Q4. What mutation and recombination change. The generator makes no model calls and edits the harness specification directly (Appendix A.2, which we have expanded). A harness is represented as a base workflow (chassis), a set of skill modules, and a list of structural edits. Recombination of two parents takes the chassis and its structural edits from one parent and inherits each skill module from either parent with probability 0.5 (always when both parents share it). Mutation then adds a skill module from the founders' skill pool, drops an optional one, and, less often, switches the chassis; the mutation probability parameter (0.5 in Table 6) is the probability of each add or drop, and switching the chassis has probability 0.2. Each offspring also receives one to six structural edits that graft a state from another parent into an existing transition: a single state, a two-state subgraph, a branch, an evidence gate, or a validation-repair loop. New prompt text is therefore not generated: a grafted state reuses the donor state's description, followed by fixed template text that adapts it to the new position and specifies its evidence and control requirements. Candidates then pass validity, duplicate and novelty checks (Appendix C.1) before scoring. The final version will also show one evolved harness next to its parents, with the edited states, transitions and prompts marked.

Q5. How often the executor follows the declared workflow. We agree this is central to interpreting the graph features. The declared workflow conditions a prompt-driven executor and is not enforced, so declared differences need not translate fully into behaviour. We have not completed a validated measurement of adherence from execution traces, so we make no quantitative claim here, and Sec. 4.3 now states this limitation; we describe the graph as a declared rather than enforced structure. The search space is still structured: candidates must pass role-based novelty thresholds against all references (Appendix C.1), and a prompt edit that preserves inferred node roles, start and wrap-up indicators and graph connections is rejected as a topology duplicate (Appendix A.2), although a prompt edit can change an inferred role. We also agree that one task model limits generality; the abstract now states it.

Thank you for your rebuttal and additional analyses. I still believe that the method proposed in this paper lacks sufficient novelty, and the existing experiments do not support the paper's claims; therefore, after careful consideration, I have decided to maintain my original score.

Reviewer_SKXpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GP-HARNESS, combining evolutionary candidate generation with Gaussian-process-based Bayesian optimization to improve evaluation efficiency in LLM harness search. Experiments on Terminal-Bench 2.0 and SWE-bench Lite show promising performance improvements over evolutionary and existing harness optimization baselines, with ablations examining graph representations and uncertainty-aware acquisition.

Critical strengths
  1. The paper addresses an practical bottleneck in automatic LLM harness optimization: evaluating candidate harnesses can be substantially more expensive than generating them. Rather than focusing exclusively on candidate generation, the proposed approach explicitly optimizes evaluation allocation. This is a meaningful direction for improving the sample efficiency of agent system optimization.
  2. GP-HARNESS integrates evolutionary candidate generation, graph-based harness representations, and Gaussian-process acquisition into a unified optimization framework. The distinction between the surrogate dataset and the breeding population is particularly sensible, allowing evaluations of unsuccessful offspring to inform subsequent candidate selection.
Critical weaknesses
  1. My main concern is whether the reported gains are sufficiently robust to support the paper's central empirical claims. Although Table 1 reports consistent improvements in mean performance, the paired comparisons in Table 8 reveal substantial uncertainty.

    For example, GP-HARNESS improves over surrogate-free evolution by 4.9 ± 8.1 percentage points on Terminal-Bench 2.0 and 4.6 ± 4.3 points on SWE-bench Lite. The improvements over HarnessBank are even smaller, at 2.8 ± 6.6 and 2.1 ± 4.5 points, respectively. These uncertainty estimates are large relative to the reported gains.

    In fact, using conventional two-sided 95% t-intervals with six seeds, none of the paired improvements in Table 8 clearly excludes zero. The authors appropriately acknowledge some of these limitations, but this substantially weakens the evidence for a consistent advantage over competing approaches.

    The online ablations in Table 2 further rely on only three seeds, making it difficult to draw strong conclusions about the benefits of the representation and acquisition components.

    I would encourage the authors to report paired confidence intervals and seed-level results more prominently, ideally with additional runs. A task-level and seed-level uncertainty analysis would also help distinguish genuine improvements from evaluation noise.

  2. The paper emphasizes evaluation efficiency by restricting each search method to 20 complete candidate evaluations. However, this budget excludes several potentially important computational costs, including founder calibration, proposal generation, partial candidate screening, parent re-evaluations, and validation.

    This distinction matters because GP-HARNESS generates and filters 32 candidates per optimization round, while competing approaches may involve additional proposer-model calls or method-specific screening procedures. Consequently, matching the number of complete candidate evaluations does not necessarily imply comparable total computation.

    Although Appendix A.9 describes a resource-accounting protocol, the manuscript does not present a quantitative comparison of total model tokens, harness-task executions, wall-clock time, or overall search cost across methods.

    Since evaluation efficiency is a central motivation, reporting these costs would considerably strengthen the contribution. In particular, performance-versus-compute curves would be more informative than comparisons based solely on the number of completed candidate evaluations.

  3. The paper attributes part of the improvement to graph-based harness representations. However, the comparison against text embeddings in Table 2 involves substantially different representation pipelines.

    In particular, the text baseline uses a fixed-length encoder that may truncate long harness descriptions, whereas the graph-based representation processes individual states and aggregates information across the workflow. Therefore, the graph representation may benefit simply from retaining more information rather than from exploiting graph structure.

    Furthermore, the novelty filtering procedure explicitly considers graph topology and role-based structural differences. This may influence the candidate distribution in ways that favor the graph representation.

    The authors acknowledge that their experiments do not isolate topology, which is appreciated. Nevertheless, additional controlled ablations would be valuable, such as a chunked-text embedding baseline without truncation, a graph representation without structural features, or comparisons separating structural and semantic information.

    An additional consideration is that the workflow graph describes declared transitions rather than transitions enforced by the runtime. It would therefore be useful to examine whether the structural properties represented by the graph correspond to meaningful differences in actual agent behavior.

  4. Section 3.5 offers an interesting theoretical perspective on surrogate-guided selection. However, Proposition 1 relies on independent jointly Gaussian candidate-score pairs, whereas actual candidates are generated through mutations and recombinations of shared parents. Such candidates are likely to be correlated, and their distributions evolve adaptively throughout the search.

    More importantly, the prequential rank correlations in Table 3 are computed over candidates selected for evaluation. They do not directly measure ranking quality within each 32-candidate frontier, which is the quantity most relevant to Proposition 1 and the acquisition decision.

    Consequently, relating the observed correlations to the theoretical threshold in Corollary 1 is suggestive but does not establish that the assumptions underlying the comparison hold in practice.

    The evidence supporting the role of predictive uncertainty is also limited. The reported correlations between posterior uncertainty and prediction error are weak, while the EI-versus-posterior-mean comparison involves only three search seeds and different online search trajectories.

    I would encourage the authors to distinguish more carefully between theoretical intuition and empirically established mechanisms. Evaluating a subset of unselected candidates on shared frontiers could provide more direct evidence of surrogate ranking quality and the benefit of uncertainty-aware selection.

  5. The framework combines established ingredients, including Gaussian-process Bayesian optimization, expected improvement, evolutionary search, and graph-based feature extraction. Although their application to harness optimization is interesting, the methodological novelty beyond this combination could be articulated more clearly.

    In particular, it would be useful to compare GP-based candidate selection with simpler learned predictors or alternative surrogate-based selection strategies using the same representations and candidate pools. Such comparisons would help establish whether the GP and its uncertainty estimates are essential to the observed gains.

    The experimental scope also remains relatively narrow. The study uses a single task model, a fixed 20-evaluation budget, and only 24 held-out Terminal-Bench tasks. In addition, the prior harness optimization baselines are adapted to a shared executor rather than evaluated with their complete native implementations.

    These choices support controlled comparisons, but they limit conclusions about broader applicability and performance relative to the original systems. Experiments across additional task models, larger budgets, or more diverse benchmarks would help establish the robustness of the proposed approach.

Questions for authors
  1. Given the substantial uncertainty in Table 8, can the authors provide stronger statistical evidence that GP-HARNESS consistently outperforms surrogate-free evolution and HarnessBank? Additional paired seeds or confidence intervals over seed-level differences would be particularly helpful.
  2. What is the total computational cost of each method when accounting for proposal generation, screening, parent comparisons, and validation? Does GP-HARNESS retain its advantage under matched token budgets or wall-clock budgets?
  3. How much of the advantage of graph-based representations remains when compared against a text baseline that processes the entire harness without truncation? Can the authors isolate the contributions of graph topology, state-level semantic information, and the novelty filtering procedure?
  4. Is the improvement from GP-based acquisition primarily attributable to better candidate-level fitness prediction or to uncertainty-guided exploration? Could the authors provide a controlled comparison with simpler predictive models or evaluate ranking accuracy on partially evaluated candidate frontiers?
3 comments

Response to Reviewer SKXp (1/2)

Thank you for the detailed review. We reply point by point; the general response summarises the revision.

W1 / Q1. Statistical support. We agree. The revision describes the main results as observed mean differences and reports seed-level paired 95% t-intervals (Sec. 4.1, Appendix A.6). The submitted Table 8 computed standard errors as if the methods were independent; New result 4 gives the corrected paired comparison against the SELECT-selected founder (+10.4 points, [-0.9, +21.7], on Terminal-Bench 2.0; +8.0, [+0.1, +15.9], on SWE-bench Lite). For the baseline rows, the revised Table 8 reports means only while their seed-level records are re-verified against the raw run logs; the margin over HarnessBank is not established. Independently of those records, the shared-frontier experiment compares selection rules on identical candidates (New results 1-2), and Table 2 now has six paired seeds for every variant except UCB (New result 3).

W2 / Q2. Total computational cost. New result 5 reports executions (including screening, VALID and TEST), proposer-model and task-model tokens and wall-clock time per method. GP-Harness makes no proposer-model calls, its executions are within 2% of surrogate-free evolution, and HarnessBank and HarnessCompass use 13-29% more executions and 17-33% more task-model tokens. Matching tokens or time would therefore reduce the baselines' budgets, not ours. The shared founder-calibration pool costs more than any single search, and the revised Sec. 4.3 states this.

W3 / Q3. Graph versus text. We agree the gap cannot be attributed to graph structure. The text encoder sees roughly a third of each harness (New result 6). On identical histories, an untruncated chunked-text representation recovers a third to a half of the gap, and removing structural features costs only 0.03-0.04 (New result 7). The revised Sec. 4.2 describes the advantage as encoding every state's content in full and discusses the novelty filter: it is applied identically to all variants but, because it requires role-level structural differences, it shapes which candidates exist. The declared graph is not enforced, and Sec. 4.3 now states this limitation.

Response to Reviewer SKXp (2/2)

W4. Dependent candidates and adaptive search. Proposition 1 is correct under its stated assumptions; independence gives the closed-form factor e_M and is not an assumption of the algorithm. We have added an extension with explicit candidate covariance (Appendix B.3). Conditional on the history, if fitness and score vectors are jointly Gaussian with common marginals and Cov(F, S) = a Cov(S), the selection gain is exactly ρσfqM(R)\rho\sigma_f q_M(R), with qM(R)=E[max⁡jZj]q_M(R) = \mathbb E[\max_j Z_j], Z∼N(0,R)Z \sim \mathcal N(0, R); R=IR = I recovers ρσfeM\rho\sigma_f e_M, and equicorrelation c gives 1−c eM\sqrt{1-c}\,e_M. Within one explicit shared-parent model (Appendix B.4), which replaces the stylised combination behind Corollary 1 of the submitted version, every parent-only rule, even with oracle parent effects, gains at most σBeP\sigma_B e_P, so candidate-level scoring is better whenever ρ>(σB/σf) eP/qM(R)\rho > (\sigma_B/\sigma_f)\, e_P / q_M(R). The value e_3/e_32 ≈ 0.41 is therefore an independent-candidate reference, not a covariance-free cutoff, and the revision no longer compares the diagnostics with it. Adaptivity is handled by conditioning on each round's history; we have also added a distribution-free result (Appendix B.5) that conditional-mean selection is optimal at a fixed information state and that a fitted score with error at most ε loses at most 2ε. Empirically, on unselected candidates (New result 1) the graph GP reaches a within-pool correlation of 0.43 and 0.41, comparable to the independent reference, while parent fitness reaches 0.11 and 0.09. We have removed our statement (Sec. 3.5 and Appendix B.1 of the submitted version) that the Table 3 correlations likely understate the within-frontier correlation.

Q4. Prediction or uncertainty-guided exploration? These are distinct mechanisms. Conditional-mean selection maximises expected fitness given the available information, whereas EI maximises the model-expected positive improvement over the incumbent; for a Gaussian prediction, dEI/dmu = Phi(u) and dEI/dsigma = phi(u) > 0, so EI and posterior-mean selection coincide when uncertainties are equal and can differ otherwise. Empirically, on identical pools EI and posterior-mean selection are indistinguishable within a round (+0.006 ± 0.011 and +0.004 ± 0.009; New result 2), and online the VALID gap of 6.2 and 5.5 points has intervals including zero on each benchmark (New result 3). We therefore do not attribute the gains to calibrated uncertainty, and the revised Sec. 4.2 no longer reads the online EI-mean gap as a measured value of exploration.

W5. Novelty and scope. The revision states the contribution as treating the allocation of complete evaluations as a Bayesian optimisation decision inside harness search, with a protocol that separates search, confirmation and test; we do not claim a new Bayesian optimisation algorithm. The theory concerns candidate-level allocation and does not require a GP; the offline comparison with simpler predictors (New result 7) shows ranking nearly as good as the GP, with the GP's advantage in predictive density. The abstract now states that a single task model is used, and Sec. 4.3 states the scope: one task model, six seeds, the evaluated 20-evaluation budget, a 24-task Terminal-Bench subset, and prior methods evaluated as adaptations.

Thank you for the response. I appreciate the additional analyses, but several concerns remain unresolved.

  1. Following established protocols does not sufficiently address the novelty concern. The response acknowledges related work but lacks substantive comparisons with alternative preference evaluation and aggregation methods.
  2. The ablations demonstrate context-dependent responses but do not distinguish changes in underlying preferences from task-specific adaptation. Genre instructions may still influence decision criteria, even when models explicitly report their own preferences.
  3. The unfiltered analysis partially addresses my concern. However, the difference between filtered (73.7%) and unfiltered (90.7%) significance rates suggests that inconsistent judgments affect statistical inference. Their context-dependent impact remains insufficiently examined.
  4. More repetitions improve statistical precision, not representativeness. Subsampling within the restricted evaluation sets and using different models cannot establish that the original findings fail to generalize.
  5. The bootstrap analysis improves uncertainty estimation but does not establish practical significance. The reduced ratio shift (~1.9× after exclusions) and high overall rank correlation (ρ = 0.96) leave the behavioral importance of these differences unclear.

The response addresses some concerns, but the main issues remain. I will keep my original rating.

Comments on the paper

7 comments

General response

We thank all three reviewers for careful and constructive reviews. We agree with most of the concerns. This comment summarises what we have done; individual replies follow under each review, and new results, supplementary proofs and a list of changes are posted as separate comments below. We have uploaded a revised PDF.

  1. Strength of the empirical evidence (all reviewers). We agree that six search seeds leave several margins unresolved. The revision states the main results as observed differences in means rather than outperformance (abstract, introduction, Sec. 4.1, conclusion) and reports the paired comparison against the founder in Sec. 4.1. In recomputing Table 8 (now Appendix A.6) we found that its standard errors had treated GP-Harness and each baseline as independent, rather than using the seed-level paired differences; New result 4 corrects this for the comparison against the founder. For the baseline comparisons, the revised Table 8 reports means only, because their seed-level records are still being re-verified against the raw run logs; the margin over HarnessBank is not established in any case. The online VALID comparisons of New result 3 do not depend on these records.

  2. Direct evidence on candidate selection (SKXp, 2XTb). Table 3 uses only acquisition-selected candidates, and the Table 2 ablations follow separate trajectories. We therefore evaluated unselected candidates from fixed pools of completed runs (shared-frontier experiment): New result 1 reports within-pool ranking accuracy, and New result 2 compares selection rules on identical candidates.

  3. Theory (SKXp, 2XTb). Proposition 1 is correct under its stated assumptions, and we keep it. To address correlated offspring we have added (new Appendix B.3-B.8, summarised in the supplementary-proof comment): a version with explicit candidate covariance, a parent-versus-candidate comparison within one shared-parent model, a distribution-free conditional statement for adaptive search, the exact Gaussian Spearman-Pearson relation, and two results on EI and on what the uncertainty-error correlation can and cannot show. The revision no longer compares the Table 3 diagnostics with the threshold e_3/e_32 ≈ 0.41 of the submitted Corollary 1; that value is an independent-candidate reference, not a covariance-free empirical cutoff.

  4. Graph versus text representations (all reviewers). The comparison concerns complete pipelines, not topology. The text encoder retains a median of 32% (Terminal-Bench 2.0) and 29% (SWE-bench Lite) of each serialised harness, whereas the graph route retains every per-state description (New result 6). On identical histories, removing the truncation recovers a third to a half of the text route's gap, removing the structural features costs the graph route only 0.03-0.04 in rank correlation, and structure alone is weakest (New result 7). The revised Sec. 4.2 attributes the graph route's advantage to encoding every state's content in full, not to topology.

  5. Choice of surrogate (2XTb, SKXp). Offline on identical histories, ridge regression and a random forest rank candidates nearly as well as the GP; the GP's clearer advantage is its predictive density, which EI uses. A Weisfeiler-Lehman-kernel GP (NAS-BOWL style) ranks clearly worse (New result 7). We did not run online searches with the alternatives.

  6. Total cost (SKXp, nBUi). GP-Harness makes no proposer-model calls and uses the fewest task-model tokens and the least wall-clock time on both benchmarks; methods that screen proposals use 13-29% more executions (New result 5). The one-time founder-calibration pool, shared by all methods, is reported separately.

  7. Related work, operators and the declared workflow (nBUi, SKXp). Sec. 2 now discusses NAS-BOWL, Appendix A.2 specifies the mutation and recombination operators, and Sec. 4.3 states that the declared workflow conditions a prompt-driven executor and is not enforced; an evolved harness shown next to its parents will be added in the final version. We have not completed a validated measurement of adherence from execution traces and make no quantitative claim about it.

New result 1: within-frontier ranking accuracy on shared frontiers

This addresses the concern of SKXp and 2XTb that Table 3 measures prediction only on acquisition-selected candidates.

Setup. Per benchmark we took 12 candidate pools from completed GP-Harness runs (six seeds, two rounds each). From each pool we sampled uniformly five candidates that the search did not select (the EI choice is not included) and evaluated them on SELECT after the search, 60 candidates per benchmark. These evaluations were not fed back into the search. Scores are those assigned in that round, before any of these evaluations. For each score we report the Pearson correlation with SELECT fitness after centring both within each pool, and the within-pool Spearman correlation, mean ± standard error across the 12 pools.

BenchmarkSelection scorePools / candidatesWithin-pool Pearson rSpearman, mean ± SE
Terminal-Bench 2.0Graph GP posterior mean12 / 600.430.44 ± 0.07
Terminal-Bench 2.0Text GP posterior mean12 / 600.310.33 ± 0.08
Terminal-Bench 2.0Primary-parent fitness12 / 600.110.12 ± 0.09
SWE-bench LiteGraph GP posterior mean12 / 600.410.42 ± 0.06
SWE-bench LiteText GP posterior mean12 / 600.290.30 ± 0.05
SWE-bench LitePrimary-parent fitness12 / 600.090.10 ± 0.04

Reading.

  1. On unselected candidates the graph GP reaches r = 0.43 and 0.41, comparable to the independent-candidate reference e_3/e_32 ≈ 0.41 of the submitted Corollary 1 but not clearly above it. Because offspring share parents, the relevant comparison also involves the covariance term q_M(R) (supplementary proofs), so we do not present this as a test of a covariance-free threshold.
  2. Primary-parent fitness, the ranking used by surrogate-free evolution, reaches only 0.11 and 0.09.
  3. The graph GP's within-pool Spearman correlations (0.44, 0.42) are close to the prequential values in Table 3 (0.48, 0.44); we have removed our statement that the latter likely understate it. These values now appear in Table 3 (column rho_pool).
  4. The text representation is lower (0.31, 0.29), but the gap to the graph GP is about one combined standard error and is not established here.

New result 2: selection rules compared on the same candidate pools

On the same 12 pools per benchmark, each rule selects one candidate and we report its SELECT fitness. All rules choose from identical pools, so there are no trajectory differences. W/T/L counts pools where the EI choice is higher, equal (including the same candidate) or lower.

BenchmarkSelection ruleSELECT fitnessEI minus rule ± SEW / T / L
Terminal-Bench 2.0EI (GP-Harness)0.618--
Terminal-Bench 2.0Posterior mean0.612+0.006 ± 0.0114 / 5 / 3
Terminal-Bench 2.0Primary-parent fitness0.583+0.035 ± 0.0138 / 2 / 2
Terminal-Bench 2.0Uniform random0.569+0.049 ± 0.01210 / 0 / 2
SWE-bench LiteEI (GP-Harness)0.364--
SWE-bench LitePosterior mean0.360+0.004 ± 0.0094 / 6 / 2
SWE-bench LitePrimary-parent fitness0.336+0.028 ± 0.0108 / 3 / 1
SWE-bench LiteUniform random0.323+0.041 ± 0.00810 / 1 / 1

Reading.

  1. EI chooses better candidates than uniform choice on both benchmarks (+0.049 and +0.041), and this survives Holm correction over the three comparisons per benchmark even with seed-level degrees of freedom.
  2. EI also chooses better candidates than primary-parent ranking (+0.035 and +0.028; unadjusted 95% intervals exclude zero). The 12 pools come from six seeds, so we treat this conservatively: with seed-level degrees of freedom the margin does not survive Holm correction, and we report it as consistent with, not proof of, a selection advantage.
  3. Within a round, EI and posterior-mean selection are indistinguishable (same candidate in 5 and 6 of 12 pools; differences well within one standard error). Any advantage of EI can only arise across rounds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.