acceptodds
Under review as a conference paper at ICLR 2027

Hypothesis-Conditioned Evaluation for Efficient Harness Optimization

Abstract

Harness optimization improves an LLM agent by editing its prompts, tools and helper code with model weights fixed. Evaluating every edit on every training task is expensive, but a small random sample can miss the failure that motivated the edit. We introduce Hypothesis-Conditioned Evaluation (\method), which uses the editor's explanation to guide evaluation. Before any candidate rollout, the editor declares an evaluation contract: the failure it targets, the tasks it expects to fix or might break, an activation predicate specifying where the edit applies, and a runtime event showing that it fired. Conditional prompt rules are inserted only when their predicates hold. The contract's targets and risks are always evaluated; a stratified probability sample fills the remaining budget, including tasks outside the declared scope. An augmented inverse-probability-weighted estimator combines accepted task history with sampled outcomes to compare candidates on the full training-score scale. Retention requires a higher estimated score, observed firing within scope, and improvement on a named target where the edit fired. In two ten-edit comparisons with Agentic Harness Engineering (AHE), using GPT-5.4, used \EvaluationTokenSaving% fewer scored search-evaluation tokens on Terminal-Bench (30 training, 58 held-out tasks) and \SoftwareEvaluationTokenSaving% fewer on Software100 (33 training, 67 held-out). The training-selected harnesses achieved held-out three-attempt mean success rates () of \HCEFinal% versus \AHEFinal% on Terminal-Bench, and \SoftwareHCETestScore% versus \SoftwareAHETestScore% on Software100. Earlier SWE-bench studies also used fewer evaluations with matching held-out quality. The reported searches establish evaluation savings, but do not establish general quality superiority or isolate the benefit of hypothesis-based task selection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, Harness optimization, Hypothesis-conditioned evaluation, Adaptive task selection, Regression testing, Augmented inverse-probability weighting

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_NyhVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Hypothesis-Conditioned Evaluation (HCE), a framework for reducing the evaluation cost of LLM harness optimization. Before evaluating a candidate harness, the editor declares an evaluation contract consisting of predicted fixes, regression risks, an activation predicate, and a runtime observable. HCE uses these declarations to construct a stratified task sample, estimates the candidate's full-training score using an augmented inverse-probability-weighted (AIPW) estimator, and applies additional retention checks based on observed firing, scope compliance, and target-task improvement.

Critical strengths
  1. The paper addresses a meaningful bottleneck in iterative harness optimization, where evaluating every candidate on every training task can be expensive.

  2. The AIPW estimator is accompanied by a clear explanation of its conditional unbiasedness assumptions and limitations. The authors appropriately distinguish unbiased estimation from guarantees about single-round accuracy or candidate-selection quality.

  3. Runtime firing and scope checks provide useful safeguards against accepting edits that do not execute as intended or that act outside their declared scope.

Critical weaknesses
  1. The AIPW estimator is unbiased under the stated sampling assumptions. However, unbiasedness does not imply low variance or reliable candidate ranking in an individual search round. The authors should report candidate-ranking agreement and the frequency of incorrect retention decisions relative to full evaluation. They should also evaluate whether uncertainty-aware acceptance criteria improve the quality-cost trade-off.

  2. The paper distinguishes HCE from prior work on active testing, regression-test selection, and adaptive task selection for harness optimization. However, the paper does not experimentally establish the incremental value of this contract over existing adaptive sampling approaches.

  3. The current experiments change task selection, the AIPW estimator, and the retention checks simultaneously, making it impossible to attribute the performance/cost gains to any specific component. How much do task targeting, the AIPW estimator, the firing check, the target check, and the scope check contribute?

  4. Why are fixed 0.5/0.3/0.2 strata weights used? Has a sensitivity analysis been performed on these weights?

Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer NyhV

W1 — How reliable are the rankings and retention decisions?

We now report both against full-training evaluation (Appendix C, Table 11). The retention decisions are reliable; ranking is good where candidates differ by more than rollout noise.

Terminal-BenchSoftware100
Decisions changed by full scores1/10 (R2)1/10 (R6)
Accepted edits scoring below their parent0/20/1
Spearman, AIPW estimate vs. full score0.050.72
Spearman, raw subset mean vs. full score−0.310.55
SD of candidates' full scores / SE of one full evaluation2.76 / 2.704.63 / 3.27

Both changed decisions are rejections whose full-score gains (+4.44 ± 3.85 and +4.04 ± 4.63 points) are within 1.3 standard errors; no accepted edit is worse than its parent. The estimate ranks candidates better than the raw subset mean on both benchmarks. On Terminal-Bench the ten candidates' full scores spread no more than one full evaluation's own rollout noise, so the low correlation reflects candidates that no single evaluation could order, not an estimator failure.

Software100 R6 is the instructive case: R1's stored estimate (89.75% against a full score of 77.78%) lowered R6's probability of passing from 78% to 4.1%. Completing each accepted candidate on its unsampled tasks removes this at 45–48 rollouts per acceptance (Table 12 and the following paragraph).

Would uncertainty-aware acceptance improve this? We tested it on the 12 recorded score decisions. Lower-confidence-bound gates at z = 0.5, 1.0 and 1.645 agree with full evaluation in 9, 9 and 8 decisions, against 10 for HCE's rule; a Hoeffding bound agrees in 7 because it rejects every accepted edit. Against a second reference (Hybrid30) the counts are 10, 10 and 9 against 9. Added conservatism therefore buys no measurable reliability here. A confirmation rule that reruns candidate and incumbent together when they are close would trigger in 3–10 decisions at 282–960 extra rollouts; we describe it as an option for users who want fewer close calls.

W2 — What does the contract add over adaptive sampling?

It adds what outcome-based sampling cannot know: which tasks this particular edit is supposed to change. We ran Task-CoEvolve's variance weights at our budget on the recorded candidates (Appendix B, Tables 5 and 7). The weights lower estimation error (RMSE 2.46 versus 3.48 points on Terminal-Bench), but they include every declared fix in only 90% of Terminal-Bench and 73% of Software100 draws, where HCE includes them always. Adding HCE's named tasks to the variance weights gives the highest agreement with full-panel decisions of any design (95.1%, versus 93.2% for the weights alone and 92.7% for HCE's strata). The two are complementary, and the contract also supplies the firing, scope and target checks, which no sampler provides. We revised Related Work accordingly.

W3 — Which components account for savings and quality?

Savings come from the task budget: 450 versus 900 rollouts on Terminal-Bench and 510 versus 990 on Software100. Quality at that budget comes from three components, each now tested:

  • Estimator. AIPW lowers RMSE against Horvitz–Thompson from 7.7 to 3.5 points on Terminal-Bench and from 7.4 to 5.2 on Software100 (Table 5).
  • Target check. Recomputing every decision with one check removed, only the target check changes outcomes: without it Terminal-Bench R6 and R10 would have been kept, although their full-training scores are 5.6 and 3.3 points below the retained harness (Table 19).
  • Contract. In new ablated searches from the same initial harness, removing the three vetoes gives 144–145/201 on held-out tasks and removing the contract gives 142/201, against 147/201 for full HCE, which also has the lowest search cost (1.22B tokens against 1.32–1.97B; Table 17).

W4 — Why 0.5/0.3/0.2, and are the results sensitive?

The weights are a fixed heuristic, and the results are insensitive to them (Appendix H, Table 20). They affect only the allocation, never the expectation of the estimate. In 13 of 20 rounds only one allocation is feasible. Across all 20 rounds, equal weights, 0.25/0.55/0.20, 0.8/0.1/0.1 and 56.8% of a 20,301-point grid reproduce the recorded allocation exactly. In the seven rounds with a choice, five rejections hold under every feasible allocation and the other two pass probabilities move by at most 0.072. The analysis also shows a possible improvement: an audit-heavier weighting such as 0.2/0.2/0.6 would lower the estimate's standard deviation by 0.3–0.8 points.

For completeness: the Software100 held-out score is corrected to ordinary mean@3, 147/201 (73.13%) versus AHE's 133/201 (66.17%), +6.97 points [1.49, 12.44]; the submitted 78.61% used the best three of up to six attempts.

Reviewer_judrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Hypothesis-Conditioned Evaluation (HCE), a method that uses the editor model's explanation to guide evaluation for efficient harness optimization. The key idea is to declare an evaluation contract before running each edit, specifying the expected fixes and the scope of the edit. The contract allows the method to evaluate fewer tasks and therefore use fewer training tokens. On Terminal-Bench 2.1 and Software100 with GPT-5.4, the method achieves a higher score than the baseline with significantly fewer tokens.

Critical strengths
  • The motivation of selecting evaluation tasks based on the editor's hypothesis makes sense. The efficiency improvement is significant, and it is good to see that the method even improves the score.
  • The proposed HCE method is evaluated on two benchmarks, with similar trends on both.
Critical weaknesses
  • The design choices and components of the HCE method are not ablated, analyzed, or explained well. Many components of the HCE design are proposed without clear justification or comparison to alternatives, for example, the stratified probability sample and the augmented inverse-probability-weighted estimator in Eq. 2. The results do not include any ablations of each component and design choice, making it unclear what their contributions are.
  • The writing of the paper needs to be improved. The terminology used in this paper is not well explained, and it is hard to understand what it means, for example, "the activation predicate".
  • More baselines are needed to justify the results. The method samples tasks based on the editor's hypothesis and a stratified sampling strategy, and it is unclear how this compares to other sampling methods, such as random sampling or other heuristic-based sampling. The authors need to include these baselines.
  • It is unclear whether HCE achieves the best trade-off between test score and training tokens. For example, the self-evolving methods achieve worse results but with significantly fewer tokens, so HCE does not necessarily give the best trade-off on this Pareto frontier.
Questions for authors
  • What is the role of the activation predicate A? Are these just the tasks not in F and R? What is the motivation for including them beyond the predicted fixes F and regression risks R?
  • Is the AIPW estimator accurate enough for the retention decision? The authors should include more justification/discussion.
2 comments
AuthorsOpenReview ↗

Reply to Reviewer judr - I

W1/W3 — Which design choices help, and how do simpler samplers compare?

We now compare HCE with uniform sampling, plain and normalized IPW, a carry-forward estimator and Task-CoEvolve's variance weights at the same budget, and we ran three ablated searches. Each design choice has a measurable role.

Estimator (Appendix B, Table 5a). On HCE's samples, AIPW has RMSE 3.5 points on Terminal-Bench and 5.2 on Software100, against 7.7 and 7.4 for Horvitz–Thompson and 8.6 and 6.0 for the subset mean. With fixed quotas normalized IPW equals Horvitz–Thompson, so history is what halves the error. Carry-forward has lower RMSE (2.0 and 4.3) but is biased (+3.4 points on Software100) and cannot detect changes on unsampled tasks. Appendix B derives the variance and the exact condition under which history helps.

Sampler (Table 5b–c). Guaranteed coverage of the named fixes is what improves decisions. Uniform sampling has similar RMSE (3.42 versus 3.48) but contains every named fix in only 42% of Terminal-Bench draws, so it agrees with the full-panel decision in 84.0% of draws versus HCE's 92.7%; 13% of its decisions reject a good candidate because its fix was never sampled. Task-CoEvolve's variance weights give lower RMSE than HCE's strata (2.46), and adding HCE's named tasks to them gives the best agreement of all designs (95.1%). We therefore describe the contract as complementary to adaptive sampling, and report the precision cost of our strata in Section 3.2.

Ablated searches (Appendix G, Table 17). Same initial harness and budget, each generating its own edits:

ProcedureSearch tokensEdits keptHeld-out (Test67)
Full HCE1.22B1147/201
No hard-regression, inertness or target-gain veto1.42B3144–145/201
No evaluation contract1.32B3142/201
Raw subset mean instead of AIPW1.97B2not run (training 72/99 vs. 77/99)

Full HCE has the highest held-out score at the lowest search cost. Removing the contract yields the highest training score (79/99) and the lowest held-out score (−2.49 points [−6.97, 1.49]). Each arm is one search; Table 18 lists exactly what each removes.

W2/Q1 — What is the activation predicate, and what does A add to F and R?

Thank you for pointing out that this was unclear; we rewrote the definitions and added a worked round and a glossary (Section 3.1, Appendix A).

The activation predicate α is a test on task information, evaluated before any rollout: "does this edit apply to this task?" The activation set A = {i : α(i) ≠ false} is every training task where the edit may act. F (predicted fixes) is a subset of A; R (regression risks) may lie inside or outside A. A is therefore not the complement of F∪R: it is the edit's declared scope, of which F is the part the editor expects to fix.

Example. In Terminal-Bench round 1 the edit adds a primer-checking rule with the condition "instruction contains primer". A = F = two DNA tasks; R is empty. Both are evaluated together with 13 of the other 28 tasks. The rule is inserted in all six target rollouts and in none of the 39 audit rollouts; dna-insert improves from 0/3 to 1/3, the estimate rises from 67.78% to 68.89%, and the edit is kept.

What A adds beyond F and R.

  1. Coverage of tasks the editor did not name. In 9 of 20 rounds A contained 36 unnamed task-rounds; HCE evaluated all 36, where uniform sampling of the same slots would cover 15.4 in expectation. In round 4, an added rule widened A from three named tasks to nine, and all nine were scored.
  2. A boundary for the scope check. Firing outside A is a violation: round 7's helper acted on three tasks outside A and was rejected, and the editor's next proposals narrowed it.
  3. Strata for sampling. Tasks in A are tested for improvement or regression; tasks outside A are audited for unintended effects.

This coverage has a precision cost, which we now report: the estimate's standard deviation is higher than under uniform sampling in those 9 rounds (3.54 versus 2.61 points in round 4). We also corrected the description of global edits: all 20 reported contracts are local.

AuthorsOpenReview ↗

Reply to Reviewer judr - II

Q2 — Is AIPW accurate enough for retention?

It is accurate to a few points at half the evaluation cost, and the recorded decisions agree with full evaluation in all but one round per benchmark.

Accuracy (Appendix B). With candidates and histories fixed, RMSE is 3.5 points on Terminal-Bench and 5.2 on Software100, less than half of Horvitz–Thompson's on Terminal-Bench. Nominal 95% intervals cover the full-training score in 88% of draws, so we use the estimate as a common-scale score together with the mechanism checks, not as a significance test.

Decisions (Appendix C, Tables 11–12). Replacing the online scores with post-search full-training scores changes 1 of 10 decisions on each benchmark, both from rejection to acceptance and both within 1.3 standard errors; no accepted edit has a negative full-score gain. On Software100 the estimate ranks candidates well (Spearman 0.72, against 0.55 for the raw subset mean). On Terminal-Bench the candidates' full scores differ by no more than one evaluation's rollout noise (SD 2.76 versus SE 2.70), so no single evaluation, full or sampled, could rank them.

Would a confidence bound help? We re-decided the 12 score-check decisions with lower-confidence-bound gates. They agree with full evaluation in 9, 9 and 8 decisions (z = 0.5, 1.0, 1.645) against 10 for HCE's rule; a Hoeffding bound agrees in 7. Extra conservatism does not improve these decisions. The one systematic issue we found, an optimistic stored threshold, has a direct remedy costing 45–48 rollouts per accepted edit (Appendix C).

W4 — Does HCE have the best quality–cost trade-off?

Against full-evaluation AHE under one protocol, HCE is cheaper with equal or higher held-out scores on both benchmarks: 36.2% and 45.3% fewer tokens including the initial evaluation, editing and debugging (Table 8), with held-out mean@3 of 83.33% versus 79.31% (+4.02 [−2.87, 10.34]) and 73.13% versus 66.17% (+6.97 [1.49, 12.44]). Among the ablated variants, full HCE also has the lowest cost and the highest held-out score (Table 17). Across all compared methods (Tables 2–3), HCE has the highest held-out score on both benchmarks; ReasoningBank, ACE and HarnessLens use fewer training tokens but score 5.2–6.3 points lower on Terminal-Bench and 1.5–3.0 lower on Software100.

The Software100 figure corrects the submitted 78.61%, which used the best three of up to six attempts; the gain still excludes zero and stays between 5.97 and 7.21 points under every answer-access and timeout adjustment (Table 16).

Reviewer_Pd74OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Hypothesis-Conditioned Evaluation (HCE) to reduce the evaluation cost of LLM agent harness optimization. HCE uses an editor's declared hypothesis to select evaluation tasks, estimates full-training performance using an augmented inverse-probability-weighted (AIPW) estimator, and verifies whether the proposed edits actually execute and improve their intended targets. Experiments on TerminalBench2.1 and Software100 show 39.9% and 49.3% reductions in scored evaluation tokens with competitive or even better task performance.

Critical strengths

Evaluation cost is a main bottleneck in iterative harness optimization. This paper addresses this practical issue and demonstrates substantial savings in evaluation tasks.

In addition, HCE goes beyond naive subset evaluation by combining hypothesis-guided task selection, statistically motivated score AIPW estimation, and explicit retention checks. Particularly, verifying whether an edit actually fires within its intended scope is a useful addition to score-based evaluation.

Critical weaknesses

As acknowledged in the limitations section, the empirical evidence is too limited (I'm not trying to repeat the limitations section; I need more empirical evidence to be convinced). The main experiments report only one optimization search per method on each benchmark, without independent repetitions (+ using only a single model, GPT-5.4). In addition, HCE introduces many components, including the evaluation contract, task sampling, AIPW estimation, and retention checks, without any ablation. This makes it difficult to distinguish the actual benefit of the proposed method.

On the other hand, HCE relies on the editor's predictions about "which tasks an edit should improve and where it should activate". However, these hypotheses may be inaccurate (Maybe that is why the editor's reasoning effort is max, while the agent's reasoning effort is high?). Curious whether it happens or it matters, which could potentially affect later decisions.

Questions for authors

The reported token savings exclude other parts, including initialization, editor calls, debugging, etc. What proportion of the total optimization cost do these components account for? How much efficiency gain remains?

How were the evaluation budgets (15/30, 17/33 tasks) chosen? How sensitive are the results to these choices?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer Pd74 - I

W1 — What additional empirical evidence supports the method?

The revision adds four kinds of evidence: component ablations, repeated shorter studies with other models, a second ten-edit search, and audits of the main result.

Main result, corrected and audited. HCE uses 39.9% and 49.3% fewer scored evaluation tokens than AHE. Held-out mean@3 is 83.33% versus 79.31% on Terminal-Bench (+4.02 [−2.87, 10.34]) and 73.13% versus 66.17% on Software100 (+6.97 [1.49, 12.44]). The Software100 figure replaces the submitted 78.61%, which used the best three of up to six attempts; the gain still excludes zero. Counting every trial flagged for reading reference material as a failure keeps the Software100 gap at 5.97–7.21 points with intervals above zero, and raises the Terminal-Bench gap to 5.46–5.75 (Appendix F, Table 16).

Ablations (Appendix G, Table 17). Three new Software100 searches each remove one component, with the same initial harness and budget:

ProcedureSearch tokensEdits keptHeld-out (Test67)
Full HCE1.22B1147/201
No hard-regression, inertness or target-gain veto1.42B3144–145/201
No evaluation contract1.32B3142/201
Raw subset mean instead of AIPW1.97B2not run (training 72/99 vs. 77/99)

Full HCE has the highest held-out score at the lowest cost. The contract-free search scores highest in training (79/99) and lowest on held-out tasks (−2.49 points [−6.97, 1.49]).

Repeats and other models (Appendix E, Table 15). Four earlier studies give twelve shorter repeats on four task panels with opus-4-8 and haiku-4-5. HCE used fewer development rollouts in ten of twelve (28 vs. 48, 72 vs. 180, 200 vs. 300, 139 vs. 270 in total); its held-out pass count was higher in four repeats, equal in six and lower in two. A second ten-edit Software100 search on a different split gives 146/201 versus 144/201, the same direction as the main result.

Estimator (Appendix B, Table 5). On HCE's samples, AIPW has RMSE 3.5 points versus 8.6 for the subset mean on Terminal-Bench and 5.2 versus 6.0 on Software100, while guaranteeing that every named fix is observed.

One disclosure: AHE's editor read HCE-specific prompt text in 10 of 20 main rounds. AHE still evaluated all tasks and kept edits by plain-mean gain in every round (Table 13). Each main comparison remains one search per method; the additions above are what we could run to test the result from independent directions.

AuthorsOpenReview ↗

Reply to Reviewer Pd74 - II

W2 — Do inaccurate hypotheses matter?

They are common, and HCE is designed so that a wrong hypothesis costs an edit its acceptance rather than corrupting the score. Two properties do this.

First, the estimate does not depend on the declaration being right: it is unbiased whenever inclusion probabilities are positive and known, which held in every round (Appendix B). A wrong declaration changes which tasks are sampled, not the expectation.

Second, the checks catch wrong declarations. Across 13 primary target declarations on Terminal-Bench, search samples show 5 gains, 6 ties and 2 losses (Figure 7). The target check rejected four candidates whose named target did not improve, including R6 and R10, which had the two highest estimates of the search; their later full-training scores were 5.6 and 3.3 points below the retained harness. Recomputing every decision with one check removed confirms that the target check is what blocks these two (Table 19). The scope check caught R7, whose helper acted on three tasks outside its declared condition; the editor's next proposals cite that rejection and narrow the helper.

Q1 — How much saving remains after other optimization costs?

Almost all of it. Adding the shared initial evaluation, editor and observed debugger tokens (Appendix C, Table 8):

Tokens (M)TB AHETB HCESW AHESW HCE
Scored evaluation604.12363.202409.341221.22
Initial evaluation (shared)54.9954.99212.13212.13
Editor32.4126.2132.7926.96
Debugger (observed)26.5313.9836.7012.33
Subtotal718.05458.372690.961472.63

The subtotal is 36.2% and 45.3% lower, and the absolute saving grows (259.7M and 1,218.3M tokens) because HCE's editor and debugger also use fewer tokens. Some debugger attempts lack usage records; assigning each such HCE attempt the largest recorded count, and nothing to AHE, still leaves 35.6% and 44.8%.

Q2 — Why these budgets, and how sensitive are decisions?

The budget is half the training tasks (15 of 30, 17 of 33), fixed in advance. Table 21 (Appendix I) varies it with candidates, histories and thresholds fixed:

Agreement with full-panel decision1/3 of tasks1/2 (paper)2/3
Terminal-Bench91.0%92.7%95.6%
Software10093.2%98.0%98.7%

Decisions are stable across the range: going from half to two thirds adds at most 3.6 points, and a third of the tasks still agrees in over 91% of samples. At every budget, designs that always include the named tasks agree 7–11 points more often than uniform sampling on Terminal-Bench.

Reviewer_C22fOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Hypothesis-Conditioned Evaluation (HCE), a framework for reducing the evaluation cost of automatic LLM agent harness optimization. Rather than evaluating every proposed harness modification on the full training set, HCE uses the editor's stated hypothesis about which tasks the modification should improve or potentially harm to guide task selection. It combines targeted evaluation with statistical score estimation and runtime checks to determine whether modifications should be retained.

The authors validate HCE against Agentic Harness Engineering (AHE) on Terminal-Bench and Software100, using ten editing rounds per method. HCE reduces scored search-evaluation tokens by 39.9% and 49.3%, respectively, while achieving held-out mean@3 success rates of 83.33% versus 79.31% on Terminal-Bench and 78.61% versus 66.17% on Software100.

Critical strengths
  1. The paper addresses an important computational bottleneck in automatic LLM agent optimization, where repeatedly evaluating candidate harnesses can be prohibitively expensive. HCE reduces scored search-evaluation tokens by approximately 40–49% across two benchmarks while achieving competitive or better held-out performance in the reported experiments.

  2. The proposed framework connects an editor's explicit predictions about a modification's effects to task selection, runtime verification, and retention decisions. Its use of inverse probability weighted estimation provides a mathematically justified way to compare candidates evaluated on different task subsets.

Critical weaknesses
  1. The paper compares HCE against full-evaluation AHE but does not include a random-subset baseline with the same evaluation budget or isolate the effects of hypothesis-conditioned sampling, statistical estimation, and retention checks. Consequently, the experiments demonstrate that evaluating fewer tasks reduces computational cost, but do not establish whether conditioning task selection on the editor's hypothesis provides a meaningful advantage over simpler alternatives. This substantially weakens the empirical support for the paper's primary contribution.

  2. The main experiments use only one optimization search per method on each benchmark, making it difficult to assess variability across stochastic editing trajectories. Moreover, the Software100 results reveal a potential weakness in the retention mechanism: an overly optimistic estimated score for the retained R1 candidate (89.75%) establishes a threshold that later candidates cannot exceed, even though R6 achieves a higher post-search full-training diagnostic score. I have overall concerns about selection bias and the reliability of the optimization procedure, which are not adequately investigated.

Questions for authors
  1. Can the authors clarify whether they have comparisons against random task subsampling or existing adaptive selection methods (e.g., Task-CoEvolve) under matched evaluation budgets? Without these comparisons, it is difficult to determine whether hypothesis conditioning provides benefits beyond simply evaluating fewer tasks.

  2. In Software100 (Figure 6), the retained R1 candidate has an AIPW estimate of 89.75%, while R6 achieves a higher post-search Full33 score. Can the authors explain how optimistic retained-score estimates affect subsequent acceptance decisions, and whether this behavior occurs in other runs?

  3. Can the authors provide results across multiple independent search seeds to establish whether the observed quality differences and selection behavior are robust to stochastic variation?

1 comment
AuthorsOpenReview ↗

Replies to Reviewer C22f

W1/Q1 — What does hypothesis conditioning add beyond using fewer tasks?

It adds two things a smaller random sample cannot provide: every task the edit claims to fix is always tested, and an edit is kept only if it acted where it was declared to act and improved a named target. New ablated searches and fixed-budget controls show that both matter.

Ablated searches (Appendix G, Table 17). Three new Software100 searches each remove one component and generate their own edits from the same initial harness and budget (ten rounds, 17 of 33 tasks, three attempts):

ProcedureSearch tokensEdits keptHeld-out (Test67)
Full HCE1.22B1147/201
No hard-regression, inertness or target-gain veto1.42B3144–145/201
No evaluation contract1.32B3142/201
Raw subset mean instead of AIPW1.97B2not run (training 72/99 vs. 77/99)

Full HCE has the highest held-out score and the lowest search cost. Without the contract, the search keeps three edits and reaches the highest training score (79/99) but the lowest held-out score (−2.49 points, paired interval [−6.97, 1.49]): the contract's checks filter out edits that fit the training sample. Each arm is one search, so we report these as observed outcomes.

Same budget, random tasks (Appendix B, Table 5). With each recorded candidate, history and budget fixed, a uniform sample contains every named fix in only 42% of Terminal-Bench and 12% of Software100 draws. On Terminal-Bench, 13% of uniform decisions reject a candidate that the full panel accepts solely because its fix was never sampled; HCE makes no such error and agrees with the full-panel decision in 92.7% of draws versus 84.0%. The estimator contributes separately: HCE's score has RMSE 3.48 points versus 7.54 for a uniform subset mean on Terminal-Bench (5.21 versus 5.95 on Software100).

Relation to adaptive sampling. Task-CoEvolve's variance weights lower estimation error further (2.46 points), and adding HCE's named tasks to them gives the highest agreement of all designs (95.1%). The contract is therefore complementary to outcome-based sampling rather than a substitute for it; we state this in Related Work and Appendix B (Tables 5–7).

W2/Q2 — Can an optimistic retained score block a better candidate?

Yes, and we now measure how often and price the fix. Table 12 (Appendix C) lists every edited decision with its stored threshold and a post-search full-training score. In the two main searches, 18 of 20 decisions match the direction of the full score, and no accepted edit scores below its parent. The two exceptions are rejections, Software100 R6 and Terminal-Bench R2, whose full scores exceed their parents' by 4.04 and 4.44 points, each within 1.3 standard errors.

R6 is the case you describe. R1 stored 89.75% against a full-training score of 77.78%; two up-weighted audit tasks explain 6.93 points of that gap. The stored value lowers R6's probability of passing from 78% to 4.1%, and it also blocks the worse R4 (25% to 0%).

The remedy is to complete each accepted candidate on its unsampled tasks, so the stored score is a full score. This costs 48 (Software100) or 45 (Terminal-Bench) rollouts per acceptance: 48 and 90 rollouts for the acceptances in our searches, against budgets of 510 and 450, so the saving over AHE (990 and 900) remains. On the recorded candidates it admits R6. It also admits some candidates below their parents, so we present it as an option whose effect on search quality needs new searches. Section 3.3 now states that unbiasedness holds for a fixed candidate and not for a selected threshold.

W2/Q3 — Independent searches. A second ten-edit Software100 search (different split, 48 of 67 test tasks shared, one training attempt) gives HCE 146/201 versus AHE 144/201 on three-attempt held-out testing, the same direction as the main result. Four earlier studies add twelve shorter repeats on four task panels with two other models: HCE used fewer development rollouts in ten, and its held-out pass count was higher in four, equal in six and lower in two (Appendix E, Table 15).

Metric correction and audits. The submitted Software100 score of 78.61% used the best three of up to six attempts. The ordinary mean@3 is 147/201 (73.13%) versus AHE's 133/201 (66.17%): +6.97 points [1.49, 12.44], still excluding zero. We audited every final-test trial for access to reference material; under every adjustment the Software100 gap stays between 5.97 and 7.21 points with an interval above zero, and the Terminal-Bench gap grows from 4.02 to 5.46–5.75 because the only flagged trials are three AHE successes (Table 16). We also found that AHE's editor read HCE-specific prompt text in 10 of 20 main rounds; AHE's evaluation and retention rule were unchanged in every round (Table 13).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.