- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Hypothesis-Conditioned Evaluation (HCE), a framework for reducing the evaluation cost of LLM harness optimization. Before evaluating a candidate harness, the editor declares an evaluation contract consisting of predicted fixes, regression risks, an activation predicate, and a runtime observable. HCE uses these declarations to construct a stratified task sample, estimates the candidate's full-training score using an augmented inverse-probability-weighted (AIPW) estimator, and applies additional retention checks based on observed firing, scope compliance, and target-task improvement.
Critical strengths
-
The paper addresses a meaningful bottleneck in iterative harness optimization, where evaluating every candidate on every training task can be expensive.
-
The AIPW estimator is accompanied by a clear explanation of its conditional unbiasedness assumptions and limitations. The authors appropriately distinguish unbiased estimation from guarantees about single-round accuracy or candidate-selection quality.
-
Runtime firing and scope checks provide useful safeguards against accepting edits that do not execute as intended or that act outside their declared scope.
Critical weaknesses
-
The AIPW estimator is unbiased under the stated sampling assumptions. However, unbiasedness does not imply low variance or reliable candidate ranking in an individual search round. The authors should report candidate-ranking agreement and the frequency of incorrect retention decisions relative to full evaluation. They should also evaluate whether uncertainty-aware acceptance criteria improve the quality-cost trade-off.
-
The paper distinguishes HCE from prior work on active testing, regression-test selection, and adaptive task selection for harness optimization. However, the paper does not experimentally establish the incremental value of this contract over existing adaptive sampling approaches.
-
The current experiments change task selection, the AIPW estimator, and the retention checks simultaneously, making it impossible to attribute the performance/cost gains to any specific component. How much do task targeting, the AIPW estimator, the firing check, the target check, and the scope check contribute?
-
Why are fixed 0.5/0.3/0.2 strata weights used? Has a sensitivity analysis been performed on these weights?
Questions for authors
See weaknesses.
Reply to Reviewer NyhV
W1 — How reliable are the rankings and retention decisions?
We now report both against full-training evaluation (Appendix C, Table 11). The retention decisions are reliable; ranking is good where candidates differ by more than rollout noise.
Both changed decisions are rejections whose full-score gains (+4.44 ± 3.85 and +4.04 ± 4.63 points) are within 1.3 standard errors; no accepted edit is worse than its parent. The estimate ranks candidates better than the raw subset mean on both benchmarks. On Terminal-Bench the ten candidates' full scores spread no more than one full evaluation's own rollout noise, so the low correlation reflects candidates that no single evaluation could order, not an estimator failure.
Software100 R6 is the instructive case: R1's stored estimate (89.75% against a full score of 77.78%) lowered R6's probability of passing from 78% to 4.1%. Completing each accepted candidate on its unsampled tasks removes this at 45–48 rollouts per acceptance (Table 12 and the following paragraph).
Would uncertainty-aware acceptance improve this? We tested it on the 12 recorded score decisions. Lower-confidence-bound gates at z = 0.5, 1.0 and 1.645 agree with full evaluation in 9, 9 and 8 decisions, against 10 for HCE's rule; a Hoeffding bound agrees in 7 because it rejects every accepted edit. Against a second reference (Hybrid30) the counts are 10, 10 and 9 against 9. Added conservatism therefore buys no measurable reliability here. A confirmation rule that reruns candidate and incumbent together when they are close would trigger in 3–10 decisions at 282–960 extra rollouts; we describe it as an option for users who want fewer close calls.
W2 — What does the contract add over adaptive sampling?
It adds what outcome-based sampling cannot know: which tasks this particular edit is supposed to change. We ran Task-CoEvolve's variance weights at our budget on the recorded candidates (Appendix B, Tables 5 and 7). The weights lower estimation error (RMSE 2.46 versus 3.48 points on Terminal-Bench), but they include every declared fix in only 90% of Terminal-Bench and 73% of Software100 draws, where HCE includes them always. Adding HCE's named tasks to the variance weights gives the highest agreement with full-panel decisions of any design (95.1%, versus 93.2% for the weights alone and 92.7% for HCE's strata). The two are complementary, and the contract also supplies the firing, scope and target checks, which no sampler provides. We revised Related Work accordingly.
W3 — Which components account for savings and quality?
Savings come from the task budget: 450 versus 900 rollouts on Terminal-Bench and 510 versus 990 on Software100. Quality at that budget comes from three components, each now tested:
W4 — Why 0.5/0.3/0.2, and are the results sensitive?
The weights are a fixed heuristic, and the results are insensitive to them (Appendix H, Table 20). They affect only the allocation, never the expectation of the estimate. In 13 of 20 rounds only one allocation is feasible. Across all 20 rounds, equal weights, 0.25/0.55/0.20, 0.8/0.1/0.1 and 56.8% of a 20,301-point grid reproduce the recorded allocation exactly. In the seven rounds with a choice, five rejections hold under every feasible allocation and the other two pass probabilities move by at most 0.072. The analysis also shows a possible improvement: an audit-heavier weighting such as 0.2/0.2/0.6 would lower the estimate's standard deviation by 0.3–0.8 points.
For completeness: the Software100 held-out score is corrected to ordinary mean@3, 147/201 (73.13%) versus AHE's 133/201 (66.17%), +6.97 points [1.49, 12.44]; the submitted 78.61% used the best three of up to six attempts.