- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the evaluation bottleneck in iterative LLM harness optimization. Existing methods evaluate every candidate harness on the full validation task set at every iteration, which can be expensive, especially for long-horizon terminal tasks. The paper proposes Task-CoEvolve, which jointly evolves the harness and the validation task subset. At each iteration, it samples tasks using weights based on the historical Bernoulli variance of their success outcomes, estimates the candidate's full-set performance using sampling-aware estimators, and selects the candidate with the highest estimated full-set score. Experiments on online text classification and Terminal-Bench 2.1 show that the method can approach full-set search performance while substantially reducing the number of task evaluations, input-token usage, and wall-clock time.
Critical strengths
-
Meaningful improvement in iterative evaluation efficiency. The paper targets an important bottleneck in harness evolution: repeatedly executing expensive candidates on the entire validation set. On Terminal-Bench 2.1, Task-CoEvolve achieves performance close to full-set search while reducing search-phase input-token usage by approximately 67--80% and wall-clock time by roughly half. On online text classification, it reaches comparable performance with only 7% of the per-iteration validation budget and improves over full search in the reported 20% setting.
-
The method addresses both task selection and score comparability. Merely evaluating a changing subset would make raw scores incomparable across iterations. The use of sampling-aware full-set estimation is therefore a sensible and necessary component of the proposed pipeline. The paper correctly separates the problem of selecting informative tasks from the problem of comparing candidates evaluated on different subsets.
-
The approach is orthogonal to improvements in candidate generation. Task-CoEvolve can in principle be combined with existing harness optimizers, so the focus on reducing evaluation cost is practically relevant and complementary to methods that improve the quality or number of generated candidates.
Critical weaknesses
-
The novelty is limited. The core approach combines adaptive subset sampling, importance/sampling-aware estimation, and evolutionary harness search. These are natural combinations of established ideas from active testing, importance sampling, and sample-efficient model evaluation.
-
The proposed weight is maximized when historical outcomes are balanced, but a task can have high variance for reasons that do not make it useful for harness selection. For example, outcomes may be noisy, execution may be unstable, the task may be sensitive to random seeds, or the observed success/failure pattern may reflect infrastructure failures rather than meaningful differences between harnesses. Conversely, a task with low historical variance may still be highly informative if it exposes a consistent but important difference between candidates or if it is close to becoming solvable after a particular harness change. The paper does not provide a sufficiently principled argument that marginal Bernoulli variance approximates information gain, ranking utility, or expected improvement in the harness search.
-
At the 20% budget, uniform resampling improves the average score from 47.2 to 48.2, full-set estimation raises it to 48.8, and adding variance-weighted selection raises it to 49.3. Thus, the largest improvement comes from resampling, while the proposed variance-based selection contributes a relatively small additional gain. This weakens the claim that variance-weighted task co-evolution is the central source of the method's effectiveness.
-
The paper states that the same 89 tasks are used for search and final evaluation because the benchmark is small and expensive. This makes the reported final pass rate less independent as evidence of generalization and may favor methods that adapt strongly to the benchmark task pool.
-
The method may place more probability mass on difficult, long-running tasks because such tasks are often discriminative. This can reduce the number of evaluations while still producing high token cost per evaluation. The reported results acknowledge that equal evaluation budgets can have very different token costs. A cost-aware objective should trade off discriminative power against runtime, token usage, and failure probability.
Questions for authors
-
What formal notion of informativeness is the variance-weighted sampling rule intended to approximate: candidate disagreement, uncertainty reduction, ranking accuracy, or expected improvement in the selected harness?
-
Can the authors compare Bernoulli-variance sampling against candidate-aware disagreement sampling, uncertainty sampling, cost-aware active testing, and an oracle that selects tasks using their actual contribution to candidate ranking?
-
How stable are the selected task subsets across random seeds? Do high-variance tasks consistently produce better candidate rankings, or does the method rely on occasional informative samples?
-
Can the authors report confidence intervals or paired statistical tests for the gains over Random-Resample and Full Search, especially given that the variance-weighted component provides only a modest incremental improvement in Table 4?
-
How is the estimator chosen in a new benchmark without using full validation results? Can one estimator and one sampling protocol be used across different task-pool geometries?
-
Could the authors add an early-stopping or sequential-elimination baseline that allocates more evaluations to promising candidates and fewer to clearly inferior ones?
-
Since Task-CoEvolve focuses on evaluation efficiency, can the authors report total end-to-end cost including meta-agent candidate generation, subset-selection overhead, retries, and estimator computation for all experimental settings?