acceptodds
Withdrawn from ICLR 2027

Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present a novel approach to efficient LLM harness optimization through adaptive validation task selection. Harness optimization iteratively rewrites the harness code based on validation performance, enabling substantial performance gains without updating the underlying model weights. Existing approaches, however, evaluate a fixed validation set in full at every iteration, incurring substantial evaluation costs even on tasks that become less discriminative as the harness evolves. We propose , which co-evolves the validation tasks with the harness by addressing two challenges: selecting informative tasks and estimating full-set performance from partial evaluations. Task-CoEvolve builds on the observation that tasks on which candidate harnesses disagree are more informative for distinguishing among them than tasks that are consistently solved or failed. It uses variance-weighted sampling based on past outcomes to focus evaluation on tasks near the capability frontier, with the sampling distribution adapting as the harness evolves. It then estimates full-set scores from the sampled tasks by accounting for their sampling probabilities, enabling comparisons on a common estimated full-set scale across iterations despite evaluating different subsets. Experiments on online text classification and Terminal-Bench 2.1 show that Task-CoEvolve improves average performance over subset-based baselines and achieves a similar pass rate on the same 89-task benchmark of full-set search while reducing the number of evaluations during optimization by 80%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
harness optimization, efficient optimization, recursive self-improvement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_voTyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the evaluation bottleneck in iterative LLM harness optimization. Existing methods evaluate every candidate harness on the full validation task set at every iteration, which can be expensive, especially for long-horizon terminal tasks. The paper proposes Task-CoEvolve, which jointly evolves the harness and the validation task subset. At each iteration, it samples tasks using weights based on the historical Bernoulli variance of their success outcomes, estimates the candidate's full-set performance using sampling-aware estimators, and selects the candidate with the highest estimated full-set score. Experiments on online text classification and Terminal-Bench 2.1 show that the method can approach full-set search performance while substantially reducing the number of task evaluations, input-token usage, and wall-clock time.

Critical strengths
  1. Meaningful improvement in iterative evaluation efficiency. The paper targets an important bottleneck in harness evolution: repeatedly executing expensive candidates on the entire validation set. On Terminal-Bench 2.1, Task-CoEvolve achieves performance close to full-set search while reducing search-phase input-token usage by approximately 67--80% and wall-clock time by roughly half. On online text classification, it reaches comparable performance with only 7% of the per-iteration validation budget and improves over full search in the reported 20% setting.

  2. The method addresses both task selection and score comparability. Merely evaluating a changing subset would make raw scores incomparable across iterations. The use of sampling-aware full-set estimation is therefore a sensible and necessary component of the proposed pipeline. The paper correctly separates the problem of selecting informative tasks from the problem of comparing candidates evaluated on different subsets.

  3. The approach is orthogonal to improvements in candidate generation. Task-CoEvolve can in principle be combined with existing harness optimizers, so the focus on reducing evaluation cost is practically relevant and complementary to methods that improve the quality or number of generated candidates.

Critical weaknesses
  1. The novelty is limited. The core approach combines adaptive subset sampling, importance/sampling-aware estimation, and evolutionary harness search. These are natural combinations of established ideas from active testing, importance sampling, and sample-efficient model evaluation.

  2. The proposed weight is maximized when historical outcomes are balanced, but a task can have high variance for reasons that do not make it useful for harness selection. For example, outcomes may be noisy, execution may be unstable, the task may be sensitive to random seeds, or the observed success/failure pattern may reflect infrastructure failures rather than meaningful differences between harnesses. Conversely, a task with low historical variance may still be highly informative if it exposes a consistent but important difference between candidates or if it is close to becoming solvable after a particular harness change. The paper does not provide a sufficiently principled argument that marginal Bernoulli variance approximates information gain, ranking utility, or expected improvement in the harness search.

  3. At the 20% budget, uniform resampling improves the average score from 47.2 to 48.2, full-set estimation raises it to 48.8, and adding variance-weighted selection raises it to 49.3. Thus, the largest improvement comes from resampling, while the proposed variance-based selection contributes a relatively small additional gain. This weakens the claim that variance-weighted task co-evolution is the central source of the method's effectiveness.

  4. The paper states that the same 89 tasks are used for search and final evaluation because the benchmark is small and expensive. This makes the reported final pass rate less independent as evidence of generalization and may favor methods that adapt strongly to the benchmark task pool.

  5. The method may place more probability mass on difficult, long-running tasks because such tasks are often discriminative. This can reduce the number of evaluations while still producing high token cost per evaluation. The reported results acknowledge that equal evaluation budgets can have very different token costs. A cost-aware objective should trade off discriminative power against runtime, token usage, and failure probability.

Questions for authors
  1. What formal notion of informativeness is the variance-weighted sampling rule intended to approximate: candidate disagreement, uncertainty reduction, ranking accuracy, or expected improvement in the selected harness?

  2. Can the authors compare Bernoulli-variance sampling against candidate-aware disagreement sampling, uncertainty sampling, cost-aware active testing, and an oracle that selects tasks using their actual contribution to candidate ranking?

  3. How stable are the selected task subsets across random seeds? Do high-variance tasks consistently produce better candidate rankings, or does the method rely on occasional informative samples?

  4. Can the authors report confidence intervals or paired statistical tests for the gains over Random-Resample and Full Search, especially given that the variance-weighted component provides only a modest incremental improvement in Table 4?

  5. How is the estimator chosen in a new benchmark without using full validation results? Can one estimator and one sampling protocol be used across different task-pool geometries?

  6. Could the authors add an early-stopping or sequential-elimination baseline that allocates more evaluations to promising candidates and fewer to clearly inferior ones?

  7. Since Task-CoEvolve focuses on evaluation efficiency, can the authors report total end-to-end cost including meta-agent candidate generation, subset-selection overhead, retries, and estimator computation for all experimental settings?

Reviewer_BRhFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Task-CoEvolve, which reduces harness optimization costs by adaptively selecting informative validation tasks and estimating full-set performance from partial evaluations. Experiments on text classification and Terminal-Bench 2.1 show performance close to full-set search with fewer evaluations.

Critical strengths
  1. The paper addresses a practical problem of reducing evaluation costs in iterative harness optimization. The proposed method is lightweight and does not require modifying the underlying LLM.
  2. The experiments show substantial evaluation cost savings while maintaining performance close to full-set search. Reporting actual token usage, monetary cost, and runtime is also valuable.
Critical weaknesses
  1. Limited methodological novelty. The method mainly combines existing techniques, including variance-based adaptive sampling and classical statistical estimators. The methodological contribution beyond applying them to harness optimization appears limited.
  2. Insufficient evidence for adaptive sampling. Table 4 shows only a 0.5-point improvement from variance-weighted selection, and Terminal-Bench lacks a corresponding ablation. Moreover, Task-CoEvolve consumes substantially more tokens than random sampling, without comparisons under matched computational or monetary budgets. It remains unclear whether the proposed strategy offers meaningful efficiency advantages over simpler alternatives.
  3. Limited Terminal-Bench evaluation. The experiments use only 89 tasks, a fixed 20% sampling budget, and ten evolution iterations, without repeated independent search runs. More importantly, the same tasks are used for optimization and final evaluation, leaving generalization to unseen tasks unclear.
Questions for authors

See Weaknesses.

Reviewer_nuugOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets the evaluation cost of automated harness optimization. Task-CoEvolve uses variance-weighted sampling, based on the Bernoulli variance of each task's past outcomes, to focus evaluation on tasks near the capability frontier. It then estimates full-set scores from the sampled tasks by accounting for their sampling probabilities, so that candidates evaluated on different subsets can be compared on a common scale.

On online text classification and Terminal-Bench 2.1, Task-CoEvolve outperforms fixed and randomly resampled subsets and performs close to full-set search, while reducing the number of evaluations during search by 80%.

Critical strengths

[C1] Well-motivated problem,. Evaluation dominates the cost of harness search, and most validation tasks are uninformative: over 70% are never or mostly solved. Separating task selection from cross-subset comparability, and handling the latter with design-based estimation, is simple and reusable.

[C2] Useful diagnostics. The paper compares estimated with true scores, reports actual token and time costs, and documents a failure of the Hájek estimator. These make the method's limits visible.

Critical weaknesses

[W1] The evidence is too small to support the claimed gains. Text classification uses three runs. The gaps to full search and Random-Resample (0.6–1.1 points) are comparable to the run-to-run standard deviations (0.5–2.2), and variance-weighted selection adds only 0.5 points in the ablation. Terminal-Bench appears to be a single run with one rollout per task, so differences of 1–4 tasks cannot be separated from noise, and full search is scored by its own selection-time validation score. At ρ = 7% the estimated scores correlate only 0.13 with true scores (Table 6), yet accuracy barely moves, so the benchmark may not discriminate between protocols.

[W2] Estimator choice is decisive, and costs are not matched. With the other estimator, Task-CoEvolve falls below both baselines on text classification (46.0 and 44.0, Table A), and Hájek is off by 14.6 points on average on Terminal-Bench (A.2). The rule for choosing the estimator has no threshold and was effectively fitted to these two benchmarks. The budget also counts tasks, not cost: at the same ρ, Task-CoEvolve uses 579M input tokens (30 USD), against 124M (8 USD) for Random-Resample (Table 3), and no cost-matched comparison is given.

Questions for authors

[Q1] Can you repeat the Terminal-Bench experiments, at least three search runs per method and model and report std in addition to the mean results?

[Q2] How should the estimator be chosen for a new benchmark? Please state the rule with an explicit threshold. Please also report Task-CoEvolve with a single, stabilized estimator on both benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.