- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether performance gains from agent harness optimization (without weight updates) stem from 'elicitation' (more reliable success on already-solvable tasks) or 'expansion' (broadening the set of solvable tasks). Using pass@k analyses across multiple attempt budgets, the authors compare optimized harnesses against baselines and stronger reference systems (larger models or higher reasoning effort) on static text-classification and dynamic QA/agentic benchmarks. They find that pass@1 advantages often reverse at larger k, aggregate gains conceal consistent shortfalls on hard tasks, and optimized harnesses fail to transfer to out-of-distribution domains due to learned domain-specific heuristics.
Critical strengths
- The application of pass@k curves to framework optimization effectively distinguishes between elicitation and scaling, surpassing the misleading nature of the aggregate pass@1 metric.
- The experimental design is comprehensive, covering both static and dynamic tasks across multiple model families (e.g., GPT-OSS, Qwen3), with comparisons to model scaling and test-time compute scaling.
- The task-level success set decomposition and direct code inspection provide clear explanations for the source of gains and the reasons for failure in OOD (Out-of-Distribution) transfer.
- The paper demonstrates that optimization targeting hard tasks increases frequency but not coverage, establishing clear boundary conditions for harness optimization.
Critical weaknesses
- Relying solely on a single optimization framework (Meta-Harness) limits the generalizability of the conclusions regarding harness optimization; different search strategies might achieve better scaling effects.
- The work is primarily diagnostic. Although it points out limitations, it does not propose or evaluate new optimization objectives specifically designed to promote the scaling of hard tasks.
Questions for authors
- Regarding optimization for hard tasks, is the validation set also restricted to hard tasks, or was a standard validation set used? How does the selection criterion interact with the training data mixture for hard-task-oriented training?
- Only two independent optimization runs were conducted for each setting. Have you considered attempting more runs to ensure statistical significance?
7 comments
Response to Reviewer PQX5 [2/4]
[W2] Algorithmic solution to extend the hard task coverage.
Revised manuscript: Section 5.2 (p. 8) and Appendix C.5 (Figure 17, p. 21).
While our study primarily focuses on investigating the gains and limitations of harness optimization rather than developing new algorithms, we agree that extending hard task coverage is an important direction. To resolve this concern, we conduct additional experiments and discover that directly optimizing pass@4 improves hard task coverage in our HotpotQA experiments. Specifically, we implement a variant of Meta-Harness that scores candidates by the fraction of questions answered correctly in at least one of four attempts. We apply this criterion to both candidate evaluation and final selection, aligning optimization with multi-attempt success. Related pass@k objectives have also been explored in reinforcement learning [1, 2].
Compared with the original objective, this modification improves mean pass@128 by 2.67 percentage points overall and 6.50 percentage points on hard questions, with similar mean pass@1. Mean hard pass@128 reaches 78.86%, compared with 78.05% for the Qwen3-32B reference. To separate the effect of the coverage criterion from that of scoring each candidate on four rollouts, we also run a control objective that uses the same four attempts per question but scores candidates by the mean accuracy over the attempts. The accuracy@4 control does not reproduce the gain of the pass@4 runs. Its mean pass@128 is close to that of the original objective.
For experimental details, we report mean ± sample standard deviation across three optimization runs per objective. We include the full pass@k curves in Figure 17 of Appendix C.5.
| System | Overall pass@1 | Overall pass@128 | Hard pass@1 | Hard pass@128 |
|---|---|---|---|---|
| Qwen3-8B baseline | 61.53% | 84.00% | 46.89% | 70.73% |
| Qwen3-8B optimized (original) | 72.50 ± 1.43% | 87.78 ± 1.39% | 57.34 ± 1.96% | 72.36 ± 1.41% |
| Qwen3-8B optimized (accuracy@4) | 71.55 ± 1.73% | 88.22 ± 1.84% | 54.89 ± 2.82% | 71.54 ± 6.14% |
| Qwen3-8B optimized (pass@4) | 72.30 ± 1.70% | 90.44 ± 0.51% | 57.20 ± 2.43% | 78.86 ± 1.41% |
| Qwen3-32B baseline | 69.14% | 87.67% | 59.51% | 78.05% |
References.
[1] Walder and Karkhanis. Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems. 2025.
[2] Tang et al. Optimizing Language Models for Inference Time Objectives using Reinforcement Learning. ICML, 2025.
Response to Reviewer PQX5 [3/4]
[Q1] Difficulty distribution of hard task optimization.
The validation set is also hard-focused in the optimization for hard tasks. It uses 20%/20%/60% easy/medium/hard proportions, matching the training and final selection sets where all are separately sampled, disjoint sets. Validation scores guide candidate optimization, while the final harness is selected by mean accuracy on the separate selection set. Each question received equal weight, so hard questions accounted for 60% of both the validation and selection objectives.
To fully answer to reviewer's question, we additionally tested a hard-only configuration with 0%/0%/100% proportions across all three splits, keeping split sizes and the test set fixed. As shown in the table below, hard-focused optimization increases hard pass@1 from 56.88% to 59.28% relative to the original mixture, but leaves hard pass@128 unchanged at 73.17%. Increasing the hard question proportion to 100% (hard-only) lowers hard pass@128 to 70.73%. Both remain below the Qwen3-32B reference's 78.05%. Thus, maintaining the hard task emphasis through validation and selection did not improve observed hard task coverage in these experiments.
| System | Overall pass@1 | Overall pass@128 | Hard pass@1 | Hard pass@128 |
|---|---|---|---|---|
| Qwen3-8B baseline | 61.53% | 84.00% | 46.89% | 70.73% |
| Qwen3-8B, original mixture | 72.43% | 89.33% | 56.88% | 73.17% |
| Qwen3-8B, hard-focused | 73.63% | 88.67% | 59.28% | 73.17% |
| Qwen3-8B, hard-only | 71.45% | 89.00% | 55.49% | 70.73% |
| Qwen3-32B baseline | 69.14% | 87.67% | 59.51% | 78.05% |
Response to Reviewer PQX5 [1/4]
[W1] Additional optimization algorithms.
We additionally evaluate three alternative approaches on HotpotQA: GEPA [1] for prompt optimization, SkillOpt [2] for skill document optimization, and AHE [3] for harness code evolution. These additional results show that our finding is not unique to Meta-Harness, demonstrating similar patterns when optimization targets prompts, skill documents, or harness codes. As shown in the table below, overall gains over each method's initial control decrease from pass@1 to pass@128: +9.01 to +1.33 percentage points for GEPA, +3.38 to +1.67 for SkillOpt, and +10.45 to +2.00 for AHE. On hard questions, the pattern is more pronounced: all three improve pass@1 but reduce pass@128, turning single-attempt improvements into coverage losses. We include the full pass@k curves in Figure 18 of Appendix D.1.
References.
[1] Agrawal et al. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. ICLR, 2026.
[2] Yang et al. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. 2026.
[3] Lin et al. Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses. 2026.