acceptodds
Under review as a conference paper at ICLR 2027

Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization

Abstract

Recent studies in harness optimization have shown substantial performance gains in language model agents without updating model weights. A well-designed harness more effectively leverages the underlying model's capabilities by shaping context, tool use, and execution workflows. However, naïve success rates alone do not fully characterize whether optimized harness yields more reliable success on tasks already within reach (*elicitation*) or a broader range of solvable tasks (*expansion*). This distinction is crucial for recursive self-improvement and evolutionary search, as expansion may provide a broader range of successful experiences to push the capability boundaries. Despite the growing significance, this distinction remains underexplored in harness optimization. We conduct a systematic analysis to study the effect of gains from harness optimization. Building on pass@k analyses in the reinforcement learning (RL) literature that compare models before and after RL training, we compare optimized harnesses with their pre-optimization counterparts. Specifically, we assess changes in reachable task coverage, compare harness optimization with model scaling and increased inference-time compute, and examine how gains vary with task difficulty. Across static prediction and dynamic tasks, we find that pass@1 advantages of harness over stronger interventions can reverse at larger attempt budgets and aggregate gains conceal shortfalls on hard tasks, underscoring the need for a fine-grained diagnosis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM agents, harness optimization, harness evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PQX5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether performance gains from agent harness optimization (without weight updates) stem from 'elicitation' (more reliable success on already-solvable tasks) or 'expansion' (broadening the set of solvable tasks). Using pass@k analyses across multiple attempt budgets, the authors compare optimized harnesses against baselines and stronger reference systems (larger models or higher reasoning effort) on static text-classification and dynamic QA/agentic benchmarks. They find that pass@1 advantages often reverse at larger k, aggregate gains conceal consistent shortfalls on hard tasks, and optimized harnesses fail to transfer to out-of-distribution domains due to learned domain-specific heuristics.

Critical strengths
  1. The application of pass@k curves to framework optimization effectively distinguishes between elicitation and scaling, surpassing the misleading nature of the aggregate pass@1 metric.
  2. The experimental design is comprehensive, covering both static and dynamic tasks across multiple model families (e.g., GPT-OSS, Qwen3), with comparisons to model scaling and test-time compute scaling.
  3. The task-level success set decomposition and direct code inspection provide clear explanations for the source of gains and the reasons for failure in OOD (Out-of-Distribution) transfer.
  4. The paper demonstrates that optimization targeting hard tasks increases frequency but not coverage, establishing clear boundary conditions for harness optimization.
Critical weaknesses
  1. Relying solely on a single optimization framework (Meta-Harness) limits the generalizability of the conclusions regarding harness optimization; different search strategies might achieve better scaling effects.
  2. The work is primarily diagnostic. Although it points out limitations, it does not propose or evaluate new optimization objectives specifically designed to promote the scaling of hard tasks.
Questions for authors
  1. Regarding optimization for hard tasks, is the validation set also restricted to hard tasks, or was a standard validation set used? How does the selection criterion interact with the training data mixture for hard-task-oriented training?
  2. Only two independent optimization runs were conducted for each setting. Have you considered attempting more runs to ensure statistical significance?
7 comments
AuthorsOpenReview ↗

Response to Reviewer PQX5 [1/4]

[W1] Additional optimization algorithms.

Revised manuscript: Appendix D.1 (Figure 18, p. 22).

We additionally evaluate three alternative approaches on HotpotQA: GEPA [1] for prompt optimization, SkillOpt [2] for skill document optimization, and AHE [3] for harness code evolution. These additional results show that our finding is not unique to Meta-Harness, demonstrating similar patterns when optimization targets prompts, skill documents, or harness codes. As shown in the table below, overall gains over each method's initial control decrease from pass@1 to pass@128: +9.01 to +1.33 percentage points for GEPA, +3.38 to +1.67 for SkillOpt, and +10.45 to +2.00 for AHE. On hard questions, the pattern is more pronounced: all three improve pass@1 but reduce pass@128, turning single-attempt improvements into coverage losses. We include the full pass@k curves in Figure 18 of Appendix D.1.

MethodOverall pass@1Overall pass@128Hard pass@1Hard pass@128
baseline → selectedbaseline → selectedbaseline → selectedbaseline → selected
GEPA61.53% → 70.54%84.00% → 85.33%46.89% → 53.73%70.73% → 65.85%
SkillOpt62.22% → 65.60%85.33% → 87.00%47.18% → 51.96%70.73% → 68.29%
AHE61.61% → 72.06%85.67% → 87.67%46.72% → 52.61%70.73% → 68.29%

References.

[1] Agrawal et al. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. ICLR, 2026.

[2] Yang et al. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. 2026.

[3] Lin et al. Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses. 2026.

AuthorsOpenReview ↗

Response to Reviewer PQX5 [2/4]

[W2] Algorithmic solution to extend the hard task coverage.

Revised manuscript: Section 5.2 (p. 8) and Appendix C.5 (Figure 17, p. 21).

While our study primarily focuses on investigating the gains and limitations of harness optimization rather than developing new algorithms, we agree that extending hard task coverage is an important direction. To resolve this concern, we conduct additional experiments and discover that directly optimizing pass@4 improves hard task coverage in our HotpotQA experiments. Specifically, we implement a variant of Meta-Harness that scores candidates by the fraction of questions answered correctly in at least one of four attempts. We apply this criterion to both candidate evaluation and final selection, aligning optimization with multi-attempt success. Related pass@k objectives have also been explored in reinforcement learning [1, 2].

Compared with the original objective, this modification improves mean pass@128 by 2.67 percentage points overall and 6.50 percentage points on hard questions, with similar mean pass@1. Mean hard pass@128 reaches 78.86%, compared with 78.05% for the Qwen3-32B reference. To separate the effect of the coverage criterion from that of scoring each candidate on four rollouts, we also run a control objective that uses the same four attempts per question but scores candidates by the mean accuracy over the attempts. The accuracy@4 control does not reproduce the gain of the pass@4 runs. Its mean pass@128 is close to that of the original objective.

For experimental details, we report mean ± sample standard deviation across three optimization runs per objective. We include the full pass@k curves in Figure 17 of Appendix C.5.

SystemOverall pass@1Overall pass@128Hard pass@1Hard pass@128
Qwen3-8B baseline61.53%84.00%46.89%70.73%
Qwen3-8B optimized (original)72.50 ± 1.43%87.78 ± 1.39%57.34 ± 1.96%72.36 ± 1.41%
Qwen3-8B optimized (accuracy@4)71.55 ± 1.73%88.22 ± 1.84%54.89 ± 2.82%71.54 ± 6.14%
Qwen3-8B optimized (pass@4)72.30 ± 1.70%90.44 ± 0.51%57.20 ± 2.43%78.86 ± 1.41%
Qwen3-32B baseline69.14%87.67%59.51%78.05%

References.

[1] Walder and Karkhanis. Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems. 2025.

[2] Tang et al. Optimizing Language Models for Inference Time Objectives using Reinforcement Learning. ICML, 2025.

AuthorsOpenReview ↗

Response to Reviewer PQX5 [3/4]

[Q1] Difficulty distribution of hard task optimization.

The validation set is also hard-focused in the optimization for hard tasks. It uses 20%/20%/60% easy/medium/hard proportions, matching the training and final selection sets where all are separately sampled, disjoint sets. Validation scores guide candidate optimization, while the final harness is selected by mean accuracy on the separate selection set. Each question received equal weight, so hard questions accounted for 60% of both the validation and selection objectives.

To fully answer to reviewer's question, we additionally tested a hard-only configuration with 0%/0%/100% proportions across all three splits, keeping split sizes and the test set fixed. As shown in the table below, hard-focused optimization increases hard pass@1 from 56.88% to 59.28% relative to the original mixture, but leaves hard pass@128 unchanged at 73.17%. Increasing the hard question proportion to 100% (hard-only) lowers hard pass@128 to 70.73%. Both remain below the Qwen3-32B reference's 78.05%. Thus, maintaining the hard task emphasis through validation and selection did not improve observed hard task coverage in these experiments.

SystemOverall pass@1Overall pass@128Hard pass@1Hard pass@128
Qwen3-8B baseline61.53%84.00%46.89%70.73%
Qwen3-8B, original mixture72.43%89.33%56.88%73.17%
Qwen3-8B, hard-focused73.63%88.67%59.28%73.17%
Qwen3-8B, hard-only71.45%89.00%55.49%70.73%
Qwen3-32B baseline69.14%87.67%59.51%78.05%
Reviewer_b1pLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper aims at studying whether agent harness optimization methods have more benefits in eliciting a model or expanding the capability of the model. Following prior work, this study focuses on using pass@1 and pass@k metrics to compare and contrast a model with a non-optimized harness, the same model with an optimized harness, and a larger reference model. The experimental results show that optimized harness leads to different levels of benefits or regression on pass@1 against pass@k, on different task levels, and on OOD tasks against in-distribution tasks.

Critical strengths

This paper aims at studying an ambitious and important question about the benefits of agent harness optimization. The experiments show some results which to my knowledge are not highlighted in prior work:

  1. agent harness optimization can result in lower pass@k compared to non-optimized harness
  2. agent harness optimization can result in lower pass@1 and pass@k on out-of-distribution tasks compared to non-optimized harness.
Critical weaknesses

The key subjects in this research, elicitation and expansion, do not have explicit and formal definitions. The paper does not establish how pass@1 and finite-budget pass@k distinguish elicitation from expansion, since improved reliability on already solvable tasks can also increase observed coverage.

Questions for authors
  1. Can you justify why finite-budget pass@k indicates capability expansion? I think finite-budget pass@k measures both success probability and task coverage. Increasing an already nonzero success probability can substantially increase pass@k without making any previously impossible task solvable. Conversely, enabling very rare successes on previously impossible tasks may barely affect measured pass@k. It might be more reasonable to measure task coverage by continuing to increase the pass@k budget until the pass rate converges. However, many of pass@k curves in Figure 3, 5, and 8 are yet to converge.
  2. For Figure 1 and Section 5.2, could you be more explicit about the difficulty distribution used in the training data of the harness optimization and in the test data? It is unclear whether the observed benefit differences on different difficulty levels are caused by the distribution shift between the training and test datasets.
  3. While the paragraph about Figure 2 in Section 3.2 acknowledged that pass@128 and pass@1 broadly track each other, the statement that "the candidate with the highest estimated pass@1 differs from the one with the highest observed pass@128" is not well justified. The metric (pass@1 or pass@128) differences of candidates in Figure 2 are quite small, within 5% for most of the evaluated candidates. Given that the test set only contains 300 instances, it would need a rigorous statistical test to establish with statistical significance that the pass@128 leader differs from pass@1 leader in terms of pass@128 and pass@1.
8 comments
AuthorsOpenReview ↗

Response to Reviewer b1pL [1/4]

Dear reviewer b1pL,

We appreciate your valuable feedback and insightful comments, which have greatly strengthened our work. We have addressed your concerns below and incorporated the definition and experimental results into the revised manuscript.


[W1] Clear definition of elicitation and expansion.

Revised manuscript: Sections 1 (p. 2) and 3.2 (p. 3); Appendix A (operational definitions and Eqs. 2–4, p. 14).

We thank the reviewer for this helpful comment, which has improved the clarity of our terminology. In short, we do not equate elicitation with pass@1 nor expansion with pass@kk on a one-to-one basis. Rather, we define these effects at task level to examine whether the accuracy gains translate into broader observed task coverage. For clarity, we provide explicit operational definitions under the same evaluation budget below. We have incorporated this distinction into Sections 1 and 3 of the revised manuscript, with the formal definitions and their connection to coverage detailed in Appendix A.

Def. For a benchmark \mathcal D\=\\\{x_i\\\}_\{i\=1\}^N, we evaluate each fixed solver-harness pair with nn independent, identically distributed executions per task. Let c{H i}c_\{H\,i\} denote the number of observed successful executions on xix_i, and p{H i}p_\{H\,i\} its true success probability; the latter can be estimated via sample mean, p^{H i}cˉ{H i}\/n\hat p_\{H\,i\}\=c_\{H\,i\}\/n. Then, with base harness H0H_0 and optimized harness H⋆H^\star, we define the two effects on task xix_i as follows:

{Empiricalelicitation}  ⟺  0<p^{H0 i}<p^{H⋆ i} \text\{Empirical elicitation\} \iff 0<\hat p_\{H_0\,i\}<\hat p_\{H^\star\,i\}\,

\text\{Empirical expansion\} \iff \hat p_\{H_0\,i\}\=0<\hat p_\{H^\star\,i\}\.

Both effects can occur on different tasks within the same harness update.

By defining the observed success sets as \hat\{\mathcal S\}_\{H\, n\}\:\=\\\{x_i\in\mathcal D\:c_\{H\,i\}\>0\\\}\, we denote the newly observed and lost success sets as \hat\{\mathcal E\}_n \:\= \hat\{\mathcal S\}_\{H^\star\, n\}\setminus\hat\{\mathcal S\}_\{H_0\, n\} and \hat\{\mathcal L\}_n \:\= \hat\{\mathcal S\}_\{H_0\,n\}\setminus\hat\{\mathcal S\}_\{H^\star\,n\}\, respectively. At knˉk\=n, the Appendix A estimator equals the fraction of tasks with at least one observed success. Therefore,

\Delta_n \:\= \hat\{\mathrm\{pass\@\}n\}�0� - \hat\{\mathrm\{pass\@\}n\}�1� \= \frac\{\|\hat\{\mathcal S\}_\{H^\star\,n\}\|-\|\hat\{\mathcal S\}_\{H_0\,n\}\|\}\{N\} \= \frac\{\|\hat\{\mathcal E\}_\{n\}\|-\|\hat\{\mathcal L\}_\{n\}\|\}\{N\} \le \frac\{\|\hat\{\mathcal E\}_n\|\}\{N\}\.

Thus, a positive coverage difference Δn\Delta_n lower-bounds the fraction of tasks with newly observed successes. Conversely, a small or non-positive difference does not establish the absence of expansion on individual tasks. The task count columns of Table 4 in the revised draft (Table 2 in the original draft) illustrate this observed set decomposition at nkˉ6ˉ4n\=k\=64. On LawBench, the optimized harness retains 49 baseline covered tasks, loses 4, and newly covers 7, increasing the observed success set from 53 to 56 tasks. Table 4 additionally reports how these task sets contribute to the observed pass@1 gain.

Our formulation uses instance level probabilistic perspective of Wang et al. [1] and adapts finite-budget empirical support [2] to harness optimization. These definitions and the bound depend on the sampled executions and their budget. Zero observed successes do not establish p{H0 i}0ˉp_\{H_0\,i\}\=0: under independent attempts, a task with p{H0 i} 0p_\{H_0\,i\}\>0 still yields no successes with probability �2�n�2�^n. Thus, empirical expansion does not establish strict support expansion, p{H0 i}0ˉ<p{H⋆ i}p_\{H_0\,i\}\=0<p_\{H^\star\,i\}, and an observed frequency increase does not certify a true probability increase. The definitions describe observed task level behavior rather than internal reasoning mechanisms. The practical value of this analysis is to diagnose success frequency and task coverage at a specified attempt budget. Also, we address the relationship to underlying task solvability and the choice of evaluation budget in [Q1].


References.

[1] Wang et al. New Skills or Sharper Primitives? A Probabilistic Perspective on the Emergence of Reasoning in RLVR. COLM, 2026.

[2] Wu et al. The Invisible Leash: Why RLVR May or May Not Escape Its Origin. 2026.

AuthorsOpenReview ↗

Response to Reviewer b1pL [2/4]

[Q1] Justification of capability expansion.

Revised manuscript: Section 3.2 (p. 3), Appendix A (finite-budget interpretation, p. 14) and Appendix C.3 Table 7, p.20)

We agree with the reviewer that finite-budget pass@kk cannot, by itself, distinguish increased success probability on already solvable tasks from expansion of the set of tasks with nonzero success probability. We instead interpret it as the expected task coverage at a specified execution budget, following the definitions in [W1]:

�0� \.$$ Nevertheless, we consider this budget-specific coverage, rather than its limit, to be the relevant quantity for two reasons. First, it is what downstream uses depend on: self-training, verifier-based selection, and recursive self-improvement can only exploit successes that appear within the attempts they can afford, so a task that succeeds once in a million attempts is solvable in the strict sense but unreachable in practice. Second, the limit $k\to\infty$, i.e., the fraction of tasks with $p_\{H\,i\}\>0$, cannot be identified from finite samples, and under stochastic decoding it also counts arbitrarily rare or chance successes. Prior work likewise reports coverage at finite budgets of 128 to 1,024 attempts [1, 2, 3]. Because each attempt in our setup requires a complete execution of the solver-harness system, which make repeated evaluation computationally expensive, we set $k\=64$ or $k\=128$, considering limited computational budget. The remaining question is whether conclusions drawn at these budgets depend on the particular choice of $k$. To this end, we conduct additional experiments by increasing the budget to 1,024 complete executions per question on all 300 HotpotQA test questions and, due to resource constraints, to 2,048 on the 41 hard questions only. Each subset's estimates use its full rollout pool, with optimized results averaged across three optimization runs. The answer is that each system's pass@$k$ value keeps rising with $k$, but the gaps between systems are stable. The curves do not saturate: the final doubling still adds 0.67–0.81 pp overall and 0.10–1.22 pp on hard questions. However, the optimized-minus-baseline gap falls from 10.96 pp at pass@1 to 3.03 pp at $k\=64$ and then stays within 2.92–3.14 pp through $k\=1024$. Likewise, the optimized harness's +3.43 pp pass@1 advantage over Qwen3-32B is gone by $k\=8$, and the deficit grows to 1.00 pp at $k\=1024$; on hard questions, the optimized system never exceeds the 32B reference at every budget up to $k\=2048$. The observations, we reported at $k\=128$, therefore hold across the extended budgets: large pass@1 gains shrink to a small, stable coverage gain, and the pass@1 advantage over the stronger reference does not persist. **Overall** | System | pass@1 | pass@4 | pass@16 | pass@64 | pass@128 | pass@256 | pass@512 | pass@1024 | pass@2048 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3-8B baseline | 61.53% | 72.56% | 79.49% | 83.32% | 84.71% | 85.74% | 86.52% | 87.33% | -- | | Qwen3-8B optimized | 72.50% | 79.32% | 83.30% | 86.35% | 87.63% | 88.74% | 89.66% | 90.33% | -- | | Qwen3-32B baseline | 69.07% | 78.81% | 83.48% | 86.53% | 87.93% | 89.27% | 90.52% | 91.33% | -- | **Hard** | System | pass@1 | pass@4 | pass@16 | pass@64 | pass@128 | pass@256 | pass@512 | pass@1024 | pass@2048 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3-8B baseline | 47.02% | 58.71% | 67.32% | 70.10% | 71.00% | 71.84% | 72.87% | 74.39% | 75.61% | | Qwen3-8B optimized | 57.04% | 63.32% | 67.66% | 70.80% | 72.27% | 73.78% | 75.40% | 76.92% | 78.05% | | Qwen3-32B baseline | 59.46% | 68.24% | 72.60% | 75.76% | 77.66% | 79.86% | 81.84% | 82.83% | 82.93% | --- **References.** [1] Yue et al. [*Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?*](https://openreview.net/forum?id=4OsgYD7em5). NeurIPS, 2025. [2] Wen et al. [*Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs*](https://openreview.net/forum?id=jGbRWwIidy). ICLR, 2026. [3] Zhang et al. [*On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models*](https://openreview.net/forum?id=TBaUfO9znF). ICML, 2026.
AuthorsOpenReview ↗

Response to Reviewer b1pL [3/4]

[Q2] Difficulty distribution of dataset.

Revised manuscript: Appendix B.1 (Table 2, p. 15); Section 5.2 (p. 8) and Appendix C.5 (Figure 17, p. 21).

Thank you for the constructive comments. As presented in the table below, our main experiments follow established benchmark setups, using GEPA’s setup for HotpotQA and the official partitions for AppWorld. However, we agree that differences in difficulty composition across splits could affect optimization and selection. We therefore conducted additional sensitivity analyses, finding that the HotpotQA hard task coverage gap persists when training, validation, and selection use the same difficulty proportions, while reweighting AppWorld's final selection scores preserves all three original harness selections. We have included this explicit distribution table in the revised draft (Appendix B.1, Table 2).

BenchmarkSplitEasyMediumHard
HotpotQATraining19.33%58.00%22.67%
Validation20.00%65.00%15.00%
Selection18.67%65.67%15.67%
Test15.33%71.00%13.67%
AppWorldTraining40.00%40.00%20.00%
Validation40.00%40.00%20.00%
Selection52.63%42.11%5.26%
Test33.93%28.57%37.50%

HotpotQA. Simply adjusting the difficulty mixture does not improve hard task coverage under both test-aligned and hard-only optimization. We ran Meta-Harness once per condition with Qwen3-8B using same split sizes and test set. The test-aligned condition approximately matched the test easy/medium/hard proportions of 15.33%/71.00%/13.67% across training, validation, and selection. The hard-only condition uses exclusively hard questions across all three splits. All splits are disjoint.

SystemOverall pass@1Overall pass@128Hard pass@1Hard pass@128
Qwen3-8B baseline61.53%84.00%46.89%70.73%
Qwen3-8B, original mixture72.43%89.33%56.88%73.17%
Qwen3-8B, test-aligned71.34%85.67%57.89%68.29%
Qwen3-8B, hard-only71.45%89.00%55.49%70.73%
Qwen3-32B baseline69.14%87.67%59.51%78.05%

AppWorld. All three original selections remain unchanged under alternative difficulty weights among the original eligible candidate harnesses. As all official non-test tasks of AppWorld have already been allocated, it is infeasible to resample the training and validation splits for additional harness optimization run with matched difficulty distribution. We therefore assess selection sensitivity post hoc using existing outcomes on the official development (selection) set. For each candidate, we first average the selection scores separately over easy/medium/hard tasks and recompute its selection score by giving these three averages equal weight or weighting them according to the test proportions of 33.93%/28.57%/37.50%. The same harness receives the highest score under each weighting in all three runs. Parentheses show the second-highest score among the finalists under each weighting.

RunSelected harnessOriginalEqual-weightedTest-aligned
0answer_arity_gate84.2% (71.9%)78.9% (70.3%)78.2% (70.1%)
1doc_view_compaction73.7% (63.2%)61.4% (53.3%)59.9% (52.8%)
2preflight_write_gate66.7% (66.7%)65.8% (64.7%)66.5% (65.4%)

Practical solution. While the suggested distribution adjustment was not effective, we find that directly optimizing multi-attempt success improves hard task coverage while retaining the original difficulty mixture. On HotpotQA with Qwen3-8B, we replace the accuracy objective with pass@4 for both candidate evaluation and final selection, keeping solver weights fixed. Across three optimization runs per objective, this modification improves mean pass@128 both overall and on hard questions. We include the full pass@k curves with additional control experiments in Figure 17 of Appendix C.5.

SystemOverall pass@1Overall pass@128Hard pass@1Hard pass@128
Qwen3-8B baseline61.53%84.00%46.89%70.73%
Qwen3-8B optimized (original)72.50 ± 1.43%87.78 ± 1.39%57.34 ± 1.96%72.36 ± 1.41%
Qwen3-8B optimized (pass@4)72.30 ± 1.70%90.44 ± 0.51%57.20 ± 2.43%78.86 ± 1.41%
Qwen3-32B baseline69.14%87.67%59.51%78.05%
Reviewer_TmJbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper conducts a multi-faceted diagnosis of performance gains in LLM agent harness optimization, using pass@k analysis to distinguish whether improvements stem from eliciting higher reliability on already solvable tasks or expanding coverage to previously unreachable problems. By evaluating on static classification, multi-hop QA, and interactive agent benchmarks, the authors show that pass@1 gains often diminish or reverse at higher attempt budgets k and that harness improvements consistently lag behind model scaling or inference compute scaling on hard tasks.

Critical strengths

The paper provides a clear conceptual distinction between capability elicitation and expansion for agent harness engineering, using systematic pass@k curves and difficulty stratification to expose how headline pass@1 numbers can obscure coverage loss and shortfalls on challenging problems.

Critical weaknesses

The empirical scope is somewhat limited, relying heavily on a single optimization framework (Meta-Harness) across a narrow set of tasks and models, and while the diagnosis is insightful, the paper lacks constructive algorithmic solutions to overcome the identified hard-task coverage bottleneck.

Questions for authors

Could the authors provide evidence on whether alternative optimization algorithms (e.g., iterative prompt-only search or agentic skill evolution) exhibit this same coverage saturation, and clarify whether the hard-task ceiling is primarily constrained by the fixed solver model's fundamental reasoning limits rather than the harness search space?

5 comments
AuthorsOpenReview ↗

Response to Reviewer TmJb [1/3]

Dear reviewer TmJb,

We appreciate your valuable feedback and insightful comments, which have greatly strengthened our work. We have addressed your concerns below and incorporated the experimental results into the revised manuscript.


[W1 & Q1] Extended empirical scope.

Revised manuscript: Appendix D.1 (Figure 18, p. 22), D.2 (Figure 19, pp. 22–23), and D.3 (Figure 20, p. 23).

We thank the reviewer for suggesting additional experiments that enrich the quality of our work. While we considered various tasks and models in the manuscript, we agree that more evaluations across different axes would strengthen the evidence for our findings. To this end, we conduct new experiments including additional task, model families, and optimization algorithms showing that aggregate pass@1 gains alone do not fully characterize the benefits of harness optimization, as they coexist with smaller gains at pass@128 and remaining coverage gaps on hard tasks.

Additional task. We evaluate Meta-Harness on OfficeQA [1] with Qwen3-8B and observe a pattern consistent with our main-paper findings: a harness optimization improves single-attempt success while coverage gains diminish at larger attempt budgets. Optimization narrows the overall pass@1 gap to Qwen3-32B from 13.34 to 3.53 percentage points (pp), but the optimized system's gap is larger at pass@128 (5.23 points). We include the full pass@k curves in Figure 19 of Appendix D.2.

BenchmarkSystemOverall pass@1Overall pass@128Hard pass@1Hard pass@128
OfficeQAQwen3-8B baseline20.09%47.67%11.22%36.56%
Qwen3-8B optimized29.91%55.81%17.43%41.94%
Qwen3-32B baseline33.44%61.05%20.78%45.16%

Additional models. We evaluate Meta-Harness with Gemma 3-4B [2] and DeepSeek-V4-Flash-0423 [3] on HotpotQA, observing smaller overall gains at pass@128 than at pass@1 with neither model improving hard task coverage. For Gemma, optimization improves overall pass@1 by 1.73 pp but pass@128 by only 0.33 points. On hard questions, both metrics decline. For DeepSeek, the overall gain similarly decreases from 8.58 points at pass@1 to 2.67 points at pass@128. On hard questions, pass@1 increases from 56.04% to 66.18%, while pass@128 remains unchanged. We include the full pass@k curves in Figure 20 of Appendix D.3.

BenchmarkSystemOverall pass@1Overall pass@128Hard pass@1Hard pass@128
HotpotQAGemma 3-4B baseline48.94%78.00%38.78%63.41%
Gemma 3-4B optimized50.68%78.33%37.60%60.98%
Gemma 3-27B baseline64.32%83.33%53.68%75.61%
DeepSeek-V4-Flash-0423 baseline70.34%88.33%56.04%75.61%
DeepSeek-V4-Flash-0423 optimized78.92%91.00%66.18%75.61%
DeepSeek-V4-Flash-0731 baseline75.81%89.00%65.99%80.49%

Additional optimization algorithms. We evaluate three alternative approaches on HotpotQA: GEPA [4] for prompt optimization, SkillOpt [5] for skill document optimization, and AHE [6] for harness code evolution. These results show that our finding is not unique to Meta-Harness, depicting similar patterns when optimization targets prompts, skill documents, or harness codes. As shown in the table below, overall gains over each method's initial control decrease from pass@1 to pass@128: +9.01 to +1.33 pp for GEPA, +3.38 to +1.67 for SkillOpt, and +10.45 to +2.00 for AHE. On hard questions, the pattern is more pronounced: all three improve pass@1 but reduce pass@128, turning single-attempt improvements into coverage losses. We include the full pass@k curves in Figure 18 of Appendix D.1.

MethodOverall pass@1Overall pass@128Hard pass@1Hard pass@128
baseline → selectedbaseline → selectedbaseline → selectedbaseline → selected
GEPA61.53% → 70.54%84.00% → 85.33%46.89% → 53.73%70.73% → 65.85%
SkillOpt62.22% → 65.60%85.33% → 87.00%47.18% → 51.96%70.73% → 68.29%
AHE61.61% → 72.06%85.67% → 87.67%46.72% → 52.61%70.73% → 68.29%

References.

[1] Opsahl-Ong et al. OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning. 2026.

[2] Gemma Team. Gemma 3 Technical Report. 2025.

[3] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. 2026.

[4] Agrawal et al. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. ICLR, 2027.

[5] Yang et al. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. 2026.

[6] Lin et al. Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses. 2026.

AuthorsOpenReview ↗

Response to Reviewer TmJb [2/3]

[W2] Algorithmic solution to extend the hard task coverage.

Revised manuscript: Section 5.2 (p. 8) and Appendix C.5 (Figure 17, p. 21).

While our study primarily focuses on investigating the gains and limitations of harness optimization rather than developing new algorithms, we agree that extending hard task coverage is an important direction. To resolve this concern, we conduct additional experiments and discover that directly optimizing pass@4 improves hard task coverage in our HotpotQA experiments. Specifically, we implement a variant of Meta-Harness that scores candidates by the fraction of questions answered correctly in at least one of four attempts. We apply this criterion to both candidate evaluation and final selection, aligning optimization with multi-attempt success. Related pass@k objectives have also been explored in reinforcement learning [1, 2].

Compared with the original objective, this modification improves mean pass@128 by 2.67 percentage points overall and 6.50 percentage points on hard questions, with similar mean pass@1. Mean hard pass@128 reaches 78.86%, compared with 78.05% for the Qwen3-32B reference. To separate the effect of the coverage criterion from that of scoring each candidate on four rollouts, we also run a control objective that uses the same four attempts per question but scores candidates by the mean accuracy over the attempts. The accuracy@4 control does not reproduce the gain of the pass@4 runs. Its mean pass@128 is close to that of the original objective.

For experimental details, we report mean ± sample standard deviation across three optimization runs per objective. We include the full pass@k curves in Figure 17 of Appendix C.5.

SystemOverall pass@1Overall pass@128Hard pass@1Hard pass@128
Qwen3-8B baseline61.53%84.00%46.89%70.73%
Qwen3-8B optimized (original)72.50 ± 1.43%87.78 ± 1.39%57.34 ± 1.96%72.36 ± 1.41%
Qwen3-8B optimized (accuracy@4)71.55 ± 1.73%88.22 ± 1.84%54.89 ± 2.82%71.54 ± 6.14%
Qwen3-8B optimized (pass@4)72.30 ± 1.70%90.44 ± 0.51%57.20 ± 2.43%78.86 ± 1.41%
Qwen3-32B baseline69.14%87.67%59.51%78.05%

References.

[1] Walder and Karkhanis. Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems. 2025.

[2] Tang et al. Optimizing Language Models for Inference Time Objectives using Reinforcement Learning. ICML, 2025.

AuthorsOpenReview ↗

Response to Reviewer TmJb [3/3]

[Q2] Source of the hard task ceiling.

Our experiments do not establish that the hard task shortfall is primarily caused by the solver’s fundamental reasoning limits. An additional oracle analysis reveals broader observed coverage across the candidate pool using the same solver than under the selected harness alone.

Specifically, we analyze the initial harness and all 40 candidates from one Meta-Harness run, each evaluated with Qwen3-8B on the same test set with 128 executions per question. Then, we consider the oracle setup that one knows which harness succeeded on each question and can choose a different harness for each question; namely, the oracle's' perfomrnace is equal to a retrospective union of observed success sets across multiple harnesses. This oracle covers 92.68% of hard questions, compared with 73.17% for the selected harness. Therefore, the selected harness’s finite-budget coverage should not be interpreted as an intrinsic capability ceiling of the solver.

SystemEvaluated harness poolsOverall observed coverageHard observed coverage
Qwen3-8B baseline184.00%70.73%
Qwen3-8B optimized (run 0)187.33%73.17%
Qwen3-8B oracle (run 0)4197.00%92.68%
Qwen3-32B baseline187.67%78.05%

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers and AC,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript.

As reviewers highlighted, our work addresses an important research question regarding the benefits of harness optimization (b1pL), provides a systematic diagnostic perspective through pass@k and task difficulty analyses (TmJb, PQX5), and reveals insightful empirical findings on coverage limitations and out-of-distribution transfer (b1pL, PQX5).

To respond to your constructive feedback on our manuscript, we have carefully updated and improved the manuscript, adding further analyses and experimental results as follows:

  • Formal definitions of empirical elicitation and expansion (Appendix A)
  • Difficulty distribution analysis (Appendix B.1)
  • Additional optimization runs for variability analysis (Appendix C.1)
  • Validation for the leader comparison (Appendix C.3)
  • Extended pass@k evaluation up to 2,048 attempts (Appendix C.3)
  • Algorithmic solution for improving hard task coverage (Appendix C.5)
  • Additional optimization algorithms: GEPA, SkillOpt, and AHE (Appendix D.1)
  • Additional benchmark : OfficeQA (Appendix D.2)
  • Additional model families: Gemma and DeepSeek (Appendix D.3)

In the revised manuscript, all changes are temporarily highlighted in blue\textcolor{blue}{\text{blue}} for ease of review. We hope these updates strengthen our contributions and adequately address the reviewers' concerns.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.