acceptodds
Under review as a conference paper at ICLR 2027

Rethinking the Evaluation of Harness Evolution for Agents

Abstract

Harness evolution is an iterative search procedure that repeatedly evaluates and revises candidate harnesses used for LLM agents using task feedback. We revisit the evaluation of such automatic harness evolution procedures and identify two fundamental issues in the protocol. First, prior work does not compare these approaches with simple task-level search baselines under matched feedback and inference budgets. Second, prior work searches for harness configurations using verification signals (e.g., unit test cases) drawn from the same benchmarks on which it reports the final performance of the evolved harnesses, violating the standard separation between training and test data. To address this, we compare automatic harness evolution with simple test-time scaling and discovery baselines under comparable feedback and inference budgets, and evaluate evolved harnesses on held-out tasks to assess generalization. Following prior work, we experiment on Terminal-Bench 2.1 and find that automatic harness evolution fails to outperform simple test-time scaling methods both with and without test cases, and exhibits limited generalization. However, we find that long-horizon games are a promising setting for automatic harness evolution, as they are difficult enough to leave headroom, rely heavily on adaptation to out-of-distribution dynamics, and provide granular feedback by design. In these settings, task-specific harness evolution improves over the search baseline by 80% on ARC-AGI-3 and by 10.9% on EdgeBench under matched budgets. Together, these findings highlight the need for matched-budget baselines and held-out evaluation to distinguish genuine harness improvements from benchmark-specific search and overfitting, and point to a more careful characterization of when automatic harness evolution is actually useful.

Primary area:
foundation or frontier models, including LLMs
Keywords:
harness evolution, llm agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_xykkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether updating an agent’s harness produces reusable capabilities or merely gives the agent more opportunities to solve tasks it has already seen. It compares parallel sampling, sequential refinement, cross-task harness evolution, and task-specific harness updates.

The main finding is that cross-task harness evolution provides little or no improvement on held-out Terminal-Bench tasks, while task-specific updates can improve performance in longer-horizon settings such as ARC-AGI-3 and EdgeBench. The paper argues that harness updates are more useful for adapting to a specific task than for learning capabilities that transfer broadly across tasks.

Overall, I view this paper’s main contribution as an evaluation perspective rather than a new harness-evolution algorithm. It helps readers distinguish genuine transferable capability from gains caused by additional search, task- specific adaptation, or unpriced internal computation, and it argues for matched baselines and held-out tasks when evaluating agent improvement. This is a useful framing for the field because it makes a common source of overclaiming more visible.

At the same time, the paper’s empirical conclusions are relatively narrow and preliminary: the strongest evidence shows limited transfer in the tested Terminal-Bench setting and benefits from task-specific adaptation in selected long- horizon environments. Because the AHE configuration differs from the original method, some metric definitions remain unclear, and the cost accounting is incomplete in parts of the evaluation, the paper should be read as evidence for a conditional and more cautious interpretation of harness evolution. The broader conclusion regarding "harness updates produce transferable capabilities or not" still remains unclear.

Based on current readability, presentation and concerns, I am conservative on recommending for acceptance, and gives score-2.

update: I recommend acceptance after author rebuttal.

Critical strengths

The paper addresses a central issue in agent evaluation: whether measured improvement reflects learning, search, or repeated task-specific adaptation.

Critical weaknesses
  • The AHE comparison may not represent the original method fairly. AHE is run with only one rollout per task per iteration, although multiple rollouts help its debugger identify partial successes and failures. Exploration is also disabled. These choices may remove mechanisms that are important to AHE’s performance, so the negative result may apply only to this controlled variant.

    • The meaning of pass@1 is unclear. Unit tests are used both to provide feedback and to select successful answers. As a result, the success rate of raw attempts may differ from the success rate of the selected final answer. Because the table reports identical pass@1 and pass@5 values in some cases, it is difficult to determine what the metrics actually measure or whether they support the claim about single-attempt ability.

    • The overfitting claim is stronger than the reported evidence. Held-out performance remains flat, but the paper does not report the corresponding training improvement from the same split and run. Without that comparison, the result could indicate severe overfitting, weak learning overall, or both.

    • The ARC cost comparison is incomplete. ARC limits environment actions, but internal reasoning and harness-tool calls do not consume that budget. The reported improvement therefore establishes better action efficiency, but not necessarily better total inference efficiency.

    • The game results have narrower scope than the broader conclusions suggest. The game experiments adapt a harness separately within each game. They demonstrate task-specific adaptation, but they do not test whether one learned harness transfers across games or explain why cross-task transfer fails in Terminal-Bench.

Questions for authors

Clarifications on Table-2 metrics are recommended. why parallel sampling pass@1=pass@5

4 comments
AuthorsOpenReview ↗

response (1/2)

Dear Reviewer xykk,

Thank you for your detailed and careful review. We appreciate your recognition that our work offers a useful framing for the field, making a common source of overclaiming more visible. We have run additional experiments to make our conclusions stronger and more thorough, and we address each of your concerns below.


W1: The AHE comparison may not represent the original method fairly. AHE is run with only one rollout per task per iteration, although multiple rollouts help its debugger identify partial successes and failures. Exploration is also disabled.

Rollouts per task. We set m=1 in the submission so that all methods use the same rollout budget. To check that this does not handicap AHE, we reran it with m=2 rollouts per task per iteration, which is the configuration in the original AHE paper. Since this doubles AHE's budget to 10 rollouts per task, we compare it with the test-time scaling baselines at K=10 (Claude Opus 4.6).

MethodWithout unit tests: pass@1With unit tests: pass@1With unit tests: pass@10
Parallel Sampling (K=10)73.688.788.7
Sequential Refinement (K=10)71.689.693.1
AHE* (m=2)68.672.894.4

*Run on 87 of the 89 Terminal-Bench 2.1 tasks; qemu-startup and qemu-alpine-ssh can no longer be built as E2B sandboxes because the Debian 11 package repositories were retired upstream.

With its original number of rollouts, AHE still does not outperform test-time scaling in single-attempt performance: its pass@1 is below both baselines without unit tests, and about 16 points below them with unit tests. Its best-of-10 score becomes comparable to Sequential Refinement, which matches the pattern discussed in Section 4.3, where the benefit appears only when selecting among multiple attempts and not as a better harness. The same holds for generalization. With m=2, the evolved harness shows no change in pass@1 on held-out tasks relative to the initial harness (+0.0).

Disabling the explore agent. This was a deliberate choice, and it is central to what we want to measure. As explained in Appendix A.1, the explore agent retrieves harnesses from external sources that are already tuned for the benchmark under evaluation. It imports benchmark-specific solutions instead of discovering improvements from feedback, so keeping it would conflate harness evolution with retrieval of harnesses already fitted to the evaluation tasks, which is the very train-test leakage our paper sets out to examine. Reviewer jVB7 mentioned the same point in their review, noting that "disabling the AHE explore agent is the right call" because retrieval of benchmark-fitted harnesses would otherwise contaminate the comparison.

W2/Q1: The meaning of pass@1 is unclear. Because the table reports identical pass@1 and pass@5 values in some cases, it is difficult to determine what the metrics actually measure or whether they support the claim about single-attempt ability.

Thank you for raising this. The definitions are given in Appendix A.1 (Evaluation Metrics): pass@1 is the average success rate of a method's returned trajectory, and pass@k is the fraction of tasks solved by at least one of the k trajectories the method produces. In other words, pass@1 measures the final answer a method commits to after spending its budget K, and pass@5 measures whether any of its five attempts succeeds.

Why pass@1 equals pass@5 for Parallel Sampling. Parallel Sampling independently generates K=5 candidate trajectories with a fixed harness. When unit test cases are available, it uses them to select its final answer, returning a candidate that passes the tests whenever one exists (Section 3.2). Its final answer is therefore correct exactly when at least one of the five candidates is correct, which is the definition of pass@5. The two metrics coincide by construction for this method in this setting.

W3: Held-out performance remains flat, but the paper does not report the corresponding training improvement from the same split and run. Without that comparison, the result could indicate severe overfitting, weak learning overall, or both.

This is an interesting point. The table below reports pass@1 on the evolution (training) tasks and on the held-out test tasks, from the same split and the same runs as Table 3.

MethodClaude Opus 4.6: trainClaude Opus 4.6: testGPT-5.4: trainGPT-5.4: testGPT-5.4 mini: trainGPT-5.4 mini: test
Direct sampling using initial harness70.063.370.072.154.554.4
Harness Evolution77.8 (+7.8)64.5 (+1.2)77.8 (+7.8)72.1 (+0.0)67.7 (+13.2)53.0 (-1.4)
AuthorsOpenReview ↗

response (2/2)

The evolved harness does learn on the tasks it is optimized on: it improves by 7.8 to 13.2 points on the training tasks for all three models. None of this improvement carries over to the held-out tasks, where the change is between -1.4 and +1.2 points. This rules out the weak-learning explanation and supports overfitting: the harness edits capture something real about the training tasks, but what they capture is specific to those tasks. The pattern is most pronounced for the weakest model, GPT-5.4 mini, which shows the largest training gain and the only drop on the test set. It also agrees with our qualitative analysis in Appendix A.2, where most edits encode task-specific fixes instead of general strategies.

W4: The ARC cost comparison is incomplete.

We now report the full API cost (as a proxy for token usage, covering all internal reasoning and harness-tool calls) and the wall-clock time of both conditions with GPT-5.6 Terra, averaged per game over the 25 public games, for three seeds.

Seed[VISTA*] RHAE[full harness] RHAE[VISTA*] cost[full harness] cost[VISTA*] time[full harness] time
Seed 121.138.2$35.37$29.353h46m2h56m
Seed 218.141.2$36.89$30.443h15m2h27m
Seed 319.441.2$35.24$29.283h14m2h27m
Mean19.540.235.8329.693h25m2h37m

Harness Scaling [full harness] is better in total inference efficiency as well as in action efficiency. In every seed it costs less (about 17% on average) and finishes sooner (about 24% on average) than Sequential Refinement [VISTA*], while roughly doubling RHAE. Writing tools adds some cost up front, but this is outweighed by the savings later, since the agent solves levels in fewer actions and often finishes before exhausting the action budget.

To compare at matched total cost directly, we also report the average RHAE reached within each time or cost limit:

Time budget[VISTA*][full harness]
30 min12.121.4
60 min16.932.3
120 min18.637.9
Final19.540.2
Cost budget[VISTA*][full harness]
$512.018.3
$1016.330.5
$2019.037.7
Final19.540.2

Harness Scaling is ahead at every matched time and cost budget. We have added these results, together with per-game cost and time, to the paper.

W5: The game experiments adapt a harness separately within each game. They demonstrate task-specific adaptation, but they do not test whether one learned harness transfers across games or explain why cross-task transfer fails in Terminal-Bench.

We agree with this characterization, and it is the claim we intend to make. By task design, the goal in these benchmarks is to make use of the rich environmental feedback and optimize performance on each individual task, not to find a single harness that transfers. ARC-AGI-3 is built to measure adaptation to a novel environment: each game has its own rules and goals, the agent is given no instructions, and it must discover them through interaction. EdgeBench likewise measures how far an agent can improve on one task over a long run. Adapting within the task is therefore the capability these benchmarks set out to evaluate, and the online feedback is part of the task itself, so the train-test separation concern we raise for Terminal-Bench does not arise in the same form. Our comparison there is between two task-level methods under the same budget and the same feedback (Sequential Refinement and Harness Scaling), which is why we do not include cross-task Harness Evolution in Section 5.

On Terminal-Bench, our analysis in Appendix A.2 shows that most edits record fixes for particular tasks (known bugs, file paths, command sequences) and do not distill general strategies, and the new training-set results in our response to W3 confirm this: the evolved harness gains 7.8 to 13.2 points on the tasks it was evolved on and nothing on held-out tasks. On the games, the artifacts that matter are equally specific, for example a glyph-matching tool for tr87 or a permutation solver for lp85, and they would be of no use in another game. Harness updates thus appear to be valuable as a way of adapting to a specific task when there is headroom and rich feedback, and much less so as a way of learning capabilities that transfer across tasks.


We have incorporated all the changes into the revised manuscript (highlighted in red in the PDF). If anything remains unclear or if you have additional questions, please don't hesitate to reach out. We look forward to hearing from you. Thank you!

I appreciate the rebuttal and have no further questions. I would raise the score, but I still expect the authors to clarify the below question, is it my misread or it is true ambiguity?

Consider a single task with five sampled trajectories whose outcomes are [failure, failure, success, failure, failure]. Unit tests identify the successful trajectory, which the method returns.

Under Appendix A.1, pass@1 is the average reward over all rollouts, so this example should yield pass@1 = 20% and pass@5 = 100%. Under your response, pass@1 measures the success of the selected final trajectory, giving pass@1 = 100% instead. Which values should this example receive under the paper’s evaluation protocol?

The ambiguity seems to concern about whether “pass@1” evaluates an individual sampled trajectory or the final output of the entire five-attempt generation-and-selection procedure.

Thank you for reading our rebuttal and for the follow-up.

Your example. With a single task and unit test cases available, this example receives pass@1 = 100% and pass@5 = 100%. pass@1 evaluates the final output of the entire five-attempt generation-and-selection procedure, not an individual sampled trajectory. In Appendix A.1, "the average reward over all rollouts" should read "the average reward of the final output over all tasks". We have updated the metric definitions in Appendix A.1 and the method descriptions in Section 3 to make this clear. Thank you for pointing this out!

How pass@1 and pass@5 differ when unit test cases are available. Consider two tasks and a method that runs five iterations (Sequential Refinement, Harness Evolution or Harness Scaling), with outcomes

  • Task 1: [success, failure, failure, success, success]
  • Task 2: [failure, success, failure, failure, failure]

pass@1 scores the final outcome returned by the algorithm. These algorithms return the result of a single iteration, the same one for all tasks, so we take the iteration with the best overall performance. The per-iteration success rates are 50%, 50%, 0%, 50% and 50%, so pass@1 = 50%.

pass@5 no longer depends on which iteration a result comes from: a task counts as solved if it succeeds at least once. Both tasks do, so pass@5 = 100%.

Why the two coincide for Parallel Sampling. Parallel Sampling has no iterations. It draws five independent rollouts per task and returns one final rollout per task, selected with the unit test cases. Its final output for a task is therefore correct exactly when at least one of the five rollouts is correct, which makes pass@1 and pass@5 identical by construction.

*side note: We agree that using unit test cases on the same tasks for both search and evaluation is not a natural setting. In practice one would have either (a) the same tasks for search and evaluation but no unit test cases, as in Section 4.2, or (b) unit test cases on a search set that is disjoint from the evaluation set, as in Section 4.4. We include this setting because it follows the protocol of existing harness evolution work, which is the protocol our paper sets out to examine. In those algorithms, what is returned is an iteration, that is, a single evolved harness that is then applied to all tasks, and not a separate choice for each task. We therefore score the iterative methods in the same way, by the best single iteration.

We hope this clarifies the metrics, and we are happy to answer any further questions.

Reviewer_4pipOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies two key gaps in evaluating harness evolution: the lack of comparisons with test-time scaling under matched computational budgets, and the use of the same tasks for both harness search and evaluation. To address these limitations, it establishes a unified budget framework for comparing four approaches: parallel sampling, sequential refinement, harness evolution (AHE, with its exploration agent disabled), and the harness scaling. On Terminal-Bench 2.1, the automatic harness does not outperform the test-time scaling methods. On long-horizon games tasks like ARC-AGI-3, the automatic harness evolution outperforms the baseline methods.

Critical strengths

The work is clear and the findings are meaningful to the harness research direction. The analysis is in-depth, for example, The ARC-AGI-3 ablation (VISTA* → VISTA → full) separates how much of the gain comes from tools and how much from memory.

Critical weaknesses

The comparison is a bit simplified, as the evaluation methods only considers the four categories, like parallel sampling, sequential refinement, harness evolution. There should be different methods under each category, and it will be interesting to see more comparison results from each category.

Questions for authors

Please add confidence intervals or more seeds, plus at least one more evolution method (e.g., Meta-Harness).

4 comments

Dear Reviewer 4pip,

Thank you for your feedback. We appreciate your recognition that the work is clear, that the findings are meaningful for harness research, and that the analysis is in-depth. We address your remaining questions below.


Q1: confidence interval or more seed

We added seeds on both benchmarks. Rows and tables marked with * are rebuttal experiments run on 87 of the 89 Terminal-Bench 2.1 tasks; the two excluded tasks (qemu-startup, qemu-alpine-ssh) can no longer be built as E2B sandboxes because the Debian 11 package repositories were retired upstream, which is independent of any method we compare.

Terminal-Bench 2.1, new seed (Claude Opus 4.6, 87 tasks*).

Joint split, without unit test cases (reproduces Table 1):

Methodpass@1
Direct sampling using initial harness72.4
Parallel Sampling83.6
Sequential Refinement81.2
Harness Evolution71.6
Harness Scaling78.8

Joint split, with unit test cases (reproduces Table 2):

Methodpass@1pass@5
Direct sampling using initial harness70.8-
Parallel Sampling89.589.5
Sequential Refinement94.396.7
Harness Evolution75.290.7
Harness Scaling80.096.7

Disjoint split (reproduces Table 3):

MethodInitial harnessEvolved harness
Harness Evolution62.561.0 (-1.5)

As in the submission, neither Harness Evolution nor Harness Scaling outperforms the best test-time scaling baseline on any metric, and the evolved harness shows no gain on held-out tasks.

ARC-AGI-3, three seeds (GPT-5.6 Terra, average RHAE over all 25 public games).

Seed 1Seed 2Seed 3Mean ± std95% CI
Sequential Refinement [VISTA*]21.118.119.419.5 ± 1.5[15.7, 23.3]
Harness Scaling [full harness]38.241.241.240.2 ± 1.7[35.9, 44.5]

The ranking is the same in every seed, the run-to-run standard deviation is below 2 points against a gap of about 21 points, and the 95% confidence intervals (t-distribution, n=3) do not overlap. Our conclusions on both benchmarks therefore hold under the additional seeds.

Q1: at least one more evolution method (e.g., Meta-Harness)

We ran Meta-Harness, as suggested, and also GEPA as a representative of the prompt-optimizer family, with Claude Opus 4.6 under the same rollout budget K.

Joint search and evaluation set, with unit test cases:

Methodpass@1pass@5
Parallel Sampling84.884.8
Sequential Refinement83.190.4
AHE73.083.2
Meta-Harness*72.889.6
GEPA*77.686.0

Disjoint search and evaluation sets (pass@1 on held-out tasks):

MethodInitial harnessEvolved harness
AHE63.364.5 (+1.2)
Meta-Harness*62.557.8 (-4.7)
GEPA*62.561.4 (-1.1)

The results agree with our findings for AHE. None of the three harness evolution methods exceeds the best test-time scaling baseline on either metric, and none yields a meaningful gain on held-out tasks.


We have incorporated all the changes into the revised manuscript (highlighted in red in the PDF). Thank you so much again and we welcome any further questions.

AuthorsOpenReview ↗

we are looking forward to your response

Dear Reviewer 4pip,

Thank you so much for being our reviewer! We have expanded our results with multi-seed experiments and two additional evolution methods (Meta-Harness and GEPA). We have also revised the manuscript to include ablation studies and additional experiments. We hope these updates further strengthen our conclusions, and we very much look forward to your reply!

Best,

Authors

Thanks for answering the questions, and I am good with the replies.

Dear Reviewer 4pip,

Thank you very much for your response and for taking the time to go through our replies. We are glad to hear that our answers have addressed your questions.

If you feel that your concerns have been resolved, we would be sincerely grateful if you could consider raising your score accordingly. If anything remains unclear, we would be happy to clarify further before the discussion period ends.

Thank you again for your valuable feedback, which has helped us improve the paper.

Best regards,

The Authors

Reviewer_BUEFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether automatically improving an agent’s harness, including its prompts, tools, memory, and execution logic, offers benefits beyond simply making more attempts. On Terminal-Bench, harness evolution generally underperforms simpler search methods and shows no average improvement on held-out tasks. However, task-specific harness adaptation delivers substantial gains on long-horizon game benchmarks. The findings emphasize the need for fair budget comparisons and separate optimization and evaluation tasks, while suggesting that harness adaptation is particularly useful in unfamiliar environments with rich feedback.

Critical strengths
  1. The paper provides a detailed investigation of when harness scaling can help, along with case studies to examine its benefits and limitations.
  2. The paper identifies a promising direction for harness evolution: adapting tools, memory, and workflows to unfamiliar, long-horizon tasks with rich feedback.
Critical weaknesses
  1. The generalization experiment tests one AHE-based method with five optimization rounds and a single split of Terminal-Bench. This supports a conclusion about the tested setting, but broader claims would require comparisons across different methods, budgets, and data splits.
  2. The budget comparisons do not capture all computational costs. Terminal-Bench excludes the cost of editing the harness, while ARC-AGI-3 limits game actions without matching the computation spent on reasoning and creating or running tools. Using fewer game actions therefore does not necessarily mean the approach is cheaper overall.
  3. The paper provides useful examples, but it still remains unclear which changes drive the improvements. Skills, tools, and hooks are enabled together in the full-harness experiments. I think giving the baseline the same information would help explain whether the gains come from changes to the harness or from additional experience and computation.
Questions for authors

None

6 comments
AuthorsOpenReview ↗

response (1/3)

Dear Reviewer BUEF,

Thank you for your thoughtful review. We are glad you found our investigation of when harness scaling helps and the accompanying case studies useful, and that you see adapting the harness to long-horizon tasks as a promising direction. We address your questions below.


W1: broader claims would require comparisons across different methods, budgets, and data splits

We ran additional experiments under three settings with Claude Opus 4.6. Rows marked with * are rebuttal experiments run on 87 of the 89 Terminal-Bench 2.1 tasks; the two excluded tasks (qemu-startup, qemu-alpine-ssh) can no longer be built as E2B sandboxes because the Debian 11 package repositories were retired upstream.

Different methods. In addition to AHE, we ran Meta-Harness and GEPA (as a representative of the prompt-optimizer family) under the same rollout budget K.

Joint search and evaluation set, with unit test cases:

Methodpass@1pass@5
Parallel Sampling84.884.8
Sequential Refinement83.190.4
AHE73.083.2
Meta-Harness*72.889.6
GEPA*77.686.0

Disjoint search and evaluation sets (pass@1 on held-out tasks):

MethodInitial harnessEvolved harness
AHE63.364.5 (+1.2)
Meta-Harness*62.557.8 (-4.7)
GEPA*62.561.4 (-1.1)

None of the three methods exceeds the best test-time scaling baseline on either metric, and none yields a meaningful gain on held-out tasks.

Different budgets. We doubled the budget in two ways. First, we increased the number of rounds from K=5 to K=10 for all methods.

Without unit test cases (pass@1):

MethodK=5K=10
Direct sampling using initial harness69.969.5
Parallel Sampling74.773.6
Sequential Refinement73.071.6
Harness Evolution71.471.9
Harness Scaling76.073.6

With unit test cases:

MethodK=5 pass@1K=5 pass@5K=10 pass@1K=10 pass@10
Direct sampling using initial harness69.9-68.5-
Parallel Sampling84.884.888.788.7
Sequential Refinement83.190.489.693.1
Harness Evolution73.083.273.686.5
Harness Scaling83.189.990.594.4

Disjoint split (pass@1 on held-out tasks):

MethodK=5K=10
Direct sampling using initial harness63.363.3
Harness Evolution64.5 (+1.2)63.0 (-0.3)

Second, we kept five rounds but sampled m=2 rollouts per task in each round, which is the original AHE configuration, and compared it with the test-time scaling baselines at the same total of 10 rollouts.

Joint search and evaluation set:

MethodWithout unit tests: pass@1With unit tests: pass@1With unit tests: pass@10
Parallel Sampling (K=10)73.688.788.7
Sequential Refinement (K=10)71.689.693.1
AHE* (m=2)68.672.894.4

Disjoint search and evaluation sets (change in pass@1 on held-out tasks, evolved harness versus initial harness):

MethodChange on held-out tasks
AHE* (m=2)+0.0

With either form of doubled budget, Harness Evolution stays well below the test-time scaling baselines on pass@1, and it shows no gain on held-out tasks: the small gain at K=5 disappears at K=10, and with m=2 the evolved harness scores exactly the same as the initial one. With m=2 its best-of-10 score becomes comparable to Sequential Refinement, while its pass@1 remains about 16 points lower. This matches the pattern discussed in Section 4.3: the benefit appears only when selecting among multiple attempts, not as a better harness.

Different data splits. We repeated the generalization experiment on a newly shuffled split with the same proportions.

SplitInitial harnessEvolved harness
Split 1 (original)63.364.5 (+1.2)
Split 2* (reshuffled)72.772.7 (+0.0)

The evolved harness again shows no gain on held-out tasks under the new split. We add all of these results to the paper.

AuthorsOpenReview ↗

response (2/3)

W2: Terminal-Bench excludes the cost of editing the harness, while ARC-AGI-3 limits game actions without matching the computation spent on reasoning and creating or running tools.

We now report the full API cost of each method (as a proxy for token usage), which includes all reasoning, harness editing, and tool writing and execution, together with wall-clock time.

Terminal-Bench 2.1 (Claude Opus 4.6, 87 tasks*, K=5). Wall-clock time is listed for reference, although it mainly reflects the degree of parallelism in our implementation (5 tasks in parallel for Parallel Sampling, 3 for Sequential Refinement and Harness Scaling, and 10 for the evaluation phase of Harness Evolution, whose trace-analysis phase is sequential).

MethodAPI costWall-clock time
Parallel Sampling$413376 min
Sequential Refinement$3931,773 min
Harness Evolution$5581,272 min
Harness Scaling$7242,152 min

Once harness editing is counted, the two harness methods cost 35% to 84% more than the test-time scaling baselines at the same rollout count. Including this cost therefore strengthens our conclusion that harness evolution does not outperform test-time scaling on Terminal-Bench.

ARC-AGI-3 (GPT-5.6 Terra, average per game over the 25 public games, three seeds).

Seed[VISTA*] cost[full harness] cost[VISTA*] time[full harness] time
Seed 1$35.37$29.353h46m2h56m
Seed 2$36.89$30.443h15m2h27m
Seed 3$35.24$29.283h14m2h27m
Mean35.8329.693h25m2h37m

Harness Scaling [full harness] is cheaper overall, not only in game actions. It costs about 17% less and finishes about 24% sooner than Sequential Refinement [VISTA*] in every seed, while roughly doubling RHAE (40.2 versus 19.5 on average over the three seeds). Writing tools adds some cost up front, but this is outweighed by the savings later, since the agent solves levels in fewer actions and often finishes before exhausting the action budget.

We also compare the two methods at matched budgets, reporting the average RHAE reached within each time or cost limit:

Time budget[VISTA*][full harness]
30 min12.121.4
60 min16.932.3
120 min18.637.9
Final19.540.2
Cost budget[VISTA*][full harness]
$512.018.3
$1016.330.5
$2019.037.7
Final19.540.2

Harness Scaling is ahead at every matched time and cost budget, so its advantage on ARC-AGI-3 does not come from additional computation. We add these results, together with per-game cost and time, to the paper.

AuthorsOpenReview ↗

response (3/3)

W3: it still remains unclear which changes drive the improvements. Skills, tools, and hooks are enabled together in the full-harness experiments. I think giving the baseline the same information would help explain whether the gains come from changes to the harness or from additional experience and computation.

Which components drive the improvement? We ran an additional tools-only ablation with GPT-5.6 Terra on 10 games, which together with the memory-only condition from the paper separates the components.

GameFixed harness [VISTA*]Memory only [VISTA]Tools onlyFull harness
ar2525.389.8100100
bp351.81.70.54.9
cd8241.765.595.186.6
cn0435.945.45.945.2
dc2225.130.617.136.5
ft0968.590.679.765.2
g50t1.01.88.310.1
ka5942.721.542.457.7
lf524.18.023.718.3
lp856.99.076.247.5
Average25.336.444.947.2

Tools are the main driver. Evolving tools alone recovers most of the gain of the full harness (44.9 versus 47.2), while evolving memory alone gives a smaller improvement (36.4). This agrees with the usage statistics in Table 24: with GPT-5.6 Terra, tool operations account for 217 of the 260 harness operations per game, whereas hooks are almost never written and skills are written only 3.8 times per game.

Harness changes versus additional experience and computation. On computation, as shown in our response to W2, the full harness condition uses about 17% less API cost and 24% less wall-clock time than the fixed harness, and it is ahead at every matched time and cost budget. On experience, all conditions have access to the same environment-feedback mechanism under the same 2,000-action budget, although their interaction trajectories can differ. The memory-only condition allows the agent to retain observations and learned information in prose while keeping its tools fixed. Its lower performance (36.4 versus 44.9 for tools only and 47.2 for the full harness) suggests that tool evolution provides benefits beyond retaining experience in textual memory. The tr87 case study in Appendix B.4 illustrates this mechanism: with memory only, the agent spends 1,320 actions on level 2 without solving it, whereas with tool writing, it constructs perception tools that address the underlying misperception and clears the same level in 32 actions.


We have incorporated the changes into the revised manuscript (highlighted in red in the PDF). We thank the reviewer for providing valuable comments on our paper. In light of these responses, we hope we have addressed your questions and sincerely hope you consider raising your score. Many thanks for your time; we are extremely grateful!

Reviewer_jVB7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits how automatic harness evolution for LLM agents is evaluated and raises two protocol problems. Harness Evolution is a recent new topic without too many prior works. Some prior works does not compare harness evolution with simple task level search baselines under matched feedback and inference budgets. Authors formalize four methods under one budget view, including parallel sampling, sequential refinement, harness evolution across a task batch, and harness scaling that adapts the harness per instance, and they separate what each method updates from what feedback it sees. Authors bring a instresting evaluation pipeline which may be fair for harness evolution.

Critical strengths
  1. The budget view that separates what is updated from what feedback is observed. This turns a vague comparison into a controlled one and makes the point that gains attributed to harness evolution may come from repeated sampling. The Terminal-Bench results under this protocol are a useful corrective, and disabling the AHE explore agent is the right call, since otherwise retrieval of benchmark fitted harnesses contaminates the comparison.

  2. The positive regime analysis and the mechanism study. Counting harness operations and showing that tool writes are amortized over many run_tool calls, together with the tr87 and lp85 and ls20 case studies, explains why full harness adaptation helps in a way that a leaderboard number cannot. This is the part of the paper that gives the reader something to build on.

Critical weaknesses
  1. The critique is aimed at a class of methods, but the evidence comes from one of them. The paper names Meta-Harness, AHE and AEVO, and only AHE is run, with the explore agent removed. A reader cannot tell whether the outcome is a property of harness evolution or a property of this particular implementation at K=5. Related but narrower optimizers are also left out. DSPy, GEPA and TextGrad optimize prompts and programs from execution feedback and are in effect harness evolution over a smaller surface, and they are cited but never compared. Finally, the strongest form of the paper's own critique is quantitative and is not attempted. If same task search inflates reported gains, the authors could re-analyze the numbers reported by prior work, or reproduce one setup twice, once with the split used there and once with a disjoint split, and report the difference.

  2. The experimental scale is not strong enough for the claims. Terminal-Bench 2.1 has 89 tasks, and every configuration is run twice with no interval reported, so one to three point differences are not resolved. As reported by terminal bench2.1 leaderboard, the accuracy of this variance is not ignorable. ARC-AGI-3 covers 25 games but each configuration is run once, and the per game scores swing by tens of points. EdgeBench covers 5 tasks with 2 runs. There is no strong human engineered harness as a reference point on Terminal-Bench, so the negative result may partly reflect that the seed harness is a single bash tool. I suggest to give more repeat for each benchmark for a more stable evaluation. Besides, more advanced benchmarks such as TerminalBench4.0 can also make this paper better.

Questions for authors
  1. Can you report the total token and wall clock cost of each method, and repeat the main comparisons at matched cost rather than matched rollout count? Especially the costs for strong but expensive models such as GPT-family and Opus-Family.

Update: 20206/11/16. Authors do not update their rebuttal so far, but I still believe this paper is up the acceptance line.

6 comments
AuthorsOpenReview ↗

response (1/4)

Dear Reviewer jVB7,

Thank you for your thorough and constructive review. We are glad you found that our protocol turns a vague comparison into a controlled one, that the Terminal-Bench results support the point that gains attributed to harness evolution may come from repeated sampling, and that the long-horizon game analysis and mechanism study give readers something to build on. We address your questions below.


W1: The paper names Meta-Harness, AHE and AEVO, and only AHE is run. DSPy, GEPA and TextGrad are cited but never compared.

Thank you for raising this. These methods share the same iterative search loop: sample trajectories, evaluate them, revise the harness, and resample. Our goal was to test the strongest harness evolution method, and we selected AHE because we consider it the strongest by design: it exposes a clear interface over the editable harness components, adds a dedicated trajectory-diagnosis step before each edit, and pairs every edit with a self-declared prediction that the next round verifies.

To test whether our conclusions depend on this choice, we additionally ran Meta-Harness and GEPA (as a representative of the prompt-optimizer family) with Claude Opus 4.6 under the same rollout budget K.

Joint search and evaluation set, with unit test cases

Methodpass@1pass@5
Parallel Sampling84.884.8
Sequential Refinement83.190.4
AHE73.083.2
Meta-Harness*72.889.6
GEPA*77.686.0

Disjoint search and evaluation sets (pass@1 on held-out tasks)

MethodInitial harnessEvolved harness
AHE63.364.5 (+1.2)
Meta-Harness*62.557.8 (-4.7)
GEPA*62.561.4 (-1.1)

Note on rebuttal experiments: all new results are reported on 87 of the 89 Terminal-Bench 2.1 tasks. We exclude qemu-startup and qemu-alpine-ssh because their official Debian 11 images can no longer be built as E2B sandboxes following the upstream retirement of the Debian 11 package repositories. This is an infrastructure issue independent of any method we compare.

The new results are consistent with our original findings. On the joint set, none of the three harness evolution methods exceeds the best test-time scaling baseline on either metric. On held-out tasks, none yields a meaningful gain over the initial harness, and AHE is the only one that does not degrade. We have added the Meta-Harness and GEPA results to the revised paper.

W1: the authors could reproduce one setup twice, once with the split used there and once with a disjoint split, and report the difference.

We agree this is the most direct test, and our paper already contains this comparison, following the AHE setup. Section 4.3 is the joint-split experiment, where the same tasks are used for both search and evaluation, as in prior work. Section 4.4 is the disjoint-split experiment, where search, validation, and evaluation tasks are separated. The table below reports the gain of Harness Evolution over direct sampling with the initial harness (pass@1) under each split.

SplitMetricClaude Opus 4.6GPT-5.4GPT-5.4 mini
Joint (Table 2)pass@1+3.1+2.7+0.5
Joint (Table 2)pass@5+13.3+13.4+7.7
Disjoint (Table 3)pass@1+1.2+0.0-1.4

The gap between the joint and disjoint rows is the portion of the reported gain that does not transfer to unseen tasks. Under the joint split, Harness Evolution appears to improve over the initial harness for all three models, most visibly on pass@5, which is the setting closest to how prior work reports results. Under the disjoint split, the gain shrinks to at most +1.2 and is zero on average.

W2: The experimental scale is thin. There is no strong human engineered harness as a reference point on Terminal-Bench.

We agree that more runs strengthen the claims. We reran all three Terminal-Bench 2.1 settings with Claude Opus 4.6 using a new seed, and reran all 25 public ARC-AGI-3 games with GPT-5.6 Terra using two additional seeds. All of our conclusions hold under the new seeds.

Terminal-Bench 2.1, new seed (Claude Opus 4.6, 87 tasks*)

Joint split, without unit test cases (reproduces Table 1):

Methodpass@1
Direct sampling using initial harness72.4
Parallel Sampling83.6
Sequential Refinement81.2
Harness Evolution71.6
Harness Scaling78.8

Joint split, with unit test cases (reproduces Table 2):

Methodpass@1pass@5
Direct sampling using initial harness70.8-
Parallel Sampling89.589.5
Sequential Refinement94.396.7
Harness Evolution75.290.7
Harness Scaling80.096.7

Disjoint split (reproduces Table 3):

MethodInitial harnessEvolved harness
Harness Evolution62.561.0 (-1.5)
AuthorsOpenReview ↗

response (2/4)

As in the submission, neither Harness Evolution nor Harness Scaling outperforms the best test-time scaling baseline on any metric, and the evolved harness again shows no gain on held-out tasks.

Human-engineered reference on Terminal-Bench. For reference, the official Terminal-Bench 2.1 leaderboard reports the following results for vendor-engineered harnesses:

ModelHarnessScore
Claude Opus 4.6Claude Code70.1
GPT-5.4Codex CLI77.3
GPT-5.4 miniCodex CLI66.1

We have added these reference points to the paper.

ARC-AGI-3, three seeds (GPT-5.6 Terra, average RHAE over 25 public games)

SeedSequential Refinement [VISTA*]Harness Scaling [full harness]
Seed 1 (Table 4)21.138.2
Seed 218.141.2
Seed 319.441.2
Mean ± std19.5 ± 1.540.2 ± 1.7

Although individual games vary considerably between seeds, the average is stable: the run-to-run standard deviation is below 2 points, against a gap of more than 20 points. Full-harness evolution roughly doubles the average RHAE in every seed, and it scores higher than [VISTA*] on 22 of 25 games in seed 2 and 21 of 25 in seed 3. Per-game results for the new seeds are below.

GameSeed 2 [VISTA*]Seed 2 [full harness]Seed 3 [VISTA*]Seed 3 [full harness]
ar2517.510029.3100
bp351.56.81.41.2
cd8234.866.451.0100
cn0440.240.328.747.6
dc2230.242.410.938.9
ft0977.023.445.485.0
g50t0.117.60.310.0
ka5932.672.775.054.5
lf522.124.51.818.9
lp857.147.48.056.3
ls202.67.90.94.7
m0r08.180.014.377.0
r11l4.826.62.710.0
re866.985.958.377.8
s5i50.729.60.525.2
sb2698.347.484.686.0
sc2520.828.615.128.6
sk480.99.816.72.8
sp804.839.50.637.6
su1543.342.85.827.1
tn368.620.322.222.7
tr874.01004.892.1
tu934.213.47.024.3
vc330.049.10.40.0
wa300.48.10.00.7
Average18.141.219.441.2

We have added all of these results to the revised paper.

W2/Q1: There is no ablation over K, over the number of evolution rounds, or over the meta agent model, and no cost reporting.

Ablation over K and the number of evolution rounds. In our formulation these are the same quantity: K is the compute budget, which for Harness Evolution equals the number of evolution rounds. We doubled the budget from K=5 to K=10 on Terminal-Bench 2.1 with Claude Opus 4.6, the model on which Harness Evolution performs best, averaging over two independent runs.

Without unit test cases (pass@1):

MethodK=5K=10
Direct sampling using initial harness69.969.5
Parallel Sampling74.773.6
Sequential Refinement73.071.6
Harness Evolution71.471.9
Harness Scaling76.073.6

With unit test cases:

MethodK=5 pass@1K=5 pass@5K=10 pass@1K=10 pass@10
Direct sampling using initial harness69.9-68.5-
Parallel Sampling84.884.888.788.7
Sequential Refinement83.190.489.693.1
Harness Evolution73.083.273.686.5
Harness Scaling83.189.990.594.4

Disjoint split (pass@1 on held-out tasks):

MethodK=5K=10
Direct sampling using initial harness63.363.3
Harness Evolution64.5 (+1.2)63.0 (-0.3)

Doubling the budget does not change the picture. Harness Evolution remains well below both test-time scaling baselines when unit tests are available, and its small held-out gain at K=5 disappears at K=10. Harness Scaling becomes comparable to Sequential Refinement at K=10, with differences of about one point that are within run-to-run variance.

Meta agent model. We considered this when designing the experiments and deliberately use the same model for the task agent and the meta agent, because this is the realistic setting: if a stronger model is available, a practitioner would use it for both roles rather than only for the meta agent.

Cost on Terminal-Bench 2.1. We report the total API cost, as a proxy for token usage, for each method with Claude Opus 4.6 (87 tasks*, K=5). We also list wall-clock time for reference, although it mainly reflects the degree of parallelism in our implementation (5 tasks in parallel for Parallel Sampling, 3 for Sequential Refinement and Harness Scaling, and 10 for the evaluation phase of Harness Evolution, whose trace-analysis phase is sequential).

AuthorsOpenReview ↗

response (3/4)

MethodAPI costWall-clock time
Parallel Sampling$413376 min
Sequential Refinement$3931,773 min
Harness Evolution$5581,272 min
Harness Scaling$7242,152 min

At a matched rollout count, the two harness methods cost 35% to 84% more than the test-time scaling baselines, so matching on cost instead of rollouts would further favor test-time scaling.

Cost on ARC-AGI-3. We report the average API cost and wall-clock time per game with GPT-5.6 Terra over the 25 public games, for each of the three seeds.

Seed[VISTA*] cost[full harness] cost[VISTA*] time[full harness] time
Seed 1$35.37$29.353h46m2h56m
Seed 2$36.89$30.443h15m2h27m
Seed 3$35.24$29.283h14m2h27m
Mean35.8329.693h25m2h37m

Full-harness evolution is cheaper and faster than the fixed-harness baseline in every seed, by about 17% in cost and 24% in time on average, while roughly doubling RHAE. The cost of writing tools is more than recovered because the agent solves levels in fewer actions and often finishes before exhausting the action budget.

We also compare the two methods at matched budgets, reporting the average RHAE reached within each time or cost limit (averaged over three seeds):

Time budget[VISTA*][full harness]
30 min12.121.4
60 min16.932.3
120 min18.637.9
Final19.540.2
Cost budget[VISTA*][full harness]
$512.018.3
$1016.330.5
$2019.037.7
Final19.540.2

Full-harness evolution is ahead at every matched time and cost budget, so its advantage on ARC-AGI-3 does not come from consuming more compute. We add these results, together with per-game cost and time, to the paper.

W3: In ARC-AGI-3, the two conditions do not consume the same compute. The gain could come from having write_tool and run_tool at all rather than from evolution. The abstract does not make the 80% figure clear.

Compute. As shown in our response to W2, we now report API cost (as a proxy for token usage) and wall-clock time for both conditions. Harness Scaling [full harness] outperforms Sequential Refinement [VISTA*] while costing about 17% less and finishing about 24% sooner on average, and it is ahead at every matched time and cost budget. Its advantage therefore does not come from consuming more compute.

Interface versus evolution. We would like to clarify that write_tool and run_tool do not ship any tools. Tools are part of the harness, and these operations are only the interface through which the agent modifies it: every tool the agent uses (e.g. frame_diff, match_symbols, the BFS solvers) is written by the agent itself during the run, in response to what it observes in that game. There is also no separate meta agent in this setting. Following VISTA's free update mechanism, the agent decides when to update its own harness. Writing a tool is therefore itself an act of harness evolution.

To separate the contributions further, we ran two ablations with GPT-5.6 Terra on 10 games: one where only the tool interface is available, and one where the full interface is available but the prompt does not instruct the agent to evolve its harness.

Game[VISTA*]Tools onlyFull interface, unpromptedFull interface, prompted [full harness]
ar2525.310094.7100
bp351.80.51.04.9
cd8241.795.183.186.6
cn0435.95.929.545.2
dc2225.117.139.536.5
ft0968.579.710065.2
g50t1.08.38.410.1
ka5942.742.48.757.7
lf524.123.72.518.3
lp856.976.271.247.5
Average25.344.943.947.2

Three observations follow. First, all three conditions in which the agent can modify its harness score far above the fixed harness (43.9 to 47.2 versus 25.3), so the gain comes from evolution during the run. Second, tool evolution alone recovers most of the gain, which agrees with our analysis in Appendix B.3 that tool operations dominate harness usage. Third, the agent makes use of the interface even when it is not prompted to do the evolution. We will state this more precisely in the paper.

The 80% figure. Thank you for pointing this out. The 80% improvement is measured against Sequential Refinement with the same starting harness ([VISTA*]), and we will make this clear in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.