- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether updating an agent’s harness produces reusable capabilities or merely gives the agent more opportunities to solve tasks it has already seen. It compares parallel sampling, sequential refinement, cross-task harness evolution, and task-specific harness updates.
The main finding is that cross-task harness evolution provides little or no improvement on held-out Terminal-Bench tasks, while task-specific updates can improve performance in longer-horizon settings such as ARC-AGI-3 and EdgeBench. The paper argues that harness updates are more useful for adapting to a specific task than for learning capabilities that transfer broadly across tasks.
Overall, I view this paper’s main contribution as an evaluation perspective rather than a new harness-evolution algorithm. It helps readers distinguish genuine transferable capability from gains caused by additional search, task- specific adaptation, or unpriced internal computation, and it argues for matched baselines and held-out tasks when evaluating agent improvement. This is a useful framing for the field because it makes a common source of overclaiming more visible.
At the same time, the paper’s empirical conclusions are relatively narrow and preliminary: the strongest evidence shows limited transfer in the tested Terminal-Bench setting and benefits from task-specific adaptation in selected long- horizon environments. Because the AHE configuration differs from the original method, some metric definitions remain unclear, and the cost accounting is incomplete in parts of the evaluation, the paper should be read as evidence for a conditional and more cautious interpretation of harness evolution. The broader conclusion regarding "harness updates produce transferable capabilities or not" still remains unclear.
Based on current readability, presentation and concerns, I am conservative on recommending for acceptance, and gives score-2.
update: I recommend acceptance after author rebuttal.
Critical strengths
The paper addresses a central issue in agent evaluation: whether measured improvement reflects learning, search, or repeated task-specific adaptation.
Critical weaknesses
-
The AHE comparison may not represent the original method fairly. AHE is run with only one rollout per task per iteration, although multiple rollouts help its debugger identify partial successes and failures. Exploration is also disabled. These choices may remove mechanisms that are important to AHE’s performance, so the negative result may apply only to this controlled variant.
-
The meaning of pass@1 is unclear. Unit tests are used both to provide feedback and to select successful answers. As a result, the success rate of raw attempts may differ from the success rate of the selected final answer. Because the table reports identical pass@1 and pass@5 values in some cases, it is difficult to determine what the metrics actually measure or whether they support the claim about single-attempt ability.
-
The overfitting claim is stronger than the reported evidence. Held-out performance remains flat, but the paper does not report the corresponding training improvement from the same split and run. Without that comparison, the result could indicate severe overfitting, weak learning overall, or both.
-
The ARC cost comparison is incomplete. ARC limits environment actions, but internal reasoning and harness-tool calls do not consume that budget. The reported improvement therefore establishes better action efficiency, but not necessarily better total inference efficiency.
-
The game results have narrower scope than the broader conclusions suggest. The game experiments adapt a harness separately within each game. They demonstrate task-specific adaptation, but they do not test whether one learned harness transfers across games or explain why cross-task transfer fails in Terminal-Bench.
-
Questions for authors
Clarifications on Table-2 metrics are recommended. why parallel sampling pass@1=pass@5
4 comments
response (2/2)
The evolved harness does learn on the tasks it is optimized on: it improves by 7.8 to 13.2 points on the training tasks for all three models. None of this improvement carries over to the held-out tasks, where the change is between -1.4 and +1.2 points. This rules out the weak-learning explanation and supports overfitting: the harness edits capture something real about the training tasks, but what they capture is specific to those tasks. The pattern is most pronounced for the weakest model, GPT-5.4 mini, which shows the largest training gain and the only drop on the test set. It also agrees with our qualitative analysis in Appendix A.2, where most edits encode task-specific fixes instead of general strategies.
W4: The ARC cost comparison is incomplete.
We now report the full API cost (as a proxy for token usage, covering all internal reasoning and harness-tool calls) and the wall-clock time of both conditions with GPT-5.6 Terra, averaged per game over the 25 public games, for three seeds.
| Seed | [VISTA*] RHAE | [full harness] RHAE | [VISTA*] cost | [full harness] cost | [VISTA*] time | [full harness] time |
|---|---|---|---|---|---|---|
| Seed 1 | 21.1 | 38.2 | $35.37 | $29.35 | 3h46m | 2h56m |
| Seed 2 | 18.1 | 41.2 | $36.89 | $30.44 | 3h15m | 2h27m |
| Seed 3 | 19.4 | 41.2 | $35.24 | $29.28 | 3h14m | 2h27m |
| Mean | 19.5 | 40.2 | 35.83 | 29.69 | 3h25m | 2h37m |
Harness Scaling [full harness] is better in total inference efficiency as well as in action efficiency. In every seed it costs less (about 17% on average) and finishes sooner (about 24% on average) than Sequential Refinement [VISTA*], while roughly doubling RHAE. Writing tools adds some cost up front, but this is outweighed by the savings later, since the agent solves levels in fewer actions and often finishes before exhausting the action budget.
To compare at matched total cost directly, we also report the average RHAE reached within each time or cost limit:
| Time budget | [VISTA*] | [full harness] |
|---|---|---|
| 30 min | 12.1 | 21.4 |
| 60 min | 16.9 | 32.3 |
| 120 min | 18.6 | 37.9 |
| Final | 19.5 | 40.2 |
| Cost budget | [VISTA*] | [full harness] |
|---|---|---|
| $5 | 12.0 | 18.3 |
| $10 | 16.3 | 30.5 |
| $20 | 19.0 | 37.7 |
| Final | 19.5 | 40.2 |
Harness Scaling is ahead at every matched time and cost budget. We have added these results, together with per-game cost and time, to the paper.
W5: The game experiments adapt a harness separately within each game. They demonstrate task-specific adaptation, but they do not test whether one learned harness transfers across games or explain why cross-task transfer fails in Terminal-Bench.
We agree with this characterization, and it is the claim we intend to make. By task design, the goal in these benchmarks is to make use of the rich environmental feedback and optimize performance on each individual task, not to find a single harness that transfers. ARC-AGI-3 is built to measure adaptation to a novel environment: each game has its own rules and goals, the agent is given no instructions, and it must discover them through interaction. EdgeBench likewise measures how far an agent can improve on one task over a long run. Adapting within the task is therefore the capability these benchmarks set out to evaluate, and the online feedback is part of the task itself, so the train-test separation concern we raise for Terminal-Bench does not arise in the same form. Our comparison there is between two task-level methods under the same budget and the same feedback (Sequential Refinement and Harness Scaling), which is why we do not include cross-task Harness Evolution in Section 5.
On Terminal-Bench, our analysis in Appendix A.2 shows that most edits record fixes for particular tasks (known bugs, file paths, command sequences) and do not distill general strategies, and the new training-set results in our response to W3 confirm this: the evolved harness gains 7.8 to 13.2 points on the tasks it was evolved on and nothing on held-out tasks. On the games, the artifacts that matter are equally specific, for example a glyph-matching tool for tr87 or a permutation solver for lp85, and they would be of no use in another game. Harness updates thus appear to be valuable as a way of adapting to a specific task when there is headroom and rich feedback, and much less so as a way of learning capabilities that transfer across tasks.
We have incorporated all the changes into the revised manuscript (highlighted in red in the PDF). If anything remains unclear or if you have additional questions, please don't hesitate to reach out. We look forward to hearing from you. Thank you!
I appreciate the rebuttal and have no further questions. I would raise the score, but I still expect the authors to clarify the below question, is it my misread or it is true ambiguity?
Consider a single task with five sampled trajectories whose outcomes are [failure, failure, success, failure, failure]. Unit tests identify the successful trajectory, which the method returns.
Under Appendix A.1, pass@1 is the average reward over all rollouts, so this example should yield pass@1 = 20% and pass@5 = 100%. Under your response, pass@1 measures the success of the selected final trajectory, giving pass@1 = 100% instead. Which values should this example receive under the paper’s evaluation protocol?
The ambiguity seems to concern about whether “pass@1” evaluates an individual sampled trajectory or the final output of the entire five-attempt generation-and-selection procedure.
Thank you for reading our rebuttal and for the follow-up.
Your example. With a single task and unit test cases available, this example receives pass@1 = 100% and pass@5 = 100%. pass@1 evaluates the final output of the entire five-attempt generation-and-selection procedure, not an individual sampled trajectory. In Appendix A.1, "the average reward over all rollouts" should read "the average reward of the final output over all tasks". We have updated the metric definitions in Appendix A.1 and the method descriptions in Section 3 to make this clear. Thank you for pointing this out!
How pass@1 and pass@5 differ when unit test cases are available. Consider two tasks and a method that runs five iterations (Sequential Refinement, Harness Evolution or Harness Scaling), with outcomes
- Task 1: [success, failure, failure, success, success]
- Task 2: [failure, success, failure, failure, failure]
pass@1 scores the final outcome returned by the algorithm. These algorithms return the result of a single iteration, the same one for all tasks, so we take the iteration with the best overall performance. The per-iteration success rates are 50%, 50%, 0%, 50% and 50%, so pass@1 = 50%.
pass@5 no longer depends on which iteration a result comes from: a task counts as solved if it succeeds at least once. Both tasks do, so pass@5 = 100%.
Why the two coincide for Parallel Sampling. Parallel Sampling has no iterations. It draws five independent rollouts per task and returns one final rollout per task, selected with the unit test cases. Its final output for a task is therefore correct exactly when at least one of the five rollouts is correct, which makes pass@1 and pass@5 identical by construction.
*side note: We agree that using unit test cases on the same tasks for both search and evaluation is not a natural setting. In practice one would have either (a) the same tasks for search and evaluation but no unit test cases, as in Section 4.2, or (b) unit test cases on a search set that is disjoint from the evaluation set, as in Section 4.4. We include this setting because it follows the protocol of existing harness evolution work, which is the protocol our paper sets out to examine. In those algorithms, what is returned is an iteration, that is, a single evolved harness that is then applied to all tasks, and not a separate choice for each task. We therefore score the iterative methods in the same way, by the best single iteration.
We hope this clarifies the metrics, and we are happy to answer any further questions.
response (1/2)
Dear Reviewer xykk,
Thank you for your detailed and careful review. We appreciate your recognition that our work offers a useful framing for the field, making a common source of overclaiming more visible. We have run additional experiments to make our conclusions stronger and more thorough, and we address each of your concerns below.
W1: The AHE comparison may not represent the original method fairly. AHE is run with only one rollout per task per iteration, although multiple rollouts help its debugger identify partial successes and failures. Exploration is also disabled.
Rollouts per task. We set m=1 in the submission so that all methods use the same rollout budget. To check that this does not handicap AHE, we reran it with m=2 rollouts per task per iteration, which is the configuration in the original AHE paper. Since this doubles AHE's budget to 10 rollouts per task, we compare it with the test-time scaling baselines at K=10 (Claude Opus 4.6).
*Run on 87 of the 89 Terminal-Bench 2.1 tasks;
qemu-startupandqemu-alpine-sshcan no longer be built as E2B sandboxes because the Debian 11 package repositories were retired upstream.With its original number of rollouts, AHE still does not outperform test-time scaling in single-attempt performance: its pass@1 is below both baselines without unit tests, and about 16 points below them with unit tests. Its best-of-10 score becomes comparable to Sequential Refinement, which matches the pattern discussed in Section 4.3, where the benefit appears only when selecting among multiple attempts and not as a better harness. The same holds for generalization. With m=2, the evolved harness shows no change in pass@1 on held-out tasks relative to the initial harness (+0.0).
Disabling the explore agent. This was a deliberate choice, and it is central to what we want to measure. As explained in Appendix A.1, the explore agent retrieves harnesses from external sources that are already tuned for the benchmark under evaluation. It imports benchmark-specific solutions instead of discovering improvements from feedback, so keeping it would conflate harness evolution with retrieval of harnesses already fitted to the evaluation tasks, which is the very train-test leakage our paper sets out to examine. Reviewer jVB7 mentioned the same point in their review, noting that "disabling the AHE explore agent is the right call" because retrieval of benchmark-fitted harnesses would otherwise contaminate the comparison.
W2/Q1: The meaning of pass@1 is unclear. Because the table reports identical pass@1 and pass@5 values in some cases, it is difficult to determine what the metrics actually measure or whether they support the claim about single-attempt ability.
Thank you for raising this. The definitions are given in Appendix A.1 (Evaluation Metrics): pass@1 is the average success rate of a method's returned trajectory, and pass@k is the fraction of tasks solved by at least one of the k trajectories the method produces. In other words, pass@1 measures the final answer a method commits to after spending its budget K, and pass@5 measures whether any of its five attempts succeeds.
Why pass@1 equals pass@5 for Parallel Sampling. Parallel Sampling independently generates K=5 candidate trajectories with a fixed harness. When unit test cases are available, it uses them to select its final answer, returning a candidate that passes the tests whenever one exists (Section 3.2). Its final answer is therefore correct exactly when at least one of the five candidates is correct, which is the definition of pass@5. The two metrics coincide by construction for this method in this setting.
W3: Held-out performance remains flat, but the paper does not report the corresponding training improvement from the same split and run. Without that comparison, the result could indicate severe overfitting, weak learning overall, or both.
This is an interesting point. The table below reports pass@1 on the evolution (training) tasks and on the held-out test tasks, from the same split and the same runs as Table 3.