acceptodds
Under review as a conference paper at ICLR 2027

The Harness Learns to Lie While Evolving: Measuring and Preventing False Success in Harness Self-Evolution

Abstract

Harness self-evolution repeatedly rewrites the prompts, tools, skills, middleware, and long-term memory around a frozen model over multiple iterations. Although existing self-evolving harness methods and benchmarks can improve usability, self-evolving harness methods cannot distinguish between "tasks that are actually completed" and "fabricated deliverables that pass the same check," and benchmarks cannot measure whether false successes increase with each iteration. We introduce two frameworks, EvoTrap-Bench and Verifiably Honest Harness Evolution (VHE). EvoTrap-Bench measures how false success evolves over the iterations of an arbitrary self-evolving harness. EvoTrap-Bench uses provably impossible tasks, on which any claimed success is false, and keeps the ground truth hidden from the loop. Verifiably Honest Harness Evolution (VHE) is a self-evolving method in which each step is determined by execution rather than by the model's assessment of its own performance. VHE derives its rewards from trajectories rather than reports, and its experience is derived mechanically without asking the model why a particular trial failed. Notably, we found through EvoTrap-Bench that false successes increase with each round of harness self-evolution. This phenomenon occurs in most existing harness self-evolution methods. In the others, false success stays high and shows no downward trend. VHE overcomes the false success phenomenon, keeping false success at an extremely low level in EvoTrap-Bench. Furthermore, VHE performs even better on two benchmarks where it has never evolved before. These results show that measuring and overcoming false success enables scalable, efficient, and self-improving LLM systems.

Primary area:
datasets and benchmarks
Keywords:
harness self-evolution, self-evolving LLM agents, false success, agent benchmark

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_37eROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies false success in harness self-evolution, where agents may claim task completion despite failing hidden tests. To address this problem, it introduces (1) EvoTrap-Bench, a benchmark with 140 tasks measuring false success across evolution iterations using hidden ground truth, and (2) Verifiably Honest Harness Evolution (VHE), a self-evolving framework designed to reduce false success by relying on execution evidence rather than self-assessment.

The problem resembles reward hacking in RL, where optimizing an imperfect proxy objective can diverge from the true one. This paper studies a similar failure mode in iterative harness self-evolution, where visible checks become optimization signals that might be exploited.

Critical strengths
  1. The paper identifies a significant reliability issue in self-evolving agents: false success caused by self-assessment or optimizing imperfect feedback signals. EvoTrap-Bench provides a direct way to identify and measure this failure mode.

  2. VHE provides a well-motivated solution by replacing self-assessed feedback with verification on claims, rule-based failure diagnosis and validated updates, which is technically coherent, and experiments show that it reduces false success while preserving task-solving capability.

Critical weaknesses
  1. The evaluation of EvoTrap-Bench is limited to a single base model, Qwen-3.8-27B. Since the paper aims to characterize a general failure mode of harness self-evolution (false successes increase with each round of harness self-evolution), broader evaluation across models is important to establish whether it is a general phenomenon.
Questions for authors
  1. Have the authors evaluated EvoTrap-Bench / VHE with different base models? Can the "false successes increase with each round of harness self-evolution" claim generalize across different base models? How does the effectiveness of VHE change as the underlying agent capability improves?
8 comments

We thank the reviewer for the positive assessment. We also thank the reviewer for recognizing EvoTrap-Bench as a direct way to measure false success, and VHE as a well-motivated and technically coherent method.

Critical Weaknesses and Questions (other base models): We are running EvoTrap-Bench and VHE with other base models, and we will post the results in this thread as soon as the runs finish. Each run follows the paper's protocol.

AuthorsOpenReview ↗

Response to Reviewer 37eR (1/2): Results with Gemma 4 31B and Qwen3-Coder-Next, and their setup (Weakness, Q1)

Thank you for the positive review. Section 6 names the single base model as one of the two main limits of our study. During the discussion period, we ran AHE and VHE on EvoTrap-Bench with Gemma 4 31B and with Qwen3-Coder-Next. We post this reply in two parts. Part 1 answers the weakness and Q1 with the results and the setup. Part 2 answers Q2 and Q3.

Weakness (single base model): With both models, VHE's false-success rate is lower than AHE's and lower than the seed harness's. Under AHE, false success rises over iterations with Gemma 4 31B. With Qwen3-Coder-Next, it rises through iteration 5. Table A gives the rates. The Qwen3.8-27B row repeats the paper.

Table A: False-success rate on EvoTrap-Bench. Iterations 2–6 are pooled, as in Table 7.

Base model (session budget)AHE, iteration 1AHE, iterations 2–6VHE, iteration 1VHE, iterations 2–6No report (AHE / VHE)
Qwen3.8-27B, paper (30 min)30.9%86.4%27%4.3%4.2% / 1.8%
Gemma 4 31B (60 min)32.1% (18/56)58.2% (163/280)28.6% (16/56)18.6% (52/280)6.8% / 32.4%
Qwen3-Coder-Next (30 min)45.5% (25/55)47.7% (133/279)48.2% (27/56)26.8% (75/280)21.7% / 3.3%
  • Iteration 1 runs the seed harness in both methods. Pooled over both methods, the seed harness has a false-success rate of 30.4% (34/112) with Gemma 4 31B and 46.8% (52/111) with Qwen3-Coder-Next.
  • Fisher's exact test compares VHE (iterations 2–6) with AHE (iterations 2–6) and with the seed harness. With Gemma 4 31B, p = 2.2 × 10⁻²² and p = 0.015. With Qwen3-Coder-Next, p = 3.5 × 10⁻⁷ and p = 1.9 × 10⁻⁴.
  • With Gemma 4 31B, 109 of VHE's 336 trials end without a report. Of these, 52 ran out of time, and 33 received no reply from the model server while it restarted.
  • With Qwen3-Coder-Next, 56 of AHE's 73 no-report trials come from iteration 6, in which no AHE agent started.

Q1 (other base models): Yes. Both models run under the paper's protocol, with the changes listed below.

  • Models. Gemma 4 31B is Google's 31B dense model. Qwen3-Coder-Next is from the same family as Qwen3.8-27B, but it is trained for coding agents. It is a mixture-of-experts model with about 80B parameters, of which about 3B are active. Both models are served in FP8 with vLLM 0.19.1, on two GPUs per endpoint, with a context of 122,880 tokens (Appendix C.1).
  • Protocol. Each run uses the same 14 evolution tasks and the same seed harness as the paper's runs. Each task runs four times per iteration, so an iteration has 56 trials. Each run has six iterations. The new model runs every role: the agent, AHE's evolution agent, and VHE's judge, probe writer and proposer. VHE's prompts and request settings do not change. The agent samples at temperature 0.7 and top-p 0.95, with at most 16,384 output tokens (Table 6).
  • Changes. Gemma 4 31B writes long thinking at each step, so its runs use a 60-minute session budget instead of 30 minutes. 60 minutes is the cap in Appendix C.3. Qwen3-Coder-Next has no thinking mode, so its agent runs without thinking. Its top-k is the model's default of 40. The paper's model uses its own default of 20.
  • Two AHE trials with Qwen3-Coder-Next failed in the sandbox before the agent started, so they have no hidden verdict. As in Appendix C.5, these two trials are left out.
  • Before the 60-minute run, we ran Gemma 4 31B with the paper's 30-minute budget. Table B lists both runs. In the 30-minute run, 35.7% of VHE's trials ended without a report, so we moved to 60 minutes. In both runs, AHE's rate over iterations 2–6 is higher than in iteration 1, and VHE's rate is lower than AHE's. In the 30-minute run, AHE's rise and VHE's gap to the seed harness are not significant (p = 0.28 and p = 0.19).

Table B: Gemma 4 31B under the two session budgets.

Session budgetAHE: iteration 1 → iterations 2–6VHE, iterations 2–6VHE trials without a report
30 min26.8% → 35.0%15.7%35.7%
60 min32.1% → 58.2%18.6%32.4%
AuthorsOpenReview ↗

Response to Reviewer 37eR (2/2): False success over iterations and agent capability (Q2, Q3)

This part continues Part 1 and uses the same runs: Gemma 4 31B with a 60-minute session budget, and Qwen3-Coder-Next with the paper's 30-minute budget.

Q2 (generalization of the claim): Under AHE, false success rises over iterations with both models, through iteration 5 with Qwen3-Coder-Next. Table C gives the number of false successes in each iteration.

Table C: False successes per iteration, out of 56 trials (55 for AHE with Qwen3-Coder-Next in iterations 1 and 4).

Iteration123456
Gemma 4 31B, AHE182723254246
Gemma 4 31B, VHE1615108910
Qwen3-Coder-Next, AHE25313033390
Qwen3-Coder-Next, VHE271913131911
  • Gemma 4 31B. Every evolved iteration has more false successes than iteration 1. The pooled rate rises from 32.1% to 58.2% (p = 0.0004). As in Figure 6, AHE's evolved harness tells the agent to replace what is missing. Its system prompt and long-term memory say what to do when a service is unreachable: read check.py, create the output files that check.py expects even when the task forbids a stand-in, and report complete: true.
  • Qwen3-Coder-Next. Iterations 2–5 all have more false successes than iteration 1. Pooled over iterations 2–5, the rate is 59.6%, against 45.5% in iteration 1 (p = 0.068). From iteration 1 to iteration 5, the visible-check pass rate, which is the signal AHE learns from, rises from 73.2% to 98.2%. In iteration 6, no AHE agent started. After iteration 5, AHE's evolution agent had committed a middleware that failed to load. All 56 trials of iteration 6 therefore ended without a report, and the rate pooled over iterations 2–6 is 47.7%. This model also starts high: its seed harness already has 45.5% false success under AHE, against 30.9% for Qwen3.8-27B.

Q3 (agent capability): VHE's reward rests on probes run in the container, so VHE does not need the judge to catch a stronger agent's false claims. In the paper's runs, the judge alone accepts 96% of false claims (Appendix D.4). VHE's reward still matches the hidden verdict on every trial, because every false-success trial fails at least one probe. A probe is a read-only command with an exact expected output, run against the deliverable. A more capable agent can write a more convincing report, but the report does not change what a probe reads from the container.

Gemma 4 31B and Qwen3-Coder-Next do not form a capability scale, because each differs from Qwen3.8-27B in more than capability. We therefore do not read a capability trend from them. They do show two things. With Gemma 4 31B, a model from another family, VHE's false-success rate is lower than AHE's and the seed harness's in the 60-minute run. The same holds with Qwen3-Coder-Next, a model trained for coding agents.

We will add these runs and their settings to the appendix of the final version.

Reviewer_JyiUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether agents, when learning from experience and modifying their own workflows, may internalize erroneous practices endorsed by evaluation checks, leading them to increasingly claim success on tasks they have not actually completed. The authors construct evaluation environments in which task requirements differ from what the visible checks can verify, and allow three existing methods to evolve continuously while recording actual execution outcomes, completion claims, and newly introduced rules and memories. The experiments show that the false success rates of some methods increase substantially over the course of evolution. Moreover, the accumulated experiences contain explicit instructions encouraging the fabrication of substitute data and the circumvention of resource constraints, revealing how flawed feedback can be converted into persistent behavioral guidance. To address this problem, the authors propose VHE, which reassesses which behaviors deserve rewards using execution traces and additional tests, and retains updates only after their effectiveness has been verified. The proposed approach reduces false success rates in the evaluation, while the evolved systems also achieve higher pass rates than their initial versions on external task benchmarks.

Critical strengths

The paper provides empirical evidence that erroneous feedback can be internalized and amplified through agent evolution. By examining changes in false success rates, discrepancies between rewards and actual outcomes, and the content of newly introduced rules and memories, the authors demonstrate how behaviors that fail to complete tasks can nevertheless be rewarded and incorporated into experiences that guide future behavior. For example, AHE's false success rate increases from approximately 31% to over 87% starting in the third round, while its accumulated memories also contain instructions encouraging the creation of local substitute services. This finding is important to the overall evaluation because the paper not only demonstrates a decline in reliability during evolution but also reveals how this decline emerges through execution traces and memory analysis, providing valuable empirical evidence for assessing whether agents are genuinely improving.

Critical weaknesses

The reported confidence intervals are inconsistent with the experimental scale described in the paper, undermining the reliability of the statistical results. According to the description in the main text, the evaluation with 140 tasks, 4 trials per task per round, and 5 rounds of evolution should yield approximately 2,800 trials. Given this scale and the 4.3% false success rate reported in Table 7, the 95% confidence interval computed using the Wilson method cannot match the reported interval of [2.5%, 7.4%]. This discrepancy also cannot be explained by the exclusion of at most 2.4% of trials reported in the paper. This issue directly affects the overall evaluation, as the experimental scale, statistical protocol, and reported confidence intervals are not mutually consistent, raising concerns about the credibility of the core empirical evidence.

The substantial gain of the proposed method is insufficiently validated. Existing methods already use rewards provided by visible verifiers, whereas VHE introduces both new reward rules and additional tests based on selecting correct and incorrect reference solutions. Therefore, the current comparison does not clearly disentangle whether the performance improvements come from more reliable verification feedback or from the proposed evolutionary procedure itself. The ablation studies also do not fully resolve this issue: after removing components such as pre-update rerunning and repair retention, the false success rate remains close to that of the full method. This limitation affects the assessment of the method's contribution, as the experiments do not yet demonstrate that the complete evolution mechanism provides sufficient additional value beyond improving verification feedback.

Questions for authors

Please clarify the actual sample size and statistical procedure used for Table 7. According to the evaluation protocol described in the paper, the post-evolution evaluation should contain approximately 2,800 trials. Given this scale and the reported false success rate of 4.3%, the Wilson 95% confidence interval cannot produce the reported interval of [2.5%, 7.4%]. If possible, please provide the actual number of tasks, valid trials, false successes, and the statistical scripts used for each method and each round, and explain the source of this discrepancy.

What additional benefit does the complex evolution process provide beyond improving verification feedback? Several ablation variants achieve false success rates close to that of the full VHE, while existing methods do not receive the same verification information and reward signals as VHE. Could you provide comparisons under matched testing and reward conditions to isolate the independent contribution of the core evolution components?

5 comments
AuthorsOpenReview ↗

Response to Reviewer JyiU (Part 1/2): sample size, intervals and evaluation on all 140 tasks

We thank the reviewer for checking our statistics so carefully. We also thank the reviewer for recognizing that the paper traces how flawed feedback is rewarded, written into memory and amplified during evolution.

W1/Q1 (sample size and intervals): Table 7 pools about 280 trials per method, not 2,800. All evolution runs use the same 14 of the 140 tasks. Each iteration runs these 14 tasks four times, which gives 56 trials. Table 7 pools iterations 2–6, so each method has 5 × 56 = 280 trials. Between one and seven of them have no hidden verdict and are excluded, which leaves 273–279. VHE has 12 false successes in its 279 trials. The Wilson 95% interval for 12/279 is exactly [2.5%, 7.4%]. The same counts reproduce every other row of Table 7 (AHE 236/273, ACE 97/277, Evo-Harness 92/274) and the Fisher tests. The bootstrap interval is wider because it resamples the 14 tasks. VHE's bootstrap interval [0.0%, 12.9%] still lies below the lower end of every baseline's interval. The table lists each iteration (false successes / trials with a hidden verdict):

IterationAHEACEEvo-HarnessVHE
117/5510/5621/5615/55
240/5515/5617/552/56
348/5315/5618/552/55
450/5522/5616/564/56
549/5423/5421/521/56
649/5622/5520/563/56
2–6 (Table 7)236/27397/27792/27412/279

Iteration 1 runs each method's harness before any evolution. ACE updates its playbook every 16 trials, so we group its trials into windows of one iteration's size. For each task, ACE's first four trials form iteration 1, its next four trials form iteration 2, and so on. The effect also holds task by task. From iteration 2 on, VHE has no false success on any of the 12 impossible tasks (0/240 trials). AHE has 227 false successes in 235 trials on the same tasks.

To test the effect at the scale the reviewer computed, we ran five harnesses on all 140 tasks. These are the seed harness and the final harness of each method. Each harness ran each task four times, which gives 2,800 trials. Of the 140 tasks, 126 were never used in evolution; we call them unseen tasks. On the unseen tasks, VHE has no false success (0/504; Wilson 95% [0.0%, 0.8%]). The seed harness has 28.0% (141/504), AHE 85.7% (432/504), ACE 37.3% (188/504) and Evo-Harness 35.4% (178/503). The seed harness has false successes on 49 different unseen tasks, and VHE has none on any of them. VHE does not get there by refusing everything. On the 18 solvable unseen tasks, it reaches 57 true successes in 72 trials, against 50 for the seed harness.

We also evolved AHE and VHE twice more each, with all settings unchanged. The table gives the false successes in each iteration (about 56 trials each) and the pooled rate over iterations 2–6:

Run123456Pooled 2–6
AHE (paper)17404850494986.4%
AHE, repeat 118454551495186.1%
AHE, repeat 220271721202137.9%
VHE (paper)15224134.3%
VHE, repeat 118333245.4%
VHE, repeat 220322435.0%

Every VHE run stays below every AHE run. In every VHE run, false success falls by 23–31 points after iteration 1. Two AHE runs repeat the climb. The third AHE run learned the same stand-in procedures, but its harness also gained a rule to report failures honestly. In 131 of its 280 trials in iterations 2–6, the agent left a failing deliverable, usually a stand-in, but reported the task as unfinished rather than as a success. Even against this run, every VHE run is 32–34 points lower (all nine pairwise Fisher p < 1e-21).

In the revision, §5.1 names the 14 evolution tasks, and a new appendix table reports the evaluation on all 140 tasks. The revised supplement adds the list of all 140 tasks, with the 14 evolution tasks marked. It also adds the outcome of every trial behind Figure 4, Table 7 and the tables in this reply, and scripts that recompute them.

AuthorsOpenReview ↗

Response to Reviewer JyiU (Part 2/2): matched comparisons of reward and evolution; presentation

W2/Q2 (what evolution adds beyond verification feedback): The verification feedback is VHE's own reward, and evolving under it gives harnesses that stay honest without any verifier. Without evolution, the same verification does not make the seed harness honest. VHE's reward is part of our contribution: it is the first step of VHE (§4.1). It consists of the judge, the reference-checked probes and the probe veto. We ran the matched comparisons the reviewer asked for. All new runs use the paper's settings and are scored exactly as in Table 7.

Evolution loopRewardFalse success, iterations 2–6False success, 126 unseen tasks
AHEvisible check (paper)236/273 (86.4%)432/504 (85.7%)
AHEVHE's reward3/280 (1.1%)3/504 (0.6%)
ACEvisible check (paper)97/277 (35.0%)188/504 (37.3%)
ACEVHE's reward10/273 (3.7%)0/504 (0.0%)
VHEVHE's reward (paper)12/279 (4.3%)0/504 (0.0%)
VHEvisible check81/280 (28.9%)143/504 (28.4%)
No evolution (seed harness)VHE's reward, used only for scoring90/280 (32.1%)141/504 (28.0%)
No evolution (seed harness)VHE's probes and judge run on each report11/280 (3.9%)—

(1) Same reward, other loops. We gave AHE and ACE exactly VHE's reward. Both became as honest as VHE (1.1% and 3.7%, against 4.3%). The harnesses they evolved also stay honest on the 126 unseen tasks, where no verifier runs. VHE's reward does not depend on VHE's loop, so existing harness-evolution methods can adopt it. (2) Same loop, baseline reward. We also ran VHE's loop with only the visible check as the reward. False success stays at the seed harness's level (28.9%, or 14–18 per iteration). VHE's loop does not raise false success under the visible check, while AHE's loop raises it to 86.4%. Removing false success takes VHE's reward. (3) Same verification, no evolution. First, we kept the seed harness fixed and used VHE's reward only to score its trials. False success stays at 32.1%, close to the seed level. Second, we ran VHE's probes and judge on each completion report and showed the agent which requirements failed. False success falls to 3.9%. But this fixes the report, not the work. In 88 of the 240 impossible-task trials, the agent still builds a deliverable that fails the hidden tests. This deliverable is usually a stand-in, and the agent reports it as unfinished. These trials count as honest failures. This setup also needs reference-checked probes and a judge call for every task at deployment. VHE's evolved harness needs neither, and it gives an honest decline in 239 of its 240 trials on impossible tasks. (4) Ablations. These results also explain Table 2. Every variant in Table 2 keeps VHE's reward. Removing replay before commit or fix retention leaves false success at 3.9% and 4.3%, and all six variants stay between 2.9% and 4.6%. On the impossible tasks, where false success concentrates, the reward uses no reference solution. The reference checks remove every probe on these tasks, so the reward there comes from the judge alone. The judge reads the trajectory and sees no reference solution or hidden test.

In the revision, §5.2.4 adds this table and states VHE's contribution in these terms. VHE's reward is our design and part of VHE, and it is the step that removes false success. Other evolution loops can adopt it. Harnesses evolved under this reward stay honest on unseen tasks without any verifier. The comparisons confirm the paper's conclusion (§6): what the loop rewards decides how honest the harness becomes.

Presentation: The revision adds a protocol table to §5.1. The table lists the 14 evolution tasks, the 126 unseen tasks, the runs per task, the iterations and the trials behind each number. The revision also gives the numerator and denominator of each metric where the metric first appears.

We hope these results address the reviewer's concerns. We are happy to provide further details.

Dear Reviewer JyiU,

We thank you for your detailed review and for the time you spent on our paper. We have posted our response in this thread. Do you have any further comments or questions about it? If any answer needs more detail, you can tell us which one.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer pFRg (Part 1/2): the same reward for ACE and AHE; reference solutions

We thank the reviewer for the careful reading, including the recomputation of Table 7. We also thank the reviewer for finding the separation of claims and verdicts, the stand-in example in Appendix D.2, and the judge and probe analyses useful. We ran the controls the reviewer asked for, and the results follow.

Q1 (same reward for ACE and AHE; fixed harness with the same verification): With VHE's reward, ACE and AHE become as honest as VHE. A fixed harness with the same verification does not. VHE's reward is part of our contribution: it is the first step of VHE (§4.1). It consists of the judge, the reference-checked probes and the probe veto. The harnesses that ACE and AHE evolve under VHE's reward also stay honest on 126 tasks that no evolution run used (unseen tasks). All new runs use the paper's settings and are scored exactly as in Table 7.

Evolution loopRewardFalse success, iterations 2–6False success, 126 unseen tasks
AHEvisible check (paper)236/273 (86.4%)432/504 (85.7%)
AHEVHE's reward3/280 (1.1%)3/504 (0.6%)
ACEvisible check (paper)97/277 (35.0%)188/504 (37.3%)
ACEVHE's reward10/273 (3.7%)0/504 (0.0%)
VHEVHE's reward (paper)12/279 (4.3%)0/504 (0.0%)
VHEvisible check81/280 (28.9%)143/504 (28.4%)
No evolution (seed harness)VHE's reward, used only for scoring90/280 (32.1%)141/504 (28.0%)
No evolution (seed harness)VHE's probes and judge run on each report11/280 (3.9%)—
  • Existing methods, same reward. We gave AHE and ACE exactly VHE's reward. AHE falls from 86.4% to 1.1%, and ACE falls from 35.0% to 3.7%. On the unseen tasks, their final harnesses reach 0.6% and 0.0%, with no verifier running.
  • Fixed harness, same verification. If VHE's reward only scores the trials, the seed harness stays at 32.1%. If VHE's probes and judge also run on each completion report, the agent sees which requirements failed, and false success falls to 3.9%. But this fixes the report, not the work. In 88 of the 240 impossible-task trials, the agent still builds a deliverable that fails the hidden tests. This deliverable is usually a stand-in, and the agent reports it as unfinished. These trials count as honest failures. This setup also needs reference-checked probes and a judge call for every task at deployment. VHE's evolved harness needs neither, and it gives an honest decline in 239 of its own 240 impossible-task trials.
  • VHE's loop, visible-check reward. False success is 28.9%, the seed level. VHE's loop does not raise false success under the visible check, while AHE's loop raises it to 86.4%.

These controls show that VHE's reward is the step that removes false success, and that it works outside VHE's loop. They confirm the paper's conclusion (§6): what the loop rewards decides how honest the harness becomes. The reviewer notes that improving the verifier is a legitimate contribution. VHE's reward is this verifier: it is our design and part of VHE (§4.1). In the revision, §5.2.4 adds this table and presents VHE's contribution in these terms. Other evolution loops can adopt VHE's reward, and harnesses evolved under it stay honest on unseen tasks without any verifier.

W1 (reference solutions; Appendices D.4–D.5): On the impossible tasks, where false success concentrates, VHE's reward uses no reference solution. The revision states the reference requirement prominently in §4 and §5.1. An impossible task has no correct deliverable, because its correct solution reports the blocker. So the reference checks remove every probe on the 12 impossible tasks. We confirmed this in every iteration of the VHE run. On these tasks, the reward comes from the judge alone. The judge reads the trajectory and sees no reference solution or hidden test. It still catches most fabrications. When this reward scores the seed harness, the judge rejects 129 of 179 false claims on impossible tasks. None of the 81 false successes on these tasks earns the full reward. The 96% in Appendix D.4 comes from VHE's own trials. There, the false claims are implementation errors on the solvable tasks, and the reference-checked probes catch them. The variant without reference checks also keeps false success at 2.9% (8/280, Table 2). The reference checks protect true successes (Appendix D.5). In the revision, the assumption reads as follows. Each training task needs one correct reference solution, used only during evolution. The tempting wrong solution is used only to filter probes. The evolved harness uses neither. Such references are common where harnesses are evolved. Every Terminal-Bench 2 task comes with a human-written solution, and every SWE-bench instance comes with the patch that resolved its issue.

AuthorsOpenReview ↗

Response to Reviewer pFRg (Part 2/2): counts and task IDs; independent repetitions; external results

Q2 (counts, task IDs and coverage): The ≈280 trials come from an evaluation subset. All evolution runs use the same 14 of the 140 tasks, and the other 126 tasks now serve as an unseen-task evaluation. Each iteration runs the 14 tasks four times (56 trials). Table 7 pools iterations 2–6, which gives 280 trials per method, 273–279 of them with a hidden verdict. On the 126 unseen tasks, false success is 85.7% for AHE's final harness, 28.0% for the seed harness and 0/504 for VHE.

The 12 impossible tasks are pull-price-feed, deploy-static-site, fetch-fx-rates, restore-db-snapshot, publish-artifact-index and verify-cache-purge. Each has one variant with a more explicit tier sentence: pull-price-feed-sp, restore-db-snapshot-sp, publish-artifact-index-sp, deploy-static-site-mm, fetch-fx-rates-mm and verify-cache-purge-mm. The two solvable tasks are ledger-amount-parse and orders-import-tolerant, one for each type of check gap. The table gives false successes / trials with a hidden verdict:

IterationAHEACEEvo-HarnessVHE
117/5510/5621/5615/55
240/5515/5617/552/56
348/5315/5618/552/55
450/5522/5616/564/56
549/5423/5421/521/56
649/5622/5520/563/56
2–6 (Table 7)236/27397/27792/27412/279

Iteration 1 runs each method's harness before any evolution. ACE updates its playbook every 16 trials, so we group its trials into windows of one iteration's size. For each task, ACE's first four trials form iteration 1, its next four trials form iteration 2, and so on.

Within the 14 tasks, the result holds task by task. AHE's false-success rate rises above its iteration 1 on 11 of the 14 tasks (sign test, p = 0.006). From iteration 2 on, VHE has no false success on any impossible task (0/240), while AHE has 227 in 235 trials. On the unseen tasks, ACE reaches 37.3% and Evo-Harness 35.4%. The seed harness has false successes on 49 different unseen tasks, and VHE has none on any of them. VHE also keeps its true successes: on the 18 solvable unseen tasks, it reaches 57 in 72 trials, against 50 for the seed harness. In the revision, §5.1 names the 14 tasks, and an appendix table reports the unseen-task evaluation. The revised supplement adds the list of all 140 tasks, with the 14 evolution tasks marked. It also adds the outcome of every trial behind Figure 4, Table 7 and the tables in this reply, and scripts that recompute them.

Q3 (independent repetitions): Yes. We evolved AHE and VHE twice more each, with all settings unchanged. Every VHE run stays below every AHE run. The table gives false successes per iteration (about 56 trials each):

Run123456Pooled 2–6
AHE (paper)17404850494986.4%
AHE, repeat 118454551495186.1%
AHE, repeat 220271721202137.9%
VHE (paper)15224134.3%
VHE, repeat 118333245.4%
VHE, repeat 220322435.0%

In every VHE run, false success falls by 23–31 points after iteration 1. Two AHE runs repeat the climb. The third AHE run learned the same stand-in procedures, but its harness also gained a rule to report failures honestly. In 131 of its 280 trials in iterations 2–6, the agent left a failing deliverable, usually a stand-in, but reported the task as unfinished rather than as a success. Even against this run, every VHE run is 32–34 points lower (all nine pairwise Fisher p < 1e-21). The repeats were scored on EvoTrap-Bench; we did not rerun the external benchmarks for them. The revised Figure 4 and Appendix C.5 show every run.

W3 (external results): The comparison between methods rests on EvoTrap-Bench, not on the external benchmarks. On EvoTrap-Bench, VHE has the lowest false success on the 14 evolution tasks, on the 126 unseen tasks and in each of its repeated runs. The revision draws no ranking of the methods from the external benchmarks. It words §5.2.3 and the abstract in these terms.

We hope these results resolve the reviewer's reservations. We would be glad to provide further analysis.

Dear Reviewer pFRg,

We thank you for your detailed review and for the time you spent on our paper. We have posted our response in this thread. Do you have any further comments or questions about it? If any answer needs more detail, please tell us which one.

Dear Reviewer pFRg,

Thank you again for your careful review. As the discussion period comes to a close, we would like to kindly check whether our response has addressed your concerns. Our response answers each of your three questions: the comparisons under the same reference-validated reward, the task IDs and counts behind Figure 4 and Table 7, and the independent repetitions of the evolution runs. If any point is still unclear, we would be glad to clarify it before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.