- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies false success in harness self-evolution, where agents may claim task completion despite failing hidden tests. To address this problem, it introduces (1) EvoTrap-Bench, a benchmark with 140 tasks measuring false success across evolution iterations using hidden ground truth, and (2) Verifiably Honest Harness Evolution (VHE), a self-evolving framework designed to reduce false success by relying on execution evidence rather than self-assessment.
The problem resembles reward hacking in RL, where optimizing an imperfect proxy objective can diverge from the true one. This paper studies a similar failure mode in iterative harness self-evolution, where visible checks become optimization signals that might be exploited.
Critical strengths
-
The paper identifies a significant reliability issue in self-evolving agents: false success caused by self-assessment or optimizing imperfect feedback signals. EvoTrap-Bench provides a direct way to identify and measure this failure mode.
-
VHE provides a well-motivated solution by replacing self-assessed feedback with verification on claims, rule-based failure diagnosis and validated updates, which is technically coherent, and experiments show that it reduces false success while preserving task-solving capability.
Critical weaknesses
- The evaluation of EvoTrap-Bench is limited to a single base model, Qwen-3.8-27B. Since the paper aims to characterize a general failure mode of harness self-evolution (false successes increase with each round of harness self-evolution), broader evaluation across models is important to establish whether it is a general phenomenon.
Questions for authors
- Have the authors evaluated EvoTrap-Bench / VHE with different base models? Can the "false successes increase with each round of harness self-evolution" claim generalize across different base models? How does the effectiveness of VHE change as the underlying agent capability improves?
8 comments
Response to Reviewer 37eR (1/2): Results with Gemma 4 31B and Qwen3-Coder-Next, and their setup (Weakness, Q1)
Thank you for the positive review. Section 6 names the single base model as one of the two main limits of our study. During the discussion period, we ran AHE and VHE on EvoTrap-Bench with Gemma 4 31B and with Qwen3-Coder-Next. We post this reply in two parts. Part 1 answers the weakness and Q1 with the results and the setup. Part 2 answers Q2 and Q3.
Weakness (single base model): With both models, VHE's false-success rate is lower than AHE's and lower than the seed harness's. Under AHE, false success rises over iterations with Gemma 4 31B. With Qwen3-Coder-Next, it rises through iteration 5. Table A gives the rates. The Qwen3.8-27B row repeats the paper.
Table A: False-success rate on EvoTrap-Bench. Iterations 2–6 are pooled, as in Table 7.
| Base model (session budget) | AHE, iteration 1 | AHE, iterations 2–6 | VHE, iteration 1 | VHE, iterations 2–6 | No report (AHE / VHE) |
|---|---|---|---|---|---|
| Qwen3.8-27B, paper (30 min) | 30.9% | 86.4% | 27% | 4.3% | 4.2% / 1.8% |
| Gemma 4 31B (60 min) | 32.1% (18/56) | 58.2% (163/280) | 28.6% (16/56) | 18.6% (52/280) | 6.8% / 32.4% |
| Qwen3-Coder-Next (30 min) | 45.5% (25/55) | 47.7% (133/279) | 48.2% (27/56) | 26.8% (75/280) | 21.7% / 3.3% |
- Iteration 1 runs the seed harness in both methods. Pooled over both methods, the seed harness has a false-success rate of 30.4% (34/112) with Gemma 4 31B and 46.8% (52/111) with Qwen3-Coder-Next.
- Fisher's exact test compares VHE (iterations 2–6) with AHE (iterations 2–6) and with the seed harness. With Gemma 4 31B, p = 2.2 × 10⁻²² and p = 0.015. With Qwen3-Coder-Next, p = 3.5 × 10⁻⁷ and p = 1.9 × 10⁻⁴.
- With Gemma 4 31B, 109 of VHE's 336 trials end without a report. Of these, 52 ran out of time, and 33 received no reply from the model server while it restarted.
- With Qwen3-Coder-Next, 56 of AHE's 73 no-report trials come from iteration 6, in which no AHE agent started.
Q1 (other base models): Yes. Both models run under the paper's protocol, with the changes listed below.
- Models. Gemma 4 31B is Google's 31B dense model. Qwen3-Coder-Next is from the same family as Qwen3.8-27B, but it is trained for coding agents. It is a mixture-of-experts model with about 80B parameters, of which about 3B are active. Both models are served in FP8 with vLLM 0.19.1, on two GPUs per endpoint, with a context of 122,880 tokens (Appendix C.1).
- Protocol. Each run uses the same 14 evolution tasks and the same seed harness as the paper's runs. Each task runs four times per iteration, so an iteration has 56 trials. Each run has six iterations. The new model runs every role: the agent, AHE's evolution agent, and VHE's judge, probe writer and proposer. VHE's prompts and request settings do not change. The agent samples at temperature 0.7 and top-p 0.95, with at most 16,384 output tokens (Table 6).
- Changes. Gemma 4 31B writes long thinking at each step, so its runs use a 60-minute session budget instead of 30 minutes. 60 minutes is the cap in Appendix C.3. Qwen3-Coder-Next has no thinking mode, so its agent runs without thinking. Its top-k is the model's default of 40. The paper's model uses its own default of 20.
- Two AHE trials with Qwen3-Coder-Next failed in the sandbox before the agent started, so they have no hidden verdict. As in Appendix C.5, these two trials are left out.
- Before the 60-minute run, we ran Gemma 4 31B with the paper's 30-minute budget. Table B lists both runs. In the 30-minute run, 35.7% of VHE's trials ended without a report, so we moved to 60 minutes. In both runs, AHE's rate over iterations 2–6 is higher than in iteration 1, and VHE's rate is lower than AHE's. In the 30-minute run, AHE's rise and VHE's gap to the seed harness are not significant (p = 0.28 and p = 0.19).
Table B: Gemma 4 31B under the two session budgets.
| Session budget | AHE: iteration 1 → iterations 2–6 | VHE, iterations 2–6 | VHE trials without a report |
|---|---|---|---|
| 30 min | 26.8% → 35.0% | 15.7% | 35.7% |
| 60 min | 32.1% → 58.2% | 18.6% | 32.4% |
Response to Reviewer 37eR (2/2): False success over iterations and agent capability (Q2, Q3)
This part continues Part 1 and uses the same runs: Gemma 4 31B with a 60-minute session budget, and Qwen3-Coder-Next with the paper's 30-minute budget.
Q2 (generalization of the claim): Under AHE, false success rises over iterations with both models, through iteration 5 with Qwen3-Coder-Next. Table C gives the number of false successes in each iteration.
Table C: False successes per iteration, out of 56 trials (55 for AHE with Qwen3-Coder-Next in iterations 1 and 4).
| Iteration | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| Gemma 4 31B, AHE | 18 | 27 | 23 | 25 | 42 | 46 |
| Gemma 4 31B, VHE | 16 | 15 | 10 | 8 | 9 | 10 |
| Qwen3-Coder-Next, AHE | 25 | 31 | 30 | 33 | 39 | 0 |
| Qwen3-Coder-Next, VHE | 27 | 19 | 13 | 13 | 19 | 11 |
- Gemma 4 31B. Every evolved iteration has more false successes than iteration 1. The pooled rate rises from 32.1% to 58.2% (p = 0.0004). As in Figure 6, AHE's evolved harness tells the agent to replace what is missing. Its system prompt and long-term memory say what to do when a service is unreachable: read check.py, create the output files that check.py expects even when the task forbids a stand-in, and report complete: true.
- Qwen3-Coder-Next. Iterations 2–5 all have more false successes than iteration 1. Pooled over iterations 2–5, the rate is 59.6%, against 45.5% in iteration 1 (p = 0.068). From iteration 1 to iteration 5, the visible-check pass rate, which is the signal AHE learns from, rises from 73.2% to 98.2%. In iteration 6, no AHE agent started. After iteration 5, AHE's evolution agent had committed a middleware that failed to load. All 56 trials of iteration 6 therefore ended without a report, and the rate pooled over iterations 2–6 is 47.7%. This model also starts high: its seed harness already has 45.5% false success under AHE, against 30.9% for Qwen3.8-27B.
Q3 (agent capability): VHE's reward rests on probes run in the container, so VHE does not need the judge to catch a stronger agent's false claims. In the paper's runs, the judge alone accepts 96% of false claims (Appendix D.4). VHE's reward still matches the hidden verdict on every trial, because every false-success trial fails at least one probe. A probe is a read-only command with an exact expected output, run against the deliverable. A more capable agent can write a more convincing report, but the report does not change what a probe reads from the container.
Gemma 4 31B and Qwen3-Coder-Next do not form a capability scale, because each differs from Qwen3.8-27B in more than capability. We therefore do not read a capability trend from them. They do show two things. With Gemma 4 31B, a model from another family, VHE's false-success rate is lower than AHE's and the seed harness's in the 60-minute run. The same holds with Qwen3-Coder-Next, a model trained for coding agents.
We will add these runs and their settings to the appendix of the final version.
We thank the reviewer for the positive assessment. We also thank the reviewer for recognizing EvoTrap-Bench as a direct way to measure false success, and VHE as a well-motivated and technically coherent method.
Critical Weaknesses and Questions (other base models): We are running EvoTrap-Bench and VHE with other base models, and we will post the results in this thread as soon as the runs finish. Each run follows the paper's protocol.