- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DiVE, a diagnostic multi-agent framework for automated harness evolution. The framework divides the improvement process into three stages: failure diagnosis, evolution planning, and patch implementation. It further employs structured intermediate artifacts, modification constraints, historical experience, and evaluation feedback to guide iterative harness revisions.
The authors evaluate DiVE on τ²-Bench and Terminal-Bench 2.1, separating evolution tasks from held-out final test tasks. The evaluation uses GLM-5.2 as the evolution backbone and DeepSeek-V4-Flash as the task-execution model. DiVE reports a full average of 91.95% on τ²-Bench, compared with 91.44% for the base harness and 90.95% for AHE. On Terminal-Bench 2.1, it increases the overall Pass Rate from 47.8% for the base harness to 59.4%. The paper also compares three-stage evolution with an end-to-end alternative and explores learning from structured evolution trajectories.
Critical strengths
-
Dividing the process into diagnosis, planning, and patch implementation makes intermediate decisions explicit and potentially easier to inspect. Modification boundaries and historical feedback are also practically useful safeguards for iterative editing of operational agent systems. These design choices address realistic concerns about uncontrolled or difficult-to-debug automatic revisions.
-
The use of disjoint evolution and held-out test splits strengthens the credibility of the reported task-level improvements. In particular, the Terminal-Bench 2.1 Pass Rate increases from 47.8% to 59.4%, and the comparison with an end-to-end evolution variant shows a 4.87-percentage-point advantage in the τ²-Bench full average. These results suggest that structured evolution can improve harness quality under the evaluated conditions.
Critical weaknesses
-
The framework combines role specialization, structured diagnosis and planning, constrained modifications, historical experience, and iterative evaluation. The comparison with end-to-end evolution is useful, but it does not establish whether the gains arise from multiple specialized agents or simply from enforcing an explicit diagnosis–plan–patch sequence. A simpler single-agent method with the same intermediate artifacts, tools, and execution budget might obtain similar benefits. This distinction is consequential because multi-agent orchestration is central to the proposed methodological contribution.
-
The overall τ²-Bench improvement over the base harness is 0.51 percentage points, with a slight decline in Telecom, while Terminal-Bench shows larger gains. Small or negative improvements in particular domains do not invalidate the framework, but they limit claims of consistently superior harness evolution. Evidence about optimization-run variability and the circumstances under which the method succeeds or fails would help establish the generality of the reported advantages.
Questions for authors
-
Can the authors distinguish the benefit of multi-agent specialization from the benefit of structured intermediate reasoning? In particular, is there an existing comparison against a single-agent diagnosis–plan–patch procedure with the same model, tools, and optimization budget?
-
How stable are the τ²-Bench results across independent evolution runs, particularly given the relatively small overall gain over the base harness and the slight decrease in Telecom performance?
-
What evidence directly supports the value of the historical experience mechanism? For example, does it reduce repeated unsuccessful edits, improve candidate efficiency, or prevent previously observed regressions? Clarifying this contribution would strengthen the causal interpretation of the framework.
3 comments
Response to Reviewer wmvH (Part II)
Weakness 2 & Question 2. Clarification on Effectiveness
To address reviewer concerns about execution variability and the magnitude of performance gains, we supplement Table 2 with means and standard deviations across repeated evaluations. For τ²-Bench, which exhibits larger result fluctuations, we conduct 4 rounds of evaluation with 4 trials per round and compute the standard deviation across rounds. For Terminal-Bench 2.1, which is relatively stable, the standard deviation is computed from 2 rounds.
-
Performance gains. τ²-Bench starts from a high baseline, with Telecom already reaching 99.85%, leaving limited room for further improvement. We therefore focus our analysis on Retail, where gains remain achievable, and examine whether Telecom exhibits any performance degradation. Trajectory analysis shows that Retail gains primarily come from fixing a clear and recurring failure pattern where the model terminates or escalates to a human agent after the user confirms an action without invoking the corresponding tool. The evolved harness addresses this by introducing bounded retries to remind the model to execute confirmed actions. For example, on Task 32, the evolved successful trajectories invoke the return and cancel-order tools and pass database state checks, with the number of passes increasing from 4/16 under Base to 10/16 across four evolved rounds. In contrast, Telecom failures appear only in a small number of trials, with the failing tasks varying across runs and no task showing consistent degradation across all four rounds. This indicates that DiVE fixes recurring failure patterns on Retail, while the sporadic failures on Telecom do not yet constitute a stable degradation pattern.
-
Execution variability. The supplemented standard deviation data further demonstrates that DiVE produces highly consistent results across runs. In particular, DiVE achieves a standard deviation of 0.00 on Terminal-Bench Full and the Telecom task, indicating exceptional stability, whereas AHE exhibits a standard deviation of 4.17 on the Airline task, reflecting substantial random fluctuation. The gains of DiVE therefore reflect stable methodological improvements rather than random variation.
Overall, DiVE not only improves performance but also substantially reduces result variability, a property of considerable practical value in real-world deployment.
Table 2. Execution Variability on τ²-Bench (Avg.) and Terminal-Bench 2.1 (Pass@1)
| τ²-Bench | Terminal-Bench 2.1 | ||||||
|---|---|---|---|---|---|---|---|
| Method | Airline | Retail | Telecom | Full | Easy | Med. | Hard |
| Base | 83.74±2.01 | 86.88±1.08 | 99.85±0.30 | 39.9±5.07 | 75.0±0.00 | 46.5±4.65 | 20.5±6.82 |
| AHE | 81.90±4.17 | 87.33±1.11 | 99.10±0.81 | 44.9±1.45 | 75.0±0.00 | 50.0±1.16 | 29.5±2.27 |
| DiVE | 85.03±2.12 | 87.95±2.13 | 99.40±0.00 | 46.4±0.00 | 75.0±0.00 | 52.3±1.16 | 29.5±2.27 |
Response to Reviewer wmvH (Part III)
Question 3. Effect of Historical Experience
To further analyze the effect of historical experience, we supplement Table 3 with a behavioral comparison between settings with and without historical experience.
Table 3. Effect of Historical Experience
| Metric | w/o experience | w/ experience |
|---|---|---|
| Proposal Repetition Rate | 100% | 0% |
| Patch Repetition Rate | 70% | 0% |
| Experience Reference Rate | 0% | 56% |
As shown in Table 3, injecting historical experience substantially reduces the proportion of repeated proposals and patches, indicating that historical experience effectively suppresses homogeneous exploration. Moreover, the role of historical experience goes beyond this. More direct evidence comes from the evolution logs during candidate generation, where past failures are explicitly referenced to rule out invalid directions and drive strategy shifts. This manifests in two patterns:
- Explicit negation of historically invalid paths. For example:
"Round 1's custom middleware failed … this proposal switches to a fundamentally different strategy"
- Cross-round pattern recognition. For example:
"Previous rounds repeatedly tried modifying ContextCompactionMiddleware and StallBreakerMiddleware … but tasks kept timing out"
These records demonstrate that historical experience transforms past failures into hard constraints and prior guidance that directly shape the generation of subsequent candidates, helping the search avoid ineffective directions and explore more promising ones.
We hope these clarifications address your concerns.
Response to Reviewer wmvH (Part I)
Thank you very much for your positive recognition of DiVE's multi-agent orchestration design, the practical value of modification constraints and historical feedback, and the credibility of our evaluation methodology with disjoint evolution and held-out test splits. We hope the following responses address your remaining concerns.
Weakness 1 & Question 1. Contributions of the multi-stage decision flow and multi-agent orchestration
To disentangle whether performance gains stem from the multi-stage decision flow or multi-agent orchestration, we introduce a DiVE (single-agent) control experiment. In this setting, a single agent sequentially performs diagnosis, planning, and patching, while retaining the same intermediate artifacts, tools, historical experience, and iterative evaluation mechanism as the multi-agent version DiVE, with the base model and total execution budget aligned.
Contribution of the multi-stage decision flow. As shown in Table 1, compared to End-to-End, DiVE (single-agent) improves the overall average score on τ²-Bench from 87.08 to 91.30 and the overall Pass@1 on Terminal-Bench 2.1 from 32.6% to 35.5%. This shows that the structured diagnosis-planning-patching flow improves overall performance on both benchmarks, even when executed by a single agent.
Additional contribution of multi-agent orchestration. Building on this, DiVE (multi-agent) achieves further significant improvements over DiVE (single-agent). On Terminal-Bench 2.1, the overall Pass@1 increases from 35.5% to 46.4%, a gain of 10.9 percentage points, and Pass% increases from 44.9% to 59.4%, a gain of 14.5 percentage points. On the Airline task of τ²-Bench, the multi-agent version also outperforms the single-agent version in both average and maximum scores (85.03/88.70 vs. 81.23/85.00). This demonstrates that assigning dedicated agents to each decision stage allows each agent to focus more exclusively on its current phase, yielding additional gains that a single agent cannot achieve even under the same process structure.
In summary, the multi-stage decision flow and multi-agent orchestration each contribute independently to performance improvement, and their combination enables DiVE to surpass the performance ceiling of single-agent approaches.
Table 1. Ablation Study on Single-Agent vs. Multi-Agent Orchestration