acceptodds
Under review as a conference paper at ICLR 2027

Divide and Evolve:Orchestrating Agent Harness Evolution with Diagnostic Multi-Agent System

Abstract

LLM-driven agents increasingly rely on well-designed harnesses to execute complex long-horizon tasks, yet harness design flaws can recur across tasks and require costly human intervention to resolve. Harness evolution addresses this by automatically searching for and implementing harness modifications based on execution traces and task feedback, but effective evolution demands sound multi-stage decision coordination and principled control of modification boundaries and search directions, challenges that existing approaches fail to address simultaneously. To this end, we propose DiVE, a diagnosis-driven multi-agent harness evolution framework that integrates constraint-based boundary control and experience-guided search. DiVE assigns dedicated roles to harness analysis, repair planning, and patch implementation, propagating decision rationale through explicit intermediate outputs to support error localization, while incorporating program constraints, historical experience, and task feedback to jointly govern modification scope and search direction across iterations. Extensive experiments on -Bench and Terminal-Bench demonstrate that DiVE consistently improves harness performance across diverse task environments, and that small LLMs trained on complete evolution trajectories can also produce effective harness improvements, suggesting that structured evolution trajectories provide useful supervision for learning evolution capabilities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Harness Evolution, Self-Improvement, Agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wmvHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DiVE, a diagnostic multi-agent framework for automated harness evolution. The framework divides the improvement process into three stages: failure diagnosis, evolution planning, and patch implementation. It further employs structured intermediate artifacts, modification constraints, historical experience, and evaluation feedback to guide iterative harness revisions.

The authors evaluate DiVE on τ²-Bench and Terminal-Bench 2.1, separating evolution tasks from held-out final test tasks. The evaluation uses GLM-5.2 as the evolution backbone and DeepSeek-V4-Flash as the task-execution model. DiVE reports a full average of 91.95% on τ²-Bench, compared with 91.44% for the base harness and 90.95% for AHE. On Terminal-Bench 2.1, it increases the overall Pass Rate from 47.8% for the base harness to 59.4%. The paper also compares three-stage evolution with an end-to-end alternative and explores learning from structured evolution trajectories.

Critical strengths
  1. Dividing the process into diagnosis, planning, and patch implementation makes intermediate decisions explicit and potentially easier to inspect. Modification boundaries and historical feedback are also practically useful safeguards for iterative editing of operational agent systems. These design choices address realistic concerns about uncontrolled or difficult-to-debug automatic revisions.

  2. The use of disjoint evolution and held-out test splits strengthens the credibility of the reported task-level improvements. In particular, the Terminal-Bench 2.1 Pass Rate increases from 47.8% to 59.4%, and the comparison with an end-to-end evolution variant shows a 4.87-percentage-point advantage in the τ²-Bench full average. These results suggest that structured evolution can improve harness quality under the evaluated conditions.

Critical weaknesses
  1. The framework combines role specialization, structured diagnosis and planning, constrained modifications, historical experience, and iterative evaluation. The comparison with end-to-end evolution is useful, but it does not establish whether the gains arise from multiple specialized agents or simply from enforcing an explicit diagnosis–plan–patch sequence. A simpler single-agent method with the same intermediate artifacts, tools, and execution budget might obtain similar benefits. This distinction is consequential because multi-agent orchestration is central to the proposed methodological contribution.

  2. The overall τ²-Bench improvement over the base harness is 0.51 percentage points, with a slight decline in Telecom, while Terminal-Bench shows larger gains. Small or negative improvements in particular domains do not invalidate the framework, but they limit claims of consistently superior harness evolution. Evidence about optimization-run variability and the circumstances under which the method succeeds or fails would help establish the generality of the reported advantages.

Questions for authors
  1. Can the authors distinguish the benefit of multi-agent specialization from the benefit of structured intermediate reasoning? In particular, is there an existing comparison against a single-agent diagnosis–plan–patch procedure with the same model, tools, and optimization budget?

  2. How stable are the τ²-Bench results across independent evolution runs, particularly given the relatively small overall gain over the base harness and the slight decrease in Telecom performance?

  3. What evidence directly supports the value of the historical experience mechanism? For example, does it reduce repeated unsuccessful edits, improve candidate efficiency, or prevent previously observed regressions? Clarifying this contribution would strengthen the causal interpretation of the framework.

3 comments
AuthorsOpenReview ↗

Response to Reviewer wmvH (Part I)

Thank you very much for your positive recognition of DiVE's multi-agent orchestration design, the practical value of modification constraints and historical feedback, and the credibility of our evaluation methodology with disjoint evolution and held-out test splits. We hope the following responses address your remaining concerns.

Weakness 1 & Question 1. Contributions of the multi-stage decision flow and multi-agent orchestration

To disentangle whether performance gains stem from the multi-stage decision flow or multi-agent orchestration, we introduce a DiVE (single-agent) control experiment. In this setting, a single agent sequentially performs diagnosis, planning, and patching, while retaining the same intermediate artifacts, tools, historical experience, and iterative evaluation mechanism as the multi-agent version DiVE, with the base model and total execution budget aligned.

Contribution of the multi-stage decision flow. As shown in Table 1, compared to End-to-End, DiVE (single-agent) improves the overall average score on τ²-Bench from 87.08 to 91.30 and the overall Pass@1 on Terminal-Bench 2.1 from 32.6% to 35.5%. This shows that the structured diagnosis-planning-patching flow improves overall performance on both benchmarks, even when executed by a single agent.

Additional contribution of multi-agent orchestration. Building on this, DiVE (multi-agent) achieves further significant improvements over DiVE (single-agent). On Terminal-Bench 2.1, the overall Pass@1 increases from 35.5% to 46.4%, a gain of 10.9 percentage points, and Pass% increases from 44.9% to 59.4%, a gain of 14.5 percentage points. On the Airline task of τ²-Bench, the multi-agent version also outperforms the single-agent version in both average and maximum scores (85.03/88.70 vs. 81.23/85.00). This demonstrates that assigning dedicated agents to each decision stage allows each agent to focus more exclusively on its current phase, yielding additional gains that a single agent cannot achieve even under the same process structure.

In summary, the multi-stage decision flow and multi-agent orchestration each contribute independently to performance improvement, and their combination enables DiVE to surpass the performance ceiling of single-agent approaches.

Table 1. Ablation Study on Single-Agent vs. Multi-Agent Orchestration

τ²-BenchTerminal-Bench 2.1
MethodAirlineRetailTelecomFullFullEasyMed.Hard
End-to-End77.18/81.2079.70/82.5099.40/100.0087.08/89.2432.6/39.175.0/75.039.5/46.511.4/18.2
DiVE (single-agent)81.23/85.0088.40/88.7099.23/100.0091.30/92.4835.5/44.975.0/75.038.4/46.522.7/36.4
DiVE (multi-agent)85.03/88.7087.95/90.0099.40/99.4091.95/93.5046.4/59.475.0/75.052.3/65.129.5/45.5
AuthorsOpenReview ↗

Response to Reviewer wmvH (Part II)

Weakness 2 & Question 2. Clarification on Effectiveness

To address reviewer concerns about execution variability and the magnitude of performance gains, we supplement Table 2 with means and standard deviations across repeated evaluations. For τ²-Bench, which exhibits larger result fluctuations, we conduct 4 rounds of evaluation with 4 trials per round and compute the standard deviation across rounds. For Terminal-Bench 2.1, which is relatively stable, the standard deviation is computed from 2 rounds.

  1. Performance gains. τ²-Bench starts from a high baseline, with Telecom already reaching 99.85%, leaving limited room for further improvement. We therefore focus our analysis on Retail, where gains remain achievable, and examine whether Telecom exhibits any performance degradation. Trajectory analysis shows that Retail gains primarily come from fixing a clear and recurring failure pattern where the model terminates or escalates to a human agent after the user confirms an action without invoking the corresponding tool. The evolved harness addresses this by introducing bounded retries to remind the model to execute confirmed actions. For example, on Task 32, the evolved successful trajectories invoke the return and cancel-order tools and pass database state checks, with the number of passes increasing from 4/16 under Base to 10/16 across four evolved rounds. In contrast, Telecom failures appear only in a small number of trials, with the failing tasks varying across runs and no task showing consistent degradation across all four rounds. This indicates that DiVE fixes recurring failure patterns on Retail, while the sporadic failures on Telecom do not yet constitute a stable degradation pattern.

  2. Execution variability. The supplemented standard deviation data further demonstrates that DiVE produces highly consistent results across runs. In particular, DiVE achieves a standard deviation of 0.00 on Terminal-Bench Full and the Telecom task, indicating exceptional stability, whereas AHE exhibits a standard deviation of 4.17 on the Airline task, reflecting substantial random fluctuation. The gains of DiVE therefore reflect stable methodological improvements rather than random variation.

Overall, DiVE not only improves performance but also substantially reduces result variability, a property of considerable practical value in real-world deployment.

Table 2. Execution Variability on τ²-Bench (Avg.) and Terminal-Bench 2.1 (Pass@1)

τ²-BenchTerminal-Bench 2.1
MethodAirlineRetailTelecomFullEasyMed.Hard
Base83.74±2.0186.88±1.0899.85±0.3039.9±5.0775.0±0.0046.5±4.6520.5±6.82
AHE81.90±4.1787.33±1.1199.10±0.8144.9±1.4575.0±0.0050.0±1.1629.5±2.27
DiVE85.03±2.1287.95±2.1399.40±0.0046.4±0.0075.0±0.0052.3±1.1629.5±2.27
AuthorsOpenReview ↗

Response to Reviewer wmvH (Part III)

Question 3. Effect of Historical Experience

To further analyze the effect of historical experience, we supplement Table 3 with a behavioral comparison between settings with and without historical experience.

Table 3. Effect of Historical Experience

Metricw/o experiencew/ experience
Proposal Repetition Rate100%0%
Patch Repetition Rate70%0%
Experience Reference Rate0%56%

As shown in Table 3, injecting historical experience substantially reduces the proportion of repeated proposals and patches, indicating that historical experience effectively suppresses homogeneous exploration. Moreover, the role of historical experience goes beyond this. More direct evidence comes from the evolution logs during candidate generation, where past failures are explicitly referenced to rule out invalid directions and drive strategy shifts. This manifests in two patterns:

  1. Explicit negation of historically invalid paths. For example:

"Round 1's custom middleware failed … this proposal switches to a fundamentally different strategy"

  1. Cross-round pattern recognition. For example:

"Previous rounds repeatedly tried modifying ContextCompactionMiddleware and StallBreakerMiddleware … but tasks kept timing out"

These records demonstrate that historical experience transforms past failures into hard constraints and prior guidance that directly shape the generation of subsequent candidates, helping the search avoid ineffective directions and explore more promising ones.

We hope these clarifications address your concerns.

Reviewer_HjpjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a framework named DiVE to automatically improve the harness of large language model (LLM) agent. To be specific, DiVE is a framework that utilizes multiple agents to better help evolve harness based on diagnosis. Experimental results on two datasets validate the DiVE method is effective in enhancing harness performance.

Critical strengths

This work has these critical strengths:

  • Harness evolution is a trending and popular topic. The studied problem about automatically repairing harness failures is meaningful.
  • This work is well-written which makes it not hard to understand. The proposed multi-agent system effectively separate the roles of different agents.
Critical weaknesses

This work has these critical weaknesses:

  • The performance on some data subsets looks marginal which might the effectiveness not very apparent.
  • Some results reported in the figures seem inconsistent with those described in the main text. For example, in section 4.3, the text shows that 'with maximum-score gains of 2.80 and 1.93 percentage points on Airline and Retail, respectively'. According to the figure 2 (a), on Airline, the score is increased from 86.30 to 87.50. According to the figure 2 (b), on Retail, the score is increased from 87.50 to 90.06.
Questions for authors

I have the following suggestions and questions:

  • The setting of transferability experiments in Section 4.5 is not very clear. Is the harness used by Kimi-K2.6 evolved by DeepSeek-V4-Flash or by Kimi-K2.6?
  • I would appreciate it if the authors could double-check the reported results and clarify any inconsistencies, or provide further explanation of the calculation methodology in case I have misunderstood it.
  • There are some typos/minor format suggestions:
    • Around Line 035, there is no space between 'tasks’ and ‘(Jimenez et al'
    • Around Line 625, 'To train a small LLMs' -> 'To train a small LLM'
    • Around Line 700, what is the meaning of this sentence: ‘booktabs tabularx array seqsplit’
    • Figure 4 appears earlier than Figure 3.
3 comments
AuthorsOpenReview ↗

Response to Reviewer Hjpj

Thank you very much for your valuable feedback and for recognizing our clear writing. We hope the following responses address your remaining concerns.

Weakness 1. Clarification on Effectiveness

We understand that the reviewer's concern mainly stems from the lack of performance gain on the Telecom subset of τ²-Bench. However, this result has a reasonable explanation. To further verify the effectiveness of our method, we report the mean and standard deviation across repeated evaluations on all datasets in Table 1.

Table 1. Execution Variability on τ²-Bench (Avg.) and Terminal-Bench 2.1 (Pass@1)

τ²-BenchTerminal-Bench 2.1
MethodAirlineRetailTelecomFullEasyMed.Hard
Base83.74±2.0186.88±1.0899.85±0.3039.9±5.0775.0±0.0046.5±4.6520.5±6.82
AHE81.90±4.1787.33±1.1199.10±0.8144.9±1.4575.0±0.0050.0±1.1629.5±2.27
DiVE85.03±2.1287.95±2.1399.40±0.0046.4±0.0075.0±0.0052.3±1.1629.5±2.27

On the τ²-Bench Telecom Subset. The performance of Base harness on this subset already reaches 99.85, leaving extremely limited room for further improvement. Under such a high baseline, the ability of the evolved Harness to maintain performance stably near the original level itself demonstrates the effectiveness of the evolution method, at least indicating that the evolution process does not introduce regression. As shown in Table 1, DiVE achieves a standard deviation of 0.00 on Telecom across repeated runs, indicating near-zero performance fluctuation. In contrast, AHE and Base show standard deviations of 0.81 and 0.30 respectively, reflecting a certain degree of instability.

On Overall Effectiveness. Beyond the Telecom subset, DiVE achieves consistent performance improvements on the remaining subsets of τ²-Bench and on Terminal-Bench 2.1. Notably, AHE degrades relative to Base on the Airline task (83.74 to 81.90), whereas DiVE improves over Base by 1.29 points and over AHE by 3.13 points. On Terminal-Bench 2.1, DiVE outperforms AHE by 1.5 points on Full Pass@1 and by 2.3 points on Med. difficulty. The standard deviation of DiVE on Full is 0.00, compared to 1.45 for AHE, further confirming the consistency of the observed gains.

In summary, DiVE demonstrates consistent and stable performance advantages across multiple tasks. The absence of notable gains on a small number of high-baseline tasks reflects the ceiling effect inherent to those tasks, rather than a limitation in the effectiveness of our method.

Weakness 2 & Question 2. Clarification on Results in Section 4.3

We have carefully checked the experimental results and corrected the reported performance improvements for the highest-scoring methods on the Airline and Retail tasks from 2.80 and 1.93 percentage points to 1.20 and 2.56 percentage points respectively. This correction does not affect the main conclusions of the section.

Question 1. Further clarification of transferability to Kimi-K2.6

The transferability experiment in Section 4.5 uses a Harness evolved with GLM-5.2 as the evolution model and DeepSeek-V4-Flash as the agent execution model. The evolved Harness is then evaluated directly on Kimi-K2.6 without any additional adaptation to the target execution model (Kimi-K2.6). This design aims to verify that the Harness evolved by DiVE is transferable across different execution models. We have provided a clearer description of this setup in Section 4.5 of the revised manuscript.

Question 3. Correction of Typos and Formatting Issues

We thank the reviewer for the careful proofreading and feedback. We have confirmed and addressed all the issues listed above. Specifically, we added the missing space between "tasks" and "(Jimenez et al)", corrected "To train a small LLMs" to "To train a small LLM", removed the LaTeX package names "booktabs tabularx array seqsplit" that were mistakenly left in the main text, and reordered Figure 3 and Figure 4 to ensure they appear in the correct sequential order.

We hope these clarifications address your concerns.

Dear Authors,

Thank you for your rebuttal. Some of my concerns have been addressed. However, I would like to seek further clarification regarding my original concerns about the results. Could you explain why the numerical values reported in the original manuscript were inconsistent?

Besides, I noticed that reviewer CbLR raised some concerns about method novelty. Could you have a few extra sentences to summary the novelty and put them in the revised manuscript in blue?

AuthorsOpenReview ↗

Response to Reviewer Hjpj

Dear Reviewer Hjpj,

Thank you for your response. We sincerely apologize for the inconsistent results in the original manuscript. These were due to writing errors on our part. The results in Figure 2 are correct, and the errors occurred inadvertently in section 4.3. We have carefully verified all numbers against our experimental records and corrected them in the revised manuscript.

Regarding your concerns about novelty, we further clarify the novelty and contributions of DiVE as follows. To the best of our knowledge, DiVE is the first to model harness evolution as a traceable three-stage decision workflow comprising failure diagnosis, repair planning, and code implementation, and the first to unify positive feedback and negative constraints in a three-stage harness evolution mechanism with historical experience reuse, while also supporting stage-level learning of the three evolution agents. Unlike existing methods that rely on predefined operator libraries or fixed harness frameworks, DiVE allows flexible repair strategies that are adaptable to new defects. The maintained decision chain enables precise error attribution across stages, and accumulated evolution experience supports both iterative improvement and supervised training of the three evolution agents, thereby achieving co-evolution of the evolver and the harness. We have added this clarification to the introduction of the revised manuscript.

We hope these clarifications further address your concerns.

Reviewer_CbLROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies automatic harness improvement while keeping the task-execution model fixed. DiVE assigns trajectory analysis, repair planning, and patch implementation to three separate agents, passing explicit diagnoses and plans between stages. Candidate modifications undergo programmatic checks and task evaluation, while records of previous attempts guide subsequent iterations. Experiments on τ²-Bench and Terminal-Bench compare the evolved harnesses with initial harnesses, manually designed systems, and AHE, and include cross-model evaluations. The authors also use the diagnosis, planning, and implementation trajectories to fine-tune Qwen3.5-9B, whose resulting harness modifications improve task performance over the initial harness.

Critical strengths
  1. The experiments cover multi-turn tool interactions in τ²-Bench and terminal tasks in Terminal-Bench. The evolved harnesses improve performance on held-out tasks, and cross-model evaluations show that the modifications also benefit different execution models.

  2. The authors train Qwen3.5-9B on diagnosis, planning, and code modification trajectories. The trained model produces harness improvements over the initial harness, showing that structured evolution trajectories can help smaller models learn harness evolution.

Critical weaknesses
  1. The paper lacks sufficient discussion and comparison with closely related work on harness evolution. DGM [1] and HGM [2] study feedback-driven code modification and evolutionary search, while Meta-Harness [3] and AHE [4] likewise analyze execution results, modify harnesses based on that analysis, and evaluate the changes. AHE is particularly close to this paper, with trajectory diagnosis, recorded modification rationales, modification boundaries, and outcome verification. The uncited MOSS [5] explicitly separates diagnosis, planning, and implementation, while HarnessFix [6] connects diagnosis, patch generation, and validation through repair specifications, using validation feedback and repair memory to guide subsequent iterations. These methods already study the stage decomposition and modification control emphasized by DiVE. TTHE [7] also evolves executable harnesses from execution traces. The authors should discuss these works and explain the differences in modification procedures, use of feedback, and candidate evaluation.

  2. The paper lacks sufficient novelty and substantive contributions. Diagnosis-driven modification, recorded modification rationales, and boundary control are already present in AHE; the separation of diagnosis, planning, and implementation is explicit in MOSS; and constrained repair, feedback-based validation, and experience reuse have been studied in HarnessFix (see Weakness 1). These overlaps concern the paper’s main claimed contributions. Relative to AHE, the clearest change is separating repair planning and patch implementation into two roles. Given the existing staged repair approaches, however, changing the role assignment and intermediate output format does not appear to constitute a substantial methodological contribution.

  3. The improvements over AHE are limited, and the paper does not analyze execution variability. On τ²-Bench, the Full average and maximum scores improve by only 1.00 and 0.28 percentage points, respectively; on Terminal-Bench, Pass@1 improves by approximately 1.5 points. Agent performance on these tasks often varies substantially across runs. I would appreciate means and standard deviations from repeated evaluations to establish whether these gains are consistent and reflect the proposed improvements rather than random variation in agent behavior.

  4. The authors should clarify how the best harness is selected during evolution, specifically whether selection is based on the evolution split, a separate validation split, or the test split.

  5. The transferability of the evolved harness requires further analysis. Evolution on Terminal-Bench uses only 20 tasks, and it remains unclear whether the resulting modifications apply to other tasks or primarily patch the failure cases encountered during evolution. I would appreciate an analysis of the actual harness modifications, supported by cross-dataset experiments, to clarify their applicability and whether they overfit the evolution tasks.

[1] Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents

[2] Huxley-Gödel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine

[3] Meta-Harness: End-to-End Optimization of Model Harnesses

[4] Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

[5] MOSS: Self-Evolution through Source-Level Rewriting in Autonomous Agent Systems

[6] From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws

[7] TTHE: Test-Time Harness Evolution

Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer CbLR (Part I)

Thank you for your constructive and thoughtful feedback. We sincerely appreciate the time and care you devoted to reviewing our work. Below are our point-by-point responses to your concerns.

Weakness 1. Detailed Comparison with Existing Work

We have systematically reviewed and discussed nearly 25 state-of-the-art related works in the manuscript, including the representative works you mentioned, Meta-Harness and AHE. Based on your suggestions, we provide more detailed comparisons as follows:

  1. DGM/HGM: Although DGM and HGM use feedback to guide evolution, they use it differently from our method. DGM uses evaluation logs to suggest code changes, while HGM uses task scores to select the version from which exploration continues. In contrast, DiVE divides evolution into three stages and uses feedback to identify which stage requires revision. For example, if a patch generated by the Patch Developer fails in code check, only the Patch Developer revises the patch. The decisions of the Harness Analyst and Repair Planner remain unchanged. Once the patch passes validation, DiVE uses the task evaluation score (feedback) to determine whether diagnosis and planning need to be repeated. This approach prevents local failures from restarting the entire workflow, reducing evolution costs and improving iteration efficiency.
  2. Meta-Harness/AHE: Meta-Harness relies on a single proposer for both diagnosis and modification. AHE assigns diagnosis and implementation to two separate agents. DiVE further decomposes the process into fault diagnosis, repair planning, and code implementation, enabling the system to respond to feedback at different levels of granularity. Additionally, AHE's Evolve Agent is tightly coupled with NexAU, making it difficult to adapt to other frameworks. In contrast, DiVE is framework-agnostic and therefore more broadly applicable.
  3. MOSS/HarnessFix: MOSS uses a seven-stage review pipeline, which adds overhead to each iteration. Moreover, it only evaluates candidates against predefined keypoints across iterations and provides limited evidence that they will generalize to unseen tasks. In contrast, DiVE uses static checks and smoke tests to filter out invalid candidates early and then validates the final result on a held-out test set to provide stronger evidence of generalization. HarnessFix, meanwhile, relies on a predefined operator library and HTIR, limiting its repair space to what the library covers. Instead, DiVE requires no operator library and formulates repair operations as needed, allowing it to adapt more readily to novel defects.
  4. TTHE: TTHE evolves candidates during the test phase using unlabeled trajectories and proxy signals, which may deviate from true task performance. DiVE evolves during the development phase guided directly by task evaluation scores, providing more faithful signals, with final validation on an independent test set that is never involved in evolution, offering stronger generalization guarantees.

In summary, DiVE differs fundamentally from the above methods in terms of evolution pipeline, feedback utilization, repair flexibility, and generalization validation. By separating fault diagnosis, repair planning, and code implementation, DiVE can use feedback to refine each stage independently. It also avoids predefined operators and framework dependencies, making it more adaptable to new defects and settings. Finally, evaluation on an independent test set provides stronger evidence that the evolved harness generalizes to unseen tasks.

AuthorsOpenReview ↗

Response to Reviewer CbLR (Part II)

Weakness 2. Clarification on Novelty and Contributions

In Weakness 1, we have detailed the differences between DiVE and AHE, MOSS, and HarnessFix. Here we further clarify the novelty and main contributions of DiVE:

  1. DiVE is the first to model the harness evolution process as a traceable three-stage decision chain of failure diagnosis, repair planning, and code implementation. In DiVE, Trajectory evidence drives problem diagnosis, diagnosis guides repair planning, and planning goals constrain code implementation. When evolution falls short, errors can be precisely attributed to a specific stage for targeted correction. This design also reflects the methodological value of agent orchestration. Proper role division and responsibility specialization are essential for improving performance and controllability, as evidenced by MLC (ACL 2025) [1], Multi² (ICML 2026) [2], and ExeGen (NeurIPS 2025) [3], all of which treat this as a core contribution.
  2. DiVE is the first to unify positive feedback and negative constraints in a three-stage harness evolution mechanism with historical experience reuse. Positive feedback draws on task evaluation outcomes and accumulated experience to guide diagnosis, repair planning, and iterative refinement. Negative constraints restrict the permitted modification scope and require candidate patches to pass program checks. Together, positive feedback and negative constraints guide the evolution direction, support targeted correction, and reduce unnecessary evaluation overhead.
  3. DiVE provides a flexible and framework-agnostic evolution strategy. Unlike HarnessFix, which relies on a predefined operator library, and AHE, which depends on a specific harness framework, the Planner of DiVE flexibly formulate repair operations within modification boundaries. This design keeps the repair space free from predefined operations, allowing it to address new defects and work across different base harnesses.
  4. DiVE is the first harness evolution framework that supports both iterative evolution and stage-level learning. Linked evolution records supply experience for subsequent iterations and serve as stage-level supervision data for training the three evolution agents, achieving co-evolution of the evolver and the harness. Experiments across diverse task environments and independent test sets further confirm its effectiveness and generalization.

In summary, DiVE addresses the core challenge of controllable and generalizable harness evolution by introducing a traceable decision chain that enables precise error attribution, a unified positive-negative control mechanism that stabilizes evolution direction, and stage-level learning that achieves co-evolution of the evolver and the harness. These contributions collectively advance harness evolution from ad hoc patching toward a principled, self-improving system.

[1] Advancing Collaborative Debates with Role Differentiation through Multi-Agent Reinforcement Learning, ACL 2025

[2] Multi²: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments, ICML 2026

[3] Knowledge Starts with Practice: Knowledge-Aware Exercise Generative Recommendation with Adaptive Multi-Agent Cooperation, NeurIPS 2025

AuthorsOpenReview ↗

Response to Reviewer CbLR (Part III)

Weakness 3. Performance Gains and Stability

To address reviewer concerns about execution variability, we report mean and standard deviation across repeated evaluations in Table 1. For τ²-Bench, which exhibits higher variance, we conducted 4 rounds of evaluation (4 trials each round). For Terminal-Bench, which is more stable, we used 2 rounds.

  1. Performance Gains. As shown in Table 1, while τ²-Bench features strong baselines with limited room, DiVE still demonstrates overall improvements. On the Airline task, AHE regresses relative to Base (83.74 to 81.90). DiVE avoids this regression and achieves a +1.29 gain over Base and a +3.13 gain over AHE. On Terminal-Bench 2.1, DiVE improves Full Pass@1 by 1.5 points over AHE. This greater consistency is especially valuable in agentic settings, where reliable performance matters.
  2. Execution Variability. The additional standard deviation results in Table 1 further demonstrate that DiVE achieves strong consistency across repeated runs. Notably, DiVE attains a standard deviation of 0.00 on both the Terminal-Bench Full and τ²-Bench Telecom tasks, indicating near-deterministic behavior. In contrast, AHE exhibits a standard deviation of 4.17 on the Airline task, reflecting substantial run-to-run variability. These results suggest that DiVE's performance gains are not attributable to random fluctuation, but rather reflect stable and reliable improvements inherent to the method itself.

Table 1. Execution Variability on τ²-Bench (Avg.) and Terminal-Bench 2.1 (Pass@1)

τ²-BenchTerminal-Bench 2.1
MethodAirlineRetailTelecomFullEasyMed.Hard
Base83.74±2.0186.88±1.0899.85±0.3039.9±5.0775.0±0.0046.5±4.6520.5±6.82
AHE81.90±4.1787.33±1.1199.10±0.8144.9±1.4575.0±0.0050.0±1.1629.5±2.27
DiVE85.03±2.1287.95±2.1399.40±0.0046.4±0.0075.0±0.0052.3±1.1629.5±2.27

In summary, DiVE not only improves overall performance but also substantially reduces result variability, a property of considerable practical value in real-world deployment scenarios.

Weakness 4. Selection of the Best Harness

During evolution, candidate harnesses are evaluated on a separate evolution split at each iteration, and the highest-scoring candidate is retained as the starting point for the next round. The held-out test set is used solely for final validation after evolution is complete and plays no role in harness selection, ensuring that no test set information leaks into our reported results.

AuthorsOpenReview ↗

Response to Reviewer CbLR (Part IV)

Weakness 5. Generalizability Analysis

To further assess the generalizability of the DiVE-evolved harness, we analyzed the code differences between the evolved and original harnesses on Terminal-Bench 2.1. We found that the evolved harness introduced general execution safeguards rather than task-specific patches. The added middleware operates through NexAU hooks to prevent stalls, premature termination, and excessive context growth. Key mechanisms include:

  1. Empty-response detection (after_model): After three consecutive responses containing neither text nor tool calls, a system message prompts the agent to change its approach and inspect the environment.
if not has_text and not has_tool_calls:
    count = self._get(..., "stall_empty_count", 0) + 1
    if count >= self.max_consecutive_empty:          # 3
        self._set(..., "stall_empty_count", 0)
        msg = {"role": "system", "content": (
            "You have produced several consecutive responses with no text and no tool calls. "
            "This indicates you are stuck. You must take a different approach immediately. "
            "Use run_shell_command to inspect the environment, list files, or read logs, then proceed.")}
        return HookResult.with_modifications(messages=[msg])
  1. Repeated-failure detection (after_tool): After three consecutive run_shell_command failures (exit_code != 0), the agent is prompted to inspect errors, verify dependencies, or try an alternative approach.
if exit_code is not None and exit_code != 0:
    count = self._get(..., "stall_fail_count", 0) + 1
    if count >= self.max_repeated_failures:         # 3
        ... "Repeatedly retrying the same approach is ineffective. Change your strategy: "
            "inspect the error, verify dependencies, or try an alternative method."
  1. Pre-termination verification (after_model): On the first two responses without tool calls, the hook sets force_continue = True and reminds the agent to check required files, run verification commands, and confirm that tests pass. This reminder count accumulates across the episode.
if not has_tool_calls:
    count = self._get(..., "verification_reminders", 0)
    if count < self.max_reminders:                   # 2
        self._set(..., "verification_reminders", count + 1)
        msg = {"role": "system", "content": (
            "Before finishing, verify that all task constraints are met: "
            "check that required files exist, run the exact verification commands, "
            "and confirm tests pass. If anything is missing, continue working.")}
        parsed.force_continue = True
        return HookResult.with_modifications(messages=[msg], parsed_response=parsed)

The middleware contains no task-specific information or concrete verification commands. Its prompts provide general operational guidance, suggesting that DiVE extracts shared failure patterns from evolution tasks rather than memorizing task-specific solutions.

To further evaluate transferability, we tested the evolved harness on 70 SWE-bench Verified tasks, randomly sampling 10 tasks from each of seven projects.

Table 2. Cross-Benchmark Generalization on SWE-bench Verified

SWE-bench-verifiedBase HarnessAHEDiVE
django90%90%80%
sklearn90%90%90%
pydata70%50%70%
matplotlib60%60%60%
sympy60%60%80%
astropy60%70%70%
sphinx50%70%70%
full68.60%70.00%74.30%

As shown in Table 2, DiVE achieves an overall success rate of 74.3%, outperforming the base harness and AHE by 5.7 and 4.3 percentage points, respectively. Although gains vary across projects, these results provide evidence of cross-benchmark generalization to software repair tasks.

We hope these clarifications address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.