acceptodds
Under review as a conference paper at ICLR 2027

Harness Optimization for Multi-Agent Systems Across Node, Edge, and Graph

Abstract

Multi-agent systems (MAS) show strong potential for complex task solving, but realizing this potential requires effective agent harnesses, which impose infrastructural constraints on execution and collaboration. Existing approaches rely on either manual design, which is labor-intensive and task-dependent, or automatic generation, which can overlook inter-agent and group-level failures, while emphasizing the individual agent harness. Effective MAS harnesses must therefore cover individual execution (i.e., node), inter-agent collaboration (i.e., edge), and group coordination (i.e., graph). However, directly extending the automatic optimization loop of failure diagnosis and constraint generation across these levels introduces two challenges: failure diagnosis duplication, where one underlying failure is repeatedly diagnosed through its propagated symptoms, and constraint candidate explosion, where numerous proposed repairs incur excessive execution costs. We propose MASH, a framework for efficient MAS harness optimization across all three levels. MASH performs structured failure diagnosis by tracing observed symptoms to shared gaps between the information a decision requires and receives, reducing redundant diagnoses and repeated repairs. It then generates coordinated cross-level constraints grounded in each diagnosed failure and applies progressively more expensive quality checks to filter weak candidates before they enter the next optimization round. Experiments across four benchmarks and four MAS topologies demonstrate that MASH consistently outperforms the evaluated baselines, supporting the effectiveness of harness optimization tailored to multi-agent execution. The codes are available at [here](https://anonymous.4open.science/r/MASH-143D/README.md).

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Multi-Agent System, MAS Harness, Harness Optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TBLoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MASH, a framework for harness optimization in multi-agent systems. It diagnoses failures from execution trajectories and merges diagnoses when the evidence supports a shared root cause, reducing repeated diagnosis and repair. It then generates constraint candidates that can coordinate changes across the node, edge and graph levels, and applies three-stage filtering before deciding whether to update the current harness. The paper reports performance gains across four benchmarks and multiple workflows, and analyzes optimization costs and performance across LLM backbones.

Critical strengths
  1. The method's design and analysis are motivated by practical problems in MAS harness optimization. The same missing information or failure can appear repeatedly during individual-agent execution and inter-agent information exchange. This can lead multiple constraint candidates to propose similar repairs, increasing optimization costs and potentially reducing effectiveness. MASH reduces unnecessary optimization work by tracing shared root causes, coordinating changes across levels, and filtering candidates.

  2. The experiments cover several settings and provide supporting evidence from different analyses. The main experiments compare MASH with several recent harness optimization methods, and MASH achieves the highest mean scores among the evaluated methods on all four benchmarks. Comparisons across workflows and component ablations provide further evidence of its effectiveness.

Critical weaknesses
  1. Important implementation details and experimental artifacts are not fully provided. The current code package does not include per-task results or the harnesses before and after optimization.

  2. The stability of the performance gains and component contributions remains unclear. The paper reports mean scores but does not provide confidence intervals or results from repeated runs of the full optimization process. The same concern applies to the core component ablations.

Questions for authors
  1. What information is required by the failure diagnosis and constraint generation steps? Their inputs and outputs are not sufficiently clear in the problem formulation and the overview figure.

  2. How were the thresholds used in the three-stage filtering process chosen? In particular, what is the basis for setting the effectiveness threshold (\theta_{\mathrm{eff}}) to 0.005 and the robustness threshold (\theta_{\mathrm{gen}}) to 0.0025 on Finance-Agent?

  3. What guided the choice of baselines and comparison methods in the additional analyses? Methods such as MASPO, discussed in the related work, appear relevant for comparison with MASH. Have the authors considered these comparisons? If they are difficult to conduct under the current setting, please explain why. Also, why does the failure-pattern analysis in Section 5.3 compare against Meta-Harness without including the stronger baseline, HarnessX?

  4. How stable are the experimental results? Could the authors report uncertainty estimates for the improvements in the main results table and for the ablation results?

  5. Could the authors provide additional experimental artifacts, including the harnesses before and after optimization and the per-task test results?

7 comments
AuthorsOpenReview ↗

Response to Weaknesses (part 1)

Dear Reviewer TBLo, thank you for your careful review and valuable suggestions. We provide further clarifications and additional experiments to address your concerns. If there is any misunderstanding, please let us know, and we will respond quickly.


Weakness 1: Important implementation details and experimental artifacts are not fully provided. The current code package does not include per-task results or the harnesses before and after optimization.

Response: Thank you for your valuable comments. We provide the implementation details and experimental artifacts in our responses to your specific questions (Question 1, 2, 5). We have also added them to the current code repository.


Weakness 2 and Question 4: The stability of the performance gains and component contributions remains unclear. The paper reports mean scores but does not provide confidence intervals or results from repeated runs of the full optimization process. The same concern applies to the core component ablations.

Response: Thank you for your constructive comments. Following your suggestions, we run the full optimization process three times independently with different random seeds. The results with uncertainty estimates are reported below for both the main results and the ablation study.

Table 3.1: Test scores (×100) across three independent optimization runs.

BenchmarkMethodRun 1Run 2Run 3Mean ± SDCV (%)
Finance-AgentGEPA39.3238.2741.0339.54 ± 1.393.52
RHI43.0242.4844.2343.24 ± 0.902.07
ACE38.2337.7536.8637.61 ± 0.701.85
AHE46.8345.4647.1246.47 ± 0.891.91
Life-Harness43.4944.7241.9543.39 ± 1.393.20
Meta-Harness47.2448.7549.3748.45 ± 1.102.26
HarnessX48.7647.4749.8548.69 ± 1.192.45
Meta-Team41.8642.2743.6342.59 ± 0.932.18
AFlow42.8541.7242.1642.24 ± 0.571.35
MASPO45.1246.3043.9345.12 ± 1.192.63
MASPOB43.2841.9644.2543.16 ± 1.152.66
MASH52.8851.1152.1352.04 ± 0.891.71
BrowseComp-PlusGEPA21.0020.0019.3320.11 ± 0.844.17
RHI23.0023.6724.3323.67 ± 0.672.81
ACE20.3319.6719.0019.67 ± 0.673.39
AHE24.0023.3325.6724.33 ± 1.204.94
Life-Harness21.6722.3320.6721.56 ± 0.843.89
Meta-Harness25.0026.6724.3325.33 ± 1.204.74
HarnessX25.3327.6726.6726.56 ± 1.174.41
Meta-Team21.6720.6721.3321.22 ± 0.512.40
AFlow22.3321.6724.0022.67 ± 1.205.30
MASPO23.3324.6722.6723.56 ± 1.024.32
MASPOB21.6722.6724.0022.78 ± 1.175.14
MASH29.0028.6730.3329.33 ± 0.883.01
PlanCraftGEPA51.6751.3352.3351.78 ± 0.510.98
RHI54.3355.0055.3354.89 ± 0.510.93
ACE51.0049.6750.3350.33 ± 0.671.32
AHE55.3356.6754.6755.56 ± 1.021.83
Life-Harness54.0052.6755.0053.89 ± 1.172.17
Meta-Harness56.3358.0056.6757.00 ± 0.881.55
HarnessX56.0054.3355.6755.33 ± 0.881.59
Meta-Team55.0054.3353.6754.33 ± 0.671.22
AFlow57.0056.3355.6756.33 ± 0.671.18
MASPO56.6757.6755.3356.56 ± 1.172.07
MASPOB55.0056.0054.3355.11 ± 0.841.52
MASH61.6759.0062.0060.89 ± 1.642.70
WorkBenchGEPA34.6735.0033.3334.33 ± 0.882.57
RHI38.0037.3340.3338.56 ± 1.584.09
ACE33.6734.0035.3334.33 ± 0.882.57
AHE40.0042.0039.6740.56 ± 1.263.11
Life-Harness42.0040.6743.3342.00 ± 1.333.17
Meta-Harness42.3344.6743.6743.56 ± 1.172.69
HarnessX43.3341.6745.3343.44 ± 1.844.23
Meta-Team40.6739.3341.3340.44 ± 1.022.52
AFlow39.0039.6740.6739.78 ± 0.842.11
MASPO40.6743.3341.3341.78 ± 1.393.32
MASPOB39.6738.3341.3339.78 ± 1.503.78
MASH48.0049.0047.6748.22 ± 0.691.44

Table 3.2: Component ablation of MASH on Finance-Agent across three independent optimization runs.

VariantStructured DiagnosisCross-level GenerationOriginalRun 2Run 3Mean ± SD
Variant 1××48.6047.1849.0248.27 ± 0.96
Variant 2×✓50.5749.0451.4150.34 ± 1.20
Variant 3✓×51.4452.1450.0651.21 ± 1.06
MASH✓✓52.8851.1152.1352.04 ± 0.89
AuthorsOpenReview ↗

Response to Weaknesses (part 2) and Question 1

Observations of Table 3.1 and 3.2: With the repeated runs, the results show that our proposed method consistently outperforms the related baselines. The ablation results over multiple runs also align with the observations in the original manuscript. Structured failure diagnosis is the most important component, whereas adding cross-level constraint generation alone yields a relatively limited gain. This aligns with the observation in the manuscript. This maybe because without structured diagnosis, propagated failures can lead to ineffective constraints, which can in turn negatively affect subsequent optimization. Combining both components achieves the best performance. We will also revise the manuscript accordingly.


Question 1: What information is required by the failure diagnosis and constraint generation steps? Their inputs and outputs are not sufficiently clear in the problem formulation and the overview figure.

Response: Thank you for your constructive questions. We further clarify the detailed inputs and outputs of the failure diagnosis and constraint generation steps, together with the other steps of MASH, as follows:

Table 3.3: Inputs, outputs, and examples of each step in MASH.

StepInputOutputExample
Task executionTask instances, the current harness, and candidate changes retained from the preceding round. The initial round uses the original harness.Execution trajectories containing agent inputs and outputs, tool interactions, inter-agent messages, and information-flow links; task scores and the selected harness after candidate evaluation.A task requires comparing revenues from the same reporting year. An agent passes a revenue figure without its year, and a downstream agent uses it in a comparison across different years. The trajectory records this information flow and the resulting answer.
Failure diagnosisExecution trajectories and their task outcomes, together with the current harness, paired evaluation results, execution-interface specifications, and prior modification history.Failure groups organized by shared root cause. Each group contains a structured tuple: failed decision DD, required information RR, received information II, missing information MM, and supporting trajectory evidence EE, with the origin and propagation path retained.DD: compare revenues for the same year; RR: revenue figures with reporting years; II: a revenue figure without its year; MM: the reporting year; EE: the extraction, handoff, and comparison events. The downstream symptoms are traced to the original omission and consolidated into one failure group.
Constraint generationThe selected harness, diagnosed failure groups and their evidence, available modification interfaces and their input/output specifications, and previous modification and evaluation results.Candidate harness changes, each linked to one failure group and repair hypothesis, specifying the affected node, edge, or graph interfaces, applicability conditions, executable implementations, and behavior checks.Generate a candidate that requires agents to retain reporting years when extracting revenue figures (node), preserve those years in inter-agent messages (edge), and verify year consistency before combining the figures (graph). These changes address the same diagnosed omission.
Fast filteringCandidate changes, the current harness as the comparison baseline, execution rules and behavior checks, a small diagnosis-task subset, a separate validation set, and acceptance thresholds.Candidates passing syntax and semantics, effectiveness, and robustness checks, together with evaluation results and rejection reasons.Check that the year-preservation candidate is valid and implements its declared changes; compare its task scores with the current harness on the small subset; then retain it only if its gains also meet the threshold on the separate validation set.
AuthorsOpenReview ↗

Response to Question 2-3 (part 1)

Question 2: How were the thresholds used in the three-stage filtering process chosen? In particular, what is the basis for setting the effectiveness threshold (θeff\theta_{\mathrm{eff}}) to 0.005 and the robustness threshold (θgen\theta_{\mathrm{gen}}) to 0.0025 on Finance-Agent?

Response: Thank you for your questions. We detail how we select the thresholds and the screening subset size below. We chose these hyperparameters by considering both computational cost and effectiveness. Specifically:

  • Screening subset size. We mainly consider the computational cost, as we aim to improve the efficiency of constraint filtering. For datasets that require long test-time reasoning (Finance-Agent and BrowseComp-Plus), we select a relatively small screening subset. In contrast, for PlanCraft and WorkBench, we set the screening subset size to at least 10 at an affordable computational cost.
  • Thresholds. Due to the variation in sample and failure distributions across datasets, we select the thresholds empirically for each dataset.

We will also add these descriptions to the manuscript accordingly. The full hyperparameter settings across the four datasets are provided below:

Table 3.4: Hyperparameters of the three-stage filtering across four benchmarks. nsn_s denotes the screening subset size, θeff\theta_{\mathrm{eff}} the effectiveness threshold, and θgen\theta_{\mathrm{gen}} the robustness threshold.

Benchmarknsn_sθeff\theta_{\mathrm{eff}}θgen\theta_{\mathrm{gen}}
Finance-Agent40.0050.0025
BrowseComp-Plus80.0600.075
PlanCraft120.1000.025
WorkBench100.1500.075

Question 3: What guided the choice of baselines and comparison methods in the additional analyses? Methods such as MASPO, discussed in the related work, appear relevant for comparison with MASH. Have the authors considered these comparisons? If they are difficult to conduct under the current setting, please explain why. Also, why does the failure-pattern analysis in Section 5.3 compare against Meta-Harness without including the stronger baseline, HarnessX?

Response: Thank you for your valuable questions. Here we clarify the choice of baselines in our manuscript and add experiments comparing against MAS structure optimization baselines (MASPO and MASPOB).

(1) Choice of baselines. Related work on MAS optimization broadly falls into two types: MAS harness optimization and MAS structure optimization. In our experiments, we mainly compare with harness approaches, i.e., MAS optimization over the infrastructural space, including tool descriptions, message passing, etc. Related MAS evolution methods such as MASPO have a different focus on MAS workflow optimization, which is an orthogonal optimization direction and can be further equipped with our proposed harness method.

(2) Additional baseline experiments. Following your suggestions, we compare MASH against MASPO and MASPOB [1], two recent representative MAS evolution methods that optimize MAS through collaboration feedback and topological dependencies, respectively, to strengthen the comprehensiveness of the baseline comparisons. This comparison helps examine whether MASH remains advantageous over dedicated multi-agent optimization methods in terms of final task performance.

Experimental setup: We evaluate MASPO and MASPOB under the same setting as the other baselines, i.e., the same MAS workflow, data splits, and evaluation protocol on all four benchmarks. Following our response to Weakness 2, we run the optimization process of each method three times independently with different random seeds. The results are as follows (the per-run results are reported in Table 3.1):

Table 3.5: Test scores (×100, mean ± SD over three independent optimization runs) of MASH and the additional baselines MASPO and MASPOB.

MethodFinance-AgentBrowseComp-PlusPlanCraftWorkBench
MASPO45.12 ± 1.1923.56 ± 1.0256.56 ± 1.1741.78 ± 1.39
MASPOB43.16 ± 1.1522.78 ± 1.1755.11 ± 0.8439.78 ± 1.50
MASH52.04 ± 0.8929.33 ± 0.8860.89 ± 1.6448.22 ± 0.69

Observations: From the results, we can find that MASH consistently outperforms MASPO and MASPOB across all four benchmarks, with gains of 4.33–6.92 points over the stronger of the two. We will revise the manuscript accordingly.

Reviewer_YdTtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents MASH, a framework for optimizing multi-agent harnesses across individual execution (node), information exchange (edge), and team coordination (graph), while keeping the underlying agents and communication topology fixed. MASH structures failure diagnoses around missing information, merges symptoms attributed to a shared root cause, and generates constraints that can revise multiple levels jointly. It filters candidates through syntax and semantics checks, small-task effectiveness tests, and broader validation before further evaluation. Experiments cover four benchmarks, four fixed workflows, and transfer to two additional backbone models. The results report higher mean test scores than the evaluated baselines, with ablations examining structured diagnosis, cross-level generation, and filtering, alongside analyses of failure patterns and optimization costs.

Critical strengths

The paper develops a clear and well-motivated problem: multi-agent harness optimization must account for information exchange and team coordination alongside individual execution. The node–edge–graph formulation gives the method a coherent structure, and the writing clearly connects failure propagation to the proposed diagnosis and constraint-generation procedures. The experiments provide encouraging evidence that MASH can improve multi-agent performance with manageable optimization costs. These results, together with the clear problem formulation, support the practical value of the approach.

Critical weaknesses
  1. Limited evidence for the reliability of the reported gains. The main results report mean scores without uncertainty estimates or variation across independent optimization runs. Several baselines also use substantially fewer task executions than MASH despite sharing the same budget cap (Table 5). This makes it difficult to assess whether the gains are stable and how much the comparisons depend on search resources.

  2. Insufficient validation of the core diagnosis mechanism. MASH relies on tracing information gaps and merging failures with shared root causes, but the correctness and coverage of these diagnoses are not directly evaluated. Neither the lower diagnosis overlap in Figure 3c nor the agreement on test-time failure labels in Table 6 verifies whether search-time diagnoses were merged correctly.

Questions for authors

Main questions

  1. Can you provide uncertainty estimates for the main score differences and results from repeated optimization on the same data splits? For baselines with substantially fewer task attempts in Table 5, how does MASH compare at matched execution budgets?
  2. Can you audit a sample of search-time diagnoses for incorrect merges and missed shared causes? What fraction of observed failures does the information-gap schema cover, and how does it handle reasoning or calculation errors when the necessary information is available? Please also provide the full diagnosis and generation prompts mentioned in §5.1.
  3. What metric and scorer are used for each benchmark? Please clarify whether you use the official scoring implementations and specify any changes, including the grader models and prompts.
  4. Appendix B.3 specifies the thresholds and four-task screening subset only for the Finance-Agent centralized analyses. What thresholds and screening-set sizes are used elsewhere, and how are they selected?
  5. Could you compare the Re-optimized condition in Figure 4b with 20 optimization rounds starting from H0 on the same target backbone? This would test whether the transferred harness provides a better initialization.

Minor suggestions for improving the paper

  1. Enlarge the example text in Figure 2 and clarify the “Failure1/Failure2” labels.
  2. Replace “acceptance” with “qualification” in the Figure 3a caption. The plot reports Q/N, whereas Table 11 defines acceptance as A/N.
  3. Use consistent terminology across “test score” in Table 1 and “Acc” in Table 3.
  4. Use consistent terminology for “zero-shot transfer” and “frozen transfer,” and briefly define “Base MAS” in the Figure 4 caption.
  5. In the abstract, change “The codes are available at here” to “The code is available here.”
11 comments
AuthorsOpenReview ↗

Response to Weaknesses & Question 1 (part1)

Dear Reviewer YdTt, thank you for your careful review and valuable suggestions. To address your concerns, we add additional experiments and further clarifications. The point-to-point responses are as follows. Please don't hesitate to reach out if there is any misunderstanding. We look forward to engaging with you.


Weakness 1 & Question 1: Limited evidence for the reliability of the reported gains. The main results report mean scores without uncertainty estimates or variation across independent optimization runs. Several baselines also use substantially fewer task executions than MASH despite sharing the same budget cap (Table 5). This makes it difficult to assess whether the gains are stable and how much the comparisons depend on search resources.

Response: Thank you for your constructive comments. Following your suggestion, we ran additional experiments as below.

(1) More independent optimization runs. We ran the optimization process three times with different random seeds and report the mean scores with variations. The results are as follows:

Table 2.1: Test scores (×100) across three independent optimization runs.

BenchmarkMethodRun 1Run 2Run 3Mean ± SDCV (%)
Finance-AgentGEPA39.3238.2741.0339.54 ± 1.393.52
RHI43.0242.4844.2343.24 ± 0.902.07
ACE38.2337.7536.8637.61 ± 0.701.85
AHE46.8345.4647.1246.47 ± 0.891.91
Life-Harness43.4944.7241.9543.39 ± 1.393.20
Meta-Harness47.2448.7549.3748.45 ± 1.102.26
HarnessX48.7647.4749.8548.69 ± 1.192.45
Meta-Team41.8642.2743.6342.59 ± 0.932.18
AFlow42.8541.7242.1642.24 ± 0.571.35
MASPO45.1246.3043.9345.12 ± 1.192.63
MASPOB43.2841.9644.2543.16 ± 1.152.66
MASH52.8851.1152.1352.04 ± 0.891.71
BrowseComp-PlusGEPA21.0020.0019.3320.11 ± 0.844.17
RHI23.0023.6724.3323.67 ± 0.672.81
ACE20.3319.6719.0019.67 ± 0.673.39
AHE24.0023.3325.6724.33 ± 1.204.94
Life-Harness21.6722.3320.6721.56 ± 0.843.89
Meta-Harness25.0026.6724.3325.33 ± 1.204.74
HarnessX25.3327.6726.6726.56 ± 1.174.41
Meta-Team21.6720.6721.3321.22 ± 0.512.40
AFlow22.3321.6724.0022.67 ± 1.205.30
MASPO23.3324.6722.6723.56 ± 1.024.32
MASPOB21.6722.6724.0022.78 ± 1.175.14
MASH29.0028.6730.3329.33 ± 0.883.01
PlanCraftGEPA51.6751.3352.3351.78 ± 0.510.98
RHI54.3355.0055.3354.89 ± 0.510.93
ACE51.0049.6750.3350.33 ± 0.671.32
AHE55.3356.6754.6755.56 ± 1.021.83
Life-Harness54.0052.6755.0053.89 ± 1.172.17
Meta-Harness56.3358.0056.6757.00 ± 0.881.55
HarnessX56.0054.3355.6755.33 ± 0.881.59
Meta-Team55.0054.3353.6754.33 ± 0.671.22
AFlow57.0056.3355.6756.33 ± 0.671.18
MASPO56.6757.6755.3356.56 ± 1.172.07
MASPOB55.0056.0054.3355.11 ± 0.841.52
MASH61.6759.0062.0060.89 ± 1.642.70
WorkBenchGEPA34.6735.0033.3334.33 ± 0.882.57
RHI38.0037.3340.3338.56 ± 1.584.09
ACE33.6734.0035.3334.33 ± 0.882.57
AHE40.0042.0039.6740.56 ± 1.263.11
Life-Harness42.0040.6743.3342.00 ± 1.333.17
Meta-Harness42.3344.6743.6743.56 ± 1.172.69
HarnessX43.3341.6745.3343.44 ± 1.844.23
Meta-Team40.6739.3341.3340.44 ± 1.022.52
AFlow39.0039.6740.6739.78 ± 0.842.11
MASPO40.6743.3341.3341.78 ± 1.393.32
MASPOB39.6738.3341.3339.78 ± 1.503.78
MASH48.0049.0047.6748.22 ± 0.691.44

Observations: From the results, we can observe that our method consistently outperforms the competitive baselines. This validates the effectiveness of our proposed method across different optimization processes, which further strengthens the empirical evidence of the superiority of MASH. We will also update the manuscript accordingly.

AuthorsOpenReview ↗

Response to Weaknesses & Question 1 (part2)

(2) Optimization effectiveness under similar task attempts. Regarding the fair comparison between MASH and baselines under the similar task executions, we follow your suggestion to compare across different methods and report the results as follows:

Table 2.2: Per-method comparison of task attempts and test scores on Finance-Agent before (Original) and after (Extended) extending the task attempts.

MethodTask Attempts (Original)Task Attempts (Extended)Extended / OriginalTest Score (Original)Test Score (Extended)Δ Score
MAS-Centralized00–39.9739.970.00
GEPA19,74219,7421.00×39.3239.320.00
RHI3,48716,9114.85×43.0244.13+1.11
ACE3,11817,0385.46×38.2337.89−0.34
AHE3,27616,8545.14×46.8347.58+0.75
Life-Harness6,91416,9872.46×43.4944.75+1.26
Meta-Harness17,28417,2841.00×47.2447.240.00
HarnessX17,53317,5331.00×48.7648.760.00
AFlow13,68416,8631.23×42.8543.46+0.61
MASH16,94616,9461.00×52.8852.880.00

Observations: From the results, we can observe that our method still achieves significant improvements under similar task attempts, validating the effectiveness of MASH. Besides, it is interesting that some baseline methods (ACE, RHI, AHE, Life-Harness) perform comparably or even worse with more task attempts. This is possibly due to the exploded failure spaces across edge and graph levels, which brings a negative impact on the failure diagnosis and the constraint generation.

Comparison with Meta-Team at task attempts around 230. Note that Meta-Team is excluded from Table 2.2. This is because the native protocol of Meta-Team performs approximately one development pass over the 32 discovery tasks in each fold, triggering an evolution step after each task. Scaling it up to the ~16k task attempts used by the other methods would require roughly 70 passes, which deviates from its original design significantly and is computationally unaffordable. For a fair comparison, we carefully adapted MASH with Meta-Team under the same budget of 236 task attempts:

Table 2.3: Test scores of Meta-Team and MASH on Finance-Agent under the same budget of 236 task attempts.

MethodTask AttemptsTest Score
Meta-Team23641.86
MASH23642.49

Observations: From the results, we can observe that MASH still outperforms Meta-Team under the same budget of 236 task attempts, which shows that the superiority of MASH does not rely on a larger search budget and further validates the effectiveness of our proposed method in a low-budget setting.


Weakness 2: Insufficient validation of the core diagnosis mechanism. MASH relies on tracing information gaps and merging failures with shared root causes, but the correctness and coverage of these diagnoses are not directly evaluated. Neither the lower diagnosis overlap in Figure 3c nor the agreement on test-time failure labels in Table 6 verifies whether search-time diagnoses were merged correctly.

Response: Thank you for your valuable comments. We agree with you that the evaluation of the diagnosis results is also important to further justify our proposed method. To validate the failure merging strategy, we employ LLM-as-judge to evaluate the correctness of the merging results, accompanied by human annotators' agreement on the randomly sampled evaluated results.

Table 2.4: Search-time diagnosis audit on Finance-Agent (centralized, 5 folds): failure consolidation correctness.

MetricCountRate
Correct Merges455/49691.7%
Incorrect Merges41/4968.3%
Missed Shared-cause Pairs113/93512.1%
LLM–Human Agreement143/15592.3%

Observations: From the results, we can observe that 1) we achieve a 91.7% correctness of the merged failures, partially validating the soundness of the proposed diagnosis method for MAS harness. Nonetheless, 2) we also identified that around 12% of failures might still be missed for merging in our method, which might cause repeated constraint generation. However, 3) the final improved performance and the failure reduction in Figure 3c, Table 2, and Table 6 further justify the effectiveness of our proposed method, in both optimization effectiveness and efficiency.


AuthorsOpenReview ↗

Response to Question 2

Question 2: Can you audit a sample of search-time diagnoses for incorrect merges and missed shared causes? What fraction of observed failures does the information-gap schema cover, and how does it handle reasoning or calculation errors when the necessary information is available? Please also provide the full diagnosis and generation prompts mentioned in §5.1.

Response: Thank you for your constructive questions. Following your suggestions, we conducted three additional analyses on (1) the correctness of the merged failures and missed shared causes; (2) the information-gap schema coverage; and (3) the correctness of information-gap and information-sufficient (reasoning/calculation) errors. The results are as follows, and the detailed diagnosis and generation prompts are provided at the end of this rebuttal.

(1) Correctness of the merged failures and missed shared causes. The results are shown in Table 2.4 (in our response to Weakness 2), which we repeat here for convenience:

MetricCountRate
Correct Merges455/49691.7%
Incorrect Merges41/4968.3%
Missed Shared-cause Pairs113/93512.1%
LLM–Human Agreement143/15592.3%

Observations: From the results, we can observe that we achieve a 91.7% correctness of the merged failures, partially validating the soundness of the proposed diagnosis method for MAS harness. Nonetheless, we also identified that around 12% of failures might still be missed for merging in our method, which might cause repeated constraint generation. However, the final improved performance and the failure reduction in Figure 3c, Table 2, and Table 6 further justify the effectiveness of our proposed method, in both optimization effectiveness and efficiency.

(2) Coverage of the information-gap schema.

Following your suggestions, we categorize the failures into information-sufficient (reasoning/calculation) errors and information-gap errors. We then compute the coverage of each of these two types individually, as well as that of mixed errors.

Table 2.5: Search-time diagnosis audit on Finance-Agent (centralized, 5 folds): information-gap schema coverage.

MetricCountRate
Pure Information-gap Applicability1487/208371.4%
Information-gap Applicability (incl. Mixed)1628/208378.2%
Correct Information-gap Diagnosis1233/148782.9%
Overall Structured Diagnosis Coverage1623/208377.9%

Observations: From the results, we can observe that most of the failures are information-gap errors, which is consistent with the failure patterns of MAS, since MAS introduces a large amount of inter-agent information exchange. This indicates that, as expected, the harness of MASH mainly contributes to resolving information-gap errors.

(3) How we handle information-sufficient (reasoning/calculation) errors and how well MASH resolves such errors.

Regarding the small portion of information-sufficient (reasoning/calculation) errors, we have also taken them into account. The information-sufficient errors, though they sit beyond the common scope of harness, might potentially negatively affect the diagnosis process and might lead to unnecessary harness. Specifically, we consider information-sufficient (reasoning/calculation) errors as a type of node-level error. Therefore, in our implementation, the diagnosis reason (e.g., reasoning error or calculation error) is recorded at failure diagnosis step. Then during the merging process, if the information is sufficient and a reasoning or calculation error exists, the diagnosis reason (e.g., calculation error) is directly recorded. The detailed prompt is provided at the end of this rebuttal, and the corresponding code is in the code repository.

Table 2.6: Search-time diagnosis audit on Finance-Agent (centralized, 5 folds): information-sufficient (reasoning/calculation) error analysis.

MetricCountRate
Correct Non-gap Diagnosis308/45567.7%
False Information-gap Attribution49/45510.8%
Missed / Unsupported Diagnosis98/45521.5%

Observations: As shown in Table 2.6, among all information-sufficient (reasoning/calculation) errors, our method successfully diagnoses nearly 70% of them, although a portion of them remains difficult to identify correctly. We agree that how to disentangle the information-sufficient (reasoning/calculation) failures from the failure cases is also a good research topic worth further exploration. We will also add this discussion to the future work.


Reviewer_x8s2OpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies harness optimization for multi-agent systems (MAS), where the harness refers to the infrastructural constraints that govern tool use, information exchange and execution control. The authors mention that existing automatic harness optimization focuses on individual agents and can overlook inter-agent and group-level failures. An effective MAS harness should therefore cover individual execution (node), inter-agent collaboration (edge) and group coordination (graph). Extending the usual optimization loop of task execution, failure diagnosis and constraint generation to all three levels introduces two challenges. The first is failure diagnosis duplication, where one underlying failure is diagnosed repeatedly through its propagated symptoms. The second is constraint candidate explosion, where too many proposed repairs lead to excessive execution costs. To address these, the paper proposes MASH. It performs structured failure diagnosis by tracing symptoms to shared gaps between the information a decision requires and the information it receives. It generates coordinated cross-level constraints for each diagnosed failure, and it filters weak candidates through progressively more expensive quality checks. Experiments on four benchmarks and four MAS topologies show that MASH consistently outperforms the evaluated baselines.

Critical strengths
  1. The node/edge/graph framing is clear and useful, and it is backed by several analyses. On Finance-Agent, the strongest baseline (Meta-Harness) mostly reduces node-level problems, while MASH also tackles edge-level and graph-level problems substantially. Most of MASH’s accepted fixes touch edge or graph rules, and allowing edits at all levels beats node-only edits on all four benchmarks.

  2. It looks like that the learned harness carries over to other models. For eg, the Finance-Agent harness was optimized for Gemini 2.5 Flash-Lite. When used as-is to GPT-4o mini and Claude Haiku 4.5, it still improves results (Fig. 4b). This suggests MASH learns coordination rules that are useful in general. It’s a practical advantage, because a harness may not need re-optimizing every time the backbone model changes.

Critical weaknesses
  1. It is unclear whether the gains come from the node/edge/graph idea or just the general setup. MASH improves by up to 5.3 points even in the workflow where agents don’t exchange messages, where edge-level rules should matter little thus suggesting much of the gain may come from the general setup instead (ie. structured prompts, candidate filtering and a strong optimizer).

  2. The paper says results are averaged over three test repetitions (Sec. 5.1), which suggests the numbers are stable. But as per my understanding all three repetitions run the same final harness, with an executor set to temperature 0. That mostly checks whether the same frozen harness gives the same answers again. The main source of randomness is the optimization itself, each method appears to be optimized only once, so this variation is never measured. It is therefore unclear whether the 3.7–4.7 point gains would hold up in another run. I would recommend repeating the optimization a few times, or at least reporting per-fold Finance-Agent scores.

Questions for authors
  1. Was each method optimized once per benchmark (once per fold on Finance-Agent)? Please report per-fold Finance-Agent scores for MASH, HarnessX and Meta-Harness. Ideally, add at least three independent optimization runs on one non-Finance benchmark.

  2. I would suggest plotting score against total agent runs and total optimizer tokens, to show whether MASH reaches a given score with fewer resources?

  3. Did a human check any of the failure labels? How can one interpret the edge and node failure rates that don’t go down?

6 comments
AuthorsOpenReview ↗

Response to Weakness 1

Dear reviewer x8s2, thank you for your careful review and valuable comments. We have carefully gone through your comments and questions. Following your suggestions, we have run additional experiments and provide further clarifications to justify the effectiveness of our proposed method. If there is any misunderstanding, please let us know, and we will respond quickly.


Weakness 1: It is unclear whether the gains come from the node/edge/graph idea or just the general setup. MASH improves by up to 5.3 points even in the workflow where agents don't exchange messages, where edge-level rules should matter little thus suggesting much of the gain may come from the general setup instead (ie. structured prompts, candidate filtering and a strong optimizer).

Response: Thanks for your valuable comments. To address your concerns, we first provide explanations for the large gain on the independent multi-agent workflow. Then we present additional experimental results to justify the effectiveness of the harness on edge and graph.

(1) Performance gain on the independent multi-agent workflow validates the effectiveness of our proposed fast filtering strategy. We evaluate MASH under four agent workflows, among which the independent workflow involves the weakest information exchange: agents exchange no intermediate messages and interact only through the terminal aggregation. The performance gain in this setting indicates that candidate filtering alone substantially strengthens harness optimization at the node-level failures, which supports the effectiveness of our fast constraint filtering.

(2) Ablation on edge- and graph-level harness optimization. To verify the contribution of the edge and graph levels directly, we conduct a controlled comparison on the three workflows with agent-to-agent message exchange (centralized, decentralized, and hybrid), extending the results in Appendix C.4 (Table 15). We remove the edge- and graph-level harnesses from MASH and retain only the node level, keeping the optimizer, structured diagnosis, filtering pipeline, and budget unchanged. The results are as follows:

  • Results 1: Performance comparison between MASH with and without edge/graph-level harness. From the results on Finance-Agent, we can observe that when the MAS has more inter-agent communications, edge- and graph-level harness will further improve the performance, as expected. This validates the effectiveness of introducing agent communication and team coordination harness explicitly in MAS.

    Table 1.1: Node-only vs. Full MASH on Finance-Agent (test score ×100). Δ denotes Full MASH minus Node-only.

    WorkflowInitial MASNode-onlyFull MASHΔ
    Centralized39.9746.8252.88+6.06
    Decentralized38.2646.7548.97+2.22
    Hybrid39.5547.3950.76+3.37
  • Results 2: Statistics of node-, edge-, and graph-level harness. To analyze whether edge- and graph-level harness is truly being considered in harness optimization, we present the distribution of node-, edge-, and graph-level harness among the accepted harness across four MAS workflows. From the statistics below, we can find that our proposed method indeed considers edge- and graph-level harness that can eliminate the failures and improve the performance.

    Table 1.2: Accepted hook modifications across four workflows on Finance-Agent.

    WorkflowAccepted RevisionsNodeEdgeGraphTotal HooksEdge+Graph Count (Share)
    Centralized292415256440 (62.5%)
    Decentralized272826136739 (58.2%)
    Hybrid302318276845 (66.2%)
    Independent23370104710 (21.3%)

    Table 1.3: Hook-level shares of node, edge, and graph within each workflow on Finance-Agent.

    WorkflowNodeEdgeGraphEdge+Graph
    Centralized37.5%23.4%39.1%62.5%
    Decentralized41.8%38.8%19.4%58.2%
    Hybrid33.8%26.5%39.7%66.2%
    Independent78.7%0.0%21.3%21.3%

AuthorsOpenReview ↗

Response to Weakness 2

Weakness 2: The paper says results are averaged over three test repetitions (Sec. 5.1), which suggests the numbers are stable. But as per my understanding all three repetitions run the same final harness, with an executor set to temperature 0. That mostly checks whether the same frozen harness gives the same answers again. The main source of randomness is the optimization itself, each method appears to be optimized only once, so this variation is never measured. It is therefore unclear whether the 3.7–4.7 point gains would hold up in another run. I would recommend repeating the optimization a few times, or at least reporting per-fold Finance-Agent scores.

Response: Thank you for your constructive comments. Following your suggestions, we run the optimization with different random seeds to account for the randomness in the optimization process. The results are as follows:

Table 1.4: Test scores (×100) across three independent optimization runs.

BenchmarkMethodRun 1Run 2Run 3Mean ± SDCV (%)
Finance-AgentGEPA39.3238.2741.0339.54 ± 1.393.52
RHI43.0242.4844.2343.24 ± 0.902.07
ACE38.2337.7536.8637.61 ± 0.701.85
AHE46.8345.4647.1246.47 ± 0.891.91
Life-Harness43.4944.7241.9543.39 ± 1.393.20
Meta-Harness47.2448.7549.3748.45 ± 1.102.26
HarnessX48.7647.4749.8548.69 ± 1.192.45
Meta-Team41.8642.2743.6342.59 ± 0.932.18
AFlow42.8541.7242.1642.24 ± 0.571.35
MASPO45.1246.3043.9345.12 ± 1.192.63
MASPOB43.2841.9644.2543.16 ± 1.152.66
MASH52.8851.1152.1352.04 ± 0.891.71
BrowseComp-PlusGEPA21.0020.0019.3320.11 ± 0.844.17
RHI23.0023.6724.3323.67 ± 0.672.81
ACE20.3319.6719.0019.67 ± 0.673.39
AHE24.0023.3325.6724.33 ± 1.204.94
Life-Harness21.6722.3320.6721.56 ± 0.843.89
Meta-Harness25.0026.6724.3325.33 ± 1.204.74
HarnessX25.3327.6726.6726.56 ± 1.174.41
Meta-Team21.6720.6721.3321.22 ± 0.512.40
AFlow22.3321.6724.0022.67 ± 1.205.30
MASPO23.3324.6722.6723.56 ± 1.024.32
MASPOB21.6722.6724.0022.78 ± 1.175.14
MASH29.0028.6730.3329.33 ± 0.883.01
PlanCraftGEPA51.6751.3352.3351.78 ± 0.510.98
RHI54.3355.0055.3354.89 ± 0.510.93
ACE51.0049.6750.3350.33 ± 0.671.32
AHE55.3356.6754.6755.56 ± 1.021.83
Life-Harness54.0052.6755.0053.89 ± 1.172.17
Meta-Harness56.3358.0056.6757.00 ± 0.881.55
HarnessX56.0054.3355.6755.33 ± 0.881.59
Meta-Team55.0054.3353.6754.33 ± 0.671.22
AFlow57.0056.3355.6756.33 ± 0.671.18
MASPO56.6757.6755.3356.56 ± 1.172.07
MASPOB55.0056.0054.3355.11 ± 0.841.52
MASH61.6759.0062.0060.89 ± 1.642.70
WorkBenchGEPA34.6735.0033.3334.33 ± 0.882.57
RHI38.0037.3340.3338.56 ± 1.584.09
ACE33.6734.0035.3334.33 ± 0.882.57
AHE40.0042.0039.6740.56 ± 1.263.11
Life-Harness42.0040.6743.3342.00 ± 1.333.17
Meta-Harness42.3344.6743.6743.56 ± 1.172.69
HarnessX43.3341.6745.3343.44 ± 1.844.23
Meta-Team40.6739.3341.3340.44 ± 1.022.52
AFlow39.0039.6740.6739.78 ± 0.842.11
MASPO40.6743.3341.3341.78 ± 1.393.32
MASPOB39.6738.3341.3339.78 ± 1.503.78
MASH48.0049.0047.6748.22 ± 0.691.44

Observations: Based on the three independent optimization runs, we report the result of each run, the mean, the standard deviation, and the coefficient of variation (CV) for MASH and the baseline methods across all four datasets. The results further justify the robustness and effectiveness of our proposed method MASH. We will also revise the manuscript accordingly.


AuthorsOpenReview ↗

Response to Question 1-2

Question 1: Was each method optimized once per benchmark (once per fold on Finance-Agent)? Please report per-fold Finance-Agent scores for MASH, HarnessX and Meta-Harness. Ideally, add at least three independent optimization runs on one non-Finance benchmark.

Response: Thank you for your constructive comments. Yes, following previous work (e.g., Meta-Harness, AHE, and Life-Harness), each method was optimized once per benchmark (once per fold on Finance-Agent), and we reported the results of a single optimization run. Following your suggestions, we report the per-fold Finance-Agent scores for MASH, HarnessX, and Meta-Harness. In addition, we run three independent optimization runs with different random seeds on all four datasets.

(1) Per-fold Finance-Agent scores for MASH, HarnessX, and Meta-Harness.

Table 1.5: Per-fold test scores (×100) of MASH, HarnessX, and Meta-Harness across three independent optimization runs on Finance-Agent under the centralized workflow.

MethodRunFold 1Fold 2Fold 3Fold 4Fold 5Mean
Meta-Harness140.2553.7043.0057.5041.7547.24
240.8055.5043.3560.6043.5048.75
340.4053.8048.3058.6545.7049.37
HarnessX140.2554.9544.2061.0043.4048.76
237.1053.6546.9556.2543.4047.47
339.6056.2050.1059.8043.5549.85
MASH143.6560.2547.7565.8546.9052.88
242.1560.1549.2062.0042.0551.11
344.6559.1048.2064.2044.5052.13

(2) Performance comparison with three independent runs of MASH, HarnessX, and Meta-Harness. The full results of other baselines are reported in our response to Weakness 2 (Table 1.4).

Table 1.6: Performance gains of MASH over competing optimizers (score points). "Original gain" is computed from the original single-run results, and "Mean gain" from the means over three independent optimization runs. 95% CIs are computed with Welch's t-test over the three independent runs.

BenchmarkComparisonOriginal GainMean Gain95% CI
Finance-AgentMASH − HarnessX+4.12+3.35[+0.89, +5.81]
MASH − Meta-Harness+5.64+3.58[+1.28, +5.89]
BrowseComp-PlusMASH − HarnessX+3.67+2.78[+0.36, +5.20]
MASH − Meta-Harness+4.00+4.00[+1.52, +6.48]
PlanCraftMASH − HarnessX+5.67+5.56[+2.17, +8.94]
MASH − Meta-Harness+5.34+3.89[+0.50, +7.28]
WorkBenchMASH − HarnessX+4.67+4.78[+0.79, +8.76]
MASH − Meta-Harness+5.67+4.67[+2.27, +7.06]

Question 2: I would suggest plotting score against total agent runs and total optimizer tokens, to show whether MASH reaches a given score with fewer resources?

Response: Thanks for your suggestions. For different target scores, we count the agent runs and the total optimizer tokens required by MASH and two competitive baselines. The results are as follows:

Table 1.7: Cumulative MAS task attempts required to reach a target score on Finance-Agent (summed over five folds).

MethodScore ≥ 40Score ≥ 44Score ≥ 47
Meta-Harness2,3187,68416,294
HarnessX1,8657,02613,218
MASH2,2102,2103,480

Table 1.8: Cumulative optimizer tokens required to reach a target score on Finance-Agent (five-fold total; millions).

MethodScore ≥ 40Score ≥ 44Score ≥ 47
Meta-Harness7.9625.7359.31
HarnessX4.9619.9438.72
MASH1.131.131.73

Observations: From the results, we can observe that, when reaching the same score, MASH consumes significantly fewer LLM tokens for optimization than Meta-Harness and HarnessX. This further validates the optimization efficiency of our proposed MASH. We will add the new results to the manuscript accordingly.


Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief Summary of Discussions

We sincerely thank the three reviewers for their thoughtful and constructive feedback. We are delighted that the reviewers found the node–edge–graph formulation of MAS harness optimization clear, coherent, and well motivated (Reviewers x8s2, YdTt, TBLo). They also found the paper clear and easy to follow (Reviewers x8s2, YdTt). We are grateful that they recognized the extensive experiments and analyses supporting the effectiveness of MASH (Reviewers x8s2, YdTt, TBLo), the practical value of its manageable optimization cost (Reviewers YdTt, TBLo), and the transferability of the learned harness across LLM backbones (Reviewer x8s2).

The reviewers also raised valuable concerns that helped us strengthen the work. The major concerns included:

  • Stability of the performance gains (Reviewers x8s2, YdTt, TBLo): We repeated the full optimization process three times with different random seeds for MASH and all baselines on all four benchmarks, as well as for the component ablation. MASH consistently achieves the best performance, and its gains over the strongest baselines remain statistically reliable.
  • Contribution of edge- and graph-level harnesses (Reviewer x8s2): A controlled node-only ablation shows that edge- and graph-level harnesses bring clear additional gains on workflows with inter-agent communication, and they account for a large share of the accepted harness modifications.
  • Fair comparison under matched budgets and optimization efficiency (Reviewers YdTt, x8s2): MASH remains superior when the baselines are given matched search budgets, and it reaches the same performance with substantially fewer optimization resources.
  • Additional baselines and implementation details (Reviewers TBLo, YdTt): We added MAS optimization baselines (MASPO and MASPOB), included HarnessX in the failure-pattern analysis, and clarified the step-wise inputs and outputs, the scorers, and the hyperparameters with a sensitivity analysis. The optimized harnesses and per-task results are released in the code repository.

Following the discussion period, we are pleased that all reviewers expressed satisfaction with our responses, which addressed their major concerns, and that Reviewer YdTt raised scores. As the reviewers suggested, the revised manuscript includes all new results and analysis, as well as further clarifications. We extend our heartfelt thanks to the reviewers for their hard work and active engagement during the discussion phase. Their valuable comments have greatly helped us improve the quality of our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.