- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents MASH, a framework for harness optimization in multi-agent systems. It diagnoses failures from execution trajectories and merges diagnoses when the evidence supports a shared root cause, reducing repeated diagnosis and repair. It then generates constraint candidates that can coordinate changes across the node, edge and graph levels, and applies three-stage filtering before deciding whether to update the current harness. The paper reports performance gains across four benchmarks and multiple workflows, and analyzes optimization costs and performance across LLM backbones.
Critical strengths
-
The method's design and analysis are motivated by practical problems in MAS harness optimization. The same missing information or failure can appear repeatedly during individual-agent execution and inter-agent information exchange. This can lead multiple constraint candidates to propose similar repairs, increasing optimization costs and potentially reducing effectiveness. MASH reduces unnecessary optimization work by tracing shared root causes, coordinating changes across levels, and filtering candidates.
-
The experiments cover several settings and provide supporting evidence from different analyses. The main experiments compare MASH with several recent harness optimization methods, and MASH achieves the highest mean scores among the evaluated methods on all four benchmarks. Comparisons across workflows and component ablations provide further evidence of its effectiveness.
Critical weaknesses
-
Important implementation details and experimental artifacts are not fully provided. The current code package does not include per-task results or the harnesses before and after optimization.
-
The stability of the performance gains and component contributions remains unclear. The paper reports mean scores but does not provide confidence intervals or results from repeated runs of the full optimization process. The same concern applies to the core component ablations.
Questions for authors
-
What information is required by the failure diagnosis and constraint generation steps? Their inputs and outputs are not sufficiently clear in the problem formulation and the overview figure.
-
How were the thresholds used in the three-stage filtering process chosen? In particular, what is the basis for setting the effectiveness threshold (\theta_{\mathrm{eff}}) to 0.005 and the robustness threshold (\theta_{\mathrm{gen}}) to 0.0025 on Finance-Agent?
-
What guided the choice of baselines and comparison methods in the additional analyses? Methods such as MASPO, discussed in the related work, appear relevant for comparison with MASH. Have the authors considered these comparisons? If they are difficult to conduct under the current setting, please explain why. Also, why does the failure-pattern analysis in Section 5.3 compare against Meta-Harness without including the stronger baseline, HarnessX?
-
How stable are the experimental results? Could the authors report uncertainty estimates for the improvements in the main results table and for the ablation results?
-
Could the authors provide additional experimental artifacts, including the harnesses before and after optimization and the per-task test results?
7 comments
Response to Weaknesses (part 2) and Question 1
Observations of Table 3.1 and 3.2: With the repeated runs, the results show that our proposed method consistently outperforms the related baselines. The ablation results over multiple runs also align with the observations in the original manuscript. Structured failure diagnosis is the most important component, whereas adding cross-level constraint generation alone yields a relatively limited gain. This aligns with the observation in the manuscript. This maybe because without structured diagnosis, propagated failures can lead to ineffective constraints, which can in turn negatively affect subsequent optimization. Combining both components achieves the best performance. We will also revise the manuscript accordingly.
Question 1: What information is required by the failure diagnosis and constraint generation steps? Their inputs and outputs are not sufficiently clear in the problem formulation and the overview figure.
Response: Thank you for your constructive questions. We further clarify the detailed inputs and outputs of the failure diagnosis and constraint generation steps, together with the other steps of MASH, as follows:
Table 3.3: Inputs, outputs, and examples of each step in MASH.
| Step | Input | Output | Example |
|---|---|---|---|
| Task execution | Task instances, the current harness, and candidate changes retained from the preceding round. The initial round uses the original harness. | Execution trajectories containing agent inputs and outputs, tool interactions, inter-agent messages, and information-flow links; task scores and the selected harness after candidate evaluation. | A task requires comparing revenues from the same reporting year. An agent passes a revenue figure without its year, and a downstream agent uses it in a comparison across different years. The trajectory records this information flow and the resulting answer. |
| Failure diagnosis | Execution trajectories and their task outcomes, together with the current harness, paired evaluation results, execution-interface specifications, and prior modification history. | Failure groups organized by shared root cause. Each group contains a structured tuple: failed decision , required information , received information , missing information , and supporting trajectory evidence , with the origin and propagation path retained. | : compare revenues for the same year; : revenue figures with reporting years; : a revenue figure without its year; : the reporting year; : the extraction, handoff, and comparison events. The downstream symptoms are traced to the original omission and consolidated into one failure group. |
| Constraint generation | The selected harness, diagnosed failure groups and their evidence, available modification interfaces and their input/output specifications, and previous modification and evaluation results. | Candidate harness changes, each linked to one failure group and repair hypothesis, specifying the affected node, edge, or graph interfaces, applicability conditions, executable implementations, and behavior checks. | Generate a candidate that requires agents to retain reporting years when extracting revenue figures (node), preserve those years in inter-agent messages (edge), and verify year consistency before combining the figures (graph). These changes address the same diagnosed omission. |
| Fast filtering | Candidate changes, the current harness as the comparison baseline, execution rules and behavior checks, a small diagnosis-task subset, a separate validation set, and acceptance thresholds. | Candidates passing syntax and semantics, effectiveness, and robustness checks, together with evaluation results and rejection reasons. | Check that the year-preservation candidate is valid and implements its declared changes; compare its task scores with the current harness on the small subset; then retain it only if its gains also meet the threshold on the separate validation set. |
Response to Question 2-3 (part 1)
Question 2: How were the thresholds used in the three-stage filtering process chosen? In particular, what is the basis for setting the effectiveness threshold () to 0.005 and the robustness threshold () to 0.0025 on Finance-Agent?
Response: Thank you for your questions. We detail how we select the thresholds and the screening subset size below. We chose these hyperparameters by considering both computational cost and effectiveness. Specifically:
- Screening subset size. We mainly consider the computational cost, as we aim to improve the efficiency of constraint filtering. For datasets that require long test-time reasoning (Finance-Agent and BrowseComp-Plus), we select a relatively small screening subset. In contrast, for PlanCraft and WorkBench, we set the screening subset size to at least 10 at an affordable computational cost.
- Thresholds. Due to the variation in sample and failure distributions across datasets, we select the thresholds empirically for each dataset.
We will also add these descriptions to the manuscript accordingly. The full hyperparameter settings across the four datasets are provided below:
Table 3.4: Hyperparameters of the three-stage filtering across four benchmarks. denotes the screening subset size, the effectiveness threshold, and the robustness threshold.
| Benchmark | |||
|---|---|---|---|
| Finance-Agent | 4 | 0.005 | 0.0025 |
| BrowseComp-Plus | 8 | 0.060 | 0.075 |
| PlanCraft | 12 | 0.100 | 0.025 |
| WorkBench | 10 | 0.150 | 0.075 |
Question 3: What guided the choice of baselines and comparison methods in the additional analyses? Methods such as MASPO, discussed in the related work, appear relevant for comparison with MASH. Have the authors considered these comparisons? If they are difficult to conduct under the current setting, please explain why. Also, why does the failure-pattern analysis in Section 5.3 compare against Meta-Harness without including the stronger baseline, HarnessX?
Response: Thank you for your valuable questions. Here we clarify the choice of baselines in our manuscript and add experiments comparing against MAS structure optimization baselines (MASPO and MASPOB).
(1) Choice of baselines. Related work on MAS optimization broadly falls into two types: MAS harness optimization and MAS structure optimization. In our experiments, we mainly compare with harness approaches, i.e., MAS optimization over the infrastructural space, including tool descriptions, message passing, etc. Related MAS evolution methods such as MASPO have a different focus on MAS workflow optimization, which is an orthogonal optimization direction and can be further equipped with our proposed harness method.
(2) Additional baseline experiments. Following your suggestions, we compare MASH against MASPO and MASPOB [1], two recent representative MAS evolution methods that optimize MAS through collaboration feedback and topological dependencies, respectively, to strengthen the comprehensiveness of the baseline comparisons. This comparison helps examine whether MASH remains advantageous over dedicated multi-agent optimization methods in terms of final task performance.
Experimental setup: We evaluate MASPO and MASPOB under the same setting as the other baselines, i.e., the same MAS workflow, data splits, and evaluation protocol on all four benchmarks. Following our response to Weakness 2, we run the optimization process of each method three times independently with different random seeds. The results are as follows (the per-run results are reported in Table 3.1):
Table 3.5: Test scores (×100, mean ± SD over three independent optimization runs) of MASH and the additional baselines MASPO and MASPOB.
| Method | Finance-Agent | BrowseComp-Plus | PlanCraft | WorkBench |
|---|---|---|---|---|
| MASPO | 45.12 ± 1.19 | 23.56 ± 1.02 | 56.56 ± 1.17 | 41.78 ± 1.39 |
| MASPOB | 43.16 ± 1.15 | 22.78 ± 1.17 | 55.11 ± 0.84 | 39.78 ± 1.50 |
| MASH | 52.04 ± 0.89 | 29.33 ± 0.88 | 60.89 ± 1.64 | 48.22 ± 0.69 |
Observations: From the results, we can find that MASH consistently outperforms MASPO and MASPOB across all four benchmarks, with gains of 4.33–6.92 points over the stronger of the two. We will revise the manuscript accordingly.
Response to Weaknesses (part 1)
Dear Reviewer TBLo, thank you for your careful review and valuable suggestions. We provide further clarifications and additional experiments to address your concerns. If there is any misunderstanding, please let us know, and we will respond quickly.
Response: Thank you for your valuable comments. We provide the implementation details and experimental artifacts in our responses to your specific questions (Question 1, 2, 5). We have also added them to the current code repository.
Response: Thank you for your constructive comments. Following your suggestions, we run the full optimization process three times independently with different random seeds. The results with uncertainty estimates are reported below for both the main results and the ablation study.
Table 3.1: Test scores (×100) across three independent optimization runs.
Table 3.2: Component ablation of MASH on Finance-Agent across three independent optimization runs.