acceptodds
Under review as a conference paper at ICLR 2027

OpenMAS-GCom: A Diagnostic Benchmark for Graph-Enhanced Multi-Agent Systems

Abstract

Graph-enhanced multi-agent systems (G-MAS) coordinate large language model agents through structured communication and role assignment, which determine information exchange and collaboration patterns. As G-MAS develops toward complex collaborative reasoning, comprehensive evaluation becomes increasingly important. However, existing benchmarks provide limited diagnostic evaluation of collaboration. They mainly evaluate final task performance, providing limited insights into how G-MAS approaches handle information integration, erroneous information propagation, and execution recovery during collaboration. To address this limitation, we propose OpenMAS-GCom, a comprehensive diagnostic benchmark for G-MAS evaluation. Specifically, (1) From the evaluation perspective, OpenMAS-GCom introduces a controlled intervention framework that represents G-MAS through collaboration units, communication links, shared intermediate information, and execution strategies. It enables systematic analysis of organizational components while keeping tasks, models, and resource settings fixed. (2) From the task perspective, OpenMAS-GCom introduces G-MAS-Complex, a new benchmark containing 400 multi-document tasks that require information integration, conflict resolution, and structured answer generation. OpenMAS-GCom covers 29 datasets across six domains and evaluates 17 representative configurations, including single-agent, ordinary multi-agent, and graph-enhanced multi-agent systems. Extensive experiments reveal that graph enhancement does not uniformly improve G-MAS performance. On G-MAS-Complex, the advantage of graph-enhanced approaches is mainly achieved by a small subset of configurations, with graph-based methods showing large performance gaps on increasingly difficult collaboration tasks. Moreover, methods with comparable initial accuracy exhibit substantially different robustness under corrupted information, revealing hidden differences in collaboration reliability beyond final task scores.

Primary area:
datasets and benchmarks
Keywords:
Multi-Agent Systems, Graph-Based Collaboration, Communication Topology, Large Language Models, Benchmarking and Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mpnhOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work proposes OpenMAS-GCom, a comprehensive diagnostic benchmark for evaluating graph-enhanced multi-agent systems. It proposes a controlled intervention framework to analyze the contribution of different components. It also proposes a new benchmark, G-MAS-Complex, containing 400 multi-document tasks.

Critical strengths
  1. The motivation seems interesting: we need more controlled interventions to understand the contribution of each component of MAS systems.
  2. The evaluation is relatively comprehensive for many MAS approaches and benchmarks. The authors also propose a new benchmark, G-MAS-Complex.
Critical weaknesses
  1. The readability of equations in Sec 2.1 is limited. The intervention protocols are not well explained by the equations.
  2. What are "Test-A and Test-B sets of version v6.3.2", "rejected records", etc in Sec 3.1? The readability is limited.
  3. The takeaway is unclear. For example, in Figure 2, the authors report the averaged results of ordinary MAS and Graph-enhanced MAS; in Figure 3, the authors report separate results for different Graph-enhanced MAS results. A lot of MAS approaches are being compared, but without diving into the details of different approaches, it is very hard to understand why some of them are better at math and why some of them are better at code.
  4. It is also unclear to me what extra value G-MAS-Complex provides compared to previous benchmarks.
Questions for authors

See weakness above.

6 comments
AuthorsOpenReview ↗

Answer to w1, we have clarified the mathematical formulation in Section 2 and explicitly connected the four intervention protocols to the organizational components they modify.

(W1) Short answer. We have clarified the mathematical formulation in Section 2 and explicitly connected the four intervention protocols to the organizational components they modify.

We thank the reviewer for pointing out this issue. We appreciate the reviewer's careful attention to the presentation and the helpful suggestion to make the formulation easier to follow. We have improved the explanation of Equations (1) to (7) by clarifying each component in the organization representation and its role in collaborative execution. Equation (6) now defines the four intervention operators, and Section 3.3 links each practical protocol to its corresponding operator. We have also explained how the modified organization produces a complete execution trace and how Equation (7) measures its score difference from the reference execution.

We thank the reviewer again for helping us improve the readability of the formulation and its connection to the experimental protocols.

AuthorsOpenReview ↗

Answer to w2, we have clarified the release version, the Test-A and Test-B collections, and the definitions of candidate, governing, and rejected records.

(W2) Short answer. We have clarified the release version, the Test-A and Test-B collections, and the definitions of candidate, governing, and rejected records.

We thank the reviewer for pointing out this issue. We appreciate the reviewer highlighting the terms that would benefit from a clearer explanation for readers. Version v6.3.2 identifies the evaluated G-MAS-Complex release. Test-A and Test-B are two disjoint test collections containing 80 and 320 tasks, respectively. Their union provides 100 tasks in each of four difficulty tiers. A candidate record specifies a variable, authority level, revision number, and expression. The governing record has the highest authority, with revision used to break ties. Rejected records are the remaining candidates for that variable. They remain in the input documents as conflicting alternatives and are excluded from subsequent computation. We have clarified these terms in Section 3.1, added the selection rule in Appendix A.1, and expanded the illustrative example in Appendix B.1.

Table R3.1. Illustration of candidate-record selection for one variable.

DocumentVariableAuthorityRevisionStatus
D1v00,01v_{00,01}29Rejected
D2v00,01v_{00,01}34Selected
D3v00,01v_{00,01}27Rejected

We thank the reviewer again for helping us make the dataset description more accessible and easier to follow.

AuthorsOpenReview ↗

Answer to w3, our main takeaway is that communication graphs can improve collaboration by directing intermediate results and verification feedback to the agents that need them. Different graph designs support these exchanges differently across tasks.

(W3) Short answer. Our main takeaway is that communication graphs can improve collaboration by directing intermediate results and verification feedback to the agents that need them. Different graph designs support these exchanges differently across tasks.

We sincerely thank the reviewer for this helpful suggestion. We are grateful for the reviewer's thoughtful feedback on the discussion and the opportunity to explain the intended insights more clearly. Figure 2 compares the best-performing configurations within ordinary MAS and graph-enhanced MAS groups, while Figure 3 examines individual graph-enhanced configurations across domains. The organizational mechanisms summarized in Appendix B.5 help interpret these results in terms of how agents exchange and use intermediate information.

A communication graph specifies which agents receive a candidate result, which agents check it, and where the resulting feedback is delivered. Together with execution rules, these connections can ensure that a downstream agent receives the inputs needed for its calculation and that a corrected result reaches the agents whose work depends on it. This provides a concrete way for graph design to improve collaboration over a fixed interaction protocol that does not organize these exchanges around the task's requirements.

These requirements differ between mathematics and code generation. In arithmetic expression search, agents can explore partial expressions, check whether combinations satisfy the rules, and pass valid candidates for further composition. In code generation, useful exchanges connect implementation with checks of input requirements, interfaces, and program behavior, then return specific feedback for revision. GoAgent selects and connects task-relevant groups, providing a way to coordinate candidate exploration and integration. ARGDes jointly generates agent roles and communication links, allowing differentiated construction and checking roles to be connected according to the task. Our interpretation is that these organizational differences can favor different patterns of candidate generation and feedback. In our results, GoAgent's mathematics lead among the four configurations in Figure 3 is concentrated in Game-of-24, where it scores 74.08% compared with ARGDes's 67.11%. ARGDes exceeds GoAgent by 9.75 and 4.88 percentage points on HumanEval and HumanEval++, respectively.

The controlled interventions support the importance of these communication choices. With GDesigner's agents, roles, model, and budget limits fixed, rewiring 70% of its edges while preserving node degrees reduces the five-dataset mean by 8.19 percentage points. Thus, the same agents and number of connections produce different outcomes when information follows different routes. The practical implication is to design connections around the intermediate results and feedback required by each task, and to evaluate whether those exchanges improve the final output.

Table R3.2. Organizational mechanisms and the task requirements they are designed to address.

Mechanism familyOrganizational operationRelevant task demands
Graph construction and optimizationConstructing or optimizing communication links among agentsInformation exchange, dependency coordination, candidate integration
Dynamic selection and routingSelecting agents or routing tasks according to task characteristicsAdaptive allocation, variable task complexity, selective collaboration
Group-based coordinationOrganizing agents into groups and coordinating group communicationParallel candidate generation and multi-stage collaboration
Verification-oriented coordinationChecking candidate outputs and mapping results to supporting informationConsistency checking, factual validation, structured output verification

We thank the reviewer again for encouraging a more explicit discussion of the results and their implications for graph-based collaboration.

Reviewer_EQspOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a diagnostic benchmark for graph-enhanced multi-agent systems. It represents an organization through agents, communication edges, shared state, and execution policy, then measures score changes under topology rewiring, role removal, incorrect messages, and worker failures. The benchmark combines 29 existing datasets with 400 newly constructed multi-document tasks called G-MAS-Complex and evaluates 17 configurations. The reported results show that graph enhancement gives uneven gains across tasks and that configurations with similar original accuracy can respond differently to corrupted messages or worker failures. The paper also reports token and latency costs to separate high accuracy from efficient execution.

Critical strengths
  1. The benchmark evaluates several different failure modes through a common interface rather than relying only on final accuracy. Section 3.3 and Appendix A.4 specify degree-preserving rewiring, format-valid message corruption, and a worker-failure schedule. Figure 4c gives a useful example where similar original scores hide quite different responses to corrupted information.

  2. The empirical coverage is broad, and the resource analysis makes the main results more informative. Table 5 shows that GoAgent's highest G-MAS-Complex score comes with much higher token and latency costs, while VeriMap achieves a different operating point. Sections 4.2 to 4.4 also appropriately describe group maxima and central-score comparisons instead of claiming a uniform benefit from graph enhancement.

Critical weaknesses
  1. I am not yet convinced that the paper establishes the validity of G-MAS-Complex. Section 3.1 and Appendix A.1 give tier sizes and task-scale statistics, and Appendix B.1 gives a small illustrative task. I could not find an explanation of the generation procedure, template diversity, or how the Oracle and six required outputs were checked. Section 4.9 says component accuracy and format failures are recorded, but the paper does not show those results. Strict EM requires all six outputs to agree, so a low score could reflect source or checkpoint reporting rather than failed information integration. I see this as a fixable issue, but it affects the central new benchmark. I would like to see the existing validation procedure and component breakdown to better understand what the benchmark diagnoses.

  2. My concern is that the common adapter does not yet establish whether the method comparisons preserve the relevant original mechanisms. Appendix A.10 identifies the two direct model mappings, and Appendix B.3 lists fields for model assignments, learning checkpoints, and adapter changes. Appendix B.5 describes an official-implementation validation protocol, but does not present the corresponding validation outcomes or instantiated settings for the graph methods. Table 1 shows very large differences on G-MAS-Complex, and learned topology, routing, verification access, and backend choice can all affect them. I appreciate that the paper reports resource costs and limits its claims to evaluated configurations. I still need a concise method-by-method configuration and validation summary to interpret these configurations as a reliable benchmark comparison.

Questions for authors
  1. I would like the authors to summarize how the 400 tasks and their Oracle outputs were generated and validated. I would also like them to use the already recorded results to explain which of the six output components and which format failures account for most of the gap between GoAgent, VeriMap, and the DeepSeek reference.

  2. I would like the authors to provide the model assignments, retained learning or graph-generation mechanisms, and official-implementation validation outcomes for the main graph baselines. I would also like to know whether their policies or checkpoints were adapted using data separate from the 400 evaluation tasks.

4 comments
AuthorsOpenReview ↗

Answer to (w1,q1), we have added the task-generation and Oracle-validation procedures and reported the six-component accuracies and format-failure results. The larger component gaps between GoAgent and the two references occur in source-document attribution and critical checkpoints.

(W1, Q1) Short answer. We have added the task-generation and Oracle-validation procedures and reported the six-component accuracies and format-failure results. The larger component gaps between GoAgent and the two references occur in source-document attribution and critical checkpoints.

We sincerely thank the reviewer for identifying these important issues. We are grateful for the reviewer's careful reading and detailed suggestions on both task construction and the interpretation of the results. G-MAS-Complex v6.3.2 combines two disjoint test collections into 400 frozen tasks, with 100 tasks in each difficulty tier. For each target variable, the generator creates candidate records containing an authority level, a revision number, and an expression. It selects the governing record by authority first and revision second, retains the remaining candidates as conflicting records, and distributes and shuffles the records across source documents. The Oracle answers are computed from the governing records and deterministic task rules. The Oracle graph remains on the evaluator side.

Section 3.1 and Appendix A.1 now explain this construction process and the validation checks. Table R2.1 specifies the tier settings, including document counts, candidate counts, and minimum context lengths. The validator checks required fields, task identifiers, split assignments, prompt overlap, consistency between rendered tasks and reference answers, and variation in public task structures within each tier. These details clarify both how the tasks are generated and how the frozen release is checked.

We have also reported all six component accuracies for GoAgent, VeriMap, and DeepSeek Direct in Table R2.2 and their format-failure counts in Table R2.3, both in Appendix B.2. Section 4.9 summarizes selected results. GoAgent exceeds VeriMap by 20.50 percentage points on source-document attribution and 20.75 points on critical checkpoints, compared with 10.50 points on final-layer values. The corresponding gaps over DeepSeek Direct are 22.25, 23.50, and 10.25 points. These comparisons locate the larger component gaps in identifying supporting documents and reporting required intermediate checkpoints.

Format-failure rates are 4.75%, 4.25%, and 6.75% for GoAgent, VeriMap, and DeepSeek Direct, respectively. VeriMap produces fewer format failures than GoAgent while achieving lower Strict EM, indicating that the performance gap also involves the correctness of the returned content. Appendix B.2 explains how Strict EM evaluates the joint correctness of all six components. Since component errors can overlap within a task, their marginal accuracy gaps describe where performance differs rather than an additive allocation of the Strict EM gap.

Table R2.1. Task-generation configuration of G-MAS-Complex v6.3.2.

TierDocs.TargetsCandidates/ targetMission candidatesMin. charsVerifier nodes
Medium523310,0002
Hard727516,0003
Ultra10211824,0004
Extreme14214936,0005

Table R2.2. Component accuracy in percent on G-MAS-Complex v6.3.2.

MethodValuesMissionChecksumRankingSourcesCheckpointsStrict EM
GoAgent60.2557.7555.0061.2552.5050.2550.25
VeriMap49.7545.2536.5044.0032.0029.5029.50
DeepSeek Direct50.0048.0034.2546.7530.2526.7526.75

Table R2.3. Output-format counts over 400 tasks in four mutually exclusive categories.

MethodTasksJSON parse failureSchema failureMissing required fieldsValid format
GoAgent400649381
VeriMap400386383
DeepSeek Direct40010512373

We thank the reviewer again for these detailed comments, which helped us provide a clearer account of the benchmark and its evaluation results.

AuthorsOpenReview ↗

Answer to (w2,q2), appendix B.3 now reports the model assignments, configured organizational mechanisms, and task-level trace-audit results for all 17 configurations. Training or adaptation used data separate from the 400 evaluation tasks, while other methods used predetermined configurations.

(W2, Q2) Short answer. Appendix B.3 now reports the model assignments, configured organizational mechanisms, and task-level trace-audit results for all 17 configurations. Training or adaptation used data separate from the 400 evaluation tasks, while other methods used predetermined configurations.

We sincerely thank the reviewer for raising this important concern. We appreciate the reviewer's attention to the implementation details and the opportunity to clarify how the evaluated configurations were prepared. All 15 MAS configurations use DeepSeek-V4-Flash and the official implementations released by their authors through the common benchmark interface. DeepSeek Direct uses deepseek-v4-flash, and Qwen Direct uses qwen-plus. Table R2.4 summarizes each method's implementation, evaluated organization, backbone, and trace-audit result. The reported mechanisms include task-dependent graph construction in GDesigner, routing in MASRouter, group-based communication in GoAgent, and verification and recovery procedures in VeriMap. The common interface handles task loading, model access, answer parsing, scoring, and execution logging.

All 136 evaluation batches passed the task-level trace audit, covering 6,800 method-task traces. Appendix B.3 specifies the audit records, which link task identifiers and configurations to predictions, scores, execution information, and resource use. Unit, role, link, and message records describe the configured organizations alongside their outputs.

We have also clarified the separation between method preparation and evaluation. The 400 G-MAS-Complex tasks and their evaluation results were reserved for testing and were not used to train, fine-tune, or select policies, checkpoints, or method configurations. Methods requiring training or adaptation used separate data, while the remaining methods used predetermined configurations.

AuthorsOpenReview ↗

Answer to (w2,q2) (continued)

Table R2.4. Method configuration and validation summary for the 17 evaluated configurations on G-MAS-Complex. Each row reports the implementation, configured organization, backbone model, and task-level trace-audit result.

MethodSource / implementation statusEvaluated organization / mechanismBackboneValidation result
ARG-DesignerOfficial implementation released by the authorsRole assignment and graph generation in the official implementationDeepSeek-V4-FlashTrace audit passed
AutoGenOfficial implementation released by the authorsConfigured agent conversation through the official frameworkDeepSeek-V4-FlashTrace audit passed
BigMASOfficial implementation released by the authorsMethod-specific organization from the official implementationDeepSeek-V4-FlashTrace audit passed
DeepSeek DirectSingle-model referenceDirect call to deepseek-v4-flashDeepSeek-V4-FlashTrace audit passed
DyLANOfficial implementation released by the authorsConfigured team selection and execution policyDeepSeek-V4-FlashTrace audit passed
EIB LearnerOfficial implementation released by the authorsExpert integration with method-specific configurationDeepSeek-V4-FlashTrace audit passed
GDesignerOfficial implementation released by the authorsTask-dependent graph construction with method-specific configurationDeepSeek-V4-FlashTrace audit passed
GoAgentOfficial implementation released by the authorsGroup-based agent communication and coordinationDeepSeek-V4-FlashTrace audit passed
GPTSwarmOfficial implementation released by the authorsGraph-based agent coordination with method-specific configurationDeepSeek-V4-FlashTrace audit passed
Graph-R1Official implementation released by the authorsGraph-reasoning configuration from the official implementationDeepSeek-V4-FlashTrace audit passed
GraphSearchOfficial implementation released by the authorsGraph-based reasoning and retrieval configurationDeepSeek-V4-FlashTrace audit passed
MASRouterOfficial implementation released by the authorsTask-dependent routing configurationDeepSeek-V4-FlashTrace audit passed
MultiAgent DebateOfficial implementation released by the authorsConfigured multi-agent debate protocolDeepSeek-V4-FlashTrace audit passed
Qwen DirectSingle-model referenceDirect call to qwen-plusQwen-PlusTrace audit passed
R-GFMOfficial implementation released by the authorsGraph-learning configuration from the official implementationDeepSeek-V4-FlashTrace audit passed
Sparse Communication TopologyOfficial implementation released by the authorsConfigured sparse agent communicationDeepSeek-V4-FlashTrace audit passed
VeriMapOfficial implementation released by the authorsConfigured verification and recovery proceduresDeepSeek-V4-FlashTrace audit passed

We thank the reviewer again for this careful assessment and for helping us make the configuration and validation information more accessible.

I would like to thank the authors for their thoughtful rebuttal. I agree with the points presented, which have largely resolved my doubts, and I will maintain my positive rating as I lean towards recommending the paper for acceptance.

Reviewer_86frOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OpenMAS-GCom, a benchmark for graph-enhanced multi-agent systems. OpenMAS-GCom integrates 29 public datasets and a newly constructed multi-document task dataset, G-MAS-Complex, to evaluate multi-agent collaboration. The benchmark evaluates 17 configurations, including single-agent, ordinary multi-agent, and graph-enhanced multi-agent systems. In addition to task performance evaluation, it introduces four types of controlled interventions to analyze the effects of communication structure, agent roles, information corruption, and execution failures.

Critical strengths
  1. Comprehensive evaluation. OpenMAS-GCom provides a unified framework for evaluating 17 configurations across 29 public datasets and the newly constructed G-MAS-Complex benchmark, covering diverse tasks and collaboration methods.
  2. Diagnostic evaluation. The four types of controlled interventions enable a more fine-grained analysis of multi-agent collaboration, revealing differences in organizational sensitivity and robustness that cannot be captured by final task accuracy alone.
Critical weaknesses
  1. Limited evaluation of the proposed framework. The four interventions are evaluated on only five selected MAS methods, with no single method tested under all four interventions. Moreover, none of the interventions are evaluated on the newly introduced G-MAS-Complex benchmark, limiting the generalizability of the diagnostic findings.
  2. Insufficient control over backbone models in the main experiments. The paper does not clearly specify the backbone LLMs used by the 15 MAS configurations or establish whether they are consistent across methods. Without controlling for model differences, it is difficult to attribute performance gains to the multi-agent architectures rather than the underlying LLMs.
Questions for authors
  1. Could the authors clarify which backbone LLMs are used by each MAS configuration in Table 1? Have the authors conducted comparisons using the same backbone model to isolate the effects of multi-agent architectures?
  2. Why are the four interventions evaluated on only a small subset of MAS methods, and why are none applied to G-MAS-Complex? Could the authors provide broader intervention results to demonstrate the generalizability of the proposed diagnostic framework?
4 comments
AuthorsOpenReview ↗

Answer to (w1,q2), we have expanded the evaluation to include all four interventions for each of four representative G-MAS configurations on G-MAS-Complex, together with supplementary intervention results on six public datasets.

(W1, Q2) Short answer. We have expanded the evaluation to include all four interventions for each of four representative G-MAS configurations on G-MAS-Complex, together with supplementary intervention results on six public datasets.

We thank the reviewer for this valuable suggestion. We appreciate the reviewer taking the time to examine the breadth of our evaluation and suggest how it could be presented more fully. Following the reviewer's suggestion, we have added comparisons of GDesigner, MASRouter, GoAgent, and VeriMap under structure, node, information, and execution interventions on all 400 frozen G-MAS-Complex tasks. These methods cover graph generation, task-dependent routing, group-level communication, and verification-oriented collaboration. Appendix A.5 presents their Strict EM under the baseline and four intervention levels, specifies the intervention settings, and reports supplementary results on six public datasets.

The expanded results show how intervention responses distinguish the evaluated configurations. At the highest information-intervention level, GoAgent and VeriMap achieve similar Strict EM scores of 16.00% and 15.75%, but their decreases from the baseline are 34.25 and 13.75 percentage points, respectively. Thus, similar scores after intervention can accompany substantially different losses from the original configuration. These additions extend the diagnostic comparison across collaboration designs and task settings.

Table 7. Method coverage of the currently available expanded broad-suite intervention records.

MethodOrganizationNodeInformationExecution
GDesignerOriginalOriginalNot reportedNot reported
MASRouterExpandedExpandedNot reportedNot reported
MADebateNot reportedNot reportedOriginal and expandedNot reported
VeriMapNot reportedNot reportedOriginal and expandedNot reported
GoAgentNot reportedNot reportedNot reportedOriginal and expanded
AutoGenNot reportedNot reportedNot reportedOriginal and expanded

Table 8. MASRouter fixed-full organization results on six public datasets.

DatasetReference (%)Intervention (%)Δ (pp)
FinQA37.1437.140.00
GSM8K93.7192.72-0.99
HumanEval76.8376.22-0.61
MedQA89.2489.32+0.08
MMLU-Pro71.2070.80-0.40
WikiTableQuestions78.4078.20-0.20
Equal-weight mean74.4274.07-0.35

Table 9. MASRouter node-intervention results on six public datasets.

DatasetBaseNo criticNo specialistExpert only
FinQA37.1438.5439.8439.06
GSM8K93.7193.6393.5693.18
HumanEval76.8377.4484.1579.88
MedQA89.2489.0088.8589.79
MMLU-Pro71.2069.8071.8071.60
WikiTableQuestions78.4079.0079.4079.60
Equal-weight mean74.4274.5776.2775.52
Change (pp)N/A+0.15+1.85+1.10

Table 10. Prompt-level incorrect-hint results on six public datasets.

DatasetMAD BaseMAD Int.VeriMap BaseVeriMap Int.
FinQA39.4123.6339.6726.24
GSM8K96.4434.3496.5135.10
HumanEval85.9855.4986.5956.71
MedQA89.8731.9791.2034.96
MMLU-Pro67.6044.0068.6043.80
WikiTableQuestions86.6058.0084.6062.00
Equal-weight mean77.6541.2477.8643.14

Table 11. Incorrect-hint following rates over all assigned tasks.

MethodReference (%)Intervention (%)
MADebate0.5825.34
VeriMap0.7025.02
AuthorsOpenReview ↗

Answer to (w1,q2) (continued)

Table 12. Single-worker failure and bypass-recovery results on six public datasets.

DatasetAutoGen Δ (pp)GoAgent Δ (pp)
FinQA+0.52-0.26
GSM8K-0.30-0.23
HumanEval0.00-3.66
MedQA+0.47+0.86
MMLU-Pro+3.20+1.80
WikiTableQuestions-0.20+0.60
Equal-weight mean change+0.62-0.15

Table 13. Target intensities for each intervention in the four-method comparison.

InterventionBaseLevel 1Level 2Level 3Level 4
Structure, original edges rewired0%10%30%50%70%
Node, workers removed0%25%50%75%90%
Information, eligible messages corrupted0%20%40%60%80%
Execution, workers unavailable0%25%50%75%90%

Table 14. G-MAS-Complex Strict EM (%). Table 13 defines intervention intensities.

MethodInterventionBaseLevel 1Level 2Level 3Level 4
GDesignerStructure14.5013.0011.509.507.75
GDesignerNode14.5017.2513.0015.008.50
GDesignerInformation14.5014.0012.7510.256.00
GDesignerExecution14.5018.2518.2514.0010.25
MASRouterStructure10.7511.009.759.508.25
MASRouterNode10.7511.5010.008.508.50
MASRouterInformation10.759.007.004.503.00
MASRouterExecution10.7512.0012.2511.008.00
GoAgentStructure50.2549.2547.7540.7533.25
GoAgentNode50.2549.5044.0039.5034.25
GoAgentInformation50.2542.7536.5029.2516.00
GoAgentExecution50.2552.2550.2545.7537.50
VeriMapStructure29.5029.0028.0022.0019.25
VeriMapNode29.5028.7525.5025.2519.50
VeriMapInformation29.5029.2529.0022.7515.75
VeriMapExecution29.5031.5029.2527.0021.75

Table 15. Six-component accuracy in percent under interventions on G-MAS-Complex.

MethodInterventionValuesMissionChecksumRankingSourcesCheckpoints
GDesignerStructure52.5048.2554.2558.7547.7554.75
GDesignerNode45.0041.0037.0065.5040.7546.50
GDesignerInformation49.2546.0034.0074.0031.7537.00
GDesignerExecution46.7543.0047.2559.2547.2552.25
MASRouterStructure48.7552.7556.0052.7541.0048.50
MASRouterNode47.5049.2547.7556.2537.2543.00
MASRouterInformation43.5045.7535.0060.0024.2527.75
MASRouterExecution46.0048.2549.0048.5035.2542.75
GoAgentStructure66.7567.5078.5062.5060.2565.50
GoAgentNode63.2563.5062.0073.5054.0058.25
GoAgentInformation59.7560.2550.2580.5039.0043.00
GoAgentExecution65.7565.2573.2565.0060.0065.00
VeriMapStructure53.7554.2559.7557.2558.0067.75
VeriMapNode48.0049.5046.7562.7552.2561.00
VeriMapInformation51.5052.7548.7570.7543.7551.75
VeriMapExecution48.2549.0056.2552.2554.0060.50

We thank the reviewer again for this constructive suggestion, which helped us present the scope and purpose of the additional comparisons more clearly.

AuthorsOpenReview ↗

Answer to (w2,q1), all 15 MAS configurations use the same backbone LLM, DeepSeek-V4-Flash. The two single-agent references, DeepSeek Direct and Qwen Direct, use DeepSeek-V4-Flash and Qwen-Plus, respectively.

(W2, Q1) Short answer. All 15 MAS configurations use the same backbone LLM, DeepSeek-V4-Flash. The two single-agent references, DeepSeek Direct and Qwen Direct, use DeepSeek-V4-Flash and Qwen-Plus, respectively.

We sincerely thank the reviewer for pointing out this omission. We appreciate the opportunity to clarify this important experimental setting and make the comparison easier to assess. All 15 MAS configurations were evaluated with DeepSeek-V4-Flash, so the comparisons among MAS configurations share the same backbone model. DeepSeek Direct also uses this backbone and provides the corresponding single-agent reference. We have explicitly specified the shared MAS backbone in Section 4.1 and provided the model assignments for all 17 configurations in Appendix B.3 and Table R2.4.

Table R2.4. Method configuration and validation summary for the 17 evaluated configurations on G-MAS-Complex. Each row reports the implementation, configured organization, backbone model, and task-level trace-audit result.

MethodSource / implementation statusEvaluated organization / mechanismBackboneValidation result
ARG-DesignerOfficial implementation released by the authorsRole assignment and graph generation in the official implementationDeepSeek-V4-FlashTrace audit passed
AutoGenOfficial implementation released by the authorsConfigured agent conversation through the official frameworkDeepSeek-V4-FlashTrace audit passed
BigMASOfficial implementation released by the authorsMethod-specific organization from the official implementationDeepSeek-V4-FlashTrace audit passed
DeepSeek DirectSingle-model referenceDirect call to deepseek-v4-flashDeepSeek-V4-FlashTrace audit passed
DyLANOfficial implementation released by the authorsConfigured team selection and execution policyDeepSeek-V4-FlashTrace audit passed
EIB LearnerOfficial implementation released by the authorsExpert integration with method-specific configurationDeepSeek-V4-FlashTrace audit passed
GDesignerOfficial implementation released by the authorsTask-dependent graph construction with method-specific configurationDeepSeek-V4-FlashTrace audit passed
GoAgentOfficial implementation released by the authorsGroup-based agent communication and coordinationDeepSeek-V4-FlashTrace audit passed
GPTSwarmOfficial implementation released by the authorsGraph-based agent coordination with method-specific configurationDeepSeek-V4-FlashTrace audit passed
Graph-R1Official implementation released by the authorsGraph-reasoning configuration from the official implementationDeepSeek-V4-FlashTrace audit passed
GraphSearchOfficial implementation released by the authorsGraph-based reasoning and retrieval configurationDeepSeek-V4-FlashTrace audit passed
MASRouterOfficial implementation released by the authorsTask-dependent routing configurationDeepSeek-V4-FlashTrace audit passed
MultiAgent DebateOfficial implementation released by the authorsConfigured multi-agent debate protocolDeepSeek-V4-FlashTrace audit passed
Qwen DirectSingle-model referenceDirect call to qwen-plusQwen-PlusTrace audit passed
R-GFMOfficial implementation released by the authorsGraph-learning configuration from the official implementationDeepSeek-V4-FlashTrace audit passed
Sparse Communication TopologyOfficial implementation released by the authorsConfigured sparse agent communicationDeepSeek-V4-FlashTrace audit passed
VeriMapOfficial implementation released by the authorsConfigured verification and recovery proceduresDeepSeek-V4-FlashTrace audit passed

We thank the reviewer again for drawing our attention to this omission and helping us improve the clarity of the experimental description.

AuthorsOpenReview ↗

Dear Reviewer

Dear Reviewer,

Thank you again for your valuable comments on our submission. Your feedback has helped us improve the presentation of our work, and we have provided detailed responses to the points you raised.

As the discussion phase is approaching its end, we would like to kindly confirm whether the above clarifications and the additional experiments in our responses and the revised draft have sufficiently addressed your concerns. We sincerely hope that you will take this context into account when assessing our paper and the accompanying reviews.

We would be happy to address any remaining points you may have during the discussion phase. Your insights are valuable to us, and we are eager to improve our work further.

Thank you for your time and effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.