- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work proposes OpenMAS-GCom, a comprehensive diagnostic benchmark for evaluating graph-enhanced multi-agent systems. It proposes a controlled intervention framework to analyze the contribution of different components. It also proposes a new benchmark, G-MAS-Complex, containing 400 multi-document tasks.
Critical strengths
- The motivation seems interesting: we need more controlled interventions to understand the contribution of each component of MAS systems.
- The evaluation is relatively comprehensive for many MAS approaches and benchmarks. The authors also propose a new benchmark, G-MAS-Complex.
Critical weaknesses
- The readability of equations in Sec 2.1 is limited. The intervention protocols are not well explained by the equations.
- What are "Test-A and Test-B sets of version v6.3.2", "rejected records", etc in Sec 3.1? The readability is limited.
- The takeaway is unclear. For example, in Figure 2, the authors report the averaged results of ordinary MAS and Graph-enhanced MAS; in Figure 3, the authors report separate results for different Graph-enhanced MAS results. A lot of MAS approaches are being compared, but without diving into the details of different approaches, it is very hard to understand why some of them are better at math and why some of them are better at code.
- It is also unclear to me what extra value G-MAS-Complex provides compared to previous benchmarks.
Questions for authors
See weakness above.
6 comments
Answer to w2, we have clarified the release version, the Test-A and Test-B collections, and the definitions of candidate, governing, and rejected records.
(W2) Short answer. We have clarified the release version, the Test-A and Test-B collections, and the definitions of candidate, governing, and rejected records.
We thank the reviewer for pointing out this issue. We appreciate the reviewer highlighting the terms that would benefit from a clearer explanation for readers. Version v6.3.2 identifies the evaluated G-MAS-Complex release. Test-A and Test-B are two disjoint test collections containing 80 and 320 tasks, respectively. Their union provides 100 tasks in each of four difficulty tiers. A candidate record specifies a variable, authority level, revision number, and expression. The governing record has the highest authority, with revision used to break ties. Rejected records are the remaining candidates for that variable. They remain in the input documents as conflicting alternatives and are excluded from subsequent computation. We have clarified these terms in Section 3.1, added the selection rule in Appendix A.1, and expanded the illustrative example in Appendix B.1.
Table R3.1. Illustration of candidate-record selection for one variable.
| Document | Variable | Authority | Revision | Status |
|---|---|---|---|---|
| D1 | 2 | 9 | Rejected | |
| D2 | 3 | 4 | Selected | |
| D3 | 2 | 7 | Rejected |
We thank the reviewer again for helping us make the dataset description more accessible and easier to follow.
Answer to w3, our main takeaway is that communication graphs can improve collaboration by directing intermediate results and verification feedback to the agents that need them. Different graph designs support these exchanges differently across tasks.
(W3) Short answer. Our main takeaway is that communication graphs can improve collaboration by directing intermediate results and verification feedback to the agents that need them. Different graph designs support these exchanges differently across tasks.
We sincerely thank the reviewer for this helpful suggestion. We are grateful for the reviewer's thoughtful feedback on the discussion and the opportunity to explain the intended insights more clearly. Figure 2 compares the best-performing configurations within ordinary MAS and graph-enhanced MAS groups, while Figure 3 examines individual graph-enhanced configurations across domains. The organizational mechanisms summarized in Appendix B.5 help interpret these results in terms of how agents exchange and use intermediate information.
A communication graph specifies which agents receive a candidate result, which agents check it, and where the resulting feedback is delivered. Together with execution rules, these connections can ensure that a downstream agent receives the inputs needed for its calculation and that a corrected result reaches the agents whose work depends on it. This provides a concrete way for graph design to improve collaboration over a fixed interaction protocol that does not organize these exchanges around the task's requirements.
These requirements differ between mathematics and code generation. In arithmetic expression search, agents can explore partial expressions, check whether combinations satisfy the rules, and pass valid candidates for further composition. In code generation, useful exchanges connect implementation with checks of input requirements, interfaces, and program behavior, then return specific feedback for revision. GoAgent selects and connects task-relevant groups, providing a way to coordinate candidate exploration and integration. ARGDes jointly generates agent roles and communication links, allowing differentiated construction and checking roles to be connected according to the task. Our interpretation is that these organizational differences can favor different patterns of candidate generation and feedback. In our results, GoAgent's mathematics lead among the four configurations in Figure 3 is concentrated in Game-of-24, where it scores 74.08% compared with ARGDes's 67.11%. ARGDes exceeds GoAgent by 9.75 and 4.88 percentage points on HumanEval and HumanEval++, respectively.
The controlled interventions support the importance of these communication choices. With GDesigner's agents, roles, model, and budget limits fixed, rewiring 70% of its edges while preserving node degrees reduces the five-dataset mean by 8.19 percentage points. Thus, the same agents and number of connections produce different outcomes when information follows different routes. The practical implication is to design connections around the intermediate results and feedback required by each task, and to evaluate whether those exchanges improve the final output.
Table R3.2. Organizational mechanisms and the task requirements they are designed to address.
| Mechanism family | Organizational operation | Relevant task demands |
|---|---|---|
| Graph construction and optimization | Constructing or optimizing communication links among agents | Information exchange, dependency coordination, candidate integration |
| Dynamic selection and routing | Selecting agents or routing tasks according to task characteristics | Adaptive allocation, variable task complexity, selective collaboration |
| Group-based coordination | Organizing agents into groups and coordinating group communication | Parallel candidate generation and multi-stage collaboration |
| Verification-oriented coordination | Checking candidate outputs and mapping results to supporting information | Consistency checking, factual validation, structured output verification |
We thank the reviewer again for encouraging a more explicit discussion of the results and their implications for graph-based collaboration.
Answer to w1, we have clarified the mathematical formulation in Section 2 and explicitly connected the four intervention protocols to the organizational components they modify.
(W1) Short answer. We have clarified the mathematical formulation in Section 2 and explicitly connected the four intervention protocols to the organizational components they modify.
We thank the reviewer for pointing out this issue. We appreciate the reviewer's careful attention to the presentation and the helpful suggestion to make the formulation easier to follow. We have improved the explanation of Equations (1) to (7) by clarifying each component in the organization representation and its role in collaborative execution. Equation (6) now defines the four intervention operators, and Section 3.3 links each practical protocol to its corresponding operator. We have also explained how the modified organization produces a complete execution trace and how Equation (7) measures its score difference from the reference execution.
We thank the reviewer again for helping us improve the readability of the formulation and its connection to the experimental protocols.