- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes GraphMAS, a systematic benchmark for evaluating multi-agent coordination strategies in graph learning. The authors construct a shared pool of four graph reasoning specialists, organize coordination along two dimensions, including inter-agent interaction and runtime adaptivity, yielding four paradigms, and instantiate seven representative coordination methods. Evaluation is conducted across seven text-attributed graph (TAG) datasets on two tasks: node classification and link prediction. Key findings include: multi-agent gains arise from "reasoning decomposition" rather than "evidence accumulation"; richer inter-agent interaction does not reliably improve performance; instance-adaptive routing achieves the best accuracy–efficiency Pareto trade-off; and the coordination policy can be learned via PPO and generalizes to held-out graphs.
Critical strengths
This paper provides a summary of current agent coordination approaches and graph datasets in the graph learning domain, conducts robust experiments, and offers certain conclusions and reference results, such as training a Router via PPO and validating its transferability to unseen graphs.
Critical weaknesses
- The narrative logic in the Introduction suffers from a fundamental ambiguity. The authors first state that multi-agent coordination for graph learning "remains largely unexplored," implying that agents should be employed to solve graph learning problems (Agent-for-Graph). They then immediately claim that graph learning "provides a natural setting for studying multi-agent coordination," implying that multi-agent coordination can be studied through graph learning (Graph-for-Agent). This reversal of the service relationship makes it difficult to determine whether this paper aims to evaluate an agent-for-graph benchmark or a graph-for-agent benchmark, resulting in a confused positioning.
- The contribution is very limited. The benchmark comprises only 7 datasets, covers only two graph tasks, is restricted entirely to text-attributed graphs, and uses exclusively publicly available datasets. This cannot reasonably be called a "Systematic Benchmark" for graph learning; it should more accurately be termed a TAG benchmark.
- The attribution of "reasoning decomposition" lacks rigorous controls. In Table 3(b), the paper concludes that multi-agent gains arise from "reasoning decomposition," because distributing evidence across multiple agents for independent reasoning outperforms a single agent processing all evidence in one long context. However, the true cause of the single agent's performance degradation may simply be context window overload or attention dilution due to excessively long prompts (i.e., the "lost-in-the-middle" phenomenon), rather than any inherent "cognitive advantage" of multi-agent coordination. The authors lack critical control experiments, for example, restricting the single agent's total token input (via Top-K truncation or evidence compression), or using models with long-context optimization (e.g., Ring Attention). Without eliminating the "context length confound," one cannot rigorously demonstrate the independent contribution of "decomposed reasoning."
- The theoretical analysis is limited. Although the core contribution of this paper is to provide a benchmark that offers reference for future research directions, the paper lacks theoretical justification for why such a benchmark is needed. Furthermore, the reasons why certain methods perform poorly in the experiments are not elaborated in detail. A substantial portion of the paper is devoted to describing repetitive conclusions.
Questions for authors
- What is the positioning of this paper—is it an agent-for-graph benchmark or a graph-for-agent benchmark? This distinction is critical for evaluating the paper's contribution and value。
- In Table 3(b), is the single agent's weaker performance simply due to excessively long input context? If the single agent's total token input were strictly constrained, would the multi-agent "decomposition advantage" still persist?
- My concern is: what is the core objective of this paper? Is it to provide design guidance for multi-agent systems in graph learning? If so, the experimental conclusions presented here appear to offer limited actionable guidance. More meaningful conclusions would explain why such performance patterns emerge, identify substantial performance gaps on other domains or self-constructed datasets, thereby revealing what problems exist in current research and how future work should proceed.
6 comments
W4,Q3: Research objectives and empirical insights.
GraphMAS has several main objectives, including: establishing the effectiveness of MAS for graph prediction, providing a controlled benchmark that yields insights into coordination design, and presenting a strong learned-coordination baseline for future works. We consider such an empirical foundation necessary at this stage, as multi-agent graph learning is still emerging, with limited controlled evidence on whether, why, or when coordination helps. To our knowledge, GraphMAS is the first systematic benchmark to establish it against GNN, LLM-based, and agentic single-agent methods. We therefore see a theoretical account as valuable future work building on these findings.
In the original manuscript, we included extensive analyses of why the benchmark is needed and why methods differ. The oracle exceeds the best specialist by 16.9 points on LP, showing that specialists are complementary and leave clear headroom for coordination, as detailed in Appendix C.1. Specialized roles reduce LP accuracy by 3.1 points without matching evidence, showing that specialization requires combining evidence and roles (Finding 3). A single agent given all four evidence sources trails distributed specialists by 5.4 points, showing that decomposition provides gains beyond evidence access (Finding 4). Coordination methods differ by 4.4 points under unanimous agreement but 19.3 under strong disagreement, showing that coordination matters most where specialists disagree (Finding 6).
To give further insight into why specific methods underperform, we conducted an additional trajectory analysis in the revised manuscript. Recorded trajectories show that Debate, DyLAN, and Graph of Agents abandon an initially correct prediction on 10% to 16% of disagreeing LP instances while independent methods abandon none, indicating that specialists often give up correct answers after seeing incorrect peer responses (Appendix C.10, see also our response to Reviewer qpWz [Q1]), which provides more insights on why richer interaction does not improve over independent coordination (Finding 5).
Again, we sincerely appreciate the reviewer's thoughtful feedback. We hope our responses and revisions further clarify our contributions and address the concerns.
We thank the authors for the detailed response. The additional control experiments addressing the "reasoning decomposition" attribution are well-designed and effectively eliminate the context-length confound. This is acknowledged.
However, two fundamental issues remain unresolved:
- Limited Benchmark Scope and Lack of New Data (W2): The authors defend the narrow scope by citing other works that also focus on TAGs. However, an "appeal to common practice" is not a valid defense for a limited scope and overclaiming. The existence of similar limitations in other works does not absolve this paper of its own. Furthermore, the paper introduces no new datasets or evaluation protocols to expose the blind spots of current SOTA methods. Relying solely on public datasets to benchmark existing generic coordination templates severely limits its core contribution as a benchmark.
- Missing Analysis on Graph Topological Properties: The response entirely fails to address how graph topological properties (e.g., homophily/heterophily) influence the optimal coordination strategies. If the conclusions regarding coordination are entirely independent of graph topologies, what fundamentally distinguishes this work from a general-purpose multi-agent evaluation report? Without this graph-specific analysis, the paper offers limited actionable guidance for future algorithmic design in graph learning.
We appreciate the reviewer’s careful evaluation and suggestions. We respond to the raised concerns below:
W1,Q1: Paper positioning.
To clarify, GraphMAS is an Agent-for-Graph benchmark. We compare seven coordination strategies over the same four graph specialists to measure how specialist participation and communication affect accuracy and inference cost in graph learning tasks.
By “largely unexplored,” we mean that multi-agent coordination has received little controlled study for graph learning. LLM-based graph methods are mostly single-agent, and existing multi-agent methods propose a single architecture for a single task. GraphMAS fixes a shared pool of specialists and varies only the coordination strategy. This separates the effects of graph evidence and reasoning from those of inter-agent interaction and runtime adaptivity.
By “natural setting,” we mean that the structure of graph evidence gives a direct reason to use multiple specialized agents for graph learning. Immediate neighbors may provide the strongest evidence for one prediction, while longer-range structure or semantically similar nodes matter more for another. Assigning these perspectives to specialists lets us test which specialists to consult and how to combine their judgments to improve graph predictions. Both sentences therefore treat graph learning as the beneficiary of multi-agent coordination. These revisions are detailed in the Abstract and Section 1.
W2: Benchmark scope.
“Systematic” refers to our controlled study of multi-agent coordination across two design dimensions, four paradigms, and seven methods under a unified protocol, rather than exhaustive coverage of graph learning. Text-attributed graphs are the standard setting for LLM-based graph learning, as LLM agents reason over node text.
Within this setting, GraphMAS is diverse along several axes. It spans three domains (citation, e-commerce, and social networks), two task levels (node classification and link prediction, giving 14 dataset-task settings), and a wide range of scales and label spaces (13K to 173K nodes and 3 to 47 classes). It further tests generalization on held-out graphs and validity under two additional backbones.
This scope is consistent with established benchmarks and studies for LLM-based graph learning, which are likewise restricted to text-attributed graphs with a similar number of datasets and tasks. Chen et al. (SIGKDD Explorations 2024) [1], GLBench (NeurIPS 2024) [2], and LLMNodeBed (ICML 2025) [3] evaluate four, seven, and ten graphs on node classification alone, and Wang et al. (ACL Findings 2026) [4] evaluate four on node classification and link prediction, whereas GraphMAS covers more dataset-task settings than each. Public datasets with splits aligned with prior work also keep our results reproducible and comparable.
W3,Q2: Context length and reasoning decomposition.
We ran single-agent controls with the same settings used in Appendix C.8, while reducing the evidence budget through Top-1 truncation, proportional truncation, and LLM-based evidence compression respectively.
Shorter contexts do not close the gap. Truncation lowers accuracy, while compression retains similar accuracy to the original single-agent setting at evidence lengths close to those of each specialist. It still trails multi-agent reasoning by 5.8 points on LP and 2.3 on NC, suggesting that context length alone does not explain the gap. The controls are detailed in Appendix C.8.