- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
GraphMAC generates a query-specific multi-agent system by autoregressively selecting roles with a GRU, choosing heterogeneous self and cross-agent interactions, and assigning LLMs through relational graph convolution. The paper reports improvements over MasRouter on five benchmarks, including 5.80 percentage points on MBPP and approximately 43% lower MATH inference cost. It also presents component comparisons and routing to an additional LLM without retraining.
Critical strengths
The unified treatment of role composition, interaction operations, and heterogeneous LLM assignment is a useful problem. Graph-conditioned routing offers a coherent way to allocate models to different structural roles.
Critical weaknesses
-
The comparability of the main baselines are unclear. The 15 rows from CoT through RouterDC in Table 1 exactly reproduce the corresponding results in the published MasRouter Table 1, although that work used a different routing pool. MasRouter, Section 5.1 and Table 1. Furthermore, four entries in Vanilla (GPT) match its GPT-3.5-turbo row, while the MATH entry matches its GPT-4o-mini row; Section 5.1 identifies GPT as GPT-4o-mini.
-
The executable collaboration evidence for its specific improvements are incomplete. Section 4.2 defines local edge operations but does not specify execution order, cyclic interactions, merging multiple incoming contexts, or final aggregation. These choices determine both correctness and cost; Appendix A.2 only instructs the reader to execute the graph.
-
Table 2 replaces entire learned modules with random choices. It therefore does not isolate the claimed advantages of GRU history over mean pooling, heterogeneous operations over a single pattern, or graph-conditioned routing over a learned graph-independent router.
Questions for authors
-
Were the matching baseline results reused or rerun, and how was comparability across model pools ensured? Please clarify which model produced each Vanilla (GPT) entry.
-
How is the generated graph executed, including execution order, cyclic interactions, incoming-message merging, and final aggregation?
-
Could you provide controlled comparisons of GRU versus mean pooling, heterogeneous versus homogeneous interactions, and graph-conditioned versus learned graph-independent routing under comparable inference budgets?
-
Hetero-Designer (Zhang et al., ACL 2026) is closely related and warrants discussion. Could you compare its interleaved role/model selection and edge construction with GraphMAC, explaining the benefits of assigning models after graph construction and using typed interactions over equivalent unrolled DAGs?
Thanks for your constructive comments!
W1 & Q1. Baseline Comparability
A1: We clarify that we reused published results because these methods employed the same GPT-4o-mini (GPT) backbone as our experiments. Since the backbone was identical, rerunning these evaluations was unnecessary. For methods involving different LLM backbones, we independently reran the experiments under our settings to ensure consistent evaluation conditions.
We adopted an updated LLM pool for two reasons. It reflects recent advances in LLM capabilities rather than relying on older model generations. Second, it provides a broader range of capability–cost trade-offs, which is particularly important for evaluating GraphMAC's graph-aware LLM allocation strategy. All independently rerun comparisons follow consistent model configurations, token-counting rules, and pricing assumptions.
W2 & Q2. Executable Collaboration Evidence
A2: We clarify that GraphMAC executes graph in two stages:
The execution procedure is summarized below:
For the default single-round setting, the nominal number of LLM calls per query is: $ N_{\mathrm{LLM}}=|V_{\mathrm{active}}|+ 2|E_{\mathrm{Debate}}|+1, $ where the final call performs answer aggregation. All intermediate and aggregation calls are included in inference cost accounting.
W3 & Q3. Controlled Ablation Studies
A3: We provide targeted comparisons that replace only one design component at a time: GRU with mean pooling, heterogeneous interactions with a homogeneous interaction pattern, and graph-conditioned routing with a learned graph-independent router.
Table R1. Targeted Ablation of GraphMAC Components.
All variants retain the same LLM pool, training protocol, and cost constraints. These comparisons isolate the contributions of temporal role selection, typed communication, and topology-aware model allocation while accounting for inference-budget differences.
Q4. Comparison with Hetero-Designer
A4: Although both methods construct query-adaptive multi-agent systems, they differ in two aspects:
LLM allocation. Hetero-Designer jointly predicts LLM-role pairs and incoming edges autoregressively. In contrast, GraphMAC first constructs the collaboration graph and then assigns LLMs using R-GCN representations, allowing allocation to leverage the complete interaction topology.
Interaction modeling. Hetero-Designer primarily represents dependencies through DAG edges, whereas GraphMAC distinguishes Chain, Criticism, Debate, CoT, and Reflection, each with relation-specific transformations. This enables distinct information propagation patterns and more structured, compact interaction representations without relying solely on graph expansion.
Although unrolled DAGs offer similar execution semantics, typed interactions provide relation-specific biases for learning diverse information flows and guiding heterogeneous LLM allocation, coupling diversity-aware interactions with topology-aware routing.