- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TopoRanker, a task-adaptive communication topology selection framework for LLM-based multi-agent collaboration. It addresses the limitations of existing methods that rely solely on successful graph executions, which provide limited information about the task-specific utility of different topologies and the contributions of individual structural components. Specifically, the method first constructs a set of candidate directed acyclic graphs (DAGs) and evaluates their utility based on execution outcomes and communication costs. It then derives global preference supervision from utility differences among candidate graphs for the same task and obtains local structural intervention supervision by removing individual nodes or communication edges. Using these supervision signals, a task-conditioned Edge-aware Graph Transformer is trained by jointly optimizing utility regression, topology ranking, and intervention consistency losses to predict the utility of candidate topologies. During inference, only the graph with the highest predicted utility is executed. Experimental results demonstrate the effectiveness of the proposed method.
Critical strengths
-
The paper proposes a task-specific utility-based topology selection method, providing a more principled approach to selecting multi-agent collaboration structures.
-
The paper introduces a topology learning method that integrates global comparisons with local interventions. By assessing the contributions of different structural components to collaborative outcomes, it provides finer-grained supervision for topology selection.
Critical weaknesses
-
The proposed method shares some similarities with prior work, and its novelty and key distinctions require further clarification.
-
The construction of candidate topologies and the associated experimental settings are not sufficiently justified and require further explanation.
-
The paper provides insufficient analysis of certain experimental settings and results, which would benefit from further explanation and discussion.
Questions for authors
-
The paper proposes a task-adaptive method for selecting multi-agent communication topologies based on predicted utility, demonstrating improvements in both performance and efficiency. However, prior studies, such as AMAS and Codebook Agent, have also explored topology selection based on task performance or utility. The authors should further clarify how their approach differs from these methods, particularly the additional benefits of incorporating node- and edge-level structural supervision compared with existing graph-level topology ranking methods. Direct experimental comparisons with these closely related methods would also help establish the effectiveness and novelty of the proposed approach.
-
The paper primarily constructs candidate DAGs from predefined topology templates, such as Chain, Star, Tree, and Dense, and their randomly pruned variants. Although this strategy reduces topology search and evaluation costs, the model is restricted to selecting from a finite candidate pool, potentially limiting its performance due to insufficient coverage and structural diversity. The authors should further discuss this limitation and consider additional experiments with different candidate generation strategies, candidate pool sizes, and levels of topological diversity.
-
In the experiments, the number of executions per task–graph pair is set to (K=1). Given the stochastic nature of LLM inference, a single execution may not reliably estimate the true utility of a candidate graph, potentially affecting the reliability of graph-level preference labels and node- and edge-level intervention labels. The authors are encouraged to evaluate different values of (K) (e.g., (K=1,3,5)) to examine their impact on final performance.
-
During training, the method requires executing multiple candidate communication topologies and re-executing modified graphs after removing individual nodes or edges to construct global and local supervision signals. Although only the selected topology is executed during inference, offline data collection and intervention evaluation may incur substantial computational costs. The authors should provide a more detailed analysis of the training overhead, including the number of LLM calls, token consumption, and wall-clock time, while distinguishing the costs of supervision data construction from those of training the utility prediction model.