- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that most existing topology design methods for LLM-based multi-agent systems typically determine a complete collaboration topology for a task before execution and keep it largely fixed throughout the subsequent process. The authors argue that, rather than directly learning “what complete topology should correspond to a given task,” it is more desirable to learn a more general principle of “how agents should be organized under different task states.” To this end, they train a lightweight topology generation model that, based on the current task, execution feedback, and historical information, dynamically generates the next group of agents and their communication topology on demand during task execution, allowing the collaboration structure to adapt as the task progresses.
Critical strengths
- The idea of dynamically adapting the collaboration topology is very interesting.
- The experiments are comprehensive and cover multiple benchmarks and ablation studies.
- The story is insightful and made me rethink the design of multi-agent collaboration topologies.
- The method is lightweight while still achieving strong performance.
Critical weaknesses
- The necessity of using such a small model for topology selection is unclear, especially since GPT-4o already understands the current execution state.
- The performance gain is small, and it is unclear whether it is robust or truly caused by dynamic topology adaptation.
- The ablation results suggest that group decomposition and cross-group overlap contribute more than dynamic adaptation, which may not fully support the paper’s main story.
- Dynamically changing agent roles across groups may introduce context or role confusion, especially for weaker models.
Questions for authors
Thank you for your paper. You propose a very interesting idea, but I have several questions regarding the paper.
-
Is it sufficient to use such a small model to determine the topology? Small models are generally more likely to make mistakes than large models. Letting such a small model take the role of topology selection makes me wonder whether this could introduce problems into the results. Is it possible that this small model is actually not that important, since GPT-4o already understands what is currently happening?
-
The performance improvement is quite small. It is unclear whether this is due to variance, and it is also unclear whether this small gain really comes from the dynamic topology.
-
According to the ablation study, removing dynamic adaptation causes an average drop of only 1.76 points, while removing cross-group overlap causes a drop of around 4.5 points. Then how can we tell whether the gain really comes from dynamic adaptation? Is it possible that the improvement mainly comes from splitting the collaboration process into multiple groups? Therefore, I think there may be a gap between the paper's story and its experimental results.
-
Could dynamically changing the topology increase context confusion for the LLM agents? Is it possible that the same model plays opposite roles at different time steps, for example, acting as a proposer at one step and an opponent at the next? For less capable models, could this framework therefore introduce context confusion? If the context needs to be reset between groups, then I think this method looks more like splitting a large task into several smaller tasks and then pre-determining a collaboration topology for each small task. This seems somewhat different from the story proposed in the paper, since the paper argues that the communication topology should change dynamically rather than being determined in advance.