- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors take a framework of coupled dynamical systems from physics, namely twisted and chimera states, and attempt to analyze how multi-agent LLM systems may express such dynamics. They study it in N = 50 stateless agents that can locally learn with their peers in a day simulation process. They test different topologies and reasoning efforts as an ablation to their results.
Critical strengths
- The attempt to join dynamical system physics with LLM behaviour is admirable. Understanding behaviours of swarms of agents is an important task, and this work seems to be in the right direction using notions from physics to monitor such patterns.
- The authors design a novel task, the clock task, to measure synchronization phenomena between multi-agent language models.
- The authors report interesting experimental results regarding collective regimes in a simple MAS system organized in a clock fashion. They ablate reasoning and network topology, but the results could be better presented.
Critical weaknesses
Novelty
First, let me state that I assume that good scholarly behaviour entails that the paper should explain clearly what the contribution is and what background the paper is built upon. This doesn’t seem to be the case.
With regard to the theoretical presentation in Section 3, I need to specify that my background is not in physics. So I used LLMs to check the novelty of Lemma 1 and Proposition 1. The result below is what the LLM reports.
The fixed-point part of Lemma 1 is a known type of result applied to a slightly different update rule. The proof is the standard circulant-symmetry computation.
With regard to Proposition 1
Proposition 1 is a direct instance of standard linear-consensus theory. The proof in B.2 is the textbook eigen-decomposition argument.
With the current presentation, it seems to me the authors present those results as if it's something novel introduced in the paper. I don’t see any original theoretical contribution regarding this problem.
Method
The current experimental setting is quite limited in terms of being realistic. The authors state this in their limitations, but even with that, other researchers would benefit from understanding why your setting is interesting to be studied. It looks to me that the current setting is both unrealistic and doesn’t give much insight into phenomena of collective behaviours in LLMMAS.
In addition to that, since even baseline models seem to show non-trivial collective behaviour, how do you distinguish the experimental setting giving rise to such behaviour, instead of the autonomy or some other property of LLMs doing so? The current work doesn’t seem to use anything that is LLM-specific, e.g. the mastery of language.
Presentation
I found the paper very difficult to understand in the way it was written. Major work in presentation would be very much required for the work to be considered for acceptance. This is one of the main reasons I lean toward a rejection. I will now present a non-comprehensive list of where it can be improved.
The related work is just a list of other works and does not engage with the similarities and differences with the current work.
Absence of interdisciplinary background for CS readers. The authors never introduce the physics background needed to understand the text. I would like to highlight that this is still a machine learning conference, and such background might not be accessible immediately to such reviewers. Section 3 starts immediately assuming what “global order, local order, spatial heterogeneity, and persistence” mean.
There are also odd phrasings like line 144 “additive innovation of the fitted rule”; innovation w.r.t. what?
Absence of any explanatory figures: to show the experimental setting could be quite helpful in aiding understanding.
Excessive experimental detail: like in line 319 “experiments use base seed 20251109”, I found this detail very odd in the presence of the paper.
Excessive references to the appendix in the main paper. There are many references to results in the appendix in the whole results section, for example lines 414 and 415, 419, 346, 372, 340, 333 and many other places, etc… I would like to remind the authors that the main paper should be self-contained, and the appendix should have only extra notes on the main. This much referencing looks almost like being required to look at the appendix, which I have not done for each result.
under the assumption that greater peer interaction leads to more reliable consensus
Where is this assumption from?
Conclusion
Overall, I had great difficulty understanding the point that the paper wanted to make, and listed many ways this presentation could be improved. I showed flaws in the novelty of the approach, even given the interesting research question. I raised some methodological concerns that, in my judgement, are crucial to be addressed before we can consider this work solid, such as the role of LLMs in the MAS studied.
Questions for authors
- Lines 274: “how ring radius, reasoning effort, and network topology affect collective behaviour”. Can you tell me how you model ring radius and network topology in your experiments? I did not find it in the experimental setting section.
- What is the shared classifier you are using in line 292?
- How do you compute the R nearest neighbours? Are you assuming some spatial location of the agents? Line 268.
- Where are the tasks from the judges' task from? What are the criteria to judge from 1 to 100?
- Many multi-agent systems have unclear influences. In your setting, was set to determine who influences what. How do you think you may extend the influence zone to allow real-world multi-agent systems to be modeled? Or do you think your objective is something else?