- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper considers an LLM agent at each site of a 2D Ising model. Each agent is asked to update the spin at its site given the states of its neighbours as a prompt. The paper then studies the statistical mechanics of this model through simulations to understand collective alignment in LLMs. The results indicate that the system does not belong to the Ising universality class.
Critical strengths
The paper reports a mostly negative result and does not overclaim Ising criticality for phi4-mini.
Critical weaknesses
I think the motivation for studying this problem is flawed. It is unclear why putting the agents on a lattice and letting the Ising dynamics run would tell us anything about their collective alignment. The fact that they do or do not update their spins according to the majority of their neighbours is highly simplified aspect of their behaviour, and the reason why this is a meaningful way of studying whether LLMs can coordinate with each other is not clarified.
If the system had behaved like a 2D Ising model, it could have been interesting, but according to the claims of the paper itself, it is clear that there is no meaningful correspondence between the Ising model and the model considered. For example, the fact that coordinated behaviour is mostly explained by the local field , while the coupling plays a minimal role, suggests that there is no motivation for making this correspondence. One can simply give an agent a set of opinions and check whether it agrees with the majority or not. This reveals an LLM's bias towards agreeing with a particular opinion or moving towards the majority. But putting such agents on an Ising lattice and studying the dynamics is unmotivated and does not provide any further insight whatsoever in my opinion.
I can point out several other technical issues, but since the review requires me to point out only things that may change my opinion, I will discuss only the significance of the problem. I do not think that this problem is interesting enough for this venue in the first place.
Questions for authors
Does the estimated kernel actually depend only on k, or also on the order and positions of the neighbours in the prompt?
We thank the reviewer for the review. We agree that for two of the three models a single-agent test captures most of the behavior: on re-examination, their neighbor influence changes the answer probability by only a few percent. The lattice adds information when coupling is not negligible, as the sector asymmetry for mistral:7b suggests, and that is where we will focus future work. On your question: the prompt we used summarizes the neighbors into one of three descriptions (majority yes / tie / majority no). So, by construction, the kernel does not depend on neighbor order or position, only on this coarse-grained count. We are withdrawing the submission; please see our main comment under Reviewer Re7Q’s review.