acceptodds
Withdrawn from ICLR 2027

Population Physics, Population Problems: Safety and Emergence in LLM Societies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The collective behaviour of large language model (LLM) societies is not the sum of their individual outputs. It yields statistically distinct, sometimes-unpredictable phenomena, for which the tools we use to study single agents may not scale. Due to recent incidents involving autonomous agentic systems, however, understanding these systems is paramount. For that we introduce a framework for measuring self-organisation in LLM social systems and apply it to three such systems: a Schelling grid, a social network (Moltbook), and a Twitter-like misinformation simulation ('Rogue'). All three exhibit statistically significant self-organisation. Moreover, their relaxation dynamics vary with the environmental information available to the agents, with open-ended systems (Moltbook, Rogue) exhibiting sharp, phase-transition-like dynamics. Further results show that population-level pathologies can emerge even when the LLMs are safety-tuned or monitored, being primarily driven by the coordinated activity of a population subset. We also show when self-organisation does not emerge under two additional scenarios (a commons dilemma, GovSim, and a LLM-as-a-judge deliberation scheme, ChatEval). We argue that measuring signatures of this kind offers a lightweight, agent-agnostic diagnostic layer for detecting coordinated collective behaviour in deployed multi-agent systems without relying on natural language or model versioning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Complex systems, safety, multi-agent systems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2Qb6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces how to identify and classify the collective behavior in different LLM societies through order parameters and a measurement framework. The authors discuss 3 main LLM societies and claim to find self-organisation, information-dependent relaxation shapes, and population-level pathologies despite individual safety tuning.

Critical strengths

The paper offers a model-agnostic way to quantify self-organisation in LLM agent populations, which is a relevant problem that has received little study. Bringing order parameters and null models from complex systems theory into this setting is a reasonable direction, and the inclusion of negative cases (GovSim, ChatEval) is appreciated.

Critical weaknesses
  1. The framework is unified only in name, and the results are inconsistent. The three systems differ in time units, in the size and nature of their activity spaces, and even in the direction in which Φ moves, so Φ trajectories are not directly comparable across them. Rogue’s second metric (bad reach) is a ratio with its own null rather than “the same operation under a different partition”. In addition, the model-selection outcomes reported in the main text are not always consistent with the appendix tables under the paper’s own criteria.
  2. Section 3 is conceptually inconsistent. Φ is first introduced as happiness, which increases over time, and is then redefined as entropy, which decreases. Schelling is presented as an example of a phase transition, but “Schelling-style relaxation” is then used as the contrast to phase-transition-like dynamics. “Phase transition” is also redefined as an S-shaped curve in time rather than as behaviour with respect to a control parameter, so the core argument is hard to follow.
  3. The presentation of results is insufficient. The main text contains only two figures, both on Schelling. The Moltbook trajectory, which is the key evidence for the paper’s main claim, is not plotted anywhere, and no figure shows fitted curves or residuals. Most conclusions rest on prose reports of metrics like R2, ΔAIC and β without systematic organization.
  4. The introduction is loosely connected to the contribution. The motivation concerns detecting coordinated behaviour in deployed systems, but the paper contains no detection or monitoring experiment, and its fitting procedure is retrospective. The framework itself is introduced in a single sentence. Broader claims, such as environmental information shaping the relaxation regime, rest on a handful of heterogeneous systems with no controlled manipulation.
Questions for authors

The authors are supposed to clarify all the inconsistencies mentioned in the weaknesses and supply necessary additional results and materials to support their claims in the paper.

Reviewer_S3hAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies groups of LLM agents and asks whether they organize themselves into patterns that no single agent is aiming for. Instead of reading what the agents say, the authors measure how concentrated each agent’s activity is (for example, whether it keeps posting in the same few places). They compare this with what random behavior would produce and look at how it changes over time. They test the method on three systems: a grid where agents move to sit near similar neighbors, a real social network made only of AI agents (Moltbook), and a simulated Twitter-like platform where some agents spread misinformation. They find organised behaviour in all three. How quickly and sharply it appears seems to depend on how much information the agents have. Misinformation also spreads even when the other agents are designed to resist it. In two other systems, a shared fishing game and a panel of AI judges, they find no such patterns.

Critical strengths

The idea is good. The authors look at how the agents behave rather than what they say. That avoids reading human thoughts or intentions into model outputs, and the method should keep working as new models come out. Using one measure across very different systems makes the results easy to compare. It is also good that the authors report cases where nothing happened (the fishing game and the AI judges). The experiments cover a lot of ground. They include both simulations and real data, test several settings, repeat key results with a second model, and share the code and prompts.

Critical weaknesses

I am not convinced the method detects real self-organization. The random baseline treats all agents as identical, but these agents differ from the start, so the measure signals organization even when agents are placed at random. Some trends may also come from how things are counted, or from most Moltbook agents leaving during the study, rather than from agent behavior. The safety result is hard to interpret. The baseline for misinformation includes the agents spreading it, so ordinary agents look resistant even if they do nothing special. Some comparisons also mix in other differences, such as the number of agents, and several numbers do not match between text and tables.

Questions for authors

Would the results still hold with a baseline that accounts for agents being different from the start? For Moltbook, do the trends hold if you only look at agents active throughout the whole period?

Reviewer_sw87OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a complex-systems-inspired measurement framework for LLM multi-agent societies: a mean per-agent Shannon entropy order parameter Φ(t) over a chosen categorical "activity space", functional fitting against stretched-exponential vs. logistic reference forms, and a bipartite configuration-model null. The framework is applied to five systems (LLM-Schelling, Moltbook, Rogue, GovSim, ChatEval). The authors report: (i) statistically significant self-organisation in the first three; (ii) covariation between relaxation shape (β) and environmental information availability; (iii) population-level pathologies in Rogue despite safety-tuned agents; and (iv) absent dynamics in GovSim and ChatEval. They argue this yields a lightweight, model-agnostic monitoring layer for deployed multi-agent systems.

Critical strengths
  1. A genuinely important problem. Population-level risks of agentic systems are understudied relative to single-agent alignment, and the argument that individual-level monitoring operates at the wrong abstraction level is well taken.

  2. The negative results are informative. The finding that ChatEval deliberation never moves verdict entropy (median change 0 across 5,508 item-outcomes) is a crisp, useful characterization, and the "easily-locatable target ⇒ fixpoint" explanation is plausible.

Critical weaknesses
  1. The fit classifications contradict the paper's own numbers and decision rule. Section 4.3 says one form is preferred when ΔAIC > 2. By that rule Moltbook has no preferred form: Table 4 gives ΔAIC ≈ 1.8, not the ≈ 22 stated in the text, so the "phase-transition" label isn't supported. Uninformed Schelling is logistic-preferred by ΔAIC ≈ 5,477 and at every subsampling level, yet it is classified as classical. For Rogue engagement, β is reported as ≈ 1.00, 1.42–1.71 and 1.76 in different places. The headline Schelling β values (0.428, 1.417) also fall outside their own reported V3 stability ranges.

  2. The null may be passed trivially, and passing it doesn't show emergence. Per-agent entropy is capped at log₂(nᵢ). With Moltbook agents posting 2–7 times per window, falling below the marginal-entropy null is nearly guaranteed unless the sampled null keeps each agent's count exactly, and Appendix I suggests it only does so "in expectation." By the paper's own definition, beating the null shows per-agent preferences, which can exist without any interaction. Controls that preserve each agent's degree, or that remove interaction entirely, are needed. Moltbook's population also falls from 29,466 to 1,159 agents, so turnover could explain the trend.

  3. The safety-tuning resistance result may follow from how the null is built. The ambient null includes malicious agents' own interactions, which push it up by design. So a benign/ambient ratio below 1, falling as prop_mal grows, is expected even with no resistance at all. The malicious agents' "coordination" is also assigned rather than emergent. And with Qwen the bad-reach fit falls to R² = 0.47, so the result doesn't replicate across models.

Questions for authors
  1. Please reconcile the β, R² and ΔAIC values for Rogue engagement, Rogue bad reach and Moltbook across Section 5, Table 4, Table 5 and Table 6. Given ΔAIC ≈ 1.8 in Table 4, do you still consider Moltbook logistic-preferred?

  2. Does the sampled null keep each agent's activity count exactly? Does V1 still hold under a strictly degree-preserving null, or after restricting to agents with high counts?

  3. How do you separate emergent self-organisation from fixed per-agent preferences? Could you run an LLM-Schelling or Rogue control in which agents can't see each other?

  4. For the bad-reach result, what is the benign/ambient ratio if benign agents run on an un-tuned model, or if the null is based on exposure?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.