acceptodds
Under review as a conference paper at ICLR 2027

Flag Game: A Toy Model for Mechanistic Swarm Interpretability

Abstract

Emergent coordinated behaviors of AI agents are starting to present critical safety risks. A key phenomenon driving these behaviors is the rapid formation and spread of beliefs about the world, and mechanistic understanding is crucial for collective alignment. To this end, we introduce the Flag Game, a toy model for studying the mechanisms of collective belief formation. Concretely, a hidden country flag defines the ground truth, and each bounded agent directly observes only a private crop but can exchange beliefs and weigh social evidence from peers. Despite its simplicity, the Flag Game reproduces rich collective phenomenology: non-monotonic scaling of performance with population size, accuracy gains from social-awareness prompting and team diversity, and strong effects of organizational structure. In particular, we identify that collective belief collapse at small population sizes turns into collective belief polarization as the population grows. This polarization causes the performance decline at large population sizes, but creates diversity in collective beliefs. Finally, we dissect the mechanisms underlying collective belief collapse and polarization with two complementary approaches. We first introduce social circuit attribution, a technique to predict which agent, and what view, matters most to collective dynamics, and verify its predictions by causal interventions on agents, tracing how agent patching changes collective outcomes. However, the efficacy of causal interventions on agents decreases as the population grows. We therefore develop a statistical mechanical theory for larger populations and verify that it matches the empirical phase diagram. Together, these results take a first step toward mechanistic swarm interpretability, a science of how the properties of individual agents and their communication give rise to emergent collective behavior.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multi-agent systems, collective decision-making, consensus, belief collapse, polarization, social learning, mechanistic swarm interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HH5FOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce the Flag Game, an experimental multi-agent framework in which agents receive different crops of a flag and exchange messages to identify the correct country. The authors explore how collective performance changes with population size, communication protocols, social-awareness prompting, and model heterogeneity. The results show that collective performance scales non-monotonically with the number of agents and that, as the population grows, failures shift from incorrect consensus to polarization between competing beliefs.

To investigate these phenomena, the authors propose two approaches. First, they introduce "social circuit attribution", a method that combines crop accuracy gain and temporal closeness to predict the influence of individual agents, which is tested through causal interventions. Second, they develop a mean-field model which implies three different phases based on the number of agents, that explain the non-monotonic behavior of the population size.

Critical strengths

The attempt to adapt methods inspired by mechanistic interpretability to investigate emergent collective behavior is a relevant research direction. The authors develop a simple and controlled experimental framework that allows the study of interaction and how macro-level phenomena emerge.

The experiments cover several orthogonal dimensions (size, protocols, models, heterogeneity, social-awareness, ...) and the authors successfully isolate the key variables governing the phenomenon under study.

Critical weaknesses

The experimental protocol and information flow are not completely clear in the main manuscript. Section 3 introduces the main components of the simulation: memory, patches, etc., but some important details are missing. To exemplify: it is not clear what a memory entry is made of. Are models given a set of candidate countries, or are the options open-ended? When does a game end if there is no 100% consensus? The low-level description of the game (with prompts and textual memory) can stay in the appendix, but Section 3 should contain all the details needed to picture a high-level overview of the experiments.

The method "social circuit attribution" is presented as a main contribution; however, there is not enough evidence supporting its effectiveness in the main manuscript. In particular, Figure 6b validates the method on a single illustrative game with 8 agents. It is not clear whether the proposed attribution score reliably predicts causal influence across different games, communication schedules, and population sizes. Finally, even if the method predicts influence reliably, the outcome is a ranking of the agents for a specific communication pattern, which might be highly correlated with their temporal centrality (especially if the number of available flags is small). It is not clear if the proposed method adds something new. A comparison with two baselines based solely on temporal centrality or crop accuracy would help clarify the contribution.

The theoretical model assumes a truth-biased adoption probability. However, since agents do not know the correct country, it is unclear why they should systematically favor messages supporting the ground truth over messages supporting a rival country. This should be supported by empirical evidence.

The authors don't list which models are used for each experiment. Some figures mention specific models, but a table or description of which models have been used throughout the whole manuscript is missing.

The related work section could be significantly improved. Currently, it reads as a high-level list of loosely related papers rather than critically engaging with existing literature or contextualizing the paper's novelty.

The core takeaway of the paper is somewhat diffused. The authors would benefit from focusing on fewer aspects of the problem and analyzing each in greater depth.

A vision task might not be a suitable choice, as individual agent performance might depend on the vision capabilities of a model. In Figure 1a, for instance, the agents' initial guesses are inconsistent with the visual crops they observe. This contradicts the results of figure 3, which instead demonstrate overall high accuracy of two models.

Questions for authors

In Section 5.1, the experiment social memory probing is not clearly explained. In particular, it is not clear what the authors mean by target ratio and social alignment. Could the authors clarify these definition?

Why were social awareness and memory omitted from the mean-field model? How do you explain the existence of a zealot supporting an incorrect flag?

It seems that some experiments were conducted for a subset of models. Why?

In Figure 6d, how does the number of interactions scale with N? This could explain the weaker impact of patching when N increases.

Could the authors conduct more experiments to validate the predictive power of your circuit tracing method and compare it to the baselines described above?

The paper "Group size effects and collective misalignment in LLM multi-agent systems" (PNAS) analyzes changes in macro-behavior when N increases. How does this work connect to that paper?

Did the author consider the voter model in finite populations without zealots? This model suits the memetic drift phase, which would imply the exact consensus probabilities based on initial opinions (see, e.g., Hassin & Peleg, Information and Computation, 2001).

Reviewer_MLs5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new task for testing multi-agent system in the view of AI safety. The task proposed by authors is to have each agent have a partial view of a flag to predict the country it belongs to through collaboration. From this task, the authors claim that we can analyze the safety of the models by analyzing the conditions where the agents tend to come to the wrong consensus. One main finding by the authors is that increasing the number of ai agents, there is an increase in belief polarizations, while less ai agents leads to collective belief collapse.

Critical strengths

This paper provides a new task for testing multi-agent collaboration. Specifically, to help understand the communications between different agents to analyze their collective decision making. This is done by having multiple agents observe a small portion of the flag to predict the correct country. There are multiple strengths in this paper:

  1. Uniqueness: there has been multiple works out about using partial image observations to predict it’s content, but from my literature review, there has not been works that tried to use it strictly for analyzing AI safety.

  2. Clear story: The paper is coherent in the story it is trying to sell, and the proposed task reflects the story of trying to analyze/test the communication efficiency of swarms of LLM agents.

  3. Good ablation: the authors do a good job in testing out different factors in the framework, such as population size. Authors also do test on different LLM models seen in the appendix.

Critical weaknesses

Weakness 1: Bias in the analysis: my main concern for this work is that the author propose the framework for decision making. From my understanding, there are various methods (including the hugging face incidence), of how multiple agents come to a final decision/prediction. By using this framework for the analysis for each LLM model, this inject a designer bias that might not reflect the real-world use environment. I ask the authors to either justify their reasoning for their framework, that it is general enough for analyzing the model as a whole; or extend the framework so that it is.

Weakness 2: Lack of novelty: There are various works that have used partial images to predict the full image. Though it has been the first from the view of AI safety (at least from the ones I can find). I do personally feel like the work itself is incremental.

Weakness 3: No visual of framework despite being framework paper: from my understanding of the paper, this is partially a framework paper; where choices were made in how final decision is made: communication protocol, prompting choices, different phases.

Questions for authors

I do have couple questions for the authors:

  1. Based on weakness 1, why was this framework chosen? Explain the effects of the architectural choices would have in biasing the decision making process. How is it similar/different with SOTA decision making frameworks.

  2. Why was GPT-4o chosen as baseline?

  3. Is image prediction a good baseline to represent AI safety, which is often based around chain-of-thought?

Reviewer_oZXTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces the Flag Game to study how agents’ private evidence and communication shape collective beliefs. Agents identify a hidden country flag from partial crops, using pairwise, broadcast, or blind-manager communication. In the main GPT-4o experiments (4–128 agents, 40 seeds per population size), accuracy peaks at an intermediate population size, while wrong consensus becomes less common and polarization more frequent as populations grow. Additional experiments show gains from social-awareness prompting under broadcast communication, benefits from some mixed GPT-4o/GPT-5.4 teams, and sensitivity to manager choice.

For agent-level analysis, the authors propose social circuit attribution, which combines the estimated local accuracy gain from replacing an agent’s crop with how quickly information from that agent could reach others. They test the predicted influence ranking through crop replacements while holding the communication schedule fixed. The declining effect of patching a fixed fraction of agents at larger population sizes motivates their population-level analysis. They develop a statistical-mechanical model of evidence-anchored and socially updating agents and compare its predicted consensus and polarization regimes with the experiments.

Critical strengths

The main contribution is a controlled testbed for mechanistic swarm interpretability, supported by agent-level interventions and a population-level model of collective failure.

Clear motivation and a useful controlled setting. Section 1 motivates the problem of collective belief formation in an accessible, easy-to-follow way. The Flag Game provides verifiable ground truth and control over each agent’s private evidence, making it possible to compare interventions under otherwise matched conditions. This is a useful experimental foundation for studying the interaction between private evidence and social influence.

Informative observations within the testbed. The distinction between wrong consensus and polarization gives a more informative account of collective failure than accuracy alone. The memory probes also provide useful evidence about how models respond to conflicting private and social information. These findings are sensible and worth reporting, although their interpretation should remain bounded by the experimental setting.

Critical weaknesses

There are several major concerns

  1. One major concern is that the Flag Game does not have a social foundation from the literature. Thus it is very difficult to assess the adequacy of such toy example to evaluate the social behaviors and structure.

  2. Limited mechanistic contribution. The Flag Game provides useful experimental control, but several phenomena have precedents, as acknowledged in Section 2. The proposed analyses need stronger validation to establish a substantial mechanistic advance beyond reproducing these observations.

  3. The claimed mixed-team advantage in Contribution 2 and Section 4 needs stronger statistical support: Figure 1d does not define its shaded band or report uncertainty in the accuracy difference between mixed teams and the best homogeneous team.

  4. Insufficient validation of social circuit attribution. The direct ranking test uses a single selected eight-agent communication schedule, with one game per patch. The highest-ranked agent also has the highest temporal closeness alone, so this example does not establish the added value of the combined attribution score over a simpler baseline.

  5. The theory does not independently explain the accuracy peak. Appendix F reports no mean-accuracy peak under neutral copying at fixed evidence probabilities or in the truth-biased fixed-share example used in Figure 7. The reported peak appears after averaging predictions over empirically measured evidence shares at each population size, leaving the explanation conditional on the observed evidence composition.

  6. The mapping from experiments to theoretical assumptions needs stronger support. Binary crop-choice probes measure preferences between truth and rival labels in isolation. They do not directly establish the persistent resistance to social influence assumed for the theory’s evidence-anchored agents.

  7. Writing and logical flow in Sections 4–5 need improvement. The succession of bold-headed paragraphs often reads as separate observations rather than a connected argument, leaving the relationships between experiments and proposed explanations underdeveloped. In particular, the transition from declining patching effects to population-level theory moves too quickly. A smaller intervention effect does not establish that agent-level explanations become inadequate. Clearer logical transitions would strengthen these sections.

Questions for authors

My major questions are summarized in the critical weakness. In addition, I have the following questions:

  1. Since the fixed-share example in Appendix F has no mean-accuracy peak, what does the theory explain about the observed decline beyond changes in measured evidence shares across population sizes? Which parameters were independently measured, and which were fitted or assumed?

  2. Is the attribution ranking validated beyond the selected Germany game? How does the score compare with temporal closeness alone or local accuracy gain alone? Please distinguish validation of the ranking from evidence that crop patching improves collective accuracy.

  3. What evidence links binary crop-probe responses to persistent resistance to opposing social information? Do the memory probes or interaction traces support the theory’s assumption of truth- and rival-anchored agents?

  4. Under matched conditions, what is the paired accuracy difference, with a confidence interval, between the best-performing mixed composition and the best homogeneous team? Please also define the shaded band in Figure 1d.

  5. Table 1 lists seeds per “(model, N),” but Figure 2 reports GPT-4o results. Were other models included in the population sweep? If so, does the transition from wrong consensus to polarization replicate, particularly for GPT-5.4?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.