- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates influence and consensus formation in heterogeneous multi-agent LLM systems through a naming game. Experiments across seven model families show that stronger models generally achieve higher consensus win rates. The authors also examine the effects of population size and memory length. To explain these findings, they introduce a model-specific parameter called assertion strength and extend an existing consensus-energy framework using replicator dynamics. The framework is evaluated by fitting assertion strengths and comparing predicted dynamics with observed trajectories in three open-weight model families.
Critical strengths
-
The naming game with meaningless labels allows the authors to study influence among heterogeneous LLMs without interference from answer correctness.
-
Experiments across seven model families show that stronger models generally have higher consensus win rates, with additional experiments examining the effects of population size and memory length.
Critical weaknesses
-
The paper mainly extends the naming-game framework of Tanaka (2026) to heterogeneous LLMs. Given prior findings on LLM conformity and asymmetric influence, the observed results provide limited additional scientific insights.
-
The connection between the agent-level behavior and the proposed macroscopic dynamics relies on heuristic assumptions. Moreover, Equation (4) can yield invalid probabilities for the reported parameters, and the framework is evaluated using the same experimental runs from which its parameters are fitted. These issues weaken the proposed mechanistic explanation.
Questions for authors
-
Given prior work on LLM conformity and consensus formation, particularly Tanaka (2026), what new, non-trivial insights does this work provide beyond extending the existing naming-game framework to heterogeneous LLMs and observing higher consensus win rates for stronger models?
-
How are the macroscopic dynamics in Equation (3) justified by the agent-level model in Equation (4)? In addition, how does Equation (4) ensure valid output probabilities for the reported assertion strengths (e.g., m = (0.9, 0.1, 0), w = 3.44)?
-
Can assertion strengths be estimated independently and used to predict consensus outcomes on held-out experiments? Are these parameters stable and comparable across different model families?
4 comments
Author Response (2/3)
W2 & Q2 (from Equation 4 to Equation 3)
The connection between the agent-level behavior and the proposed macroscopic dynamics relies on heuristic assumptions.
How are the macroscopic dynamics in Equation (3) justified by the agent-level model in Equation (4)?
Equation 3 is obtained from Equation 4 under two closure approximations, which are stated in Appendix G: the agents share the population's composition, and the strength acting on a label is that of the class that brought it. We agree that this step is heuristic. For this reason, the paper treats Equation 2 as a posited energy and tests it directly, in the spirit of replicator dynamics [1].
We would like to emphasize that this test is now out of sample. With strengths measured outside the population, the flow of Equation 3 agrees with the measured motion in three families (r = 0.61–0.67), and the energy falls in 84–88% of the rounds. Additionally, the agent-level prediction in Appendix K uses neither Equation 3 nor the closure, and names the large model as the most likely winner in all five families.
To make this clearer, we revised Section 4.3 (pages 7–9) and the paragraph "Measurement" in Appendix J (page 26), and added Appendix K (pages 27–29).
[1] Hofbauer & Sigmund, Evolutionary Games and Population Dynamics. Cambridge University Press, 1998.
Author Response (3/3)
W1 & Q1
The paper mainly extends the naming-game framework of Tanaka (2026) to heterogeneous LLMs. Given prior findings on LLM conformity and asymmetric influence, the observed results provide limited additional scientific insights.
We would like to clarify the contribution of this paper relative to the prior work from three perspectives.
- Who wins is a measurable property of single models.
Tanaka [1] studies homogeneous populations, in which the winning convention is a lottery. We show that, when models are mixed, the outcome is no longer a lottery (the large model wins most often in every family; Figure 3), and that, in the three families we analysed, the order of win rates follows the order of a strength that is measured on each model alone, without using any population run.
- Outcome of a population, not susceptibility of one model.
Conformity studies measure how much one model yields to a majority [2, 3]. They do not say which convention a mixed population adopts. In LLM naming games, a committed minority of scripted agents can overturn a convention [4]; our agents are unscripted and differ only by model.
- A mechanism tested by intervention.
We tested whether in-context learning (ICL) ability causes the influence, by an intervention inside one model. In the Qwen 2.5 population, we modified only the large model (32B). First, we computed an induction score for each of its attention heads [5]: on a sequence of random tokens repeated twice, the attention from a token in the second copy to the token that followed its first occurrence. We then zeroed the output of the 128 heads with the highest score. We compared this with two controls. In the first, the intact model is run at the temperature that gives it the same output entropy as the ablated model, so that both answer equally sharply; it wins 29 of 36 trials. In the second, 128 other heads are zeroed, with the same number per layer; it wins 32 of 36 trials. With every output constrained to be a valid label, the model without its induction heads wins 11 of 36 trials (p = 2e-5 against the first control). Without this constraint it wins 6 of 36 trials and produces many invalid outputs. Therefore, the influence of this model is removed by reducing its ICL ability, and not by lowering its sharpness or by removing the same number of other heads. The ablation covers one model of one family. We searched for prior work that links ICL ability to influence in a population and found no experimental test of it. The closest work we found is a position paper that proposes to study multi-agent LLM systems with mechanistic interpretability, and gives attention heads that copy tokens between agents as a hypothetical example, without experiments [6]. Debate studies with a correct answer report that accuracy can fall even when stronger models outnumber weaker ones, because models give up correct answers in response to their peers [7]; they describe the behaviour and do not identify a mechanism.
To make the contribution clearer, we revised the abstract (page 1), the introduction and contributions (pages 2–3), Section 4.3 (pages 8–9) and the conclusion (page 10), and added [6] and [7] to the related work in Section 5 (page 10).
[1] Tanaka, When Is Collective Intelligence a Lottery? Multi-Agent Scaling Laws for Memetic Drift in LLMs. 2026. https://arxiv.org/abs/2603.24676
[2] Weng et al., Do as We Do, Not as You Think: the Conformity of Large Language Models. ICLR 2025.
[3] Zhu et al., Conformity in Large Language Models. ACL 2025.
[4] Flint Ashery et al., Emergent Social Conventions and Collective Bias in LLM Populations. Science Advances, 2025.
[5] Olsson et al., In-context Learning and Induction Heads. 2022. https://arxiv.org/abs/2209.11895
[6] Lee et al., Towards Ethical Multi-Agent Systems of Large Language Models: A Mechanistic Interpretability Perspective. 2025. https://arxiv.org/abs/2512.04691
[7] Wynn et al., Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debate. 2025. https://arxiv.org/abs/2509.05396
Again, we appreciate the detailed comments from the reviewer. Please let us know if our responses address your concerns. If needed, we will be happy to clarify them further.
Kind Reminder to Reviewer Kbez
Thank you again for your valuable review and constructive feedback on our paper. As we have not yet received a response, we would like to kindly remind Reviewer Kbez that we have thoroughly addressed your concerns. Specifically, we now estimate the assertion strengths independently of the runs, so that the framework is tested out of sample, and added a single-agent rule that always gives valid probabilities (Equation 5). We have also clarified our contribution relative to Tanaka (2026) and conducted an additional intervention experiment involving induction-head ablation. These updates have been incorporated into the revised paper, which we believe has been significantly strengthened as a result. We would greatly appreciate it if you could review these updates and kindly reconsider your score in light of our revisions to address your comments.
Author Response (1/3)
We thank the reviewer for the careful reading and precise questions. Please let us know if our responses in the following address your concerns.
We revised the paper based on the reviewers' comments, and the major edit was highlighted with coloring (red). Please also check the updated manuscript.
W2 & Q3
Yes. To address your concern, we now estimate the assertion strengths independently of the runs. Each model is queried alone, with the same prompt as in the game, on synthetic memories that occur in no population run, and the same regression as before is fitted to these isolated answers. No population run enters w, so every run is held out.
With this independent estimate, the results of the submission are reproduced for Gemma 4 and Qwen 2.5. For Gemma 4, the strengths are 2.96 (31B), 2.67 (26B-A4B) and 2.47 (E4B), in the same order as the values fitted on the runs (3.44, 3.19, 2.52). The well of the 31B model is still the deepest, the flow agrees with the measured motion at r = 0.67 (0.65 before), and the energy falls in 86% of the rounds (85% before). The order of w matches the order of win rates in every pair of classes in Gemma 4, Qwen 2.5 and Gemini.
As for stability, the strengths measured alone have the same order as those fitted on the runs for Gemma 4 and Qwen 2.5 and differ from them by at most 0.53 (table in Appendix H), and every pair of their models keeps its order in at least 99% of bootstrap resamples. For Gemini, the two estimates differ: only the one measured alone follows the ladder. Additionally, when the memory states of the runs are replayed to each model alone, the model reproduces what it said in the run: the probability of the majority label agrees within 0.03 for the six open-weight models. As for comparability, all models are measured on the same kind of memories with the same regression, so the strengths share one scale; they range from 1.18 to 3.55 across three families.
Therefore, we believe your concern is resolved: the framework is no longer evaluated on the runs from which its parameters are fitted. In the revised paper, we rewrote Section 4.3 (pages 7–9) around the independently estimated w, redrew Figure 9 and Figures 11–14 (Appendix I, pages 23–25) with it, and described the estimation in Appendix H (pages 22–23). Thank you for this question, which made the paper stronger.
W2 & Q2 (valid probabilities)
Thank you for pointing this out. You are right: Equation 4 does not ensure this. For your example it gives x = (1.148, −0.148, 0), and it is a valid distribution on the whole simplex only for w ≤ 1.
To address this, we added the rule x_k ∝ m_k exp(w m_k) (Equation 5), which is a valid distribution for every w, and we state on which states Equation 4 is invalid. Equation 4 is the first-order expansion of Equation 5 in w, and both have the same continuous-time limit (Equation 3), so the energy and the flow of Section 4 are unchanged. In the revised paper, the least-squares regression on this first-order term is kept as the estimator of w. The quantities tested in Section 4 (which well is the deepest, the direction of the flow, and whether the energy falls) do not depend on the overall scale of w. When Equation 5 is fitted directly to the utterances of the runs, the order of the strengths within Gemma 4 and within Qwen 2.5 is the same.
In the revised paper, we added the corrected rule as Equation 5 in Appendix H (page 23), together with the fraction of visited states on which Equation 4 is invalid.