acceptodds
Withdrawn from ICLR 2027

Let Them Talk: A Multi-Level Analysis of Communication Between LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A swarm of large language model (LLM) agents has demonstrated remarkable abilities in solving complex tasks by interacting and coordinating with one another at scale. In this work, we investigate how communication affects the behavior of LLMs in repeated multi-agent interactions with incomplete information, such as the Iterated Prisoner's Dilemma and Public Good Game. Our multi-level analysis of agents' signaling behavior, chain of thought, layer activations, and population dynamics reveals that communication generally fosters trust, leading to the emergence of cooperative strategies that benefit both parties in dyadic interaction. However, the same channel also makes the population of agents more exploitable to predatory “free riders" when the benefit of cooperation is non-excludable. Together, these findings underscore crucial ethical and strategic considerations for responsible deployment of multi-agent systems.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM evaluation, interpretability, game theory, inter-LLM communication, cooperative AI, multi-agent safety

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dJp4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the effects of communication on cooperative behaviour among LLM agents through two social dilemma games: two-player iterated Prisoner’s Dilemma (IPD) and four-player Public Goods Game (PGG) . The authors use Gemma-3-12B-it for all experiments, comparing 30-round games under two conditions: open (agents can communicate) and silent (no communication). Agents take one of three personas: neutral, benevolent, or deceptive, and the authors use a trembling hand mechanism, where the agent’s action is flipped with probability 0.1.

The authors examine agent behaviour at multiple levels, including message content and reasoning traces, internal representations through residual-stream probing, and population-level outcomes through replicator dynamics.

They find that communication generally improves cooperation and helps agents recover from unintended defections in IPD, while also reducing the effectiveness of deceptive strategies. However, in PGG, communication can increase the ability of deceptive agents to exploit cooperation. The authors conclude that the effects of communication depend on whether the benefits of cooperation are excludable.

Critical strengths
  1. The comparison of communication across two different social dilemmas provides an interesting perspective on multi-agent cooperation. Evaluating both IPD and PGG with and without communication allows the authors to demonstrate that communication does not always have the same effect on collective outcomes. The finding that communication can support cooperation in one setting while facilitating exploitation in another is relevant to the design and evaluation of multi-agent systems.

  2. The paper attempts to examine communication beyond directly observed actions and payoffs. The analysis of message content, agent reasoning, internal representations, and population dynamics is a potentially useful approach to understanding multi-agent interactions.

Critical weaknesses
  1. The main conclusions are not sufficiently supported by the experiments, particularly the proposed causal mechanisms. The authors conclude that communication supports cooperation when benefits are excludable but facilitates exploitation when they are non-excludable. However, this conclusion is based on comparing two games that differ in several important dimensions, including the number of players and payoff structure. It is therefore unclear whether excludability is the determining factor or whether other differences between the environments explain the results. This is consequential as identifying the mechanisms through which communication affects cooperation is a central goal of the paper.

  2. Several experimental and analytical choices are insufficiently justified, limiting the generalisability and interpretation of the results. For instance, all experiments use Gemma-3-12B-it, and Appendix B notes that "we tested a variety of models before deciding on one for fair comparison across experiments," but no further explanation or results are provided. This makes it difficult to assess whether the findings generalise beyond the selected model. Similarly, while the trembling-hand mechanism is motivated as a way of introducing uncertainty, its effects are not isolated experimentally, making it unclear how much it contributes to the reported findings. The replicator dynamics analysis is also not sufficiently motivated in relation to the central research question, and it is unclear what additional conclusions can be drawn from this analysis beyond the behavioural results already presented.

  3. The novelty of the contribution is not sufficiently established. The finding that communication can both facilitate cooperation and enable exploitation has already been explored in the literature on communication and social dilemmas, including more recent work on multi-agent learning and LLM agents. The related work section does not sufficiently engage with some of this literature, making it difficult to assess what is genuinely new about the findings. While the paper attempts to provide a multi-level analysis, it is also unclear to what extent the additional analyses provide new insights into the effects of communication, rather than further characterisations of the observed behaviour. This limits the significance of the contribution, particularly given the paper's emphasis on providing a deeper understanding of communication between LLM agents.

Questions for authors
  1. How can the authors establish that the differing effects of communication between IPD and PGG arise specifically from benefit excludability, rather than other differences between the environments? Are there controlled comparisons or additional analyses that could help distinguish these explanations?
  2. What motivated the selection of Gemma-3-12B-it, and what were the results of the preliminary experiments with other models mentioned in Appendix B? Similarly, what motivated other design choices, such as the choice of social dilemmas or number of rounds?
  3. Are the agents aware they are taking an action due to the trembling hand mechanism?
  4. How do some of the design choices add to the analysis and to answering the main research question? Particularly the trembling hand mechanism or the replicator dynamics analysis.
Reviewer_eZjGOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how natural-language communication influences cooperation, trust, and strategic behavior among LLM agents in repeated multi-agent interactions. Using game-theoretic environments, the authors examine whether 1. communication improves coordination, 2. how agents represent trustworthy and deceptive intentions internally, 3. how these behaviors influence long-term dynamics.

Critical strengths
  1. Interesting findings about the dual effects of communication in multi-agent systems. studying whether natural-language communication improves coordination and collective outcomes among LLM agents.The contrast between the Iterated Prisoner's Dilemma and the Public Good Game is interesting
  2. Comprehensive multi-level analysis connecting observable behavior, internal representations, and population dynamics.
Critical weaknesses
  1. all experiments were done on relatively tiny and old models, if possible at least some verification of partial of the conclusions on larger/newer models would benefit audience to know whether its conclusions generalize
  2. The probe results are largely explained by the prompt design. The message prompt states the agent’s already-decided action, so decoding the “upcoming action” from message-position activations reads information in the context. Lies come almost entirely from dec, whose persona is decodable at 97–100%, so the PCA separation of lies from honest promises may just separate personas. A within-persona comparison (at leaset dec’s broken vs. kept promises) is needed. also a line of prior work on lie and deception probes are missing
Questions for authors

see weakness

Reviewer_qGV7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies messages between Gemma-3-12B-it agents prompted to be neutral, benevolent, or deceptive. With and without messages, agents play an Iterated Prisoner's Dilemma with randomly flipped actions and a four-player public good game, where cleaning sustains apples that others can harvest.

Critical strengths

This paper provides some interesting insights. For example, this paper finds that the same message channel helps some non-deceptive LLM agents cooperate in the Iterated Prisoner's Dilemma but enlarges a deceptive free-rider's gain in a public good game. This cautions against assuming communication always helps.

The paper also proposes the ability to withhold benefits from non-contributors as a condition for messages to help cooperation, which gives later work a specific hypothesis about when communication is useful.

Critical weaknesses
  1. There may be some claim-evidence misalignment. Some key arguments and insights are not well-supported. For example,
    • Classifiers trained on hidden activations provide the main evidence about internal mechanisms, but the prompts already state the persona, action, and opponent history, so the classifiers may only identify prompt content. Specifically, Prompt 3 states "You have already decided to choose" the action before the message is written. Nearly all lies in Figure 5 come from the deceptive persona, whose identity the activations encode at 97–100% accuracy. So lie detection independent of persona cues remains unverified. The authors may consider classifiers using prompt fields alone, comparisons of kept and broken promises within one persona, and causal interventions.
    • This paper concludes that the ability to withhold cooperation benefits determines whether messages help, but the two games differ in other ways, so this explanation is not isolated. Specifically, it is worthy noting that the games differ in group size, message privacy, payoffs, and prompts. Human experiments show that payoffs alone can reverse the effect of unenforced messages [3]. Within the public good game, messages lowered a neutral newcomer's payoff among benevolent residents but raised a deceptive newcomer's (See Table 1). So messages have opposite effects under the same benefit-sharing rule.
    • The paper credits two-player gains to distinguishing accidental from deliberate defections. But only the condition allowing messages mentions random action flips. So the comparison may confound communication with knowledge of these flips. The message prompt says that "either player can occasionally end up taking an unintended action" (Prompt 3), and no decision prompt repeats it. Accident excuses appear nearly absent in Figure 2(a), and the case study's return to cooperation follows two deliberate defections (See Figure 1(b)).
    • Claims that communication "generally fosters trust" and makes cooperators more forgiving are based mainly on the neutral persona, so they may not hold beyond it. Cooperation appears to rise noticeably only for the neutral agent facing neutral or benevolent opponents (See Figure 2(b)), without a test for any pairing. The benevolent agent, whose prompt asks it to value the opponent's payoff as much as its own, cooperates in about a third of rounds in self-play, and its best-matching strategies under communication are mostly strategies that persistently punish defection (See Figure 4).
    • The public good section attributes the free-rider's larger gain with messages to false pledges that keep residents cleaning, but residents see every past action and the text also credits a "selfish reputation". So the role of deception is not isolated. Prompt 4 lists each agent's past choices, so false pledges seem easy to detect. When a deceptive agent joins, messages raise the group total while the residents' payoffs fall (Table 1), so the residents seem to lose through a less equal split.
  2. The generalizability of the experiment results and insights are questionable. All results come from one model, Gemma-3-12B-it, chosen after unreported tests of other models, so the conclusions about LLM agents may not transfer to other models or settings. Appendix B states that the authors "tested a variety of models before deciding on one". But no results for those models appear. Each persona also comes from one fixed prompt, with one action-flip probability and a stated 70% chance of continuing that does not match the fixed 30 rounds (Section 2 and Prompt 1). Repeating the core comparisons with two or three other model families and reporting the preliminary results would help.
  3. The paper provides no code or game transcripts and omits several settings, so the public good findings and the classifier results cannot currently be reproduced or verified. Considering the exponentially growing volume of machine learning submissions, it is strongly encouraged that the authors may submit the code along with the submission. Otherwise, it is impossible for reviewers to verify the correctness of the methodology, whether the code align with the method, whether the code can reproduce the reported results, and whether the results truly support the method.

In addition, one suggestion is that some highly related additional references seem missing. The authors may consider incorporating them into this paper such as [1,2,3]. [1,2] are among the first studies on agent simulation and agent behavior probing with behavioral economics games. [3] is also very related.

[1] Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies ICML 2023

[2] Can Large Language Model Agents Simulate Human Trust Behavior? NeurIPS 2024

[3] “I'm just a soul whose intentions are good”: The role of communication in noisy repeated games. Games and Economic Behavior 2017

Questions for authors
  1. Is the classifier's target the action decided in the same round? Which Prompt 3 places in the context, or the action of the next round?
  2. How does each classifier compare with a logistic regression on the persona, the decided action, and the opponent's visible history?
  3. Would the contrast between the two games persist in a four-player game where cleaners can exclude non-cleaners from the harvest?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.