acceptodds
Withdrawn from ICLR 2027

What Does the Mediator Learn? Mechanistic Analysis of Learned Intervention Policies in LLM Games

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When LLM agents are applied in multi-agent scenarios, such as automated negotiation and collaborative coding, they often fail to effectively coordinate. In this work, we study a third-party mediator agent trained with reinforcement learning (RL). Rather than applying fixed rules, a learned mediator can adaptively decide when and what to communicate, improving cooperation without accessing agent internals. We evaluate such learned mediators across three LLM families (Llama, Granite, and Qwen) on four games spanning social dilemmas and pricing competition, and provide a mechanistic account of the results. Our experiments demonstrate that a mediator trained on one LLM family transfers zero-shot to others, provided the target model responds monotonically to cooperation messages. A contrarian agent that inverts this response collapses cooperation despite active mediation. Moreover, we find that the value of interventions decomposes in terms of timing, which is model-agnostic and learned by the mediator, and content, reflecting the combined effect of the mediator's messages and peer communication. Interestingly, content adds value only when the population's baseline cooperation rate falls below an empirically estimable threshold . Above that threshold, timing alone suffices. An extended multi-way controller improves cooperation for susceptible populations by learning state-dependent template selection. No single message type dominates overall, suggesting the gain comes from matching message type to game state rather than from any single dominant framing. Ablations without player communication estimate this threshold empirically, requiring no retraining or access to model internals. These ablations also provide a practical criterion for predicting whether richer mediator vocabularies will help in a new deployment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Reinforcement Learning, Social Dilemmas, Zero-shot Transfer, Equilibrium Selection, Multiagent Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fawPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether an external PPO mediator trained with one LLM family can transfer to other model families without accessing player parameters. The mediator observes numerical interaction history and decides whether to send a fixed cooperation message to the least cooperative player. It is trained with Llama 3.3-70B and evaluated zero-shot with Granite 4.1-30B and Qwen 2.5-72B on IPD and Stag Hunt. The paper separates intervention timing from message content, proposes monotone response as a sufficient condition for non-harmful transfer, and uses an empirical susceptibility threshold to predict when content matters. Additional experiments examine peer communication, mixed-model persona assignments, contrarian players, multiple message types, and Bertrand pricing.

Critical strengths
  • The timing/content decomposition gives the paper a clear structure. Treating intervention timing as behavior-based and message content as population-dependent is a reasonable hypothesis, and the proposed monotonicity and susceptibility checks provide concrete quantities for evaluation.

  • The experiments include targeted tests of the proposed account. The contrarian setting examines a predicted failure case, and the mixed-population experiment keeps model composition fixed while changing persona assignment. These results show that model identity alone is insufficient to explain cooperation outcomes.

Critical weaknesses
  • The central susceptibility-threshold account is not sufficiently supported by the reported evidence. The paper brackets τ∗\tau^* using baseline cooperation rates of 42.8% and 46.6%, but these values cannot be traced to a row in the main or appendix tables under the stated never-intervene, peer-chat-active definition. The claim that Granite on Chicken falls below τ∗\tau^* is also difficult to reconcile with the 53.3% never-intervene value in Table 4, and no Chicken no-communication ablation is reported. More broadly, the propositions rely on strong assumptions, including successful detection of coordination-failure states, monotone response at the relevant states, and approximate separability of timing and content. The results are therefore better read as useful behavioral characterizations than as a general transfer theory unless these quantities and assumptions are defined and tested more directly.

  • Several internal inconsistencies and baseline results weaken confidence in the empirical conclusions. Section 4.1 states that PPO significantly improves over never-intervene in four of six cases, while the p-values in Table 1 support only three at p<0.05p<0.05. Section 4.2 reports a 2.2-point Llama IPD degradation after removing chat, whereas Table 2 gives 52.6 to 21.31, a 31.29-point drop. Table 2 and Table 4 also report substantially different Llama Stag Hunt results for the binary controller without explaining the setting difference. More importantly, the simple coop-thresh rule significantly outperforms PPO on both Llama games in Table 1. This does not show that learning is unnecessary, but the paper needs to explain what the learned mediator adds beyond a strong threshold policy, especially on its own training population.

Questions for authors
  1. How exactly are the 42.8% and 46.6% baseline cooperation rates computed? Please report the corresponding model, game, communication condition, seed aggregation, and table entries. How does the claim that Granite Chicken is below τ∗\tau^* relate to the 53.3% never-intervene value in Table 4?

  2. Could the authors reconcile the numerical inconsistencies between the prose and tables, particularly the “four of six” significance claim, the Llama IPD no-chat drop, and the different Llama Stag Hunt binary-controller results in Tables 2 and 4? Clarifying whether these rows use different checkpoints or evaluation protocols would be important.

  3. Why does coop-thresh outperform PPO on both Llama games in Table 1? A tuning-matched threshold comparison, together with intervention frequency and welfare after intervention cost, would help show what the learned timing policy contributes beyond the fixed message and a simple cooperation-rate rule.

  4. What evidence supports the claim that the 5-way controller performs state-dependent template selection? Figure 2 reports only marginal message frequencies. A state-by-message conditional analysis or an entropy-coefficient ablation would help distinguish meaningful selection from near-uniform exploration.

Reviewer_KinAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In summary, this paper uses PPO to train a simple mediator that sends messages to agents to encourage cooperation. The authors conduct experiments in social dilemma games to investigate the effectiveness of the mediator and the behavior of LLM agents in these environments.

Critical strengths

None

Critical weaknesses
  1. The theoretical arguments throughout the paper seem to rely more on a series of assumptions than on rigorous mathematical proofs. Moreover, some of the theoretical claims appear to be internally inconsistent. The susceptibility classifications of Granite and Qwen seem to be reversed according to the paper's own definition. Alao, the threshold tau^* appears to be chosen primarily to separate LLM agents into susceptible and non-susceptible categories according to the expironments. This tau should not be a threshold but should a susceptible level.

Additionally, I think the expironments setting is knid of boring. The LLMs are very strong and good at reasoning now, just using them to solve the traditional social dilemma is too easy for them. Maybe sequential social dilemma would be more interesting. Therefore I think teh theory and expironments are both below the standard of ICLR.

Questions for authors

Question

  1. How were c̄₀ = 42.8% (Granite) and 46.6% (Qwen) computed? How do you get the number of tau and c̄₀.

Just a suggestion Would you like consider some sequential social dilemma environments?

Reviewer_Lb3yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines whether an RL-trained "mediator" can help cooperation in three-player LLM-based social dilemma games. They train with Llama 3.3-70B players and test on OOD player models.

Critical strengths
  • The general idea of training models to be mediators is an interesting one and perhaps worth thinking about more broadly—e.g., should we mix in mediator training into foundation model training pipelines? If so, would that help in multi-agent cooperation games that are perhaps less toy than the ones in this paper.
  • The paper tests OOD generalization to different types of players.
Critical weaknesses
  • There are several parts of the paper that are very dense and difficult to follow (e.g., section 4.2).
  • It is unclear why one would use a small RL controller limited to fixed message templates and a fixed targeting rule, rather than an LLM trained (or simply prompted) to mediate the way a real mediator would (in natural language). The paper calls this "a deliberate design choice" (p. 9), but without even a prompted LLM mediator as a baseline it's a bit unclear how this would generalize outside of these fixed games and message types. At least in natural language, one could hope that the general mediation policy might overcome the deficiencies in the message types and improve the effectiveness of the message content.
  • The biggest weakness of this paper is the lack of a major "so what." The paper takes a relatively artificial setup, and it is hard to see what carries over to the applications it uses as motivation (pricing, coding, negotiation, cybersecurity) on page 1, none of which are tested apart from a pricing game. The mediator policy is really not learning much. Please correct me if I'm wrong, but as far as I can understand the mediator never sees what the players say to each other and whom to message is rule-based (?), so all it learns is when to send. That seems like it's closer to nudging than to mediation and we shouldn't expect it to learn optimal content anyways since it doesn't see actual ongoing conversation content. To make the case that this is useful or generalizable, I would have wanted to see the mediator in a setting closer to a real application as in the original motivation, with open-ended natural language mediation. While I appreciate that having a small MLP policy to nudge mediation is potentially useful for efficiency, it's not clear how much we're giving up by not just going straight to a prompted LLM.
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.