acceptodds
Under review as a conference paper at ICLR 2027

Group Alignment: Internal and External Incentives for Cooperation in Social Dilemmas

Abstract

AI agents are becoming increasingly autonomous and, similarly, act in more and more complex environments. In such environments, agents pursue underspecified goals and need coordinated interaction with other agents, however, how this cooperation is undertaken is still poorly understood. In this paper, we characterize the cooperation between different agents in scenarios and prove an inefficiency theorem of contracts in such complex environments. We show two orthogonal categories of solutions to the collaboration problem grounded in different economic theories of cooperation, namely contracts and "we”-frames. Frames internally shift reasoning patterns of different agents towards cooperative behaviour and enable finding the best individual mutually beneficial action for the group, while contracts align incentives with an external device for the same objective. We study the effect of a range of contract representations, from natural language to formal contracts across all frames in classical matrix-based games and common-pool resource environments. We find that in matrix games, contracts reliably improve mutual-advantageous actions. We observe that contracts of any kind perform 15.33% worse in environments where agents choose selfishly compared to when they instantiate the group frame, and 41.76% worse in common-pool resource environments. Our results suggest both self-negotiated contracts and "we”-frames improve cooperation over LLM behaviour without such devices, with "we”-frames being more robust in complex settings.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Social Dilemmas, Cooperative AI, Contracts, Common Pool Resource, AI Alignment, Incomplete Contracts

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_r1stOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work studies cooperation among LLM agents through two mechanisms: external incentive alignment via contracts and internal group alignment via proposed we-frames. The authors evaluate these mechanisms in classical matrix social dilemmas in a repeated common-pool-resource environment, GovSimContract. Five LLMs are tested under no-contract, natural-language-contract, and code-contract regimes.

Critical strengths
  1. writing is easy to follow, it also provides prompts, contract examples, construction details, model/sampling information, and code, seeds for reproduction..
  2. the qualitative contract-failure analysis is interesting.
Critical weaknesses

My main concerns are the novelty of this work.

  1. The contract-impossibility result is very close to existing contractibility/observability results. Haupt et al. [3] already formalize contracts in multi-agent systems in terms of contractible observations, prove socially optimal implementation when deviations are observable and the contract space is rich enough, and prove that more informative contractible features weakly improve equilibrium welfare. They even experimentally degrade the probability of detecting contractible behavior and show welfare falls as observability worsens.

  2. Empirically:

  • CoopEval[1] already evaluates contracts alongside other mechanisms in LLM social dilemmas and finds contracting to be one of the most effective mechanisms for achieving cooperation.
  • Wyse et al. [2] also studies LLM agents that negotiate their own contracts and explicitly compares representations ranging from natural-language contracts to contracts compiled into executable code.
  1. The empirical cooperation gap does not measure the theoretical cooperation gap. Experimentally, the paper defines cooperation gap as the welfare difference between two prompting conditions while holding the contract regime fixed. But the theoretical impossibility concerns something different: whether the first-best welfare profile can be implemented by a contract given limited observability.

[1] Tewolde, E., Zhang, X., Piedrahita, D. G., Conitzer, V., & Jin, Z. (2026). Coopeval: Benchmarking cooperation-sustaining mechanisms and llm agents in social dilemmas. arXiv preprint arXiv:2604.15267.

[2] Wyse, T., Bustos, K., Volkova, Y., & Kleiman-Weiner, M. (2026). Commitment To Cooperation With Self-Negotiated Contracts. arXiv preprint arXiv:2607.22750.

[3] Haupt, A. A., Christoffersen, P. J., Damani, M., & Hadfield-Menell, D. (2022). Formal Contracts Mitigate Social Dilemmas in Multi-Agent RL. arXiv preprint arXiv:2208.10469.

Questions for authors
  1. Can you clarify what is theoretically new relative to classical moral hazard/incomplete-contract results and prior work on contractible observability?
  2. Can you add a communication-matched baseline to isolate the effect of contracts from negotiation and cheap talk?
  3. Can you experimentally test the theorem by directly varying whether the relevant effort/action is observable to the contract?
Reviewer_z8weOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies LLM agents cooperation in social dilemma settings. The authors introduce the concept of non-contractibility and show theoretical limitations of contracts.

The paper investigates two mechanisms for improving cooperation: internal incentives through "we"-framing and external incentives through contracts, as well as their interaction. The experiments compare agents with "I", "we" or no framing under three conditions: no contract, natural-language contract and code contract. All the combinations are evaluated in classical, more toy game-theoretic settings and in a more complex common-pool resource scenario in which agents need to efficiently manage shared resources.

Critical strengths
  1. Theoretical and empirical contributions to an important problem of multi-agent cooperation. The paper is well-written and easy to read.

  2. Comparing the framing-based approaches and contract-based approaches is a valuable angle. The results on the limitations of contracts and their potential inefficiencies in complex environments are compelling to me.

Critical weaknesses
  1. Unclear if the results generalise to heterogenous multi-agent settings. Will benefits of "we"-framing extend to a group of agents with very different capabilities?
  2. "We"-framing may not be robust. It is unclear how fragile the results are if we consider more complex settings or framing prompts variations.
Questions for authors

You only use self-play in the experiments. Do you think "we"-framing benefits would hold if you have groups of agents of different capabilities and/or model families?

Reviewer_yXMmOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies cooperation among LLM agents in social dilemmas, focusing on two approaches to promoting cooperation: aligning external incentives through contracts and shifting agents' reasoning towards cooperation through "we"-frames. The authors draw on incomplete-contracts theory and team reasoning to examine the limitations of contracts when agents' actions cannot be fully observed or verified. They further provide a theoretical analysis showing that, under certain conditions of non-contractibility, contracts cannot implement the utilitarian welfare optimum as a nash equilibrium.

Critical strengths
  1. With the development of LLM-based multi-agent systems, understanding how to coordinate agents with potentially conflicting objectives and avoid collective inefficiency caused by individual rationality is an important research problem. This paper attempts to connect this problem with social dilemmas and incomplete-contracts theory in economics, providing a useful perspective on multi-agent cooperation. 
  2. This paper identifies several failure modes of contracts, including over-prudency, failure to renegotiate, and erroneous implementation of negotiated contracts. These observations provide useful insights into coordination challenges in multi-agent systems.
Critical weaknesses

I have the most direct question. Although the multi-agent cooperation problem addressed in this paper has practical significance, current research is mostly guided by existing economic theories to compare the effects of different external constraints and prompts on LLM behavior. The author did not fully clarify what scientific hypotheses about the cooperative ability of language models were previously unclear about the purpose of these experiments. Therefore, although the paper demonstrates changes in cooperative behavior, these experiments are not yet sufficient to establish a clear and explanatory scientific discovery. I think this does not meet the expectations of the ICLR conference for scientific research issues. Similar conferences on multi-agent systems such as AAMAS may be more suitable.

The paper regards Group Alignment as one of its main contributions, but its actual implementation mainly relies on adding simple target prompt words in the context. Although I do not believe that ICLR papers must propose new training algorithms or model architectures, even as an empirical study, this paper still needs to provide substantial and novel findings beyond simply comparing the effects of prompt words.

Above, I acknowledge the author's time, effort, and interesting perspective on this article. But I think this paper does not meet the requirements and expectations of ICLR. I will give a clear rejection and hope that the author can fully consider other more suitable meetings.

Questions for authors

NA

Reviewer_Vr6LOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The article is an experimental work that studies how to enforce LLMs to cooperate in social-dilemma settings (such as prisoners-dilemma) where full cooperation is not achievable by individually trying to maximize the own utility.

The authors consider two ways of enforcing cooperation. First, contracts, where agents explicitly agree on some terms to be respected, and second, frames, where LLMs are explicitly said to maximize the social welfare rather than their individual utility on the prompt.

The authors show that using both ways to enforce cooperation actually improves the LLMs cooperation, but that softer levels of enforcing lead to less cooperative agents.

Critical strengths
  • The article considers a nice problem, namely, how to encourage LLMs to cooperate
  • The considered problem is a well fit for ICLR
  • The article presents several experiments using LLMs from different families
Critical weaknesses
  • The article is difficult to read, with several concepts never explained or defined. Moreover, several times the article refers to things that will be defined later, without a proper introduction, leaving the reader to reach those posterior sections to be able to understand the past discussions. Some examples are : Slurm schedule, Ex-post non-contractible, Ex-ante non-contractible, "we"-frame, utilitarian efficiency, among others.
  • Figure 1 is never referenced on the text nor explained.
  • Several discussions are imprecise. For example, in line 140, it says "a selfish but rational agent would reserve the whole shared resource for itself ...". This is never the case in multi-agent environnements where agents have similar power. Indeed, agents might compete for the resources, or create a congestion as in congestion games, but rarely they will be able to starve the other agents.
  • Non-contractability seems a very important concept in the article. Actually, the authors claim they introduce it to the literature. Still, this is never properly defined.
  • Theorem 1 lacks a whole technical section explaining the required context to understand the statement and consequences. Still, this is left for the appendix.
  • The first time the article mentions that it is actually an experimental work is in Page 4, Line 208. This is an important fact to mention from the beginning.
  • Section 3.3 talks about months but only one of the two experiments has months as time unit.
  • The phrase "finally they remember parts of the discussion" in Line 258 requires more formalisme.
  • Figure 3 talks about "utilitarian efficiency" but this is never defined. Moreover, the article introduces the cooperation gap, the utilitarian welfare accuracy, and the survival rate, but no experiments use these measures.
  • The main conclusions of the article is that LLMs, when forced to cooperate, they cooperate. Unfortunately, either the conclusions are more complex but the current writing of the article does not allow to get them, or the conclusions are not very surprising. In either case, the contributions are not at the bar of ICLR.
Questions for authors

None

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary

Dear AC and Reviewers,

As the discussion period is ending, we kindly ask you to check our responses if you haven't already.
We hope they address the remaining concerns.

Summary of the Strengths

We would like to thank the reviewers for their valuable feedback, and we appreciate the positive assessment of the contributions of our work in the following points:

  • Recognizing the timely problem of LLM cooperation:
    • z8we “Theoretical and empirical contributions to an important problem of multi-agent cooperation.”
    • Vr6L “The article considers a nice problem, namely, how to encourage LLMs to cooperate” ,
    • yXMm “With the development of LLM-based multi-agent systems, understanding how to coordinate agents with potentially conflicting objectives and avoid collective inefficiency caused by individual rationality is an important research problem.”
  • The paper is well written:
    • r1st “writing is easy to follow”,
    • z8we “The paper is well-written and easy to read.”,
    • yXMm Presentation 3: Clear and easy to follow.
  • Empirical contribution acknowledged by all reviewers:
    • yXMm “These observations provide useful insights into coordination challenges in multi-agent systems.”
    • Vr6L “The article presents several experiments using LLMs from different families”,
    • R1st “the qualitative contract-failure analysis is interesting.”
    • Z8we “Comparing the framing-based approaches and contract-based approaches is a valuable angle.”
  • Fit for ICLR: Vr6L says “The considered problem is a well fit for ICLR” .

Rebuttal Actions

To address all the points from the reviewers, we have taken action to improve both the empirical strength and presentation of our work:

  • Clarified the presentation and the research questions (Vr6L, yXMm). We restructured the introduction and research questions to make them more explicit as requested; we marked modifications in blue.
  • Added observable effort ablation (r1st, yXMm). In the same 4×4 games, we made effort observable to the contract (Appendix G.2).
  • Added a communication baseline (r1st). In a cheap-talk condition, players exchange as many messages as in contract negotiation, but nothing is binding (Appendix G.4).
  • Tested heterogeneous pairs (z8we). We ran the frames with pairs of models of different capability (GPT-5.6 Terra, Luna and Sol) and of different families (GPT-5.4 with DeepSeek-V4-Flash).
  • Checked robustness to the frame wording (z8we, yXMm). We compared the underspecified team-reasoning prompt with an explicit-objective prompt (Figure 10).
  • Clarified novelty and the measured gap (r1st). We compared our work with Haupt et al., CoopEval and Wyse et al., and added a measure closer to the theoretical gap in the rebuttal.

All additional experiments are consistent with the claims in the main paper.
Thank you for all the time and effort in reviewing our work.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.