- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper analyses the impact of LLM capabilities and test time compute in strategic multi agent negotiations. Considered domains for this LLM benchmark are item allocation, diplomatic negotiation and public good funding. Overall 30 models are considered in 7160 runs. Highly strategic patterns like exclusion occur only in highly capable models. Further coordinated weaker agents can outperform stronger agents, however this advantage does not increase with team size.
Critical strengths
The choosen environments cover competition, capapbitly and group size, three differnt relevant axis across 30 models and 7160 runs which is a substantial amoutn of experimentation.
Consistent capability and test time compute relationships established.
Critical weaknesses
Observing the behavior of minimal coalitions and exclusions in LLM is interesting. The concepts are well know and within the training data. So is this amplified by certain methods/ optimisaiton pressure?
Models of weaker capabilities might be overwhelmed with too many items within their context. How much of their degredation is due to capabilities (in context) vs actual strategic deliberations?
Questions for authors
Thanks for the llama baseline that is a first step towards cross provider negotations (at matched elo) Can you please extend this to within lab and cross-lab negotiations.
Can you please ablate the impact of choosen prompt for different model capabilities and see when exclusion of certain agents or formation of minimum winning coalitions emerge.
Please attribute the capability payoff relationship to a) strategic decision makind and b) miscellaneous (arithmetic errors, invalid, parsing etc)
How to make teams of weak agents scale with team size?
2 comments
Response to Reviewer GyM7 [Part 2 of 2] Within-lab vs Cross-lab, Prompt Ablations, Strategic Decision-making vs Misc, Scaling with Team Size [Q1, Q2, Q3, Q4]
Response to Question 1:
Thanks for the llama baseline that is a first step towards cross provider negotations (at matched elo) Can you please extend this to within lab and cross-lab negotiations.
Yes! We run an analysis to compare the within-lab scaling trend versus the cross-lab scaling trend for all three games. Specifically, because our baseline is GPT-5 nano, we investigate the scaling trend of within-lab (GPT-5 nano vs. other OpenAI models) vs. cross-lab (GPT-5 nano vs non-OpenAI models). In Rebuttal Figure 16, we show that we see no statistically significant difference in the payoff trend across the three.
Response to Question 2:
Can you please ablate the impact of choosen prompt for different model capabilities and see when exclusion of certain agents or formation of minimum winning coalitions emerge.
Yes. Here, we ablate three different prompts. In addition to the current default prompt, we ablate a version of the prompt which contains language that encourages the model to consider being collaborative (“Pursue your own utility while also caring about each other agent's utility. Work toward an agreement that benefits everyone.”). We also use a third prompt variant where we insert language in the prompt to encourage the model to be competitive and emphasize that its goal is to maximize its own payoff without caring about others' payoffs (“Your sole goal is to maximize your own total utility after discounting; other agents' utilities do not matter to you.”).
We find a very interesting effect on the emergence of exclusionary collisions. Under the default prompt, exclusionary coalitions emerge around Elo 1422 and 1468. Under the competitive prompt, they emerge earlier and more frequently, beginning at Elo 1422. Under the collaborative prompt, the emergence of exclusionary coalitions disappears altogether. Thus, these ablations reveal that prompting can actually influence the effect of exclusionary coalitions in an interesting way.
Response to Question 3:
Please attribute the capability payoff relationship to a) strategic decision makind and b) miscellaneous (arithmetic errors, invalid, parsing etc)
We see that some percentage of it is, but it is not statistically significant. Even when removing all execution issues in terms of parsing errors or invalids, and then also when additionally removing numerical or arithmetic issues, we still see a clear positive slope. Please see Rebuttal Figure 18 for the exact results.
Response to Question 4:
How to make teams of weak agents scale with team size?
It would be hard for the performance of the weaker agent team to further scale because it already achieves near 100% performance with the 3-agent team. However, we found a way to at least ensure that the performance does not degrade as we scale up the number of agents, which is by holding the complexity of the environment roughly constant as we scale the team. In our experiment where we hold the number of items constant at 10, we do not see a degradation as we scale team size, and we actually see a small improvement scaling from 3-agents to 9-agent teams. Please see Rebuttal Figure 19 for the exact result here!
Response to Reviewer GyM7 [Part 1 of 2] Free-form Negotiation Scenarios and Teams of Weaker Agents [W1, W2]
Dear Reviewer GyM7,
Thank you for the feedback! Please see the supplementary material, ICLR 2027 Rebuttal Figures, for Rebuttal Figures 1–19 referenced below.
Response to Critical Weakness 1:
We agree that minimum-winning coalitions are established concepts that may have appeared in the training data when the models were trained. The goal and purpose of this paper was to take those finished models that have already been trained, and to study their behavior and scaling properties when placed in a wide range of negotiation settings. How the characteristics that we observe these models exhibit during negotiation came about through its training process is both (1) unobservable to us (since we do not have access or visibility to the training process and intermediary checkpoints of these models during their training/optimization), and (2) it is a question which is outside of the scope of this paper.
Response to Critical Weakness 2:
This is a great point! We find that much of the degradation is indeed driven by the increased complexity of reasoning over more items. We ran an additional experiment comparing (1) scaling the number of items alongside the number of weak agents, as in our original setup, vs. (2) holding the number of items constant at 10 as the team grows. As shown in Rebuttal Figure 15, the degradation disappears when the number of items is held constant. This supports our hypothesis that weaker agents struggle with increased game complexity, but it does reveal a new learning that a significant portion of this complexity comes from juggling more items rather than just the raw number of agents itself.
We expect that, in real-world settings, larger groups of agents will generally operate in larger, more complex environments (i.e. analogous to more items). That’s why the default setting in our environment is to also scale the number of items along with scaling the number of agents, such that each agent has the same average number of items to begin. This new experiment both supports our original hypothesis and provides a valuable new insight into the root source of the degradation.