acceptodds
Withdrawn from ICLR 2027

SidConArena: An Environment Evaluating Agents in Open-Ended, Positive-Sum Bargaining Game

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Evaluating LLM agents requires dynamic environments that go beyond static reasoning and zero-sum games. Real-world economic interaction is often open-ended and mixed-motive: agents must negotiate, create positive-sum surplus, compete for scarce assets, and plan under delayed returns. We introduce SidConArena, a new benchmark framework for evaluating LLM agents in open-ended, positive-sum bargaining. SidConArena formalizes a multi-player economy as a finite-horizon partially observable stochastic game with three coupled phases: natural-language negotiation with binding trades, deterministic converter-based production, and sealed-bid auctions for long-term assets. The framework combines structured observations, a phase-specific reference agent, a validated function-call interface, and asynchronous execution, enabling free-form interaction while preserving rule-grounded evaluation. Homogeneous and heterogeneous rollouts show variation in economic outcomes under the shared reference implementation. Selected trajectories illustrate valuation inconsistencies, passive bargaining, and limited long-horizon planning.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, LLMs, evaluation, negotiation, bargaining

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NdXnOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a multi-agent benchmark inspired by the board game Sidereal Confluence. A key property of this game is that it is positive-sum, i.e. the agents can make mutually beneficial negotiations. They observe that certain models perform better, and analyze specific examples to draw conclusions such as agents negotiating too passively, and failing at long-term planning.

Critical strengths

Multi-agent is a timely and interesting research direction, and the positive sum aspect of this game is particularly interesting. The latter property would allow us to investigate whether agents are capable of negotiating mutually beneficial terms.

Critical weaknesses

It's unclear what the precise novelty is. There are many other multi-agent benchmarks that require similar bargaining and planning ability, not all are zero-sum, and others are more aligned with the real world. There are multiple aspects of agent behavior that lead to a high score (e.g. negotiation, planning), but they are all combined into a single score and thus it's difficult to explain precisely why a certain agents perform better. The authors conduct behavioral analysis using specific examples, but this is ad hoc and a more quantitative, ablation-based approach would be better.

Questions for authors

Can you think of more rigorous ways we can explain why certain agents perform better, without having to resort to analyzing specific traces?

Reviewer_BntvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SidConArena, a multi-agent benchmark based on an economic game. It evaluates LLM agents' cooperation, long-term planning and negotiation capabilities in positive-sum games.

The paper studies natural-language bargaining, auctions and production of resources. Authors report substantial variation in model performance and present evidence of bad valuation of resources, passive bargaining and limited long-term planning capabilities.

Critical strengths

The paper introduces an important and timely evaluation. An environment for analysing natural-language negotiation capabilities as well as capabilities for economic panning by AI agents is valuable, especially given it combines competition for resources in a multi-agent setting.

The interconnected rounds of negotiation, production and auction seem very relevant for evaluating long-term economic reasoning.

Critical weaknesses

The paper presents empirical results on overall economic performance by the agents, but there is no quantitative evaluation of bargaining or long-horizon planning capabilities beyond selected qualitative examples and discussion. This makes it difficult to judge the effectiveness of the benchmark for the evaluation of these capabilities.

Questions for authors

Could the authors please provide further quantitative evaluation of agents' bargaining and long-horizon planning capabilities?

Reviewer_dKUnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SidConArena, an LLM-agent evaluation environment based on Sidereal Confluence. It combines natural-language negotiation with binding trades, converter-based production, and sealed-bid auctions for assets affecting future rounds. The contribution is an integrated environment for studying economic decisions whose consequences propagate across negotiation, production, and investment.

Critical strengths
  • The coupled economic environment provides a meaningful evaluation setting.
  • Structured actions and rule-based transitions make committed trades, production, and auction outcomes inspectable.
Critical weaknesses
  • Section 5 and Appendix D do not adequately report the numbers of games and players, evaluated horizon, species/seat assignment protocol, randomization, negotiation stopping conditions, or interaction budgets. These details matter because agents have asymmetric capabilities and negotiate asynchronously.
  • Table 1 primarily compares Ship prices against a reference accounting scale and prompt-imposed fairness bounds. However, Section 3.4 and the limitations acknowledge that Ships can rationally command a premium because they provide auction access.
Questions for authors
  • What are the numbers of independent games and players, game horizon, species/seat assignment procedure, and random seeds?
  • How does negotiation terminate, and how are action opportunities, token budgets, and asynchronous timing controlled across models?
  • Can the existing experiments quantify gains from negotiation using a matched no-trade baseline and total economic value, alongside individual rankings? Such evidence would help establish whether the measured gains reflect effective coordination rather than production alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.