- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a multi-agent benchmark inspired by the board game Sidereal Confluence. A key property of this game is that it is positive-sum, i.e. the agents can make mutually beneficial negotiations. They observe that certain models perform better, and analyze specific examples to draw conclusions such as agents negotiating too passively, and failing at long-term planning.
Critical strengths
Multi-agent is a timely and interesting research direction, and the positive sum aspect of this game is particularly interesting. The latter property would allow us to investigate whether agents are capable of negotiating mutually beneficial terms.
Critical weaknesses
It's unclear what the precise novelty is. There are many other multi-agent benchmarks that require similar bargaining and planning ability, not all are zero-sum, and others are more aligned with the real world. There are multiple aspects of agent behavior that lead to a high score (e.g. negotiation, planning), but they are all combined into a single score and thus it's difficult to explain precisely why a certain agents perform better. The authors conduct behavioral analysis using specific examples, but this is ad hoc and a more quantitative, ablation-based approach would be better.
Questions for authors
Can you think of more rigorous ways we can explain why certain agents perform better, without having to resort to analyzing specific traces?