- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies why LLM agents outperform Bayesian negotiators in a five-party negotiation benchmark. Through communication ablations, preference-declaration experiments, and LLM interfaces for algorithmic agents, the authors argue that free-form communication improves coordination by conveying preference information and helping agents converge on feasible agreements.
Critical strengths
-
The paper asks an interesting question rather than only reporting that stronger LLMs perform better.
-
The experiment testbed is well controlled. The exact solver guarantees that every instance has a feasible agreement and allows every outcome to be evaluated compared to the full feasible set.
-
The hybrid agent result is interesting.
-
The writing is easy to follow.
Critical weaknesses
-
The core causal claim is stronger than what the experiments establish. The evidence doesn't sufficiently support free-form communication is why LLMs outperform Bayesian agents in negotiation, as muted Opus agents still outperform the Bayesian agents.
-
The paper argues that the relevant payload of communication is preference information. However, the experiments does not establish that preference revelation is the causal reason why natural-language communication improves LLM performance.
-
Only one particular Bayesian reference policy is evaluated. The claim about Bayesian agents is too broad.
Questions for authors
- How do the authors justify the main causal claim with the all-muted experimental result?
- Can agents retain the same communication budget but be prohibited from revealing preferences?
- How robust is the main effect to other frontier models?