- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a benchmark to measure language models’ collaborative capabilities. The benchmark consists of puzzles adapted from the “BOM-BANANA” game. Puzzles require three players to solve the task, where each player is given different action and communication channels.The authors use their benchmark to evaluate a selection of language models, showing that teams use different communication protocols. Trace analyses further suggest that the majority of failures can be attributed to communication and instruction-following failures.
Critical strengths
[significance] The increasing use of multi-agent systems (MAS) consisting of language model agents makes studying collaborative capabilities a timely and important topic. The paper’s goal to measure collaboration under action and communication asymmetry is thus a useful objective.
Critical weaknesses
[significance] Confidence intervals are missing. Per section 4 [l262-263], only three independent runs per variant are performed per model. This is simply not enough for any claims of significance.
[clarity] The primary proposed contribution is a benchmark. Yet, reading section 3, it is unclear to this author how the different tasks actually measure the collaborative capabilities of interest.
[quality] Per appendix D, analyses of the rollouts are exclusively performed by a single AI model (Opus 5), which is also used to “escalate” to a human reviewer [l983 and l988]. This contradicts the claim made in the main text l376. Only using a single model can lead to potential confounding due to biases toward specific models. Not doing random spot checks, or sharing anything about the process that led to the “annotation” prompts further jeopardizes the validity of the presented results.
[quality]] A central framing of the paper is to measure collaboration under different action modalities. In practice, a “mute’ player is simulated by providing a limited set of “emojis.” Is this not all just the same modality but with different vocabulary subsets?
[quality] AI Agents based on language models are relatively new (2022 onwards) – research on multi-agent systems and collaboration is not. The paper could be strengthened by more clearly positioning it in prior art. Currently, the paper cites four works written prior to 2021.
[quality] The paper has strong claims that appear not quite grounded, e.g., (Conclusion) “[...] successful collaboration requires more than producing a code. The protocol must become shared, remain aligned, respect each recipient’s perceptual and communication constraints, adapt as the task changes, and ultimately guide the authorized agent’s action.” – what is this based on?
Questions for authors
-
Key experimental details are fuzzy, e.g., what is the stop condition? How were messages exchanged?
-
Section 5.2, l360-362: “On the same bombs, [...] on harder variants” and Table 4– See [A] for research describing this as the “collaboration gap” between solo and collaborative efforts.
-
Section 5.3, Perhaps useful to compare and contrast with [B].
-
Why is this framed as “functional persona consistency” and not just “instruction following”? The only “persona” part appears to be the tokens models are allowed to output?
[A] Davidson, Tim R., et al. "The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperation." arXiv preprint arXiv:2511.02687 (2025).
[B] Cemri, Mert, et al. "Why do multi-agent llm systems fail?." Advances in Neural Information Processing Systems 38 (2026).
Dear Reviewer,
Thank you again for your review. As the discussion period ends today, we would like to check whether our reply has addressed your concerns.
If it has, we would appreciate it if you could consider updating your score. If anything remains unclear, please let us know and we will answer right away.