acceptodds
Under review as a conference paper at ICLR 2027

SANZARU: Benchmarking Protocol Formation under Epistemic & Communication Asymmetry

Abstract

A language model's ability to solve a task individually does not ensure that a team can solve it under unequal access to information and communication. SANZARU examines this gap through cooperative bomb-defusal puzzles adapted from BOMBANANA! (Lefto Studio, 2026). Task information and action authority are divided among three agents: one cannot hear, another cannot speak, and the third cannot see. The benchmark contains 120 puzzle instances across seven families, each targeting a distinct coordination ability. Across 1512 episodes, 7 models achieve an overall success rate of 49%, ranging from 20% to 86% under the evaluated configurations. Teams develop or adopt diverse communication protocols, from simple gesture codes to Morse and encodings of entire manuals. Yet trace analysis attributes about 60% of 771 team losses to communication, and agents attempt to use channels unavailable to their teammates despite explicit role instructions. Together, task outcomes and interaction traces enable evaluation of both collaborative performance and perspective taking throughout cooperation.

Primary area:
datasets and benchmarks
Keywords:
LLM Benchmark, Epistemic Asymmetry, Multi-Agent LLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_j1E4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a benchmark to measure language models’ collaborative capabilities. The benchmark consists of puzzles adapted from the “BOM-BANANA” game. Puzzles require three players to solve the task, where each player is given different action and communication channels.The authors use their benchmark to evaluate a selection of language models, showing that teams use different communication protocols. Trace analyses further suggest that the majority of failures can be attributed to communication and instruction-following failures.

Critical strengths

[significance] The increasing use of multi-agent systems (MAS) consisting of language model agents makes studying collaborative capabilities a timely and important topic. The paper’s goal to measure collaboration under action and communication asymmetry is thus a useful objective.

Critical weaknesses

[significance] Confidence intervals are missing. Per section 4 [l262-263], only three independent runs per variant are performed per model. This is simply not enough for any claims of significance.

[clarity] The primary proposed contribution is a benchmark. Yet, reading section 3, it is unclear to this author how the different tasks actually measure the collaborative capabilities of interest.

[quality] Per appendix D, analyses of the rollouts are exclusively performed by a single AI model (Opus 5), which is also used to “escalate” to a human reviewer [l983 and l988]. This contradicts the claim made in the main text l376. Only using a single model can lead to potential confounding due to biases toward specific models. Not doing random spot checks, or sharing anything about the process that led to the “annotation” prompts further jeopardizes the validity of the presented results.

[quality]] A central framing of the paper is to measure collaboration under different action modalities. In practice, a “mute’ player is simulated by providing a limited set of “emojis.” Is this not all just the same modality but with different vocabulary subsets?

[quality] AI Agents based on language models are relatively new (2022 onwards) – research on multi-agent systems and collaboration is not. The paper could be strengthened by more clearly positioning it in prior art. Currently, the paper cites four works written prior to 2021.

[quality] The paper has strong claims that appear not quite grounded, e.g., (Conclusion) “[...] successful collaboration requires more than producing a code. The protocol must become shared, remain aligned, respect each recipient’s perceptual and communication constraints, adapt as the task changes, and ultimately guide the authorized agent’s action.” – what is this based on?

Questions for authors
  1. Key experimental details are fuzzy, e.g., what is the stop condition? How were messages exchanged?

  2. Section 5.2, l360-362: “On the same bombs, [...] on harder variants” and Table 4– See [A] for research describing this as the “collaboration gap” between solo and collaborative efforts.

  3. Section 5.3, Perhaps useful to compare and contrast with [B].

  4. Why is this framed as “functional persona consistency” and not just “instruction following”? The only “persona” part appears to be the tokens models are allowed to output?

[A] Davidson, Tim R., et al. "The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperation." arXiv preprint arXiv:2511.02687 (2025).

[B] Cemri, Mert, et al. "Why do multi-agent llm systems fail?." Advances in Neural Information Processing Systems 38 (2026).

1 comment

Dear Reviewer,

Thank you again for your review. As the discussion period ends today, we would like to check whether our reply has addressed your concerns.

If it has, we would appreciate it if you could consider updating your score. If anything remains unclear, please let us know and we will answer right away.

Reviewer_aTwGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper introduces a benchmark (SANZARU) for communication skills among Large Language Models under the condition where the models have asymmetric channels of communication (e.g. one actor (mute) has privileged information and can only communicate in emojis but can receive text from the others. Another (blind) can send/recieve text and take actions, but needs the actions dictated to it by the others, and cannot receive the emojis. Finally the last one (deaf) can see the emojis from "mute" and actions of "blind", and can send text, but cannot receive text from either of the others.). This is modeled off the video game BOMBANANA! released on September 2nd, 2026.

They create 120 puzzle instances (with agent assistance) spanning 7 categories: "Base", "Carving", "Flashing Light", "Flashback", "Shuffle Wires", "Haystack", "Bait". They run 1512 trials of 7 models and find an overall success rate of 49%. They create a codebook of failures and use an agent to label the trajectories. The authors find the 60% of the failures are due to communication. They also identify coordination failures by observing a gap between solo agent performance and trio performance. Finally they identify interesting ways in which models fail to understand the perspective of peer models, and highlight some interesting cases of emergent communication standards (appendix F)

Critical strengths

This work is timely since the video game was release on September 2nd, it is unlikely that frontier models will have been trained or evaluated on the exact puzzle formulation yet.

The capabilities breakdown for the benchmark (memory, grounding, relay, etc.) is thoughtful and leads to insightful discussion about skill gaps in communication between models.

Figure 1 is nicely illustrated and clarifying.

With a few exceptions mentioned in my suggestions section the writing is relatively straightforward and easy to parse. The results and analysis are logical and informative.

The benchmark (if properly validated) is a clever formulation for testing communication under asymmetric channels and measuring theory of mind in LLMs to understand the limitations of collaboration partners. This feels like a step towards building more empathetic ai -- ai that understands and adapts to the needs of the conversation partner.

Critical weaknesses

The core contribution of this paper is the benchmark for evaluating agents on communication ability under asymmetric channels. I was thus expecting a bit more of a focus on the construction and validation of the benchmark puzzles. The AI Use Statement describes that "The authors designed the puzzle types, while generative AI wrote code to generate their instances". Further, some design decisions such as varying the number of rounds based on the task type are not motivated in the paper. The paper would benefit from a more careful examination of how puzzles were generated and validated since all findings rely on the validity of the puzzles.

The process for creating the codebook is similarly not described in detail in the paper or the appendix. The codebook is presented directly as an artifact in Table 5 with limited explanation of how it was derived (and the provenance of the codebook - human designed or ai generated). Any sourcing from the literature or qualitative coding practices used to create the codebook would be appreciated to determine why these were the choices made for coding the failure modes.

Something I find personally unsatisfying is how the evaluation seems quite hackable with a simple strategy that the LLMs indeed discovered. In Figure F.4 the LLMs realize that they can communicate in morse code through thumbs up and thumbs down. The simple strategy emerges; "mute" -> share all privileged information in morse code; "deaf" -> relay decoded morse code from "mute" to "blind" and describe the full action space; "blind" perform all reasoning on the now complete information and take the appropriate actions. Essentially because the agents can communicate everything they know in morse code there is no longer any information asymmetry and this once again becomes a solo reasoning task. Since GPT 5.6 terra already discovered this, it stands to reason more capable models will continue exploiting this strategy and diminishing the utility of the benchmark for future evaluation. In the human version of the game this would be time prohibitive, but in SANZARU there is no cap the the amount of gestures you can send in a single turn.

Questions for authors

Questions:

  • One thing that was unclear to me is the "Flashback" category relies on testing models when observations leave the context window of the model. I could not find explanation of how context overflow is handled aside from 8000 token response limits. I did however see Qwen was tested at a 4096 context limit. How do you handle context window overflow?
  • Are models made aware of their context limits or in some way communicated that they need to take action to preserve conversation history? It seems reasonable that a model with 128k context limit would leave an observation more that 4096 tokens deep into it's context without suspicion of the context being pruned.
  • "The bank holds 24 variants of these families, 16 Core and 8 Plus, with five seeded instances each and 120 in all." What is a seeded instance?
  • What is the actual difference between Core and Plus? Appendix A does not really clarify this beyond what the main paper says "A Core variant changes one variable along its family’s axis" "Plus variants combine changes to separate the strongest teams".
  • KEY QUESTION: How were the puzzles generated and validated?
  • KEY QUESTION: How was the codebook generated and validated?

Suggestions:

  • The colors in Figure 4 are too similar making it difficult to parse.
  • Section 5.4 and Table 5 feel very ai generated to me and I had a hard time parsing on first read.
  • Would require redoing the benchmark so I am not actually suggesting you do this, but would be interesting to consider limiting the gestures per round to avoid the morse code hack and encourage real shared code building.
  • Perhaps a useful concurrent work to note: https://openreview.net/forum?id=qo1hehBPVf&noteId=UrdSQLghO9
2 comments

Dear Reviewer,

Thank you again for your review. As the discussion period ends today, we would like to check whether our reply has addressed your concerns.

If it has, we would appreciate it if you could consider updating your score. If anything remains unclear, please let us know and we will answer right away.

Reviewer_aTwGOpenReview ↗

Response to Authors

Thank you so much for your reply. I apologize for my delayed response.

Q5 - Unfortunately I would still love some more detail on this process as it is challenging for me to assess how sound this is from only the high level description you are providing. It might help to have an appendix section to this end. Some things that still come to mind:

Python scripts generate each instance's manual and bomb configuration

It might be helpful to get an example of one of these scripts if it is simple (this is not a strong factor in my decision). I just think it would make things a bit more clear with what you mean when you say "generated". I am beginning to interpret this as you described the benchmark to a coding agent and had it create the environments, but do correct me if this is wrong.

A reference solution is run against each generated instance

How do you produce the reference solution? Do you verify that it is possible to create this reference only with access to the details provided to the agents, and not the full eval setting? (e.g. have a solo agent solve the task without the communication bottleneck)

"Round budgets were set by the authors based on task difficulty and the minimum number of rounds required for a human team to complete the necessary communication and actions. They were not fitted to model performance"

Did you test this with people, or is this a theoretical minimum rounds? Also this doesn't give a rule for this, did you do something like human minimum + 5? human minimum + (3*difficulty)? Does this vary based on task?

Separately I feel the motivations for the evaluation families are a bit of a stretch. For instance adding environmental distractors tests "Triage" ability. Shortening the conversation history tests "memory". To some extent these feel like a set of puzzles that are orthogonal to the communicative evaluation.

Q4 - I found your reannotation quite convincing. Thank you for providing this additional human verification.

Q1 - Thank you for clarifying this, I see it in Appendix A now.

Q2 - Thanks for verifying. I suggest you come up with a different term because "seeded instance" is not very intuitive.

Q3 - Thanks for the example, this helps.

S1,2 - This is much more clear now, thank you! I am raising your presentation score to a 3.

S3 - I appreciate your honesty in this limitation. 5% seems substantial, but I am not holding this emergent behavior against you as I appreciate you documenting it transparently and it is an interesting finding.

Overall I really appreciate your responses, I am just trying to uncover if the setting is evaluating something meaningful about the underlying communicative capabilities of the models or more of a collection of puzzles. Given my uncertainty on this so far I'd like to maintain my assessment, though I am happy to increase my presentation score.

Reviewer_T5UQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SANZARU, a benchmark for evaluating multi-agent LLM collaboration under asymmetric information, communication, and action capabilities. It uses a three-agent bomb-defusal setting in which agents have complementary roles and restricted communication channels. The authors evaluate seven LLMs across 1,512 episodes, reporting success rates ranging from approximately 20% to 86%. A single-agent comparison highlights substantial performance degradation under distributed coordination. The authors analyze 771 failed episodes using eight failure categories and five perspective-taking indicators, finding that approximately 60% of failures are attributed to communication. Additional ablations investigate prompting and reasoning budgets, while qualitative traces illustrate different emergent or adopted communication protocols.

Critical strengths

The paper addresses an important limitation of existing multi-agent benchmarks by explicitly modeling asymmetric communication capabilities alongside information and action constraints. The seven diagnostic families provide a structured environment for examining distinct coordination challenges.

The combination of task success rates, single-agent comparisons, failure-mode analysis, and interaction traces provides a richer understanding of coordination failures.

Critical weaknesses

The comparison between the single-agent and three-agent settings (Table 4) demonstrates a substantial performance gap, but it does not isolate the contribution of asymmetric communication.

The settings also differ in the number of agents, distribution of information, and communication overhead.

The central finding that approximately 60% of failures are communication-related relies primarily on LLM-based annotations. Although inter-annotator agreement is reported, the annotators use the same model family, and human review is concentrated on adjudicated disagreements rather than a representative sample of all episodes. This limits confidence in the validity of the explanatory labels.

Questions for authors
  1. Could the authors evaluate a three-agent baseline with symmetric communication channels while preserving the same information distribution, action restrictions, and puzzle instances? This would help quantify the additional difficulty specifically introduced by asymmetric communication.

  2. Could the authors provide independent human validation on a representative sample of both successful and failed episodes, including cases where all LLM annotators agreed? How sensitive is the approximately 60% communication-failure estimate to annotation errors or alternative classifications?

2 comments

Dear Reviewer,

Thank you again for your review. As the discussion period ends today, we would like to check whether our reply has addressed your concerns.

If it has, we would appreciate it if you could consider updating your score. If anything remains unclear, please let us know and we will answer right away.

Sorry for my latency, authors. Thank you so much for your thoughtful replies. I just commented. thank YOU!!!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.