acceptodds
Under review as a conference paper at ICLR 2027

MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability

Abstract

Large language models (LLMs) have progressively evolved into the core of autonomous agents. Building on this progress, LLM-based multi-agent systems (MAS) coordinate multiple agents into a synergistic team to accomplish complex tasks that exceed the capabilities of individual agents. The effectiveness of such systems, depends not only on the agents themselves, but also on how collaboration mechanisms are designed and organized. Note that real-world collaboration is typically partially observable, where each agent can only access partial information about the environment due to physical or privacy-related constraints. However, many existing multi-agent benchmarks assume global observability, and leave limited support for systematically evaluating collaboration mechanisms. To bridge this gap, we introduce MASBench, a multi-agent collaboration benchmark designed under partially observable constraints. It is organized into three progressive task categories: Reasoning, Scheduling, and Game. Through this structure, we progressively evaluate three representative collaboration mechanisms: Protocol, Memory, and Routing. MASBench further provides deterministic evaluation metrics, including performance score, communication cost, and cost effectiveness, to characterize both collaboration outcomes and communication overhead. Experiments across diverse LLM backbones and mechanism configurations offer empirical guidance for effective MAS design.

Primary area:
datasets and benchmarks
Keywords:
Multi-Agent Systems, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fBXwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MASBench, a benchmark for LLM-based multi-agent collaboration under partial observability. It uses reasoning with partitioned evidence, scheduling with private calendars, and resource-collection games with local observations to examine protocols, memory, and routing. Evaluation measures task performance and communication efficiency. Experiments across multiple LLM backbones and two multi-agent frameworks illustrate how collaboration configurations affect task performance and communication efficiency.

Critical strengths
  1. The studied problem is well motivated and valuable. Evaluating collaboration when agents have different information is a useful complement to benchmarks where every agent can access the complete task context.
  2. The benchmark combines three different collaboration settings with explicit task rules and deterministic final scoring. Reporting both task outcomes and communication volume helps characterize the trade-off between effectiveness and communication overhead.
  3. The experiments cover multiple backbones and mechanism configurations, with useful supplementary analyses. In particular, the cross-round memory ablation and expanded game cost accounting help clarify the sources and limits of the reported gains.
Critical weaknesses
  1. The tasks cover relatively limited collaboration settings. Reasoning involves only two agents, scheduling considers a single meeting with fixed calendars, and the game category contains only 100 manually specified maps. This can limit the benchmark's value for studying more complex collaboration. Also, although the reasoning evidence is split between agents, the checks in Appendix D.1 do not clearly show whether either agent can answer independently. Comparisons without communication and with all evidence given to one agent would help clarify this. The authors are also suggested to explain how the dataset can be expanded, especially for game tasks with only 100 samples in total.

  2. It is not fully clear which changes account for the improvements in the mechanism experiments. For example, Protocol changes both instructions and answer handling in Reasoning, introduces a search-and-vote procedure in Scheduling, and adds message validation in Game. Similarly, Gate changes both when to communicate and whom to contact. Although the memory ablation helps explain part of the gains, the other mechanisms are not examined at the same level. More controlled comparisons would help distinguish the effects of these factors.

  3. The choice of models and multi-agent frameworks in the experiments is somewhat limited. Several backbones are lightweight variants, such as GPT-5-nano and Gemini-2.5-Flash-Lite. Although DeepSeek-V3.2 and Qwen-Plus are included, it would be useful to also evaluate stronger models such as GPT-5/6, to see whether the proposed tasks can still differentiate the task-solving abilities of frontier or near-frontier LLMs. For multi-agent frameworks, the paper only compares CAMEL and AgentVerse (from 2023/2024), while more recent methods are not evaluated. Without these comparisons, it is hard to judge whether the benchmark remains challenging for stronger systems and can distinguish their collaboration abilities under partial evidence regime.

  4. I'm not sure whether word-level F1 is sufficient for the reasoning tasks in this benchmark. Some questions require comparing several properties and explaining their differences. An answer may express the correct meaning with different words, or share many words with the reference while making an incorrect comparison. Also, answers that cannot be parsed receive zero, and 11 of the 40 sampled reasoning failures are format violations. The authors are suggested to report these failures separately and check whether F1 agrees with human judgments of answer correctness.

Questions for authors

Most of my questions are already discussed in the Weaknesses section. I briefly restate some of them here for clarity:

  1. How often can either reasoning agent answer correctly from its own evidence alone, and how does performance compare with a single agent given all evidence?
  2. Can the authors evaluate a stronger backbone such as GPT-5 and more recent multi-agent methods to test whether the benchmark remains challenging and the reported mechanism gains persist?
4 comments

Thank you for your thorough review and helpful suggestions. We address each point below.


For Weaknesses 1 and 4 and Question 1

The task settings reflect a deliberate trade-off: MASBench sacrifices some surface realism for controllability and reproducibility, while preserving realistic forms of partial observability (distributed evidence, private calendars and local spatial views), and Scheduling and Game already involve five and four to eight agents. Even these settings require collaboration mechanisms for a strong backbone: gpt-6.1-sol solves only 21/60 Scheduling instances without them versus 48/60 with Memory, and scores 58.39 versus 69.68 in Game (see Weakness 3). Each category can also be extended with generators that we will release. Reasoning: evidence chains from any multi-hop QA source are split between agents and filtered by the quality checks in Appendix D.1. Scheduling: the generator exposes days, time granularity, team size, meeting length and the number of feasible windows (Appendix D.2). Game: new maps are drawn as ASCII grids, annotated with resources, warehouse quotas and agent settings, converted by script and checked by the reference planner, so new maps can be added beyond the current 100.

To check whether either agent can answer alone, we added A-only / B-only agents (one evidence partition, no communication) and a Full-info agent (both partitions). F1 / manually verified accuracy (%) on 90 instances:

BackboneA-onlyB-onlyBaseProtocolFull-info
GPT-5-nano9.06 / 6.677.93 / 10.0051.20 / 48.8957.29 / 67.7872.44 / 91.11
Qwen-Plus11.37 / 10.0017.09 / 13.3364.27 / 75.5667.52 / 81.1176.06 / 96.67
Gemini-2.5-Flash-Lite15.38 / 17.7814.75 / 18.8959.12 / 71.1165.83 / 77.7872.65 / 92.22
DeepSeek-v3.211.41 / 13.3315.22 / 18.8965.80 / 77.7866.92 / 78.8974.31 / 93.33

Neither agent can answer reliably alone: a single agent answers only 6.7–18.9% of questions from its own evidence, versus 91.1–96.7% with all evidence, and communication closes much of this gap. F1 agrees with manual judgments in ordering, A-only / B-only ≪ Base < Protocol < Full-info for every backbone, although it under-credits correct paraphrases (Full-info: 72–76 F1 vs. 91–97% accuracy).

As for format errors, the 11/40 comes from 40 sampled failure cases; over all 300 instances in the main runs, unparsable answers are rare: below 4% in every run except Llama-3.1-8B (23% for Base, 5% for Protocol). Excluding them, Protocol still improves F1 for all five backbones.


For Weakness 2

To separate message format from interaction rules in Scheduling, a format condition keeps Memory and its preprocessing and only structures the messages. Successes on the same 60 instances:

BackboneMemoryMemory+FormatMemory+Protocol
GPT-5-nano11325
Gemini-2.5-Flash-Lite263040
DeepSeek-v3.2213233
Qwen-Plus223435

Formatting alone helps every backbone, and the search–proposal–vote rules add further gains on every backbone. For Reasoning, Protocol still improves F1 after excluding parse failures (above), so its gain goes beyond answer handling. For Game, adding Protocol under fixed routing improves all four backbones (+1.2 to +4.2 with Random, +1.1 to +2.3 with Gate). We did not separate Gate's two choices (when and with whom) and will state this.

For Weakness 3 and Question 2

Thank you for the suggestion. We evaluated gpt-6.1-sol under all main configurations on the stratified 30% subsets used for the framework comparison. Tok is communication tokens per instance (per episode for Game), as in the paper.

ReasoningBaseProtocol
F158.0060.80
Tok657370
SchedulingBaseMemoryMemory+Protocol
Successes21/6048/6049/60
Tok1,8385,2782,573

M = Memory, R = Random, G = Gate, P = Protocol.

GameBaseMM+RM+GM+R+PM+G+P
Score58.3969.6869.0870.4669.9171.84
Tok——4,3402,1893,5611,816

gpt-6.1-sol ranks first under every Scheduling and Game configuration, so the tasks still differentiate frontier backbones. The mechanism gains persist: Protocol improves Reasoning, Memory adds 27 Scheduling successes and 11.3 Game points, Gate outperforms Random, and Protocol improves Game under both routing modes. The benchmark also remains challenging: gpt-6.1-sol still fails on 11 of 60 Scheduling instances and reaches the reference score on no Game map.

We also integrated AgentScope (AS) and Microsoft Agent Framework (MAF) into all three categories, on the same subsets as CAMEL and AgentVerse. Each framework runs with its native collaboration mechanisms (agent abstraction, conversation memory and message passing) under the same task rules, information boundaries, turn limits and evaluators; Ours is our scaffold with the full mechanisms of each category (Reference in the paper's framework table).

Reasoning

BackboneCAMELAgentVerseASMAFOurs
GPT-5-nano38.1543.3148.3951.0852.07
Gemini-2.5-Flash-Lite53.9747.8758.0161.1554.08
DeepSeek-v3.257.9254.9459.8958.7361.11
Qwen-Plus57.1658.5962.8764.6960.46

Scheduling

BackboneCAMELAgentVerseASMAFOurs
GPT-5-nano6.6716.6733.3338.3341.67
Gemini-2.5-Flash-Lite6.6713.3338.3340.0066.67
DeepSeek-v3.240.0038.3340.0045.0055.00
Qwen-Plus21.6723.3340.0050.0058.33

Game

BackboneCAMELAgentVerseASMAFOurs
GPT-5-nano46.4742.0256.5752.4357.74
Gemini-2.5-Flash-Lite29.8218.9748.3649.3432.25
DeepSeek-v3.251.3633.4451.6651.6956.92
Qwen-Plus26.1129.6143.0243.0757.32

The recent frameworks match or outperform CAMEL and AgentVerse on every backbone and category. Notably, their native mechanisms even surpass Ours in several cases (Reasoning on Gemini-2.5-Flash-Lite and Qwen-Plus, Game on Gemini-2.5-Flash-Lite): under the same task rules, mechanism design substantially changes collaboration outcomes, which is what MASBench is designed to study. The mechanism gains also persist: Ours remains best in Scheduling on every backbone and in Game on three of four. The benchmark also remains challenging: no system exceeds 65 F1 in Reasoning or 67% success in Scheduling, and none reaches 60% of the reference score in Game.

Thank you for the detailed responses and additional experiments. The rebuttal addresses many of my concerns, particularly the need for collaboration under partitioned evidence and the evaluation of stronger models and newer frameworks. The additional ablations and manual accuracy checks are also helpful.

A few points remain open, especially separating the effects of different Protocol and Routing components and the limitations of word-level F1 for semantic correctness. I encourage the authors to clarify these limitations and incorporate the new results into the revised paper. Overall, the additional evidence strengthens the paper, and I will raise my score.

Dear Reviewer fBXw,

We sincerely appreciate your time and effort in reviewing our paper and response. Thank you for recognizing the additional experiments and for raising your score. Your insightful suggestions have greatly helped us improve the quality of this work.

Best regards,
The Authors

Reviewer_JCLoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MASBench, a benchmark for evaluating LLM-based multi-agent collaboration under partial observability. It contains three task categories: Reasoning, where two agents exchange distributed evidence to solve questions; Scheduling, where five agents negotiate feasible meeting times using private calendars; and Game, where multiple agents coordinate resource collection in a partially observable grid world. The benchmark evaluates three collaboration mechanisms—Protocol, Memory, and Routing—using deterministic task-performance metrics, communication-token costs, and communication efficiency. The authors evaluate five LLM backbones under different mechanism configurations and additionally compare CAMEL and AgentVerse on a subset of the benchmark. Their main findings are that structured communication, persistent memory, and selective routing generally improve task performance and communication efficiency.

Critical strengths
  1. A relevant research motivation and a clear evaluation structure. Partial observability is a meaningful challenge in multi-agent collaboration, as agents often need to coordinate without access to the complete global state. The paper organizes its evaluation around three representative mechanisms—Protocol, Memory, and Routing—and uses progressively more demanding task categories to study their effects. This provides a clear and understandable evaluation framework.
  2. Deterministic evaluation and useful controlled comparisons. The benchmark avoids relying on LLM-as-a-judge for its main outcome metrics. Scheduling uses explicitly enumerated feasible solutions, while Game uses executable environment-based scoring. The authors also provide multiple mechanism configurations, additional memory ablations, and comparisons with existing multi-agent frameworks. These efforts support reproducibility and provide a useful starting point for studying collaboration mechanisms.
Critical weaknesses
  1. Limited task complexity and insufficient evidence that the benchmark captures challenging multi-agent collaboration.

While the motivation of partial observability is well established, I find the proposed task environments relatively elementary for a new multi-agent benchmark.

The Reasoning category largely consists of two-agent evidence exchange over multi-hop QA instances. This primarily evaluates whether agents can communicate complementary facts and combine them into a final answer. Although this is a useful basic capability test, it covers only a narrow portion of multi-agent collaboration. The Simple/Complex split is mainly determined by supporting-fact count, context length, and answer length, which do not necessarily characterize coordination complexity. Moreover, the paper does not sufficiently demonstrate, through systematic single-agent controls, that each instance requires collaboration and cannot be solved reliably from one agent's information alone.

Scheduling reduces coordination to finding a common 20-minute interval among five private calendars, with difficulty mainly controlled by the number of feasible solutions. This is a well-defined and verifiable task, but the underlying problem is essentially a constrained intersection search. The benchmark does not explore many important challenges of realistic coordination, such as changing constraints, conflicting preferences, task dependencies, or long-horizon adaptation.

The Game category introduces spatial partial observability, but the environments are limited to relatively small, manually specified 15×15 and 18×18 maps with a 50-round horizon and simple resource-collection objectives. Simulated environments can certainly provide valuable controlled evaluations, but the paper does not sufficiently establish that these particular environments capture coordination challenges beyond basic exploration, information sharing, and resource delivery.

Overall, the three categories are useful as introductory tests of information exchange and coordination. However, I am not convinced that their difficulty and behavioral coverage are sufficient to support a comprehensive benchmark for contemporary multi-agent collaboration. The benchmark should demonstrate meaningful challenges involving more complex dependencies, coordination failures, adaptive planning, or longer-horizon interactions.

  1. The benchmark's discriminative power for frontier-level agents remains unvalidated, and the mechanism-level findings offer limited new insight.

My main concern is the absence of convincing evidence that MASBench remains challenging and discriminative for contemporary high-capability LLM agents.

The evaluation uses Llama-3.1-8B-Instruct, GPT-5-nano, Gemini-2.5-Flash-Lite, DeepSeek-v3.2, and Qwen-Plus, with Llama evaluated only on Reasoning. While these models cover several capability and cost regimes, the experiments lack a representative evaluation of current frontier-grade flagship models. In particular, the inclusion of smaller or efficiency-oriented models does not establish the benchmark's difficulty ceiling or its ability to distinguish stronger collaborative systems.

For a new benchmark targeting the state of multi-agent systems in 2026, testing its discriminative power on stronger contemporary backbones is important. Without such evidence, it is unclear whether the relatively simple Reasoning and Scheduling tasks would retain meaningful difficulty, whether the observed mechanism improvements would persist, or whether stronger systems would approach saturation.

The experimental findings are also largely expected: structured communication reduces redundant messages, memory helps agents retain previously observed information, and selective routing generally performs better than random communication. While experimentally confirming these effects has some value, the paper provides limited new insights into when these mechanisms fail, how they interact under challenging conditions, or which coordination problems remain unresolved by strong modern agents.

Furthermore, the mechanism comparisons introduce several confounding changes. In Scheduling, Memory includes availability preprocessing, while Protocol introduces a substantially more structured search, proposal, and voting procedure. Thus, the large improvement over Base may reflect the advantages of an explicit coordination algorithm rather than communication protocol alone. The additional cross-round memory ablation is useful, but does not fully isolate these effects. In Game, the Base condition disables communication entirely, and the structured Protocol configuration includes grounded message construction and validation. These changes make it difficult to attribute all performance gains to individual collaboration mechanisms.

The communication-efficiency metric also primarily counts transmitted message tokens rather than full inference cost. Although Appendix B.4 partially addresses message-phase overhead, the main efficiency comparisons do not capture the complete computational cost of the different mechanisms.

Taken together, these limitations weaken the paper's central claim that MASBench provides a sufficiently informative and broadly applicable evaluation of modern multi-agent collaboration mechanisms.

Questions for authors

Q1. Discriminative power on contemporary frontier models.

Can the authors evaluate MASBench on a representative set of stronger, contemporary frontier-grade LLMs? Even a carefully selected subset of strong models and benchmark instances would be informative if full-scale evaluation is computationally expensive. In particular, how do task performance, mechanism gains, and model rankings change with substantially stronger backbones? Do Reasoning and Scheduling approach saturation? If not, what are the remaining failure modes? Evidence that the benchmark remains challenging and discriminative at the current capability frontier would materially strengthen my assessment.

Q2. Task difficulty and the necessity of collaboration.

How do the authors establish that the constructed tasks genuinely require multi-agent collaboration? For Reasoning, can they report single-agent performance using each private evidence partition separately, alongside a full-information upper-bound baseline? For Scheduling and Game, can they provide stronger baselines or harder variants that test more complex coordination requirements? More generally, how do the authors justify the chosen task difficulty and demonstrate that the benchmark covers meaningful coordination capabilities beyond basic information exchange?

Q3. Isolation of collaboration mechanisms.

How much of the reported improvement is attributable specifically to Protocol, Memory, or Routing, rather than additional task-solving algorithms and preprocessing? For Scheduling, can the authors compare structured message formatting against the full multi-phase search-and-voting procedure while controlling availability preprocessing? For Game, can the authors provide stronger communication-enabled baselines and report the complete computational cost, including routing decisions and auxiliary model calls? These controls would help establish more convincing mechanism-level conclusions.

Q4. Benchmark validation and long-term usefulness.

Can the authors provide stronger evidence for the reliability and diversity of the benchmark instances? In particular, how were the Reasoning evidence partitions validated to ensure that one agent cannot independently answer the question? How was the diversity and difficulty of the manually designed Game maps assessed? Since the Game normalization uses a heuristic reference planner rather than a proven optimum, how sensitive are the results to the choice of reference? What evidence supports the benchmark's ability to remain informative as agent capabilities improve?

5 comments

Thank you for your detailed review and insightful suggestions. We address each point below.


For Weakness 1

MASBench deliberately trades some surface realism for controllability and reproducibility, while preserving realistic forms of partial observability: distributed evidence in Reasoning, private calendars in Scheduling, and local spatial views in Game. Even in this simplified form, current LLMs struggle without collaboration mechanisms: without them, Scheduling succeeds on at most 22.5% of instances (35% for gpt-6.1-sol) and Game reaches at most 42.6 normalized points. Below we summarize what makes each category challenging and where agents fail in practice, based on our failure analysis of sampled failed or low-scoring trajectories (Appendix):

  • Reasoning. No agent can answer alone, and construction ensures that neither the question nor the remaining context reveals the answer. Each agent must infer what its partner is missing and which of its own facts close the gap. In practice, agents exchange facts but fail to merge them into a complete answer: incomplete answers are the most frequent failure type.
  • Scheduling. Feasible windows are scarce (2–5), so a slot free for one agent is almost always busy for another. Since calendars stay private, the window must be found by proposing and checking candidates and remembering rejected ones within a few rounds. In practice, agents fail either to surface a feasible window or to judge a proposed one correctly.
  • Game. Information is local and arrives with delay, and constraints are coupled (one resource type per trip, per-type warehouse quotas), so a wrong type or a full warehouse wastes rounds of a fixed budget, and agents must decide what to tell whom. In practice, agents fail to turn local discoveries into deliveries, mainly through low efficiency and insufficient exploration.

For Weakness 2 and Question 1

We evaluated gpt-6.1-sol on all main configurations, using the stratified 30% subsets of the framework comparison (90 Reasoning instances, 60 Scheduling instances, 30 Game maps). Tok is communication tokens per instance (per episode for Game), as in the paper.

Reasoning

BaseProtocol
F158.0060.80
Tok657370

Scheduling

BaseMemoryMemory+Protocol
Successes21/6048/6049/60
Tok1,8385,2782,573

Game (M = Memory, R = Random, G = Gate, P = Protocol)

BaseMM+RM+GM+R+PM+G+P
Score58.3969.6869.0870.4669.9171.84
Tok——4,3402,1893,5611,816

gpt-6.1-sol ranks first under every Scheduling and Game configuration, and the mechanism trends of the original backbones persist: Protocol improves Reasoning F1 (58.00→60.80); Memory adds 27 Scheduling successes (21→48) and 11.3 Game points (58.39→69.68); Gate outperforms Random with and without Protocol (70.46 vs. 69.08; 71.84 vs. 69.91); Protocol improves Game under both routing modes; and M+G+P remains the best Game configuration. The benchmark is not saturated. Without mechanisms, gpt-6.1-sol solves only 35% of Scheduling instances; with them, it still fails on 11 of 60, and the remaining failures are all non-convergence within five rounds, concentrated on instances with only 2–3 feasible windows. In Game, it reaches the reference score on no map. The results also reveal less obvious behaviors:

  1. Memory alone increases communication. Retained state might be expected to reduce redundant exchanges, yet adding Memory increases Scheduling communication for every backbone (e.g., 999→1,805 Tok for Gemini-2.5-Flash-Lite, 515→1,168 for Qwen-Plus, 1,838→5,278 for gpt-6.1-sol). Adding Protocol structures the negotiation and reduces communication (e.g., 1,805→933 for Gemini-2.5-Flash-Lite).
  2. As capability grows, Protocol's benefit shifts from accuracy to efficiency. For the original backbones, Protocol substantially raises Scheduling success over Memory (e.g., 1→25 for GPT-5-nano, 26→40 for Gemini-2.5-Flash-Lite). gpt-6.1-sol already succeeds with Memory alone, so Protocol keeps success (49 vs. 48) while halving communication. Game shows the same shift: Protocol cuts gpt-6.1-sol's communication by 17–18% under both routing modes, with smaller score gains (+0.8 and +1.4) than the original backbones' averages (+2.5 and +1.7).

For Questions 2 and 4

To verify that Reasoning requires collaboration and that one partition cannot answer the question, we added A-only / B-only agents (one partition, no communication) and a Full-info agent (both partitions). F1 on 90 instances:

BackboneA-onlyB-onlyBaseProtocolFull-info
GPT-5-nano9.067.9351.2057.2972.44
Qwen-Plus11.3717.0964.2767.5276.06
Gemini-2.5-Flash-Lite15.3814.7559.1265.8372.65
DeepSeek-v3.211.4115.2265.8066.9274.31

A single agent reaches only 7.9–17.1 F1, versus 72.4–76.1 with both partitions, so the task cannot be solved without collaboration; communicating agents close most of this gap.

The 100 Game maps contain 94 distinct layouts up to reflection, with 4–8 agents, 3–8 warehouses and 3–40% obstacle density; the reference planner reaches 53–100% of the task quota, and per-map scores of the best configuration range from 7.5 to 95.8.

To test sensitivity to the Game reference score, we compare raw and normalized scores on the same 30 maps, including gpt-6.1-sol (raw / normalized):

BackboneBaseMM+RM+GM+R+PM+G+P
GPT-5-nano36.6 / 40.041.3 / 45.045.7 / 49.950.1 / 54.948.1 / 52.652.8 / 57.7
Gemini-2.5-Flash-Lite19.7 / 20.821.4 / 23.220.7 / 22.624.6 / 27.023.8 / 26.127.7 / 30.6
DeepSeek-v3.217.3 / 18.940.4 / 43.944.2 / 48.251.0 / 56.147.2 / 51.851.7 / 56.9
Qwen-Plus33.3 / 36.042.0 / 45.644.6 / 48.552.6 / 57.849.7 / 54.652.1 / 57.3
gpt-6.1-sol53.2 / 58.462.9 / 69.762.2 / 69.163.7 / 70.563.1 / 69.965.1 / 71.8

The configuration ordering within every backbone and the backbone ordering within every configuration are identical under both scores (Kendall's τ = 0.995 over the 30 pairs). Re-normalizing with any of 16 alternative heuristic reference scores also gives τ ≥ 0.986 and leaves all mechanism conclusions unchanged; we will report both scores. As for long-term usefulness, gpt-6.1-sol still fails on 18% of Scheduling instances and reaches the reference score on no Game map. Difficulty is controlled by the same parameters as our Simple/Complex split: Reasoning by a score combining supporting-fact count, context length and answer length (Appendix D.1), Scheduling by the number of feasible windows (2–3 for Complex vs. 4–5 for Simple), and Game by map scale (18×18 maps with more resources vs. 15×15). Harder instances can be generated along these axes, e.g., fewer feasible windows or larger maps; we will release the generators and the map-authoring pipeline.


For Question 3

For Scheduling, a format condition keeps Memory and its preprocessing and only structures the messages. Successes on the same 60 instances:

BackboneMemoryMemory+FormatMemory+Protocol
GPT-5-nano11325
Gemini-2.5-Flash-Lite263040
DeepSeek-v3.2213233
Qwen-Plus223435

Simple message structure already helps every backbone, and the full protocol helps further. For Game, our communication mechanisms are likewise simple: Random sends free-form messages to a random teammate, Gate adds an LLM decision on whether and to whom to send, and Protocol adds grounded message fields. Even these simple designs are effective: Memory+Gate improves over Memory for every backbone (+2.9 to +8.8), adding Protocol improves further (+4.7 to +10.1 over Memory), and with routing fixed, Protocol alone adds +1.2 to +4.2 (Random) and +1.1 to +2.3 (Gate). Since simple communication already yields consistent gains, stronger communication methods are expected to be effective as well, and MASBench can evaluate them under the same tasks and metrics. Output tokens per episode recorded in the gpt-6.1-sol runs (k):

SettingRouting decisionsCommunicationEnd-to-end
Base——14.6
M——11.3
M+R—5.916.2
M+G6.512.129.8
M+R+P—17.129.5
M+G+P6.96.925.3

Dear Reviewer JCLo,

We are grateful for your detailed and constructive review. In our response, we have included the evaluation of the frontier model gpt-6.1-sol that you suggested, along with several additional analyses. With the discussion period coming to a close, we would be thankful if you could review our response and consider the new results in your final assessment. We look forward to hearing from you.

Best regards,
The Authors

Reviewer_GEifOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

MASBench is a benchmark for evaluating LLM-based multi-agent collaboration under partial observability. It contains three task categories, including Reasoning, Scheduling, and Game, to check protocol, memory, and routing mechanisms. Experiments across several LLM backbones show the effects of these mechanisms on task performance and communication efficiency.

Critical strengths
  1. The evaluation under partial observability is meaningful.

  2. The paper formulates three forms of collaboration, like reasoning, scheduling, and game.

  3. The paper evaluates some LLM backbones and existing multi-agent frameworks.

Critical weaknesses
  1. The novelty of MASBench compared to existing multi-agent benchmarks is unclear, as partially observable settings exist in the current benchmarks.

  2. The benchmark covers a limited range of scenarios, with 600 instances. Scheduling and game rely on synthetic environments. It is unclear how the findings generalize to more diverse or realistic multi-agent collaboration settings.

  3. The framework comparison includes different orchestration strategies and task-specific adaptations, rather than only different framework implementations.

Questions for authors
  1. Could the authors provide an estimate of the end-to-end computational cost?

  2. How do the authors justify that the conclusions about Protocol, Memory, and Routing generalize beyond their respective task-specific implementations?

3 comments

Thank you for your careful review and constructive feedback. We address your comments below.


For Weakness 1

We agree that partial observability itself appears in existing benchmarks. MASBench's novelty is a unified testbed for comparing collaboration mechanisms and their performance-communication trade-offs across tasks:

BenchmarkPartial observabilityMultiple task familiesMechanism comparisonsDeterministic scoringCommunication cost-effectiveness
MultiAgentBench (2025)✓✓◐——
Collab-Overcooked (2025)✓——✓—
The Collaboration Gap (2025)✓—◐✓—
CooperBench (2026)✓—◐✓—
Sotopia-ToM (2026)✓—◐——
MASBench (ours)✓✓✓✓✓

Mechanism comparisons: — none; ◐ focused; ✓ systematic comparison of components and combinations.


For Weakness 2

The controlled environments reflect a deliberate trade-off. MASBench sacrifices some surface realism for experimental controllability and reproducibility, using explicit information partitions, verifiable task outcomes and deterministic scoring. This lets us compare mechanism configurations on common task instances and evaluation rules.

The tasks nevertheless preserve realistic forms of partial observability: distributed evidence in Reasoning, private calendars in Scheduling, and local spatial views in Game. They abstract away surface complexity while retaining the central coordination constraint: agents must exchange and integrate information unavailable in their own views.

MASBench contains 600 instances (300 Reasoning, 200 Scheduling and 100 Game). Its scale is comparable to recent interactive MAS benchmarks:

BenchmarkYearInstances
Collab-Overcooked202530
HiddenBench202565
MAGPIE2025200
CooperBench2026652
TickingCollabBench2026634
MASBench—600

We will release the Reasoning/Scheduling generators and validation procedures to support further scaling.


For Weakness 3

We agree that this is a system-level comparison rather than an isolation of framework code. Each framework runs with its native orchestration (agent abstraction, conversation memory, prompt templates and history serialization), since these are part of what a framework provides. Task-specific adaptations are shared: CAMEL and AgentVerse use the same adapters, information boundaries, turn limits and evaluators (Appendix C.5), so differences between them reflect the frameworks themselves. This tests how framework-backed systems collaborate under common task constraints.


For Question 1

MASBench focuses on communication tokens because they directly quantify information exchanged between agents, allowing us to assess task performance relative to communication overhead. This metric captures a central cost of collaboration under partial observability. Our second response adds per-instance end-to-end and output tokens over all LLM calls.


For Question 2

Our conclusions concern the mechanisms, i.e., what each one does, rather than particular implementations: Memory retains state across rounds, Routing decides who communicates with whom and when, and Protocol specifies message schemas and interaction rules. Their implementations differ across tasks because information needs differ, yet the gains are consistent across tasks and backbones. Protocol improves every backbone in all three categories, each with a different implementation; Memory improves every backbone in both Scheduling and Game, and even a simpler cross-round variant (Appendix B.5) recovers at least half of the Scheduling gain; Gate routing outperforms Random for all four backbones in Game while transmitting 28.0–35.6% fewer tokens. The gains also hold for the stronger gpt-6.1-sol (Reasoning F1 58.00→60.80 with Protocol; Scheduling successes 21 / 48 / 49 of 60 for Base / Memory / Memory+Protocol). Since simple, transparent implementations already yield consistent gains, these consistent gains show that the mechanisms themselves are effective, and more elaborate designs can be evaluated in MASBench under the same tasks and metrics.

For Question 1 (end-to-end cost)

The tables extend the main results (Overall). Score and Tok are as reported in the paper; End-to-end is the mean number of input plus output tokens per instance (thousands) over all LLM calls, and Output counts output tokens only. We report Output separately because it relates more closely to monetary cost: output tokens, including hidden reasoning, are priced several times higher than input tokens, whose cost is further reduced by prompt caching.

Reasoning.

BackboneSettingScoreTokEnd-to-endOutput
Llama-3.1-8BBase26.061,26121.71.64
Protocol33.4553017.30.86
GPT-5-nanoBase43.3656716.30.99
Protocol49.6615516.80.40
Gemini-2.5-Flash-LiteBase50.925897.70.98
Protocol55.533246.90.92
DeepSeek-v3.2Base54.4171610.21.15
Protocol55.232087.40.88
Qwen-PlusBase55.813824.60.64
Protocol59.241615.10.60

Scheduling.

BackboneSettingScoreTokEnd-to-endOutput
GPT-5-nanoBase11,116358.4207.9
Memory21,763529.5394.0
Memory+Protocol422,942366.1281.9
Gemini-2.5-Flash-LiteBase17999302.3196.8
Memory36.51,805715.3398.4
Memory+Protocol62.5933689.9289.3
DeepSeek-v3.2Base22.5418247.599.3
Memory38572217.8117.9
Memory+Protocol63905206.9100.5
Qwen-PlusBase13.5515377.3223.4
Memory401,168384.5276.0
Memory+Protocol63.51,044293.2197.2

Game.

BackboneSettingScoreTokEnd-to-endOutput
GPT-5-nanoBase42.61—228.317.5
Memory50.65—297.619.8
Memory+Random53.855,897455.826.4
Memory+Gate58.983,916647.425.8
Memory+Random+Protocol56.413,417519.724.6
Memory+Gate+Protocol60.712,105723.821.3
Gemini-2.5-Flash-LiteBase23.85—228.617.8
Memory25.75—302.022.2
Memory+Random26.293,081455.325.9
Memory+Gate28.641,984645.225.6
Memory+Random+Protocol27.482,683521.726.6
Memory+Gate+Protocol30.411,592628.025.5
DeepSeek-v3.2Base18.59—228.017.2
Memory50.12—298.518.7
Memory+Random54.233,581452.222.8
Memory+Gate58.952,527642.422.8
Memory+Random+Protocol56.403,312518.923.8
Memory+Gate+Protocol60.071,857725.322.8
Qwen-PlusBase38.73—250.039.2
Memory53.39—321.842.0
Memory+Random53.204,927477.047.6
Memory+Gate59.773,548666.947.3
Memory+Random+Protocol57.433,787542.547.4
Memory+Gate+Protocol62.052,167748.946.4

Dear Reviewer GEif,

We are grateful for the time you dedicated to reviewing our paper. In our response, we have added a comparison with existing benchmarks and an end-to-end cost analysis, as you suggested. With the discussion period coming to a close, we would be thankful if you could review our response and let us know whether it resolves your concerns. We look forward to hearing from you.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.