- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MASBench, a benchmark for LLM-based multi-agent collaboration under partial observability. It uses reasoning with partitioned evidence, scheduling with private calendars, and resource-collection games with local observations to examine protocols, memory, and routing. Evaluation measures task performance and communication efficiency. Experiments across multiple LLM backbones and two multi-agent frameworks illustrate how collaboration configurations affect task performance and communication efficiency.
Critical strengths
- The studied problem is well motivated and valuable. Evaluating collaboration when agents have different information is a useful complement to benchmarks where every agent can access the complete task context.
- The benchmark combines three different collaboration settings with explicit task rules and deterministic final scoring. Reporting both task outcomes and communication volume helps characterize the trade-off between effectiveness and communication overhead.
- The experiments cover multiple backbones and mechanism configurations, with useful supplementary analyses. In particular, the cross-round memory ablation and expanded game cost accounting help clarify the sources and limits of the reported gains.
Critical weaknesses
-
The tasks cover relatively limited collaboration settings. Reasoning involves only two agents, scheduling considers a single meeting with fixed calendars, and the game category contains only 100 manually specified maps. This can limit the benchmark's value for studying more complex collaboration. Also, although the reasoning evidence is split between agents, the checks in Appendix D.1 do not clearly show whether either agent can answer independently. Comparisons without communication and with all evidence given to one agent would help clarify this. The authors are also suggested to explain how the dataset can be expanded, especially for game tasks with only 100 samples in total.
-
It is not fully clear which changes account for the improvements in the mechanism experiments. For example, Protocol changes both instructions and answer handling in Reasoning, introduces a search-and-vote procedure in Scheduling, and adds message validation in Game. Similarly, Gate changes both when to communicate and whom to contact. Although the memory ablation helps explain part of the gains, the other mechanisms are not examined at the same level. More controlled comparisons would help distinguish the effects of these factors.
-
The choice of models and multi-agent frameworks in the experiments is somewhat limited. Several backbones are lightweight variants, such as GPT-5-nano and Gemini-2.5-Flash-Lite. Although DeepSeek-V3.2 and Qwen-Plus are included, it would be useful to also evaluate stronger models such as GPT-5/6, to see whether the proposed tasks can still differentiate the task-solving abilities of frontier or near-frontier LLMs. For multi-agent frameworks, the paper only compares CAMEL and AgentVerse (from 2023/2024), while more recent methods are not evaluated. Without these comparisons, it is hard to judge whether the benchmark remains challenging for stronger systems and can distinguish their collaboration abilities under partial evidence regime.
-
I'm not sure whether word-level F1 is sufficient for the reasoning tasks in this benchmark. Some questions require comparing several properties and explaining their differences. An answer may express the correct meaning with different words, or share many words with the reference while making an incorrect comparison. Also, answers that cannot be parsed receive zero, and 11 of the 40 sampled reasoning failures are format violations. The authors are suggested to report these failures separately and check whether F1 agrees with human judgments of answer correctness.
Questions for authors
Most of my questions are already discussed in the Weaknesses section. I briefly restate some of them here for clarity:
- How often can either reasoning agent answer correctly from its own evidence alone, and how does performance compare with a single agent given all evidence?
- Can the authors evaluate a stronger backbone such as GPT-5 and more recent multi-agent methods to test whether the benchmark remains challenging and the reported mechanism gains persist?
4 comments
For Weakness 3 and Question 2
Thank you for the suggestion. We evaluated gpt-6.1-sol under all main configurations on the stratified 30% subsets used for the framework comparison. Tok is communication tokens per instance (per episode for Game), as in the paper.
| Reasoning | Base | Protocol |
|---|---|---|
| F1 | 58.00 | 60.80 |
| Tok | 657 | 370 |
| Scheduling | Base | Memory | Memory+Protocol |
|---|---|---|---|
| Successes | 21/60 | 48/60 | 49/60 |
| Tok | 1,838 | 5,278 | 2,573 |
M = Memory, R = Random, G = Gate, P = Protocol.
| Game | Base | M | M+R | M+G | M+R+P | M+G+P |
|---|---|---|---|---|---|---|
| Score | 58.39 | 69.68 | 69.08 | 70.46 | 69.91 | 71.84 |
| Tok | — | — | 4,340 | 2,189 | 3,561 | 1,816 |
gpt-6.1-sol ranks first under every Scheduling and Game configuration, so the tasks still differentiate frontier backbones. The mechanism gains persist: Protocol improves Reasoning, Memory adds 27 Scheduling successes and 11.3 Game points, Gate outperforms Random, and Protocol improves Game under both routing modes. The benchmark also remains challenging: gpt-6.1-sol still fails on 11 of 60 Scheduling instances and reaches the reference score on no Game map.
We also integrated AgentScope (AS) and Microsoft Agent Framework (MAF) into all three categories, on the same subsets as CAMEL and AgentVerse. Each framework runs with its native collaboration mechanisms (agent abstraction, conversation memory and message passing) under the same task rules, information boundaries, turn limits and evaluators; Ours is our scaffold with the full mechanisms of each category (Reference in the paper's framework table).
Reasoning
| Backbone | CAMEL | AgentVerse | AS | MAF | Ours |
|---|---|---|---|---|---|
| GPT-5-nano | 38.15 | 43.31 | 48.39 | 51.08 | 52.07 |
| Gemini-2.5-Flash-Lite | 53.97 | 47.87 | 58.01 | 61.15 | 54.08 |
| DeepSeek-v3.2 | 57.92 | 54.94 | 59.89 | 58.73 | 61.11 |
| Qwen-Plus | 57.16 | 58.59 | 62.87 | 64.69 | 60.46 |
Scheduling
| Backbone | CAMEL | AgentVerse | AS | MAF | Ours |
|---|---|---|---|---|---|
| GPT-5-nano | 6.67 | 16.67 | 33.33 | 38.33 | 41.67 |
| Gemini-2.5-Flash-Lite | 6.67 | 13.33 | 38.33 | 40.00 | 66.67 |
| DeepSeek-v3.2 | 40.00 | 38.33 | 40.00 | 45.00 | 55.00 |
| Qwen-Plus | 21.67 | 23.33 | 40.00 | 50.00 | 58.33 |
Game
| Backbone | CAMEL | AgentVerse | AS | MAF | Ours |
|---|---|---|---|---|---|
| GPT-5-nano | 46.47 | 42.02 | 56.57 | 52.43 | 57.74 |
| Gemini-2.5-Flash-Lite | 29.82 | 18.97 | 48.36 | 49.34 | 32.25 |
| DeepSeek-v3.2 | 51.36 | 33.44 | 51.66 | 51.69 | 56.92 |
| Qwen-Plus | 26.11 | 29.61 | 43.02 | 43.07 | 57.32 |
The recent frameworks match or outperform CAMEL and AgentVerse on every backbone and category. Notably, their native mechanisms even surpass Ours in several cases (Reasoning on Gemini-2.5-Flash-Lite and Qwen-Plus, Game on Gemini-2.5-Flash-Lite): under the same task rules, mechanism design substantially changes collaboration outcomes, which is what MASBench is designed to study. The mechanism gains also persist: Ours remains best in Scheduling on every backbone and in Game on three of four. The benchmark also remains challenging: no system exceeds 65 F1 in Reasoning or 67% success in Scheduling, and none reaches 60% of the reference score in Game.
Thank you for the detailed responses and additional experiments. The rebuttal addresses many of my concerns, particularly the need for collaboration under partitioned evidence and the evaluation of stronger models and newer frameworks. The additional ablations and manual accuracy checks are also helpful.
A few points remain open, especially separating the effects of different Protocol and Routing components and the limitations of word-level F1 for semantic correctness. I encourage the authors to clarify these limitations and incorporate the new results into the revised paper. Overall, the additional evidence strengthens the paper, and I will raise my score.
Dear Reviewer fBXw,
We sincerely appreciate your time and effort in reviewing our paper and response. Thank you for recognizing the additional experiments and for raising your score. Your insightful suggestions have greatly helped us improve the quality of this work.
Best regards,
The Authors
Thank you for your thorough review and helpful suggestions. We address each point below.
For Weaknesses 1 and 4 and Question 1
The task settings reflect a deliberate trade-off: MASBench sacrifices some surface realism for controllability and reproducibility, while preserving realistic forms of partial observability (distributed evidence, private calendars and local spatial views), and Scheduling and Game already involve five and four to eight agents. Even these settings require collaboration mechanisms for a strong backbone: gpt-6.1-sol solves only 21/60 Scheduling instances without them versus 48/60 with Memory, and scores 58.39 versus 69.68 in Game (see Weakness 3). Each category can also be extended with generators that we will release. Reasoning: evidence chains from any multi-hop QA source are split between agents and filtered by the quality checks in Appendix D.1. Scheduling: the generator exposes days, time granularity, team size, meeting length and the number of feasible windows (Appendix D.2). Game: new maps are drawn as ASCII grids, annotated with resources, warehouse quotas and agent settings, converted by script and checked by the reference planner, so new maps can be added beyond the current 100.
To check whether either agent can answer alone, we added A-only / B-only agents (one evidence partition, no communication) and a Full-info agent (both partitions). F1 / manually verified accuracy (%) on 90 instances:
Neither agent can answer reliably alone: a single agent answers only 6.7–18.9% of questions from its own evidence, versus 91.1–96.7% with all evidence, and communication closes much of this gap. F1 agrees with manual judgments in ordering, A-only / B-only ≪ Base < Protocol < Full-info for every backbone, although it under-credits correct paraphrases (Full-info: 72–76 F1 vs. 91–97% accuracy).
As for format errors, the 11/40 comes from 40 sampled failure cases; over all 300 instances in the main runs, unparsable answers are rare: below 4% in every run except Llama-3.1-8B (23% for Base, 5% for Protocol). Excluding them, Protocol still improves F1 for all five backbones.
For Weakness 2
To separate message format from interaction rules in Scheduling, a format condition keeps Memory and its preprocessing and only structures the messages. Successes on the same 60 instances:
Formatting alone helps every backbone, and the search–proposal–vote rules add further gains on every backbone. For Reasoning, Protocol still improves F1 after excluding parse failures (above), so its gain goes beyond answer handling. For Game, adding Protocol under fixed routing improves all four backbones (+1.2 to +4.2 with Random, +1.1 to +2.3 with Gate). We did not separate Gate's two choices (when and with whom) and will state this.