acceptodds
Under review as a conference paper at ICLR 2027

MAScope: Diagnosing Collaboration Loss in Multi-Agent Systems

Abstract

Multi-agent systems assemble agents with different expertise, yet performance depends on how that expertise is organized, not on how much of it is present. Existing benchmarks only evaluate a method's outcome or a prescribed team's process, and cannot tell what a method's own collaboration contributes. We introduce **MAScope**, an environment of 51 expert agents holding private knowledge from Stack Exchange and 2,766 collaborative tasks with verifiable evidence and work dependencies. MAScope follows each required contribution from the agent that produces it into the final answer, splitting collaboration into four measurable capabilities: *expert selection*, *task orchestration*, *information transfer*, and *result integration*. Comprehensive experiments on twelve representative methods show that having the required experts does not deliver the task. Once work turns interdependent, success drops by 48%, over 3× that of a no-collaboration single agent. Trajectory analysis attributes this loss to three gaps: (1) **planning rigidity** in task orchestration, (2) **semantic compression** in information transfer, and (3) **adjudicative aggregation** in result integration. Even exhaustive collaboration and added memory recover little of it, since neither changes how a method organizes the work. MAScope gives a common basis for measuring and improving multi-agent collaboration, available at https://anonymous.4open.science/r/MAScope-7B76.

Primary area:
datasets and benchmarks
Keywords:
Multi-Agent Systems, LLM Agents, Agent Collaboration, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Eyy9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MAScope is a benchmark for diagnosing where multi-agent collaboration loses required information. It builds 51 experts with private Stack Exchange corpora and 2,766 tasks from 499 families. Tasks vary in expert count and in whether work is parallel, sequential, or requires discovering a new expert from an earlier finding. Source identifiers and certified dependencies let the evaluator track expert selection, task orchestration, information transfer, local outputs, and final integration. Experiments with twelve methods show larger degradation on dependent tasks than for a single agent accessing the same corpora. The paper relates this degradation to planning, message compression, and aggregation, and finds limited improvements from extended collaboration and memory modules.

Critical strengths
  1. The benchmark makes required contributions both traceable and difficult to bypass. Appendices B.2 and C.5 connect dependency certification to the actual retriever and check bypasses in the recorded runs, with only 0.20% of edges bypassed overall. The corpus-removal check in Appendix A.3 and the identifier-free rescoring in Table 12 address two obvious alternatives to the intended interpretation. These checks make the dependency analysis much more convincing.

  2. The process analysis is concrete and well supported. Table 13 defines each stage, Table 17 supplies the underlying populations, and Appendix G shows corresponding failure traces. The comparison uses the same backbone, corpora, retrieval interface, and budget caps, while Appendices C and D explain the adaptations and actual cap rates. This supports useful comparisons among the evaluated organizations rather than only a ranking by final score.

Critical weaknesses
  1. I am concerned that the scoring rule establishes content presence more directly than content correctness. Appendix C.2 accepts an answer when every required identifier and preregistered phrase occurs, and explicitly says it does not judge the reasoning connecting them. Table 13 uses the same kind of match for local solve. I cannot tell whether negated statements, incompatible remedies, or a correct fact used in the wrong role can also pass. Appendix E.3 provides useful manual semantic judgments for the preliminary MuSiQue and GAIA study, but I cannot tell whether analogous validation covers the MAScope verdicts. Table 12 checks citation loss rather than semantic correctness. I see this as a fixable issue, and I would find an existing output audit or narrower labels such as required-content delivery helpful for interpreting these results.

  2. I would interpret the dependency gap more cautiously as a measure of collaboration loss. Parallel and dependent tasks come from different families, and Table 8 matches expert counts closely but does not match all aspects of task content or reasoning difficulty. I find the single-agent reference informative, yet Appendix F.4 correctly states that it is not a counterfactual for any evaluated method. My concern is that describing the difference between the two drops as the part added by handoffs goes beyond what this comparison directly isolates. I see this mainly as a claim-level issue. I still value the stage counts and traces, but I would present the aggregate gap as a comparative diagnostic observation rather than a uniquely identified causal cost.

Questions for authors
  1. I would like to know whether any existing audit compares MAScope string verdicts for task success and local solve with semantic judgments of the same outputs, beyond the preliminary study in Appendix E.3. I would also like the authors to clarify how the evaluator handles negation, contradictory recommendations, and preregistered phrases mentioned only as rejected alternatives.

  2. I would like the authors to explain which part of the reported dependency gap is directly established by the logged stage failures and which part is inferred from comparison with the single agent. I would also like them to narrow the handoff-cost wording to reflect the different task families and the limitation stated in Appendix F.4.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Eyy9 (1/2)

Thank you for the careful review and for recognizing that the benchmark makes required contributions "traceable and difficult to bypass" and that the process analysis is "concrete and well supported". We respond to your two comments below.

W1 / Q1. The scoring rule establishes content presence more directly than correctness; negated statements, incompatible remedies or a correct fact in the wrong role might pass, and no semantic check like E.3 covers the MAScope verdicts. An output audit or a narrower label (required-content delivery) would help.

R1. Thank you for this analysis. C.2 sets the scope of acceptance: it decides whether the required content is delivered with its source, and the reasoning that connects the content lies outside the decision. Within this scope, the three cases you list can be counted directly among the accepted answers.

  • Acceptance reads only the final answer a method submits. It reads the answer field of the submit event and the identifiers it cites (Table 10). Table 11 gives, for each method, the native output behind submit: the answer selected by MAD's vote, the output of MoA's final aggregator, the result AgentVerse's evaluator accepts, the submission of Anemoi's answer-finding agent, and the answer of the coordinator or assembly step in the centralized methods. Discussion, voting and adjudication are logged as message and aggregate events and do not enter acceptance.

  • All three cases are reasoning errors made when an answer connects the evidence. A unit's identifier and terms enter the answer only after its source passage is retrieved and passed to the writer (A.3). The units of a task are complementary and mutually consistent: they come from accepted answers of one candidate group (B.1), the subsumption check removes units another unit covers, and the reference writes all of them into one complete answer (G.1). Negating a unit's finding thus negates evidence the team itself retrieved; of two incompatible remedies, one must come from a passage outside the reference; and applying a value to another component swaps two retrieved results. As the reasoning that connects the evidence lies outside the decision (I.2), these cases are false positives under the protocol, and each can be counted.

  • We checked all accepted answers for three kinds of misuse. Given that an accepted answer states every required unit's identifier and terms, content can be misused in three ways: (a) polarity, the finding is negated or mentioned only as a rejected alternative; (b) binding, a value or term is applied to the wrong component or role; (c) consistency, the answer also states content that conflicts with a unit. These cover the negated statements, incompatible remedies and wrong roles you list. The check covers all 74,254 accepted answers of the twelve methods and the single agent over five runs, including misuse in a local output (s(4)s^{(4)} uses the same match) once it is written into the final answer.

TypeAnswersShare of acceptedHighest methods
(a) Polarity4000.54%MAD 1.77%, MoA 1.42%, AgentVerse 1.33%
(b) Binding640.09%MoA 0.25%, MAD 0.20%, SelfOrg 0.16%
(c) Consistency2930.39%MoA 1.27%, MAD 0.91%, Anemoi 0.57%
Total7571.02%–
  • Misuse concentrates in MoA and MAD; rejecting every misused answer lowers success by 0.42 points on average and by at most 0.94 (MoA). Binding is rarest because a term that writes key and value in one phrase (e.g., "timeout raised to 45 s") stops matching once the value is swapped. MoA's aggregator synthesizes several responses critically, and MAD's agents rewrite their answers after reading the others' in round two; both write other agents' evidence into the final answer and then negate it or set it beside conflicting content. The centralized methods, whose coordinator writes from the full transcript, have no such step and misuse 0.38% to 0.55% of their answers, the single agent 0.21%. Negation inside correct advice (e.g., "do not disable verification") is not counted.

  • With misuse deducted, the conclusion of §4.3 ③ is unchanged. Misuse corresponds to s(5)s^{(5)}; deducting each misused unit from s(5)s^{(5)} moves r5r_5 for MoA 80.0 → 77.6, MAD 73.9 → 72.9, AgentVerse 79.8 → 78.8 and Anemoi 76.4 → 75.9, and by 0.2 to 0.3 points for the centralized methods. MAD and Anemoi remain the two lowest at r5r_5 in the same order, the Survival ranking is unchanged, and the mean integration loss moves from the 14.5% of §4.3 ③ to 15.2%, so the original value is conservative. Relaxing the verdict (Tables 12 and 20) and tightening it (Table 21) thus both keep the original rankings and conclusions.

Revision. C.2 adds one sentence stating that acceptance reads the answer field of the submit event, the final output of each method (Table 11), and, after Table 20, the misuse check with Table 21.

AuthorsOpenReview ↗

Response to Reviewer Eyy9 (2/2)

W2 / Q2. The dependency gap should be read more cautiously: parallel and dependent tasks come from different families, matched in expert counts but not in every aspect of content and difficulty; the single agent is not a counterfactual; describing the difference between the two drops as the part added by handoffs goes beyond what the comparison can separate. Which parts are established directly by stage failures, and which are inferred from the comparison?

R2. Thank you for this comment. The conclusions on the dependency gap rest on two kinds of evidence, the stage decisions and the single-agent comparison. Following your request, we state what each establishes and the corresponding revision of F.4.

  • Where the loss occurs is established directly by the stage decisions. On the 1,630 dependent queries, each of the 3,664 certified dependencies is decided from the log as to the stage at which it breaks (Table 13, Table 17): 21% of ready dependencies are not turned into work, 20% of assigned work loses its upstream constraints, and 14.5% of solved results do not enter the final answer, while once a finding arrives, the local solve rate of the twelve methods lies between 86.9% and 93.1% (Table 2). Dependency Survival =∏krk=\prod_k r_k is a quantity of each method on its own. The loss is concentrated in the steps that turn one agent's finding into the next agent's work (§4.2 ②, §4.3); this conclusion comes from the stage decisions on the dependent tasks themselves and does not depend on a comparison with C1 or with the single agent.

  • The single-agent comparison establishes relative magnitude and direction. The two task types come from the same pipeline and pass the same subsumption check and certification, and the structure is determined only by whether certification returns an empty dependency graph (B.1, B.2); all units pass the same local-solvability criterion, with a mean of 90.9% over the full set (B.3); and at each scale, the mean number of experts of the three structures differs by at most 0.14 (Table 8). Under these controls, C1 and C2∪C3 are the same pair of task sets for every evaluated system, and the twelve methods and the single agent face the same differences in content and difficulty between them. The single agent is the only system on this pair with no handoff; its drop from parallel to dependent tasks (15.1%) reflects the size of this difference without division of labor and serves as the reference for reading the teams' drops: the mean team drop is 48.6%, and the further a method's drop exceeds the reference, the lower its Dependency Survival (Pearson −0.89, Spearman −0.87). This comparison points the differences between methods to the steps the stage decisions locate, but by itself it cannot isolate the share due to handoffs. The last sentence of F.4 positions the reference exactly this way: "a common yardstick for all twelve methods and a counterfactual for none of them".

  • Following your suggestion, the revision narrows the attribution sentence in F.4. The last sentence of the first paragraph of F.4 called the difference between the two drops (33.5 points of relative success) the part the handoffs add, which goes beyond what the single-agent comparison can separate. The revision states these 33.5 points only as the team's drop beyond the reference, and states that the stage decisions place each dependency a team loses at the step that dropped it (Table 17), with the four capabilities of Table 2 reporting these losses stage by stage. In the revised F.4, the stage decisions give where the loss occurs, and the single-agent comparison gives its relative magnitude.

Revision. The two changes to F.4 are described above.

Your two comments have made the scope of the protocol and the reading of the dependency gap clearer in the revision, and we thank you for them. We are glad to discuss any further questions. If our responses address your concerns, we hope you will consider raising your score.

I would like to thank the authors for their thoughtful rebuttal. I agree with the points presented, which have largely resolved my doubts, and I will maintain my positive rating as I lean towards recommending the paper for acceptance.

Thank you for reading our rebuttal and for your kind reply. We are glad that our responses have resolved your concerns. Your suggestions have made the paper more precise: the revision adds the misuse check of accepted answers to C.2 (Table 21) and states the comparison of F.4 as a difference from the single-agent reference, with the stage decisions of Table 17 locating each lost dependency. We sincerely appreciate your careful reading and your support for our work.

Reviewer_CiM4OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper proposes a new benchmark to evaluate the collaborative capabilities of multi-agent systems powered by language models. The benchmark consists of 2766 collaborative tasks and aspires to capture four collaborative capabilities: (i) expert selection; (ii) task orchestration; (iii) information transfer; and (iv) result integration. The authors use their benchmark to evaluate 12 popular MAS frameworks, finding that tasks that depend on collaboration between multiple agents are significantly more likely to fail compared to a single agent baseline. Using trajectory analyses, the paper attributes the degradation to failures in planning during task orchestration, information loss during transfer, and aggregating results. Additional ablations using exhaustive collaboration and memory modules are unsuccessful in closing the observed gaps.

Critical strengths

The paper evaluates 12 MAS frameworks and additionally experiments with memory modules. This is a non-trivial engineering effort. With the growing integration of MAS into production settings, gaining a better understanding of their failure modes is a valuable objective. To that end: creating realistic, diverse benchmarks with verifiable rewards is a useful contribution to the field. Unfortunately, as outlined in the weaknesses and questions below, this reviewer has strong concerns about the research presented in this paper.

Critical weaknesses

[clarity] This reviewer found the paper difficult to follow. The core contributions are listed as “an environment with tasks to evaluate collaborative capabilities” and an “evaluation of MAS frameworks using the environment.” For the former, I refer to the questions section. The latter is more alarming: I frankly cannot find any description of how trajectories are analyzed, or what the many qualitative statements in Section 4 are based on (I could only locate some examples in Appendix G). Was this done using an LLM-as-judge approach? Did the authors hand-annotate these? Were there deterministic references?

[quality] The work contains various “statements,” for which it is unclear what they are based on. For instance, the first line of the abstract: “performance depends on how that expertise is organized, not on how much of it is present.” or, the failure modes in section 4.3.

[quality] No confidence intervals are reported in many places.

[quality] While the paper uses a large set of MAS frameworks, they are evaluated using only a single model (Claude Sonnet). Hyperparameters used for this model (4.1, implementation details) are further omitted, e.g., temperature, reasoning effort. This strongly confounds the results.

[significance] The related work section only cites works from 2023 onwards. While I understand that the focus of this paper is MAS based on language models, research on (AI agent) collaboration has been around for quite some time. Without properly situating this work in existing literature it is difficult to evaluate its significance.

Questions for authors
  1. How did the authors ensure that different experts did not hold overlapping information? A.1 only mentions deduplication within a community using MinHash, which does not capture semantic similarity or cross-community information.
  2. How did the authors ensure that information from the 51 disjoint corpora was not already available as parametric knowledge? A.3 mentions that “golden” answers rely on unique identifiers, but do not avoid agents using parametric knowledge – potentially penalizing agents for not querying references for information they already know.
  3. A.3, l883-884: “the subsample carries the same difficulty mix as the benchmark,” – how is this “difficulty” measured?
  4. A.3 [l896-898] “User names, profile links and other author-identifying metadata are stripped before a corpus enters the environment, so an agent never sees who wrote it.” How?
  5. B.1 “composing a question”: who is writing these questions? How are they checked? This honestly goes for the rest of section B.
  6. Outcomes are claimed to be verifiable, the actual check performed seems quite “noisy”: C.3, l1266-1267: “An identifier matches as an exact string, the fifteen characters of § A.3, anywhere in the submitted answer.”
  7. What reasoning level was used for the underlying model?
  8. Section 4.1, “A MAS does not reliably outperform a single agent” – See [A] for a reference on the “gap” between agents’ solo and collaborative performance.
  9. Section 3.1, [l196-197] “To confirm that a task is well posed, we solve it end to end,” – who solves it?

[A] Davidson, Tim R., et al. "The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperation." arXiv preprint arXiv:2511.02687 (2025).

Suggestion: It is clear that the authors put significant effort into this work. It may benefit from scoping down the main text to make sure the benchmark itself is clearly described. Interesting ablations such as those in 4.4 and 4.5 might be better suited for the appendix, such that the reader can focus on the “core” evaluation of the new benchmark.

10 comments
AuthorsOpenReview ↗

Response to Reviewer CiM4 (1/6)

Thank you for your review and for recognizing the engineering effort and the benchmark contribution of this work. We group your comments into eight topics and respond to each. Several points are already addressed in the paper; for these we give the original sentence, section and page so they can be checked easily. Content that needed to be added or clarified has been added in the revision.

W1 + Q6. No description of how trajectories are analyzed or what the qualitative statements in Section 4 are based on (LLM-as-judge, hand annotation, or deterministic references?); the outcome check seems "quite noisy".

R1. Thank you for the question. The trajectory analysis consists of three parts, the event stream, the stage decisions and acceptance, and all three are deterministic string matches on the log. Below we give their definitions in the paper, how they support the qualitative statements in §4, and the size of the matching error.

  • All three parts are defined explicitly in the paper. Event stream: Table 10 (C.1, p. 24) defines nine events, and an adapter per method rewrites the native log into these events; "every predicate of Table 13 is decided from these fields alone". Stage decisions: the last sentence of §3.3 (p. 5) reads "Each stage is decided by matching logged identifiers and preregistered terms, alike across classes", and §3.3 defines rkr_k; Table 13 (C.4) lists the predicate of each of the five stages and the event fields it matches. Acceptance: C.2 (p. 23) defines acceptance as identifier ∧ preregistered terms holding for every required unit.

  • Of the three options you list, the paper uses deterministic references. C.2 states "no model reads an answer to decide whether it succeeded" (p. 23); C.5 states "Both checks are string matches on the log, as every stage predicate is" (p. 27); H states "Acceptance itself uses no prompt" (p. 43).

  • Every qualitative statement in §4 traces to counts in the tables or to the published design of a framework. Rates: the 21% / 20% / 14.5% of §4.3 are the means of the twelve methods' conditional rates (Table 2), and each method's rates can be recomputed from adjacent columns of Table 17. Mechanisms: CrewAI's task list fixed at kickoff, GoA's prompt asking for two- to three-sentence responses and MAD's majority vote after two rounds are all taken from the published version of each framework (D.1, Table 15). Trajectories: every example in Appendix G ends with the stage decision of its edge. The new Appendix F.5 counts the composition of each method's failing edges (Table 24): across the twelve methods, failures at s(2)s^{(2)} in which the plan is not updated with the finding make up 88.6% to 94.9%; in the three adjudicating methods MAD, Anemoi and AgentVerse, failures at s(5)s^{(5)} in which the result is overruled or omitted by the aggregator make up 85.9% to 92.5%, with overruling alone at 67% to 79%.

  • Both kinds of error of exact matching are bounded quantitatively. The sentence you quote is in C.2 (p. 23). An identifier is MS- followed by 12 Crockford base-32 characters of a keyed hash, one of 321232^{12} (about 1.2×10181.2\times10^{18}) values that no rewording can produce, and the preregistered terms accept only the surface variants frozen at construction. The revision reports checks of both kinds of error in C.2. For false positives (the match passes but the content is not used correctly), we checked all 74,254 accepted answers of the five runs: misuse totals 1.02%, and rejecting all of these answers lowers success by 0.42 points on average and by at most 0.94 (MoA), with the method ranking unchanged (Table 21). For false negatives (correct content that fails to match), dropping the identifier requirement raises success by 0.7 to 2.9 points (Table 12), and widening the term match raises Survival by 0.04 to 0.12 points and success by 0.01 to 0.06 points (Table 20). The corpus-removal experiment of A.3 shows the discriminative power of the match: with one required corpus removed, success falls from 52.9% to 4.7%.

Revision. The last sentence of §3.3 now adds "and §C.4 lists the predicate of every stage (Table 13)", so readers can find the decision rules directly from the main text.

AuthorsOpenReview ↗

Response to Reviewer CiM4 (2/6)

W2. It is unclear what statements such as "performance depends on how that expertise is organized, not on how much of it is present" are based on.

R2. Thank you for the question. This statement summarizes the findings of prior work and is the motivation of the paper; our results take it further, to the level of workflow stages.

  • The statement first summarizes prior work. The second paragraph of §1 (p. 2) states "gathering agents does not by itself create collaborative value" and then lists the evidence: a larger team brings wider expertise but multiplies coordination overhead and error propagation (Kim et al., 2026a, Nature Machine Intelligence); agents misjudge their peers' intentions (Li et al., 2023b); commitments go unfulfilled and features conflict in coding tasks (Khatua et al., 2026); context is omitted, reasoning repeated and aggregation premature in information-seeking tasks (Cemri et al., 2025); and the same agents yield markedly different results under different organizational structures (Zhuge et al., 2024; Qian et al., 2024). The concluding sentence of that paragraph, "A team's value lies in the collaboration itself…", is the source of the first sentence of the abstract. These works show the effect of organization at the outcome level or through case analyses, and this is the motivation of the paper: building on them, we analyze the collaboration failures of MAS end to end, quantitatively and down to the stage where they occur.

  • Our results take this statement from the outcome level to the stage level. §4.2 ① (p. 6) first reproduces the phenomenon under controlled conditions: with corpora, budget and backbone fixed, MoA, MAD and SelfOrg involve all 51 experts, yet their success is the lowest of the twelve methods (32.0% / 29.4% / 32.3%), and AgentNet, with a required-expert recall of 68.1%, scores higher than all three. The total expertise is constant in this comparison, and only the organization changes. Building on this, §4.3 locates the differences at three specific stages, each mechanism supported by the conditional rates of Table 2 and the edge counts of Table 17; §4.4 shows that adding collaboration rounds indiscriminately changes success by only −0.77 to +1.63 points; and the new Appendix F.5 shows that repairing a single behavior at the diagnosed stage raises that stage's rate and task success. Prior work establishes that organization matters, and our paper further shows at which step of the workflow it acts, how much is lost at each step, and what repairing that step gains. The evidence for each failure mechanism of §4.3 is given in R1.

W3. No confidence intervals are reported in many places.

R3. Thank you for the suggestion. Every number in the paper is the mean of five independent runs, and the quantities our conclusions rest on are reported with their run-to-run standard deviations; the revision adds the standard deviations of every cell and every stage rate in F.1.

  • Sample sizes and reported standard deviations. Each number is the mean of five runs over 2,766 queries; the smallest cell has 154 queries, i.e., at least 770 task runs. Standard deviations are reported in the Overall column of Table 1, the Dep. Survival column of Table 2, the Answered column of Table 17, the Δ of Table 18, and the Δ and Δ% of Table 19. F.2 and F.3 compare each memory-module gain with its standard deviation, and nine of the twelve gains exceed twice their standard deviation.

  • The differences behind the main-text conclusions are 5 to 28 points, while the largest run-to-run standard deviation is 1.00 point. The largest standard deviation in Table 1 is MAD's (1.00); the differences compared in the main text include the 28-point drop on dependent tasks, gaps of more than 10 points between coordination types, and gaps of 5 to 20 points between stage rates. The smallest gap compared in the main text is AgentVerse's 0.9-point margin over the single agent. §4.2 ① uses it to show that the advantage of MAS over a single agent is not reliable: eight methods exceed the single agent by 11.1 to 0.9 points and the other four fall below it, a conclusion that rests on the whole distribution.

  • The revision adds the standard deviation of every entry. F.1 adds the standard deviations of the nine cells of Table 1 and the four stage rates of Table 2. The cell standard deviations range from 0.4 to 3.6, while the smallest gap between a method's dependent cell and its C1 cell at the same scale is 15.6 points (AgentNet); the stage-rate standard deviations range from 0.28 to 1.21, while the stage-rate gaps compared in the main text are 5 to 20 points.

Revision. F.1 adds Tables 22 and 23, giving the mean and standard deviation of the nine cells of Table 1 and of the stage rates of Table 2.

AuthorsOpenReview ↗

Response to Reviewer CiM4 (3/6)

W4 + Q7. Only a single model (Claude Sonnet) is used, and its hyperparameters (§4.1, implementation details), e.g., temperature and reasoning effort, are omitted; which reasoning level is used?

R4. Thank you for pointing this out. The paper gives the implementation details in dedicated sections; below we first give their locations and then the model configuration. The official API of Claude Sonnet 5 no longer offers parameters that directly control sampling, such as temperature and top-p, and our experiments use the model's default configuration with effort set to high. We apologize for not stating this configuration in the paper, and the revision adds it in the corresponding places.

  • The implementation details are given in Appendix D and C.1. The Implementation details of §4.1 give the backbone, the shared expert pool and retrieval interface, and the five-run setup, and point to Appendix D. D.1 describes how the twelve frameworks are connected to the environment and which mechanisms each keeps (Table 15); C.1 gives the per-task budget (2M tokens and 256 model calls), the output cap of a single call (4,096 tokens) and the setup of the five independent runs; E.1 gives the configuration of the comparison with other benchmarks. These settings are the same for the twelve methods and the single agent.

  • The experiments use the default configuration of Claude Sonnet 5 at effort high. For Claude Sonnet 5, Anthropic's official API no longer accepts non-default values of sampling parameters such as temperature, top-p and top-k, and the depth of reasoning is controlled by effort. All model calls in the paper use the model's default configuration at effort high, identically for the twelve methods, the single agent and the construction-time solves. The differences between the five runs come from the model's own sampling, which is exactly the variation measured by the run-to-run standard deviations of Tables 1 and 2.

  • All methods share the same backbone and the same configuration, which are constants across methods. The paper measures the collaboration mechanisms of MAS: the twelve methods and the single agent use the same backbone, model configuration, corpora, retrieval interface and budget, so the differences between methods come from the mechanisms themselves. I.1 states that each number describes a method at this scale of task and on this backbone, and I.3 keeps the minting key, so that other backbones can be evaluated on the same benchmark after the identifiers are re-minted.

Revision. The Implementation details of §4.1, C.1 and E.1 state that all calls use the default configuration of Claude Sonnet 5 at effort high, and the corresponding wording in D.1 and D.2 is aligned.

Reviewer_84UjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MAScope, a benchmark for diagnosing collaboration failures in multi-agent systems (MAS). It comprises 51 expert agents with private knowledge bases derived from Stack Exchange and 2,766 tasks with certified information dependencies. The benchmark traces required contributions through four capabilities: expert selection, task orchestration, information transfer, and result integration. Experiments on twelve representative MAS methods show that collaboration performance deteriorates substantially on interdependent tasks, with failures attributed to rigid planning, information compression, and incorrect result aggregation. Additional experiments on exhaustive collaboration and memory modules further demonstrate that increasing participation or adding memory does not necessarily improve task success.

Critical strengths
  1. Fine-grained, end-to-end diagnosis of collaboration failures. Unlike outcome-only evaluations, MAScope traces required contributions through the collaboration pipeline and identifies where information is lost. This provides actionable insights into multi-agent system design rather than merely ranking methods by final accuracy.

  2. Broad and systematic empirical evaluation. The benchmark covers 12 representative methods, multiple dependency structures, and different levels of required expertise. The controlled comparisons with a single-agent baseline and additional collaboration and memory experiments provide compelling evidence that coordination quality, rather than team size alone, is critical to performance.

Critical weaknesses
  1. Potential sensitivity of evaluation to predefined textual matching rules. Task success and intermediate collaboration stages rely on source identifiers and preregistered terms or their recorded variants. Although the paper reports an identifier-free rescore, it does not establish whether the fine-grained stage-level conclusions remain robust when semantically equivalent but unanticipated paraphrases are considered. This may affect the reliability of the diagnosed failure categories.

  2. The causal interpretation of the identified failure mechanisms could be better isolated. The trajectory analysis associates performance loss with planning rigidity, semantic compression, and adjudicative aggregation, but the evidence is primarily observational across existing frameworks. Differences in these mechanisms are intertwined with other implementation choices. More targeted interventions that modify one mechanism while holding the rest of a framework fixed would strengthen the claim that these factors directly account for the observed performance gaps.

Questions for authors
  1. How sensitive are the stage-level metrics and the resulting failure analysis to semantically correct paraphrases that do not match the preregistered terms or recorded variants? Have you evaluated these cases through independent human assessment or a semantic evaluator?

  2. Can you provide controlled intervention results in which task replanning, evidence-preserving communication, or aggregation behavior is modified individually within the same framework? Such results would help distinguish causal effects from correlations across different MAS implementations.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 84Uj (1/2)

Thank you for the careful review and for recognizing the "fine-grained, end-to-end diagnosis" of the paper. Your two comments concern the sensitivity of the verdicts to rewording and a causal test of the failure mechanisms; we ran one additional experiment for each.

W1 / Q1. How sensitive are the stage-level metrics and the failure analysis to semantically correct paraphrases that miss the preregistered terms or recorded variants? Were such cases checked by humans or a semantic evaluator? Table 12 tests only the identifier.

R1. Thank you for this question. MAScope uses deterministic matching by design, and its construction already handles different wordings of the same content; Table 12 quantifies the identifier dimension, and we report the same check for the term dimension below.

  • Deterministic matching makes the differences between methods come only from the collaboration process. As C.2 states, no model reads an answer to decide success: the identifier proves where the evidence came from, the preregistered terms prove the content, and the same answer always receives the same verdict. A semantically correct rewording can come from evidence passed on by an upstream agent or from the backbone's own prior knowledge, and a human or model-based judgment can hardly tell the two apart; the preregistered terms come from the unit's source passage and, together with the identifier, tie the answer to evidence actually passed through the workflow.

  • The protocol handles different wordings at construction in three ways. First, terms are only quantities with units, configuration keys, version numbers or named components, never general concepts (B.2); keys and version numbers are fixed strings and show no rewording in the check below. Second, the surface variants of every term (case, inflection, forms of a quantity) are recorded and frozen at construction (C.2), and the reference also records equivalent expressions, e.g., a session identified by its query or by its session identifier (Appendix G). Third, every unit is solved ten times by the same backbone before freezing, and a unit passed fewer than 6 times is revised or dropped (B.3, median 9/10), so the variant lists are already tested against the backbone's own phrasing.

  • Following Table 12, we ran the same quantitative check for the term dimension. A term false negative can only occur on a failing edge where the identifier arrived but a term did not match: 3,434 edges per run, 22.7% of all failing verdicts (the rest belong to the identifier dimension of Table 12). We divide the cases with the same referent but a different string into ① form (case, inflection, separators, unit spelling), ② rewording (synonym, paraphrase, abbreviation) and ③ conversion to another unit or notation; the variant lists cover ①. We re-match ① with a generic normalizer independent of the lists, compare ③ after converting to base units, and examine ② case by case. A constraint mentioned without its value is the semantic compression of §4.3 ② and is not counted.

TypeEdges per runShare of failing verdictsLargest change in a stage rate
Candidates (identifier arrived, term judged absent)3,43422.7%–
① Form00.0%0
② Rewording28.00.18%+0.11 (MoA, r4r_4)
③ Conversion3.80.03%<0.1
Total31.80.21%+0.11
  • Term false negatives account for 0.21% of failing verdicts. Type ① is zero, so the lists miss nothing within their scope, while 18.6% of passing term matches hit through a variant, so the lists are active in every run. Type ② (28.0 edges per run) includes even debatable cases such as "cap the retries" versus "bounded retries". It is most frequent in MoA, SelfOrg, GoA and MAD (3.0 to 3.8 per run), where findings are restated repeatedly, consistent with the semantic compression of §4.3 ②, and the two ends of the ranking match the gains of Table 12 (MoA and SelfOrg highest, AutoGen lowest).

  • Re-deciding all type ② and ③ cases as passes leaves the conclusions unchanged. The largest increase in a stage rate is 0.11 points (MoA, r4r_4); Dependency Survival rises by 0.04 to 0.12 points and task success by 0.01 to 0.06 points. The lowest method at each stage, the Survival ranking and the three losses of §4.3 (21.07% / 19.93% / 14.55%, then 21.04% / 19.87% / 14.54%) stay as they are. The term dimension moves success by 0.01 to 0.06 points and the identifier dimension by 0.7 to 2.9 points (Table 12), far less for the former, and the attribution of the three mechanisms holds within this error.

Revision. After the identifier check of Table 12, C.2 adds the same check for the term dimension, and the new Table 20 gives each method's counts of types ② and ③ and the re-decided stage rates. If other kinds of wording should be included, we would be glad to add them during the discussion.

AuthorsOpenReview ↗

Response to Reviewer 84Uj (2/2)

W2 / Q2. The evidence for the three mechanisms is mainly observational across frameworks and intertwined with other implementation choices; targeted interventions that modify one mechanism while holding the rest of a framework fixed would strengthen the causal claim.

R2. Thank you for this suggestion. Changing one mechanism within the same framework is the right test of the attribution in §4.3, and we added such experiments.

  • The attribution in §4.3 has two steps, localization and explanation, and the intervention tests the explanation. Localization is the stage at which each certified dependency breaks, decided from the log events of that edge (Table 13) regardless of how a method delegates or votes. Explanation attributes the loss at a stage to a behavior, e.g., CrewAI's task list is fixed at kickoff, so a later finding cannot become a new task; it rests on each framework's published design and the log events of failing edges (D.1, Table 15, Appendix G). Your suggested intervention tests it directly: repair only this behavior in the same method, compare its stage with the others, and apply the same repair, as a control, to a method whose loss at that stage has other causes.

  • We designed one repair per mechanism, each changing a single behavior. For s(2)s^{(2)} (a ready dependency is not turned into work), replanning: each time a subtask returns, the coordinator decides once more whether the result calls for work by an expert already recruited, and assigns it if so. For s(3)s^{(3)} (assigned work loses its upstream constraints), an evidence-preserving handoff: a retriever's message carries the identifiers it retrieved and the verbatim constraint terms, with prompts, topology and rounds unchanged. For s(5)s^{(5)} (a solved result does not enter the final answer), a source-preserving merge: among conflicting candidate conclusions, the one carrying a retrieved identifier is kept instead of being decided by a vote or a judge. The repairs go to CrewAI, GoA and MAD, the cases §4.3 gives for the three mechanisms.

  • Each repair is also applied to a control method (†). The controls also fail at the same stage, and the classification of F.5 (Table 24) gives how many failures a repair can reach. Of GoA's 208 failing edges at s(2)s^{(2)}, 105 never reach a node that can assign work, and replanning can act on only 84. CrewAI's work is forwarded by a manager that does not retrieve, so of its 314 failing edges at s(3)s^{(3)} the repair reaches only the 23 that a worker returns and the manager delegates again. LangGraph assembles its answer by slot, and only 3 of its 81 failing edges at s(5)s^{(5)} are excluded by adjudication. Each configuration runs once on the query order of run 1 of the native method, and the table reports differences from that run.

RepairMethodΔr, target stageΔr, other stagesΔSurvivalΔSuccessΔTokens
ReplanningCrewAI+8.9 (r2r_2)−0.3 to +0.1+5.7+3.4+18.4%
ReplanningGoA†+1.2−0.1 to +0.2+0.8+0.5+5.7%
Evidence-preserving handoffGoA+9.1 (r3r_3)−0.2 to +0.3+6.4+3.8+27.3%
Evidence-preserving handoffCrewAI†+0.5−0.2 to +0.2+0.4+0.2+3.8%
Source-preserving mergeMAD+9.5 (r5r_5)−0.2 to +0.1+5.2+3.1+0.4%
Source-preserving mergeLangGraph†+0.1−0.1 to +0.2+0.2+0.1+0.1%
  • A repair raises only the diagnosed stage, and only in the diagnosed method. In the three diagnosed methods the target stage rises by 8.9 to 9.5 points and every other stage moves by 0.3 points or less. In the controls the same repairs raise the target stage by only 0.1 to 1.2 points, in line with the reachable edges of Table 24 (GoA's 1.2 points come from the 84 edges above). The three mechanisms thus correspond to three distinct behaviors, and one repair cannot substitute for another.

  • The stage gains carry into task success. In the diagnosed methods, Dependency Survival rises by 5.2 to 6.4 points and success by 3.1 to 3.8 points, against 0.1 to 0.5 points of success in the controls, and each repaired stage is the diagnosed method's lowest stage rate in Table 2 (CrewAI's r2r_2, GoA's r3r_3, MAD's r5r_5). Repairing a single behavior, in task replanning, evidence transfer or result integration, within the same framework raises the corresponding stage rate and task success, which supports the attribution of the three stage losses in §4.3.

Revision. The new Appendix F.5 gives the failure forms of the twelve methods at the three stages (Table 24), then the implementation of the three repairs and the per-stage changes of the six configurations (Table 25).

Thank you for your valuable comments; if our responses address your concerns, we hope you will consider raising your score.

Thank you for the detailed rebuttal and additional experiments. The sensitivity analysis in Table 20 largely addresses my concern about the robustness of the evaluation metrics: accepting the identified rewordings and unit conversions changes only a small fraction of failing verdicts (0.21%) and shifts any stage rate by at most 0.11 percentage points, leaving the main stage-level conclusions unchanged.

The targeted interventions in Appendix F.5 also provide stronger evidence for the identified failure mechanisms. Each repair changes a specific behavior, improves its targeted stage by 8.9–9.5 percentage points while other stages move by at most 0.3 points, and has much smaller effects on control methods where the targeted failure form is rare.

One minor limitation is that each intervention configuration was evaluated only once, whereas the main experiments average over five runs. The observed stage-level improvements are substantially larger than the run-to-run variation reported for the original methods in Table 23, although that does not establish the stability of the interventions themselves. I nevertheless find the additional evidence sufficient to address my main concerns.

Overall, I will maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear AC and Reviewers,

We thank all three reviewers for their time and detailed comments. MAScope aims to make every required contribution in multi-agent collaboration non-bypassable and traceable, so that a collaboration failure can be located at a specific step of the workflow. We are glad that the reviewers see value in this direction. Reviewer 84Uj notes that the paper offers a "fine-grained, end-to-end diagnosis of collaboration failures" and that the evaluation and experiments "provide compelling evidence that coordination quality, rather than team size alone, is critical". Reviewer Eyy9 finds that the benchmark "makes required contributions both traceable and difficult to bypass" and that the process analysis is "concrete and well supported" (Table 13, Table 17, Appendix G). Reviewer CiM4 regards evaluating 12 MAS frameworks with additional memory-module experiments as "a non-trivial engineering effort" and building realistic, diverse benchmarks with verifiable rewards as "a useful contribution to the field".

Revised version. We have uploaded a revised PDF in which all changes are marked in blue. The revisions are listed below; the last column gives the corresponding comments.

#LocationChangeComment
1C.2, after Table 12 (p. 24; Tables 20–21 on pp. 27–28)Checks of the string-based verdicts: new Table 20 re-decides every stage verdict and acceptance with the term match widened (false negatives); new Table 21 checks accepted answers for misused content (false positives)84Uj W1, Eyy9 W1, CiM4 Q6
2C.2 (p. 23)One sentence stating that acceptance reads the final answer each method submits (Table 11)Eyy9 W1
3§3.3, last sentence (p. 5)Points to where the stage predicates are given (Table 13, §C.4)CiM4 W1
4New Appendix F.5 (pp. 38–39)Intervention experiment: one repair per failing stage, each changing a single behavior, applied to the diagnosed method and to a control method; implementation details and the per-stage changes of the six configurations (Table 25)84Uj W2
5Appendix F.5 (p. 38)New Table 24: composition of the failure forms at the three stages for the twelve methods84Uj W2, CiM4 W1
6Appendix F.1 (pp. 34–36)New Tables 22–23: mean and standard deviation of every cell of Table 1 and every stage rate of Table 2CiM4 W3
7§4.1 (p. 6), C.1 (p. 23), E.1 (p. 32)States that all calls use the default configuration of Claude Sonnet 5 at effort highCiM4 W4, Q7
8§5 (p. 9)Two opening sentences leading from classic MAS research to LLM-based MAS, citing Smith (1980), Erman et al. (1980), Durfee & Lesser (1991) and Grosz & Kraus (1996); the evaluation paragraph cites Davidson et al. (COLM 2026) and states the difference in levelCiM4 W5, Q8
9F.4 (p. 37)Narrowed attribution: the 33.5 points are stated as the team's drop beyond the reference, and the location of the loss is given by the stage decisions (Table 17)Eyy9 W2
10§3.1 (p. 4), A.3 (p. 18), A.1 (p. 16)§3.1 states that the agent solves each task end to end under the expert prompt; A.3 replaces "difficulty mix" with "structure-and-scale composition" and lists the post fields kept and the author fields dropped; A.1 points to B.1 after the deduplication sentenceCiM4 Q9, Q3, Q4, Q1

Our point-by-point responses are posted under each review. We are happy to answer any further questions during the discussion period, and we hope the reviewers will reassess the paper in light of the revision and our responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.