- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MAScope is a benchmark for diagnosing where multi-agent collaboration loses required information. It builds 51 experts with private Stack Exchange corpora and 2,766 tasks from 499 families. Tasks vary in expert count and in whether work is parallel, sequential, or requires discovering a new expert from an earlier finding. Source identifiers and certified dependencies let the evaluator track expert selection, task orchestration, information transfer, local outputs, and final integration. Experiments with twelve methods show larger degradation on dependent tasks than for a single agent accessing the same corpora. The paper relates this degradation to planning, message compression, and aggregation, and finds limited improvements from extended collaboration and memory modules.
Critical strengths
-
The benchmark makes required contributions both traceable and difficult to bypass. Appendices B.2 and C.5 connect dependency certification to the actual retriever and check bypasses in the recorded runs, with only 0.20% of edges bypassed overall. The corpus-removal check in Appendix A.3 and the identifier-free rescoring in Table 12 address two obvious alternatives to the intended interpretation. These checks make the dependency analysis much more convincing.
-
The process analysis is concrete and well supported. Table 13 defines each stage, Table 17 supplies the underlying populations, and Appendix G shows corresponding failure traces. The comparison uses the same backbone, corpora, retrieval interface, and budget caps, while Appendices C and D explain the adaptations and actual cap rates. This supports useful comparisons among the evaluated organizations rather than only a ranking by final score.
Critical weaknesses
-
I am concerned that the scoring rule establishes content presence more directly than content correctness. Appendix C.2 accepts an answer when every required identifier and preregistered phrase occurs, and explicitly says it does not judge the reasoning connecting them. Table 13 uses the same kind of match for local solve. I cannot tell whether negated statements, incompatible remedies, or a correct fact used in the wrong role can also pass. Appendix E.3 provides useful manual semantic judgments for the preliminary MuSiQue and GAIA study, but I cannot tell whether analogous validation covers the MAScope verdicts. Table 12 checks citation loss rather than semantic correctness. I see this as a fixable issue, and I would find an existing output audit or narrower labels such as required-content delivery helpful for interpreting these results.
-
I would interpret the dependency gap more cautiously as a measure of collaboration loss. Parallel and dependent tasks come from different families, and Table 8 matches expert counts closely but does not match all aspects of task content or reasoning difficulty. I find the single-agent reference informative, yet Appendix F.4 correctly states that it is not a counterfactual for any evaluated method. My concern is that describing the difference between the two drops as the part added by handoffs goes beyond what this comparison directly isolates. I see this mainly as a claim-level issue. I still value the stage counts and traces, but I would present the aggregate gap as a comparative diagnostic observation rather than a uniquely identified causal cost.
Questions for authors
-
I would like to know whether any existing audit compares MAScope string verdicts for task success and local solve with semantic judgments of the same outputs, beyond the preliminary study in Appendix E.3. I would also like the authors to clarify how the evaluator handles negation, contradictory recommendations, and preregistered phrases mentioned only as rejected alternatives.
-
I would like the authors to explain which part of the reported dependency gap is directly established by the logged stage failures and which part is inferred from comparison with the single agent. I would also like them to narrow the handoff-cost wording to reflect the different task families and the limitation stated in Appendix F.4.
4 comments
Response to Reviewer Eyy9 (2/2)
W2 / Q2. The dependency gap should be read more cautiously: parallel and dependent tasks come from different families, matched in expert counts but not in every aspect of content and difficulty; the single agent is not a counterfactual; describing the difference between the two drops as the part added by handoffs goes beyond what the comparison can separate. Which parts are established directly by stage failures, and which are inferred from the comparison?
R2. Thank you for this comment. The conclusions on the dependency gap rest on two kinds of evidence, the stage decisions and the single-agent comparison. Following your request, we state what each establishes and the corresponding revision of F.4.
-
Where the loss occurs is established directly by the stage decisions. On the 1,630 dependent queries, each of the 3,664 certified dependencies is decided from the log as to the stage at which it breaks (Table 13, Table 17): 21% of ready dependencies are not turned into work, 20% of assigned work loses its upstream constraints, and 14.5% of solved results do not enter the final answer, while once a finding arrives, the local solve rate of the twelve methods lies between 86.9% and 93.1% (Table 2). Dependency Survival is a quantity of each method on its own. The loss is concentrated in the steps that turn one agent's finding into the next agent's work (§4.2 ②, §4.3); this conclusion comes from the stage decisions on the dependent tasks themselves and does not depend on a comparison with C1 or with the single agent.
-
The single-agent comparison establishes relative magnitude and direction. The two task types come from the same pipeline and pass the same subsumption check and certification, and the structure is determined only by whether certification returns an empty dependency graph (B.1, B.2); all units pass the same local-solvability criterion, with a mean of 90.9% over the full set (B.3); and at each scale, the mean number of experts of the three structures differs by at most 0.14 (Table 8). Under these controls, C1 and C2∪C3 are the same pair of task sets for every evaluated system, and the twelve methods and the single agent face the same differences in content and difficulty between them. The single agent is the only system on this pair with no handoff; its drop from parallel to dependent tasks (15.1%) reflects the size of this difference without division of labor and serves as the reference for reading the teams' drops: the mean team drop is 48.6%, and the further a method's drop exceeds the reference, the lower its Dependency Survival (Pearson −0.89, Spearman −0.87). This comparison points the differences between methods to the steps the stage decisions locate, but by itself it cannot isolate the share due to handoffs. The last sentence of F.4 positions the reference exactly this way: "a common yardstick for all twelve methods and a counterfactual for none of them".
-
Following your suggestion, the revision narrows the attribution sentence in F.4. The last sentence of the first paragraph of F.4 called the difference between the two drops (33.5 points of relative success) the part the handoffs add, which goes beyond what the single-agent comparison can separate. The revision states these 33.5 points only as the team's drop beyond the reference, and states that the stage decisions place each dependency a team loses at the step that dropped it (Table 17), with the four capabilities of Table 2 reporting these losses stage by stage. In the revised F.4, the stage decisions give where the loss occurs, and the single-agent comparison gives its relative magnitude.
Revision. The two changes to F.4 are described above.
Your two comments have made the scope of the protocol and the reading of the dependency gap clearer in the revision, and we thank you for them. We are glad to discuss any further questions. If our responses address your concerns, we hope you will consider raising your score.
I would like to thank the authors for their thoughtful rebuttal. I agree with the points presented, which have largely resolved my doubts, and I will maintain my positive rating as I lean towards recommending the paper for acceptance.
Thank you for reading our rebuttal and for your kind reply. We are glad that our responses have resolved your concerns. Your suggestions have made the paper more precise: the revision adds the misuse check of accepted answers to C.2 (Table 21) and states the comparison of F.4 as a difference from the single-agent reference, with the stage decisions of Table 17 locating each lost dependency. We sincerely appreciate your careful reading and your support for our work.
Response to Reviewer Eyy9 (1/2)
Thank you for the careful review and for recognizing that the benchmark makes required contributions "traceable and difficult to bypass" and that the process analysis is "concrete and well supported". We respond to your two comments below.
R1. Thank you for this analysis. C.2 sets the scope of acceptance: it decides whether the required content is delivered with its source, and the reasoning that connects the content lies outside the decision. Within this scope, the three cases you list can be counted directly among the accepted answers.
Acceptance reads only the final answer a method submits. It reads the answer field of the submit event and the identifiers it cites (Table 10). Table 11 gives, for each method, the native output behind submit: the answer selected by MAD's vote, the output of MoA's final aggregator, the result AgentVerse's evaluator accepts, the submission of Anemoi's answer-finding agent, and the answer of the coordinator or assembly step in the centralized methods. Discussion, voting and adjudication are logged as message and aggregate events and do not enter acceptance.
All three cases are reasoning errors made when an answer connects the evidence. A unit's identifier and terms enter the answer only after its source passage is retrieved and passed to the writer (A.3). The units of a task are complementary and mutually consistent: they come from accepted answers of one candidate group (B.1), the subsumption check removes units another unit covers, and the reference writes all of them into one complete answer (G.1). Negating a unit's finding thus negates evidence the team itself retrieved; of two incompatible remedies, one must come from a passage outside the reference; and applying a value to another component swaps two retrieved results. As the reasoning that connects the evidence lies outside the decision (I.2), these cases are false positives under the protocol, and each can be counted.
We checked all accepted answers for three kinds of misuse. Given that an accepted answer states every required unit's identifier and terms, content can be misused in three ways: (a) polarity, the finding is negated or mentioned only as a rejected alternative; (b) binding, a value or term is applied to the wrong component or role; (c) consistency, the answer also states content that conflicts with a unit. These cover the negated statements, incompatible remedies and wrong roles you list. The check covers all 74,254 accepted answers of the twelve methods and the single agent over five runs, including misuse in a local output (s(4) uses the same match) once it is written into the final answer.
Misuse concentrates in MoA and MAD; rejecting every misused answer lowers success by 0.42 points on average and by at most 0.94 (MoA). Binding is rarest because a term that writes key and value in one phrase (e.g., "timeout raised to 45 s") stops matching once the value is swapped. MoA's aggregator synthesizes several responses critically, and MAD's agents rewrite their answers after reading the others' in round two; both write other agents' evidence into the final answer and then negate it or set it beside conflicting content. The centralized methods, whose coordinator writes from the full transcript, have no such step and misuse 0.38% to 0.55% of their answers, the single agent 0.21%. Negation inside correct advice (e.g., "do not disable verification") is not counted.
With misuse deducted, the conclusion of §4.3 ③ is unchanged. Misuse corresponds to s(5); deducting each misused unit from s(5) moves r5 for MoA 80.0 → 77.6, MAD 73.9 → 72.9, AgentVerse 79.8 → 78.8 and Anemoi 76.4 → 75.9, and by 0.2 to 0.3 points for the centralized methods. MAD and Anemoi remain the two lowest at r5 in the same order, the Survival ranking is unchanged, and the mean integration loss moves from the 14.5% of §4.3 ③ to 15.2%, so the original value is conservative. Relaxing the verdict (Tables 12 and 20) and tightening it (Table 21) thus both keep the original rankings and conclusions.
Revision. C.2 adds one sentence stating that acceptance reads the answer field of the submit event, the final output of each method (Table 11), and, after Table 20, the misuse check with Table 21.