- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates if LLM agents can autonomously organize collaboration when information, state and action are distributed across nodes without predefined task decomposition. It compares six organizations (Merge, Central, P2P, Relay, Role and Control), and evaluates them across four models and four task families to see the performance gain obtained by different organizations. The paper also investigates if the communication contributions converted to persistent process and the failure mode through trace analysis.
Critical strengths
- The paper studies the MAS problem under a meaningful setting where agents must coordinate across separate information. This makes collaboration necessary for task completion.
- The experiments cover four models and four task families span different domains and organizations, and the trajectory analysis systematically studies if contributions of different organizations lead to lasting progress on performance.
Critical weaknesses
- The overall narrative is hard to follow and understand.
- The organization conditions change several factors at once, including information access, communication and how agents combine their work. Also each configuration is evaluated only once, and the proposed failure mechanisms are not tested through targeted interventions. It is unclear which factors drive the observed differences or whether fixing the identified failures would improve performance.
- The evaluation depends heavily on LLM-generated annotations with limited independent validation. The evaluation relies on GPT-6-Astra, with only 96 of 2,880 trajectories rescored by one author. Thus I think perfect agreement on a sample does not establish independent inter-annotator reliability.
Questions for authors
- Could the authors separate the effects of decision making structure from differences in information access, communication, and artifact integration? For example, in Cooper, some successful runs use only one agent’s final branch. How much of the MAS advantage remains if agents work independently without communicating?
- Please see weakness 2, where repeated runs should be provided.
- The authors have already identified a forwarding rule that rejects valid evidence in Control. Why was this not fixed before comparing Control with the other organizations?
- Did the human reviewer assign labels before seeing GPT-6’s annotations? What exactly was hidden during the blind review?
3 comments
Response to Reviewer aRbF (2/2)
Q2 (continued)
Revisions: Appendix B.7 reports graded completion and native success across the submitted round and two full reruns (L1216–1240, Table 14).
Q3. Why did the main comparison keep Control’s original forwarding rule?
We kept per-hop authentication fixed across all six Core conditions so that the main comparison used a common protocol.
Common authentication protocol. Giving Control alone runtime support for authenticating references against their original producers would change the protocol for one condition. We held the per-hop authentication rule fixed across all six Core conditions.
Checks under the original rule. Control still trailed P2P when we excluded affected Core pairs and in Silo without the authentication rule.
Core matched comparison. Control still trailed P2P in Core after excluding the pairs with rejected references. We excluded the 21 pairs with reference rejections in Control. Across the remaining 75 pairs, Control completion was .075 below P2P.
Silo comparison. Control also trailed P2P in Silo, where this authentication rule does not apply. The scores were .210 for Control and .302 for P2P.
Core rejection records. Most Core rejections concerned references that had already reached the recipient. Of the 791 rejections, 332 were resends to nodes that already held the reference, 346 occurred after an earlier successful delivery, and 113 involved references with no prior observed delivery.
Two targeted changes to Control. Both the coordinator’s obligation ledger and per-reference authentication raised Core Control completion. Following your question, we tested two changes separately on the same 96 Core model–configuration pairs, keeping the coordinator and worker roles fixed. First, we kept per-hop authentication and gave the coordinator an initially empty obligation ledger to track pending work, dependencies, owners, and evidence. Second, we added runtime support that authenticates each forwarded reference against its original producer. Both changes add support designed to improve handoffs, and the table reports each against the original Control.
| Core Control condition | Mean completion | Native successes |
|---|---|---|
| Original Control | .050 | 1/96 |
| Control + coordinator-maintained obligation ledger | .302 | 23/96 |
| Control + per-reference authentication | .549 | 43/96 |
Design recommendations and revision. The two tests point to concrete design choices: track downstream obligations through completion and keep each forwarded reference verifiable. Section 4.2 reports these tests alongside the handoff analysis, and Appendix A.1 summarizes the recommendations.
Revisions: Appendix B.3 documents the common authentication rule and three checks (L948–955). The paired Control tests are in Appendix B.5 (L1080–1098, Table 9), and the design implications are in §4.2 (L424–431, L459–467) and Appendix A.1 (L723–727). The abstract summarizes the per-reference authentication result (L27–29).
Q4. Independent agreement and blinding in the human audit.
In the original audit, the annotator scored items before seeing the automated labels. The new blind audit hid the original labels, native outcomes, and model and organization names.
Original audit and order of label disclosure. In the original audit, a coauthor independently scored completion atoms for 96 stratified runs before viewing the GPT-6-Astra labels, then compared the two sets of labels. Appendix B.4 now specifies that sequence.
New blind audit and hidden information. Two annotators independently labeled all 288 runs, with three distinct runs drawn from each of 96 strata (4 models × 4 task families × 6 organizations). They used task requirements, completion criteria, final state, and action logs, with GPT-6-Astra labels, native success results, and model and organization names hidden.
Agreement between the two annotators. The annotators agreed on every completion judgment in 259 of the 288 runs (89.9%). In the remaining 29 runs, every disagreement occurred when one annotator was unsure and the other gave a definite judgment. At the completion-item level, inter-annotator agreement was Cohen’s κ = 0.972. In Core, Silo, and Cooper, the original GPT-6-Astra labels fully matched the human consensus in 191/191 runs for which every completion item had both a definite human consensus and an original label.
Revisions: Appendix B.4 clarifies the order of the original audit and reports the 288-run blind audit protocol and agreement (L1008–1020).
We hope the repeated runs, targeted interventions, and new blind audit address your concerns. We would appreciate your reconsideration of the paper’s soundness and overall assessment.
Dear Reviewer aRbF,
Thank you again for the time and effort you dedicated to reviewing our paper, as well as your constructive feedback. Following your suggestions, we have provided comprehensive additional results and follow-up discussions in our response, and we would love to engage with you further.
As the author-reviewer discussion period is coming to a close soon, we wanted to gently check in and see if our responses and new additions have addressed your concerns. If any questions remain, we would be more than happy to clarify further. We truly appreciate your time and look forward to hearing your thoughts!
Best regards, The Authors
Response to Reviewer aRbF (1/2)
We thank the reviewer for their time and for recognizing the value of the cross-node cooperation setting, the breadth of the evaluation across four models and four task families, and the analysis tracing messages to lasting progress. We appreciate the reviewer’s specific suggestions for clarifying the comparison and testing the failure mechanisms.
The paper’s main argument.
CrossMAS asks how agents organize and complete a shared task without predefined assignments when the information, state, and action authority they need are held at different nodes. Section 3 formalizes local state and message passing, then introduces six organizations that vary in access, decision placement, and coordination support. We compare them on matched tasks across four task families. Section 4 first compares completion and cost, then uses execution traces to show how local contributions advance the task and where handoffs break down. The Discussion draws implications for system design. The revised introduction makes this progression explicit, and §4.2 presents the new tests alongside the corresponding trajectory findings.
Revisions: The revised introduction connects the research questions to the matched comparison and trajectory analysis (§1, L66–89). The setup, results, and design implications are presented in §3.1 (L116–151), §§4.1–4.2 (L265–269, L378–384, L424–431, L459–468), and Appendix A.1 (L723–730).
Follow-up tests of the organizational comparison.
What each follow-up test addresses. We ran three types of follow-up tests to clarify what the original comparisons measure and what improves task completion. We tested shared execution records and independent execution (Q1), repeated the full six-condition comparison twice (Q2), and tested targeted changes for handoff failures (here and in Q3).
Targeted test of obligation tracking. An obligation ledger improved Core P2P completion while decisions and tool use stayed local. In Core, P2P agents maintained an initially empty obligation ledger while continuing to decide and act with local tools. Across all 96 matched model–configuration pairs, tracking unfinished work raised mean completion from .158 to .568 (paired gain .410, 95% CI [.328, .483]) and native successes from 7 to 42, with completion gains for three of the four models.
Revisions: The P2P ledger test is reported in §4.2 (L461–468) and Appendix B.5 (L1080–1098, Table 9). The matched controls and full reruns are reported in Appendices B.3 (L961–977) and B.7 (L1216–1240, Table 14).
Q1. Local access and execution without communication.
Without communication, independent execution retains part of P2P’s completion advantage over Central in Cooper, and sharing execution records narrows the gap to Central in Core and Silo with decisions and tools kept local.
Independent execution in Cooper. We used Cooper for this control because individual branches can yield evaluable artifacts. Across 128 matched model–task pairs, independent execution retained about 61% of P2P’s completion advantage over Central and matched P2P on native successes. Thirteen independent runs did not reach native evaluation and were counted as failures.
Shared execution records with local decisions and tools. Sharing execution records each round narrowed the gap to Central while keeping P2P decisions and tools local. Central and P2P differ in decision ownership, tool access, and history visibility. We tested shared execution records while keeping P2P decisions and tools local. Providing the records each round closed 77% of the completion gap to Central in Core (53 matched pairs) and 64% in Silo (102). In the separate matched subsets with on-demand access, completion rose from .269 to .998 in Core (43 pairs) and fell from .141 to .069 in Silo (26).
Revisions: Appendix B.3 reports the shared execution-record controls and independent execution without communication in Cooper (L956–977).
Q2. Do the main results hold across repeated runs?
Merge and Central ranked first and second on both mean completion and native successes in every round. We repeated the submitted six-condition comparison twice on the same matched model–task pairs. Each of the three rounds comprised 2,880 runs (480 per organization). The relative order of the four multi-agent conditions varied across rounds and metrics.