acceptodds
Under review as a conference paper at ICLR 2027

CrossMAS: A Deep Dive into Multi-Agent Collaboration Across Distributed Environments

Abstract

Language-model agents can complete complex tasks with tools, and multi-agent systems demonstrate collaborative execution. These advances motivate delegating tasks spanning multiple systems and participants to agents. Many systems rely on human-designed roles and workflows, with studies focusing on overall performance. Yet dependencies and failures during execution may require handoffs beyond these plans. Greater autonomy thus requires agents to organize cooperation themselves, but how they do so with local access and no predefined task assignments remains poorly understood. CrossMAS studies task completion and autonomous cooperation when information, mutable state, and action authority are distributed across nodes. Across four task families and four models, we combine matched comparisons of six organizations with trajectory analysis. Across these tasks, autonomous MAS average 0.394–0.437 completion, below the single-decision references Merge (0.604) and Central (0.538); organizational support yields no consistent gain. Agents exchange contributions and act on them, but adopting incorrect contributions, confusing evidence sources or action targets, and leaving downstream dependencies unresolved prevent local progress from becoming joint completion.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM Agents, Multi-Agent Systems, Agent Collaboration, Distributed Environments, Agent Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aRbFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates if LLM agents can autonomously organize collaboration when information, state and action are distributed across nodes without predefined task decomposition. It compares six organizations (Merge, Central, P2P, Relay, Role and Control), and evaluates them across four models and four task families to see the performance gain obtained by different organizations. The paper also investigates if the communication contributions converted to persistent process and the failure mode through trace analysis.

Critical strengths
  1. The paper studies the MAS problem under a meaningful setting where agents must coordinate across separate information. This makes collaboration necessary for task completion.
  2. The experiments cover four models and four task families span different domains and organizations, and the trajectory analysis systematically studies if contributions of different organizations lead to lasting progress on performance.
Critical weaknesses
  1. The overall narrative is hard to follow and understand.
  2. The organization conditions change several factors at once, including information access, communication and how agents combine their work. Also each configuration is evaluated only once, and the proposed failure mechanisms are not tested through targeted interventions. It is unclear which factors drive the observed differences or whether fixing the identified failures would improve performance.
  3. The evaluation depends heavily on LLM-generated annotations with limited independent validation. The evaluation relies on GPT-6-Astra, with only 96 of 2,880 trajectories rescored by one author. Thus I think perfect agreement on a sample does not establish independent inter-annotator reliability.
Questions for authors
  1. Could the authors separate the effects of decision making structure from differences in information access, communication, and artifact integration? For example, in Cooper, some successful runs use only one agent’s final branch. How much of the MAS advantage remains if agents work independently without communicating?
  2. Please see weakness 2, where repeated runs should be provided.
  3. The authors have already identified a forwarding rule that rejects valid evidence in Control. Why was this not fixed before comparing Control with the other organizations?
  4. Did the human reviewer assign labels before seeing GPT-6’s annotations? What exactly was hidden during the blind review?
3 comments
AuthorsOpenReview ↗

Response to Reviewer aRbF (1/2)

We thank the reviewer for their time and for recognizing the value of the cross-node cooperation setting, the breadth of the evaluation across four models and four task families, and the analysis tracing messages to lasting progress. We appreciate the reviewer’s specific suggestions for clarifying the comparison and testing the failure mechanisms.

The paper’s main argument.

CrossMAS asks how agents organize and complete a shared task without predefined assignments when the information, state, and action authority they need are held at different nodes. Section 3 formalizes local state and message passing, then introduces six organizations that vary in access, decision placement, and coordination support. We compare them on matched tasks across four task families. Section 4 first compares completion and cost, then uses execution traces to show how local contributions advance the task and where handoffs break down. The Discussion draws implications for system design. The revised introduction makes this progression explicit, and §4.2 presents the new tests alongside the corresponding trajectory findings.

Revisions: The revised introduction connects the research questions to the matched comparison and trajectory analysis (§1, L66–89). The setup, results, and design implications are presented in §3.1 (L116–151), §§4.1–4.2 (L265–269, L378–384, L424–431, L459–468), and Appendix A.1 (L723–730).

Follow-up tests of the organizational comparison.

What each follow-up test addresses. We ran three types of follow-up tests to clarify what the original comparisons measure and what improves task completion. We tested shared execution records and independent execution (Q1), repeated the full six-condition comparison twice (Q2), and tested targeted changes for handoff failures (here and in Q3).

Targeted test of obligation tracking. An obligation ledger improved Core P2P completion while decisions and tool use stayed local. In Core, P2P agents maintained an initially empty obligation ledger while continuing to decide and act with local tools. Across all 96 matched model–configuration pairs, tracking unfinished work raised mean completion from .158 to .568 (paired gain .410, 95% CI [.328, .483]) and native successes from 7 to 42, with completion gains for three of the four models.

Revisions: The P2P ledger test is reported in §4.2 (L461–468) and Appendix B.5 (L1080–1098, Table 9). The matched controls and full reruns are reported in Appendices B.3 (L961–977) and B.7 (L1216–1240, Table 14).

Q1. Local access and execution without communication.

Without communication, independent execution retains part of P2P’s completion advantage over Central in Cooper, and sharing execution records narrows the gap to Central in Core and Silo with decisions and tools kept local.

Independent execution in Cooper. We used Cooper for this control because individual branches can yield evaluable artifacts. Across 128 matched model–task pairs, independent execution retained about 61% of P2P’s completion advantage over Central and matched P2P on native successes. Thirteen independent runs did not reach native evaluation and were counted as failures.

Cooper conditionMean completionNative successes
Central.32832/128
P2P.58662/128
Independent execution without communication.48462/128

Shared execution records with local decisions and tools. Sharing execution records each round narrowed the gap to Central while keeping P2P decisions and tools local. Central and P2P differ in decision ownership, tool access, and history visibility. We tested shared execution records while keeping P2P decisions and tools local. Providing the records each round closed 77% of the completion gap to Central in Core (53 matched pairs) and 64% in Silo (102). In the separate matched subsets with on-demand access, completion rose from .269 to .998 in Core (43 pairs) and fell from .141 to .069 in Silo (26).

Revisions: Appendix B.3 reports the shared execution-record controls and independent execution without communication in Cooper (L956–977).

Q2. Do the main results hold across repeated runs?

Merge and Central ranked first and second on both mean completion and native successes in every round. We repeated the submitted six-condition comparison twice on the same matched model–task pairs. Each of the three rounds comprised 2,880 runs (480 per organization). The relative order of the four multi-agent conditions varied across rounds and metrics.

Round (completion / native successes)MergeCentralP2PRelayRoleControl
Submitted.604 / 238.538 / 204.430 / 120.437 / 119.435 / 121.394 / 109
Rerun 1.597 / 215.539 / 193.420 / 111.432 / 110.433 / 118.391 / 106
Rerun 2.590 / 214.534 / 174.425 / 109.416 / 100.417 / 117.385 / 104
AuthorsOpenReview ↗

Response to Reviewer aRbF (2/2)

Q2 (continued)

Revisions: Appendix B.7 reports graded completion and native success across the submitted round and two full reruns (L1216–1240, Table 14).

Q3. Why did the main comparison keep Control’s original forwarding rule?

We kept per-hop authentication fixed across all six Core conditions so that the main comparison used a common protocol.

Common authentication protocol. Giving Control alone runtime support for authenticating references against their original producers would change the protocol for one condition. We held the per-hop authentication rule fixed across all six Core conditions.

Checks under the original rule. Control still trailed P2P when we excluded affected Core pairs and in Silo without the authentication rule.

Core matched comparison. Control still trailed P2P in Core after excluding the pairs with rejected references. We excluded the 21 pairs with reference rejections in Control. Across the remaining 75 pairs, Control completion was .075 below P2P.

Silo comparison. Control also trailed P2P in Silo, where this authentication rule does not apply. The scores were .210 for Control and .302 for P2P.

Core rejection records. Most Core rejections concerned references that had already reached the recipient. Of the 791 rejections, 332 were resends to nodes that already held the reference, 346 occurred after an earlier successful delivery, and 113 involved references with no prior observed delivery.

Two targeted changes to Control. Both the coordinator’s obligation ledger and per-reference authentication raised Core Control completion. Following your question, we tested two changes separately on the same 96 Core model–configuration pairs, keeping the coordinator and worker roles fixed. First, we kept per-hop authentication and gave the coordinator an initially empty obligation ledger to track pending work, dependencies, owners, and evidence. Second, we added runtime support that authenticates each forwarded reference against its original producer. Both changes add support designed to improve handoffs, and the table reports each against the original Control.

Core Control conditionMean completionNative successes
Original Control.0501/96
Control + coordinator-maintained obligation ledger.30223/96
Control + per-reference authentication.54943/96

Design recommendations and revision. The two tests point to concrete design choices: track downstream obligations through completion and keep each forwarded reference verifiable. Section 4.2 reports these tests alongside the handoff analysis, and Appendix A.1 summarizes the recommendations.

Revisions: Appendix B.3 documents the common authentication rule and three checks (L948–955). The paired Control tests are in Appendix B.5 (L1080–1098, Table 9), and the design implications are in §4.2 (L424–431, L459–467) and Appendix A.1 (L723–727). The abstract summarizes the per-reference authentication result (L27–29).

Q4. Independent agreement and blinding in the human audit.

In the original audit, the annotator scored items before seeing the automated labels. The new blind audit hid the original labels, native outcomes, and model and organization names.

Original audit and order of label disclosure. In the original audit, a coauthor independently scored completion atoms for 96 stratified runs before viewing the GPT-6-Astra labels, then compared the two sets of labels. Appendix B.4 now specifies that sequence.

New blind audit and hidden information. Two annotators independently labeled all 288 runs, with three distinct runs drawn from each of 96 strata (4 models × 4 task families × 6 organizations). They used task requirements, completion criteria, final state, and action logs, with GPT-6-Astra labels, native success results, and model and organization names hidden.

Agreement between the two annotators. The annotators agreed on every completion judgment in 259 of the 288 runs (89.9%). In the remaining 29 runs, every disagreement occurred when one annotator was unsure and the other gave a definite judgment. At the completion-item level, inter-annotator agreement was Cohen’s κ = 0.972. In Core, Silo, and Cooper, the original GPT-6-Astra labels fully matched the human consensus in 191/191 runs for which every completion item had both a definite human consensus and an original label.

Revisions: Appendix B.4 clarifies the order of the original audit and reports the 288-run blind audit protocol and agreement (L1008–1020).

We hope the repeated runs, targeted interventions, and new blind audit address your concerns. We would appreciate your reconsideration of the paper’s soundness and overall assessment.

Dear Reviewer aRbF,

Thank you again for the time and effort you dedicated to reviewing our paper, as well as your constructive feedback. Following your suggestions, we have provided comprehensive additional results and follow-up discussions in our response, and we would love to engage with you further.

As the author-reviewer discussion period is coming to a close soon, we wanted to gently check in and see if our responses and new additions have addressed your concerns. If any questions remain, we would be more than happy to clarify further. We truly appreciate your time and look forward to hearing your thoughts!

Best regards, The Authors

Reviewer_boh2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

CrossMAS investigates how large language model (LLM) agents collaborate when the information, mutable state, and action authority required for a shared task are distributed across multiple nodes, without predefined task-level work assignments. The paper introduces a distributed execution setting and evaluates six organizational configurations: Merge, Central, P2P, Relay, Role, and Control. The experiments cover four task families, four LLMs, 120 task configurations, and 2,880 execution trajectories, with matched comparisons across organizational conditions.

The main empirical finding is that single-decision configurations perform better overall than autonomous multi-agent configurations: Merge achieves a mean completion score of 0.604 and Central achieves 0.538, compared with 0.430 for P2P, 0.437 for Relay, 0.435 for Role, and 0.394 for Control. The paper further analyzes agent messages, subsequent decisions, task actions, persistent state changes, and evaluated outcomes to identify failures in contribution selection, provenance and target preservation, and downstream task closure. Its main contribution is an empirical framework for studying not only whether agents communicate, but also whether exchanged contributions lead to valid actions and jointly completed tasks.

Critical strengths
  1. A systematic, matched comparison of organizational configurations. The paper evaluates six organizational conditions across multiple models and task families while keeping the task-level objective, underlying operations, and evaluator fixed within each matched panel. This provides a useful empirical basis for understanding how decision centralization and coordination mechanisms relate to task completion, rather than relying on isolated demonstrations or comparisons between unrelated systems.

  2. Fine-grained analysis connecting communication to execution outcomes. The analysis traces messages through receiver decisions, task actions, persistent progress, and final evaluation. This is a meaningful strength because message counts and apparent agreement alone are insufficient measures of effective collaboration. The task-specific analyses of artifact integration, candidate aggregation, dependency execution, and information–authority binding offer concrete evidence about where collaboration succeeds or breaks down.

Critical weaknesses
  1. The organizational comparisons do not fully isolate the causal contribution of individual design choices. Although the matched setup controls several important task-level factors, the organizational conditions change multiple aspects of execution together, including access scope, decision history, communication interfaces, and work organization. Consequently, the observed differences cannot always be attributed to a specific mechanism, such as decentralization, role information, or coordinator-mediated communication. This limits the strength of the conclusions that can be drawn about which organizational design choices actually cause the observed performance differences.

  2. The operational interpretation of the process annotations needs stronger validation beyond agreement on the assigned labels. The paper uses GPT-6-Astra to annotate completion atoms and cooperation episodes, with human verification on stratified samples. The reported exact agreement is encouraging, but agreement with the same annotation schema does not by itself establish that the categories capture the causal or functional role of a contribution in subsequent execution. In particular, a contribution may correlate with a later action without being decisive for it, and an observed execution divergence does not necessarily identify the mechanism responsible for the outcome. This leaves some of the paper's finer-grained explanations less conclusive than its directly measured completion results.

Questions for authors
  1. Clarification—interpretation of organizational effects: Given that each organizational condition changes multiple interface and execution properties, how should readers distinguish the effect of decision centralization from the effects of access scope, communication structure, and task-specific implementation choices? Are there controlled comparisons or ablations in the existing experiments that support attributing the observed differences to any one of these factors?

  2. Clarification—validation of process-level claims: For the claims concerning contribution uptake, execution, persistent progress, and final retention, how often do the annotations agree with an independent assessment of whether a contribution was actually instrumental to the downstream outcome, rather than merely preceding or matching a later action? Could the authors provide evidence supporting the stronger causal interpretations of the audited trajectories?

  3. Clarification—robustness of aggregate conclusions: How sensitive is the ranking of the six organizational conditions to the aggregation scheme across the four task families and 120 configurations? In particular, do the main conclusions remain unchanged under family-balanced aggregation or when each task configuration, rather than each configuration within the pooled suite, is treated as the primary unit of comparison?

5 comments
AuthorsOpenReview ↗

Response to Reviewer boh2 (1/2)

We thank the reviewer for their time and for recognizing the value of the matched six-condition comparison and of tracing messages through receiver decisions, task actions, and persistent state. The reviewer’s questions about what these comparisons can attribute and what the process labels establish helped us design the additional controls and the message-removal replay below.

Q1 · Which organizational choices explain the differences?

Sharing execution records narrowed the gap to Central with decisions and tools kept local, and targeted handoff support improved Core completion.

What the original comparison holds fixed. The original comparison held the task, model, operations, and evaluation fixed. The original Central–P2P contrast measures the combined change in decision ownership, tool access, and history. Within this panel, we added one form of handoff support at a time to an existing organization, keeping all other settings fixed. These conditions are reported separately from the six-condition comparison.

Core handoff interventions. Preserving reference provenance and tracking unfinished obligations each improved Core completion in the matched comparisons.

Core intervention, 96 matched runs per conditionMean completionPaired change [95% CI]Native successes
Control with producer authentication attached to each forwarded reference.050 → .549+.498 [.438, .554]1 → 43
Control with a coordinator-maintained obligation ledger.050 → .302+.252 [.205, .301]1 → 23
P2P with a shared obligation ledger.158 → .568+.410 [.328, .483]7 → 42

The obligation ledger tracks pending work, dependencies, owners, and evidence. The Control ledger also improves completion under the original forwarding rule and worker tool permissions.

Shared execution history with local decisions. Providing shared execution records each round narrows the gap to Central without centralizing P2P decisions or tools. Providing the history each round closes 77% of the gap to Central in Core (53 pairs) and 64% in Silo (102 pairs). On-demand access exceeds Central in its Core subset (.998 vs. .577, 43 pairs) and lowers completion relative to P2P in Silo (.069 vs. .141, 26 pairs). The two history protocols ran on separate configuration subsets, each with its own matched baselines.

Independent execution in Cooper. In Cooper, mean completion is .484 with independent execution, .586 with P2P, and .328 with Central on the same 128 matched runs per condition. Independent execution and P2P both yield 62 native successes.

The intervals resample task configurations and keep all four models together. Section 3.1 defines the original six conditions, and Section 4.1 points to the separate handoff interventions.

Revisions: The matched organizational contrasts and local-access controls are described in §3.1 (L137–151) and Appendix B.3 (L956–977). The paired handoff interventions are reported in Appendix B.5 (L1080–1098, Table 9). The abstract summarizes the Core results on obligation tracking and reference authentication (L27–29).

AuthorsOpenReview ↗

Response to Reviewer boh2 (2/2)

Q2 · How do peer contributions affect downstream execution?

Removing peer messages often changed the next action in stable Core replays.

Message removal and the next action. We selected 100 Core decision points where the receiver’s recorded tool call followed a peer message containing a dependency reference. At each point, we replayed the decision twice with the recorded message retained and twice with only that message removed. We held all other inputs and the model endpoint fixed within each comparison. At 69 points, all four replays returned valid actions and both retained-message replays reproduced the recorded action. Removing the message changed the next action in both replays at 61 of these 69 points, in one replay at 4 points, and in neither replay at 4 points. At the remaining 31 points, the retained-message replays did not consistently reproduce the original action at 24 points, and action parsing or validation failed at 7.

Section 4.2 reports this next-action result.

Recorded actions and state changes. Core traces connect peer contributions to receiver actions and verifiable state changes. We observed this chain in 59 of 384 Core multi-agent runs.

Task completion under targeted handoff interventions. Tracking unfinished obligations and preserving reference provenance both raised Core completion. Tracking unfinished obligations raised Core P2P completion from .158 to .568, and preserving verifiable provenance for each forwarded reference raised Core Control completion from .050 to .549.

If our reading of Q2 differs from what you intended, we would welcome a specific test you have in mind and will run it during the discussion period.

Revisions: The Core trace links and message-removal replay are reported in §4.2 (L478–484) and Appendix B.5 (L1101–1109). The handoff interventions are reported in Appendix B.5 (L1080–1098, Table 9).

Q3 · Does the aggregate ranking depend on weighting?

Merge and Central stay first and second, and Control stays last, under both aggregation methods and across all three rounds.

Family-balanced means and configuration ranks. The same conditions lead and trail under equal family weights and mean configuration ranks.

Submitted bare setups, graded completionMergeCentralRelayRoleP2PControl
Pooled mean.604.538.437.435.430.394
Four families weighted equally.593.538.420.415.413.372
Mean rank across 120 configurations (lower is better)2.542.923.733.753.914.15

Configuration-level comparison and the Cooper result. Merge and Central outperform the MAS average on most configurations, while all four MAS conditions outperform Central in Cooper. For the rank, we average across the four models within each configuration and give ties their average rank. Comparing the mean of Merge and Central with the mean of the four MAS conditions within each configuration gives 81 wins, 5 ties, and 34 losses. In Cooper, all four MAS conditions score above Central.

Rank stability across three rounds. Across the submitted round and two full reruns of the original six conditions (2,880 runs per round), both family-balanced aggregation and mean rank across configurations give the same overall result: Merge and Central rank first and second, and Control ranks last.

Revisions: Appendix B.7 reports the three-round checks under equal family weights and mean configuration ranks, along with the submitted-round win–tie–loss and Cooper comparisons (L1255–1260).

We appreciate the opportunity to strengthen these claims and hope the interventions, replay, and aggregation checks address your concerns. We would welcome your reassessment of the paper’s soundness and overall score.

Dear Reviewer boh2,

Thank you again for the time and effort you dedicated to reviewing our paper, as well as your constructive feedback. Following your suggestions, we have provided comprehensive additional results and follow-up discussions in our response, and we would love to engage with you further.

As the author-reviewer discussion period is coming to a close soon, we wanted to gently check in and see if our responses and new additions have addressed your concerns. If any questions remain, we would be more than happy to clarify further. We truly appreciate your time and look forward to hearing your thoughts!

Best regards, The Authors

Reviewer_cozAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CrossMAS studies how agents work together when information, state, and permission to act are split across nodes. It compares six organizations on 120 task configurations using four models. Single-agent setups perform best overall, while extra coordination support brings no consistent improvement. The Control setup uses the most tokens on average and has the lowest overall completion. Analysis of about 23,000 receiver–decision episodes finds three common failures: accepting incorrect information, losing track of its source or target, and leaving dependent work unfinished.

Critical strengths
  1. The experiments are carefully controlled: each comparison uses the same task, model, underlying tools, and evaluator. Testing several models and task families makes the comparison useful.

  2. The three failure modes help explain why communication does not always lead to task completion. The paper is clearly written, and the appendix provides detailed supporting examples.

Critical weaknesses
  1. The practical takeaway is unclear. Central keeps node interfaces separate, but one agent can still use every node's tools and share one history. This makes it hard to separate the benefit of one decision maker from global access and context. The paper also identifies dependency failures without showing how agents should track unfinished work across handoffs or testing whether this helps.

  2. Evaluation relies partly on an LLM judge. Native evaluator results help, but one author independently re-scored only 96 of 2,880 outcomes (3.3%) and checked 240 cooperation episodes. The reported agreement is reassuring; a larger independent audit would strengthen confidence.

Questions for authors
  1. Which conclusions still apply when no single agent can access all nodes or act on their behalf?

  2. Do the organizational rankings hold using only native-evaluator binary success? How sensitive are the findings to annotation errors?

  3. What concrete design change should readers take from this study? In particular, is there evidence that explicitly tracking dependencies across handoffs improves completion?

3 comments
AuthorsOpenReview ↗

Response to Reviewer cozA (1/2)

We thank the reviewer for their time and for highlighting the clarity of the presentation, the care in the matched experimental design, the reassuring agreement in the original audit, and the value of the trajectory analysis. We appreciate the reviewer’s constructive suggestions and questions about what these findings mean for multi-agent system design.

Q1 · What holds under local access?

With tool access kept local, Relay and Role show no consistent advantage over P2P, while sharing execution records each round brings P2P closer to Central.

Relay and Role compared with P2P. Relay and Role showed no consistent gains over P2P when all four MAS conditions used node-local tool permissions. Across three rounds, the mean gain was +.003 for each, with both intervals spanning 0.

Shared execution records. Sharing execution records each round narrows the gap to Central while P2P nodes continue to decide and act locally. Central combines one decision-maker, access to every node's tools and a unified history. P2P agents shared a record of task execution, and each node continued to make its own decisions and use its local tools.

FamilyShared historyMatched pairsP2PP2P + shared historyCentralGap closed
Coreprovided each round53.067.391.48577%
Corequeried on demand43.269.998.577exceeds Central
Siloprovided each round102.343.542.65464%
Siloqueried on demand26.141.069.569none

The two variants were run on separate subsets determined by the run schedule and compared with their corresponding P2P and Central baselines. Providing execution history each round closed 77% of the gap to Central in Core and 64% in Silo. On-demand access improved completion in its Core subset and reduced it in its Silo subset. The revised comparison uses Central as a reference for the full organization and shows how sharing execution records can improve completion while preserving local decisions and tool permissions.

Preserving provenance and verifying results. Preserve the producer of each forwarded reference and verify candidates or artifacts before treating agreement as evidence of completion. Attaching producer authentication raised Core Control from .050 to .549 and native successes from 1 to 43 out of 96. In Cooper, 151 of 351 runs with converged or cleanly merged artifacts still failed the task evaluation.

Revisions: The shared-record comparison is reported in Appendix B.3 (L956–971), the Relay and Role reruns in Appendix B.7 (L1228–1232), and the provenance test and verification guidance in Appendices B.5 (L1080–1098, Table 9) and A.1 (L723–730). The abstract states the finding on organizational support (L24–25).

Q2 · Native scoring and annotation robustness.

Merge and Central remain the top two under native scoring and simulated annotation errors.

Native success across three rounds. Merge and Central rank first and second on native binary success in every round, including when Ent is excluded. We repeated the submitted six-condition comparison twice, with 2,880 runs in each round (480 per organization). We also recomputed the ranking excluding Ent, whose evaluator includes a judging component. The ordering among MAS conditions varies.

Native successesRuns per conditionMergeCentralP2PRelayRoleControl
Core, round 19629337721
Silo, round 1128817520211518
Cooper, round 1128603262686870
Ent, round 1128686431233620
All families, rounds 1 / 2 / 3480238 / 215 / 214204 / 193 / 174120 / 111 / 109119 / 110 / 100121 / 118 / 117109 / 106 / 104
Excluding Ent, rounds 1 / 2 / 3352170 / 158 / 150140 / 133 / 11789 / 81 / 8196 / 89 / 7785 / 83 / 7689 / 84 / 80
AuthorsOpenReview ↗

Response to Reviewer cozA (2/2)

Q2 (continued)

Independent human audit. The new two-annotator blind audit found strong human agreement and matched the original labels wherever a definite run-level comparison was possible in Core, Silo, and Cooper. In the original audit, a coauthor independently scored completion atoms in a stratified sample of 96 runs before viewing the GPT-6-Astra labels and comparing the two sets of labels.

Two annotators independently labeled all 288 runs, with three distinct runs drawn from each of 96 strata (4 models × 4 task families × 6 organizations). They used task requirements, completion criteria, final state, and action logs, with GPT-6-Astra labels, native success results, and model and organization names hidden. The annotators agreed on every completion judgment in 259 of the 288 runs (89.9%). In the remaining 29 runs, every disagreement occurred when one annotator was unsure and the other gave a definite judgment. At the completion-item level, inter-annotator agreement was Cohen’s κ = 0.972. In Core, Silo, and Cooper, the original GPT-6-Astra labels fully matched the human consensus in 191/191 runs for which every completion item had both a definite human consensus and an original label. The submitted analysis also included human verification of 240 cooperation episodes against the recorded messages, decisions and task state.

Sensitivity to annotation errors. The leading conditions and the lowest-ranked MAS condition were unchanged under simulated annotation errors. In all 500 simulations on the submitted first-round runs, Merge and Central remained the top two and Control ranked lowest among MAS conditions when we independently flipped Core, Silo and Cooper atom labels and Ent binary success labels with 10% probability and averaged the resulting run scores (atom pass fraction or Ent 0/1) over each organization’s 480 runs.

Revisions: Appendices B.4 (L1008–1020) and B.5 (L1058–1068, Table 8) document the completion and cooperation audits. Appendix B.7 reports native scoring and label-error sensitivity (L1228–1252, Tables 14–15, and L1262–1266).

Q3 · Designing handoffs.

Tracking unfinished obligations in a shared ledger improved Core P2P completion.

Keep unfinished obligations visible across handoffs. When a task spans nodes with separate permissions, keep unfinished work visible to the nodes that can carry it out. If an upstream result enables a downstream step, record that step, its prerequisites, the responsible node and the supporting evidence, and keep the item open until the downstream work is complete.

Dependency tracking experiment. Tracking unfinished dependencies with a shared ledger raised Core P2P completion. Following your suggestion, we gave P2P agents a shared obligation ledger that starts empty and is maintained by the agents. Each node still decides and acts with its local tools. Across all 24 Core configurations and four models (96 matched runs per condition), the ledger raised mean completion by .410 (95% CI [.328, .483]).

Core conditionMean completionNative successes
P2P.1587/96
P2P + obligation ledger.56842/96

With the ledger, distributed agents averaged .568 completion, compared with .526 for Central (33/96 native successes). Gains hold for three of four models: GPT-5.6-Sol .245 → 1.000, Kimi K3 .373 → .791 and DeepSeek V4 Flash .002 → .479, while Qwen Plus stays near zero (.010 → .002). The experiment gives a concrete design rule for Core handoffs. Keep each obligation open until its downstream operation is confirmed, and carry the evidence and target forward with it. Section 4.2 reports the experiment alongside the closure analysis, and the discussion in Appendix A.1 draws out the design implication.

Revisions: The Core dependency-tracking test is reported in §4.2 (L461–468) and Appendix B.5 (L1080–1098, Table 9), and its design implication is in Appendix A.1 (L723–727). The abstract summarizes the dependency-tracking result (L27–29).

We hope the handoff experiment and the shared-history control clarify the practical takeaway, and that the native-scoring analysis and expanded human review address the robustness concerns. We would appreciate your reconsideration of the score.

Dear Reviewer cozA,

Thank you again for the time and effort you dedicated to reviewing our paper, as well as your constructive feedback. Following your suggestions, we have provided comprehensive additional results and follow-up discussions in our response, and we would love to engage with you further.

As the author-reviewer discussion period is coming to a close soon, we wanted to gently check in and see if our responses and new additions have addressed your concerns. If any questions remain, we would be more than happy to clarify further. We truly appreciate your time and look forward to hearing your thoughts!

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

Dear Area Chair and Reviewers,

We thank the Area Chair and reviewers for their time and constructive feedback, and for highlighting the value of the matched organizational comparisons and trajectory analysis. Your questions guided the follow-up comparisons, replay experiment, and independent audit reported in our individual responses.

We report the following work in our rebuttal and revised manuscript.

  • We tested how agents coordinate across nodes through matched comparisons and message replay. The experiments show how sharing execution records affects task completion and how peer messages affect the receiving agent’s next action.
  • We tested specific ways to improve handoffs through targeted interventions. The results inform the design of multi-agent systems that track unfinished obligations and pass along verifiable evidence.
  • We checked our main findings through full reruns and an independent blind human audit. The leading conditions remained the same across reruns and robustness checks, and the audit supported the reliability of the completion annotations.

The table below summarizes our responses to each concern and the corresponding manuscript revisions.

ReviewersConcernResponse and manuscript revision
cozA, boh2, aRbFWhat the organizational comparisons measureWe clarified what the contrasts measure and added matched controls. Central–P2P jointly varies decision ownership, tool access, and history. Sharing execution records each round closed 77% of the gap to Central in Core and 64% in Silo, with P2P decisions and tools local. In Cooper, independent execution without communication retained about 61% of P2P’s completion advantage over Central.<br>Revisions: Appendix B.3 reports the original contrast and shared records and independent-execution controls (L956–977); the abstract states the organizational-support finding (L24–25).
cozA, boh2, aRbFDependency tracking and evidence forwardingWe tested targeted changes to cross-node handoffs. In matched Core tests, a shared obligation ledger raised P2P completion from .158 to .568; authenticating each forwarded reference against its producer raised Control from .050 to .549. All six original Core conditions used the same per-hop rule. Under that rule, a coordinator-maintained ledger separately raised Control completion from .050 to .302.<br>Revisions: The common authentication rule appears in Appendix B.3 (L948–955), the paired interventions in Appendix B.5 (L1080–1098, Table 9), and design implications in Appendix A.1 (L723–730). The abstract summarizes Core results on obligation tracking and reference authentication (L27–29).
boh2Whether peer messages affect subsequent actionsWe replayed Core decisions with peer messages retained or removed. Among the 69 points with valid replays and successful reproduction of the recorded action when retaining the message, removal changed the next action in both repeats at 61 points. Traces also link peer messages to receiver actions and verifiable state changes.<br>Revisions: The Core trace evidence and message-removal replay are reported in §4.2 (L478–484) and Appendix B.5 (L1101–1109).
cozA, boh2, aRbFReproducibility, scoring, and aggregationWe reran the full six-organization comparison twice. Merge and Central remained first and second across all three rounds by mean completion and native binary success. The same configurations led under equal family weights and mean configuration ranks. They remained top two under simulated label errors in the submitted round.<br>Revisions: Appendix B.7 reports reruns and native scoring (L1216–1252, Tables 14–15), plus alternative aggregation and label-error sensitivity (L1255–1266).
cozA, aRbFIndependence of the completion auditWe added an independent two-annotator blind audit of 288 runs. Completion-item agreement was Cohen’s κ = 0.972. Original GPT-6-Astra labels matched human consensus in all 191 comparable Core, Silo, and Cooper runs. We clarified the original audit’s labeling and comparison order.<br>Revisions: Appendix B.4 reports the original audit sequence and independent blind audit (L1008–1020).
aRbFHow the study’s parts fit togetherWe clarified how the six-organization comparison leads to trace analysis and design recommendations. The matched comparison establishes completion and cost differences. Trajectory analysis examines how local contributions advance the task and where handoffs break down.<br>Revisions: The study questions and setup appear in §§1 and 3.1 (L66–89, L116–151); the trace transition in §§4.1–4.2 (L265–269, L373–374, L378–384); and handoff guidance in Appendix A.1 (L723–730).

Our individual responses detail experimental settings, numerical results, and section and line references for the manuscript revisions. We thank you for your time and welcome further discussion.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.