- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies coordination failure in multi-user, multi-agent systems. The authors compare four formations: solo, coordinator, silent team, and peer-to-peer team, and evaluate them in four environments (API key, merge queue, clinic, and personal assistant). The authors find that the peer-to-peer team often performs worse than the coordinator, and in some models in environments such as clinic even performs worse than the silent team; failure modes include participation decline, wasted share (tokens spent on never-finished jobs), unconsented interference, premature checkout, overrides, fabrications, etc. The authors also test mitigations such as team lead, charter, checkout guard, and owner acknowledgement. The overall idea is clear, the logic is explicit, and the problem is clear and important. However, the scale and number of scenarios in the benchmark and the study are inconsistent, and multiple failure modes coexist under the same formation, while failure modes also differ across environments, making it difficult to directly attribute the outcome gap to a single mechanism.
Critical strengths
- The problem is clearly defined and important. Multi-user, multi-agent shared-resource settings are real scenarios being rapidly deployed, while existing benchmarks mostly assume a shared goal or a single user. This paper takes “different users, competing resources, and no shared goal” as its core setting, which is of significant importance.
- The environments and scenarios are rich, and the failure-mode analysis is comprehensive. The four environments cover different resource types such as divisible tokens, time-sliced runners, discrete slots, and money/cart, as well as different time pressures, scoring methods, communication mechanisms, and safety stakes; failure modes range from participation, prioritization, consent, information propagation, and follow-through to truthfulness, covering a wide scope.
- The design of the four formations is reasonable. It forms a 2×2 contrast of “information aggregation (centralized vs. decentralized) × communication capability (present vs. absent).” 4) The failure-mode analysis has practical value. Participation decline, unconsented interference, premature checkout, fabrication, etc. are not abstract concepts but behavior patterns supported by concrete logs and counts, and are important.
Critical weaknesses
- The number of scenarios in the benchmark and the study is inconsistent, and the positioning is not clear enough. Why not frame it initially as the benchmark’s 74/3, and regard the 3 scenarios in the merge queue environment as an extension?
- The scenarios are constructed by an LLM pipeline varying jobs, priorities, team size, budget, etc., but the paper does not report the proportion of manual review, the distribution of scenario difficulty, or whether some scenarios dominate the average. More importantly, some design choices may induce failure modes. For example, scattered facts in clinic are artificially created information fragmentation; DM-as-user-message in PA is a platform design choice. The paper needs to discuss how these design choices affect the external validity of the failure modes.
- Failure modes are exploratory, not pre-registered. The paper first compares formation outcomes, then induces "which behaviors are associated with low scores" through transcripts, action logs, tool calls, and DM logs. This is reasonable exploratory analysis, but the taxonomy was not defined before the experiment, and the paper does not report "which behaviors were examined and which were excluded", posing risks of researcher degrees of freedom and confirmation bias.
- Some failure modes rely on LLM judges and have not undergone human validation. Consent (API key) and fabrication (personal assistant) are behavior claims in the paper, but the judges were not aligned with human judgment.
- Outcome failure and mechanism failure lack a unified taxonomy. "Patients dropped" and "Asks dropped or overridden" are outcomes; "participation decline" and "premature checkout" are mechanisms. The paper does not clearly distinguish these two levels.
- The coordinator naturally has all user requests and global information, while the peer-to-peer team only has local information. This comparison can show that "centralized information aggregation + a single decision-maker" is better in terms of group outcome, but it cannot separate "information advantage" from "decision-structure advantage".
Questions for authors
- The number of scenarios in the benchmark and the study is inconsistent, and the positioning is not clear enough. Why not frame it initially as the benchmark’s 74/3, and regard the 3 scenarios in the merge queue environment as an extension?
- The scenarios are constructed by an LLM pipeline, but the paper does not report the manual review.
- The coordinator naturally has all user requests and global information, while the peer-to-peer team only has local information. This comparison can show that "centralized information aggregation + a single decision-maker" is better in terms of group outcome, but it cannot separate "information advantage" from "decision-structure advantage".
Thank you for a detailed and constructive review.
[W1, Q1] Scenario counts and positioning
Adopted. The abstract and introduction now present 74 scenarios in three environments, with the merge queue as a three-scenario extension of the API key environment. Later sections refer to all 77 scenarios in four environments.
[W2, first part; Q2] Manual review, scenario difficulty, whether some scenarios dominate
New Appendix F. It describes each authoring pipeline and its checks, and reports a hand review made for this revision: one author read a random sample of 23 of the 77 scenarios, drawn within each environment (7 of 25 API key, 6 of 24 clinic, 7 of 25 personal assistant, all 3 merge queue), against four fixed questions and judged all 23 sound. Table 15 gives the minimum, median and maximum per-scenario score for each model's coordinator and peer-to-peer team, the number of scenarios the coordinator wins (for example 22 of 22 and 18–22 of 25 in the API key environment, 22–24 of 25 in the personal assistant), and the gap with each scenario left out in turn and with the three largest-gap scenarios removed. It also states where the gap is not robust: four-user API key scenarios for models other than Opus 5, and assembled clinic scenarios for three models.
[W2, second part] Design choices that may induce failure modes
[W3] Failure modes are exploratory
Correct. The setup section describes the behavior analysis as exploratory, and Appendix E lists the behaviors we measured and how each was measured (Table 14). The mitigations then test the leading explanations by intervention (Section 7).
[W4] LLM judges without human validation
Each behavior is now labeled by two LLM judges, and the main text counts only the cases on which both agree. For consent, GPT-6.1-terra confirms 93% of the 20,561 cuts that Claude Sonnet 5 counted; for fabrication, Claude Opus 5.5 confirms 86% of the 5,413 claims that Claude Opus 5 counted (these two are from one model family). Counting agreed cases only, cuts without the owner's prior agreement by GPT and Qwen agents go from 8.9–16.4 to 8.5–15.0 per episode, and fabrication from 58% and 53% of Opus 5 and Sonnet 5 team episodes to 51% and 47%. The revision reports cuts made without the owner's prior agreement in place of unconsented cuts, because judges agreed less on whether the owner was told afterwards. One author also spot-checked a small sample of the labels by hand and found them correct; we did not collect systematic human labels. Neither behavior enters an outcome score, and fabrication is reported as prevalence only.
[W5; W6, Q3] Outcomes against mechanisms; information advantage against decision structure
We agree the two levels were mixed. The behavior analysis is exploratory, and Section 7 with Table 1 addresses the second point. Outcomes are the mechanical scores in each environment, the behaviors are mechanisms found by exploratory analysis (listed in Appendix Table 14), and the mitigations are the interventions that test them. Read as a mediation analysis, they separate the coordinator's two advantages, with each arm's caveats given in Appendix E. The shared-information arm, which we ran in the three benchmark environments for this revision, closes the gap in the personal assistant and for Sonnet 5 in the clinic, and all or most of it for Opus 5 in the API key environment. A single decision-maker without the information closes it only in the API key environment, at four and eight users.