acceptodds
Under review as a conference paper at ICLR 2027

Worse Together: How Performance Breaks Down in Multi-User Multi-Agent Teams

Abstract

People are increasingly delegating tasks to AI agents, and those agents are increasingly encountering other people's agents over shared resources such as a codebase, a calendar, or a budget. When each agent acts for a different user with different goals, coordination often fails, and the group ends up worse off than if a single agent had acted for everyone. We study this multi-user, multi-agent setting across five frontier models and 77 scenarios in four environments: an API key environment in which agents share a compute budget, a clinic in which they share a calendar, a personal assistant environment in which they share a group order or booking, and a merge queue in which they share a release cutoff. In each scenario, we compare a single agent that serves every user (a coordinator) to a team in which each agent serves one user, with and without a communication channel between the agents. Teams deliver worse group outcomes than the coordinator in every environment: without a channel, they completely collapse in two environments, and even with one, coordination overhead creates substantial gaps. For example, in the personal assistant environment, the coordinator fulfills a targeted user request about twice as often as teams. We identify distinct behaviors associated with this poor group-level performance, ranging from stalling as teams grow to overriding each other's actions and fabricating claims. We find effective but environment-specific mitigations, such as a team lead, explicit procedural instructions, and a platform check that makes an agent read its peers' messages before committing. We release the API key, clinic, and personal assistant environments as MAMUBench, comprising 74 scenarios for evaluating multi-user, multi-agent coordination.

Primary area:
datasets and benchmarks
Keywords:
multi-agent, multi-user, language models, agent coordination, agent behavior, evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KyRpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies coordination failure in multi-user, multi-agent systems. The authors compare four formations: solo, coordinator, silent team, and peer-to-peer team, and evaluate them in four environments (API key, merge queue, clinic, and personal assistant). The authors find that the peer-to-peer team often performs worse than the coordinator, and in some models in environments such as clinic even performs worse than the silent team; failure modes include participation decline, wasted share (tokens spent on never-finished jobs), unconsented interference, premature checkout, overrides, fabrications, etc. The authors also test mitigations such as team lead, charter, checkout guard, and owner acknowledgement. The overall idea is clear, the logic is explicit, and the problem is clear and important. However, the scale and number of scenarios in the benchmark and the study are inconsistent, and multiple failure modes coexist under the same formation, while failure modes also differ across environments, making it difficult to directly attribute the outcome gap to a single mechanism.

Critical strengths
  1. The problem is clearly defined and important. Multi-user, multi-agent shared-resource settings are real scenarios being rapidly deployed, while existing benchmarks mostly assume a shared goal or a single user. This paper takes “different users, competing resources, and no shared goal” as its core setting, which is of significant importance.
  2. The environments and scenarios are rich, and the failure-mode analysis is comprehensive. The four environments cover different resource types such as divisible tokens, time-sliced runners, discrete slots, and money/cart, as well as different time pressures, scoring methods, communication mechanisms, and safety stakes; failure modes range from participation, prioritization, consent, information propagation, and follow-through to truthfulness, covering a wide scope.
  3. The design of the four formations is reasonable. It forms a 2×2 contrast of “information aggregation (centralized vs. decentralized) × communication capability (present vs. absent).” 4) The failure-mode analysis has practical value. Participation decline, unconsented interference, premature checkout, fabrication, etc. are not abstract concepts but behavior patterns supported by concrete logs and counts, and are important.
Critical weaknesses
  1. The number of scenarios in the benchmark and the study is inconsistent, and the positioning is not clear enough. Why not frame it initially as the benchmark’s 74/3, and regard the 3 scenarios in the merge queue environment as an extension?
  2. The scenarios are constructed by an LLM pipeline varying jobs, priorities, team size, budget, etc., but the paper does not report the proportion of manual review, the distribution of scenario difficulty, or whether some scenarios dominate the average. More importantly, some design choices may induce failure modes. For example, scattered facts in clinic are artificially created information fragmentation; DM-as-user-message in PA is a platform design choice. The paper needs to discuss how these design choices affect the external validity of the failure modes.
  3. Failure modes are exploratory, not pre-registered. The paper first compares formation outcomes, then induces "which behaviors are associated with low scores" through transcripts, action logs, tool calls, and DM logs. This is reasonable exploratory analysis, but the taxonomy was not defined before the experiment, and the paper does not report "which behaviors were examined and which were excluded", posing risks of researcher degrees of freedom and confirmation bias.
  4. Some failure modes rely on LLM judges and have not undergone human validation. Consent (API key) and fabrication (personal assistant) are behavior claims in the paper, but the judges were not aligned with human judgment.
  5. Outcome failure and mechanism failure lack a unified taxonomy. "Patients dropped" and "Asks dropped or overridden" are outcomes; "participation decline" and "premature checkout" are mechanisms. The paper does not clearly distinguish these two levels.
  6. The coordinator naturally has all user requests and global information, while the peer-to-peer team only has local information. This comparison can show that "centralized information aggregation + a single decision-maker" is better in terms of group outcome, but it cannot separate "information advantage" from "decision-structure advantage".
Questions for authors
  1. The number of scenarios in the benchmark and the study is inconsistent, and the positioning is not clear enough. Why not frame it initially as the benchmark’s 74/3, and regard the 3 scenarios in the merge queue environment as an extension?
  2. The scenarios are constructed by an LLM pipeline, but the paper does not report the manual review.
  3. The coordinator naturally has all user requests and global information, while the peer-to-peer team only has local information. This comparison can show that "centralized information aggregation + a single decision-maker" is better in terms of group outcome, but it cannot separate "information advantage" from "decision-structure advantage".
1 comment

Thank you for a detailed and constructive review.

[W1, Q1] […] Why not frame it initially as the benchmark’s 74/3, and regard the 3 scenarios in the merge queue environment as an extension?

[W1, Q1] Scenario counts and positioning

Adopted. The abstract and introduction now present 74 scenarios in three environments, with the merge queue as a three-scenario extension of the API key environment. Later sections refer to all 77 scenarios in four environments.

[W2, first part; Q2] […] the paper does not report the proportion of manual review, the distribution of scenario difficulty, or whether some scenarios dominate the average.

[W2, first part; Q2] Manual review, scenario difficulty, whether some scenarios dominate

New Appendix F. It describes each authoring pipeline and its checks, and reports a hand review made for this revision: one author read a random sample of 23 of the 77 scenarios, drawn within each environment (7 of 25 API key, 6 of 24 clinic, 7 of 25 personal assistant, all 3 merge queue), against four fixed questions and judged all 23 sound. Table 15 gives the minimum, median and maximum per-scenario score for each model's coordinator and peer-to-peer team, the number of scenarios the coordinator wins (for example 22 of 22 and 18–22 of 25 in the API key environment, 22–24 of 25 in the personal assistant), and the gap with each scenario left out in turn and with the three largest-gap scenarios removed. It also states where the gap is not robust: four-user API key scenarios for models other than Opus 5, and assembled clinic scenarios for three models.

[W2, second part] […] some design choices may induce failure modes. […]

[W2, second part] Design choices that may induce failure modes

  • Scattered facts. Yes, and it is by design: scattering a patient's facts across calls is what tests coordination. Every scattered scenario has an assembled control, and the paper reports both.
  • Message delivery is a platform choice. Section 6 now describes it where the behavior is reported: a message is delivered when the recipient's current run ends, the queueing mode that coding agents such as Codex offer alongside steering (cited). We did not test steering-mode delivery. The guard result shows the choice matters: a platform check recovers 73% of failed Opus 5 episodes.

[W3] Failure modes are exploratory, not pre-registered. […]

[W3] Failure modes are exploratory

Correct. The setup section describes the behavior analysis as exploratory, and Appendix E lists the behaviors we measured and how each was measured (Table 14). The mitigations then test the leading explanations by intervention (Section 7).

[W4] Some failure modes rely on LLM judges and have not undergone human validation. […]

[W4] LLM judges without human validation

Each behavior is now labeled by two LLM judges, and the main text counts only the cases on which both agree. For consent, GPT-6.1-terra confirms 93% of the 20,561 cuts that Claude Sonnet 5 counted; for fabrication, Claude Opus 5.5 confirms 86% of the 5,413 claims that Claude Opus 5 counted (these two are from one model family). Counting agreed cases only, cuts without the owner's prior agreement by GPT and Qwen agents go from 8.9–16.4 to 8.5–15.0 per episode, and fabrication from 58% and 53% of Opus 5 and Sonnet 5 team episodes to 51% and 47%. The revision reports cuts made without the owner's prior agreement in place of unconsented cuts, because judges agreed less on whether the owner was told afterwards. One author also spot-checked a small sample of the labels by hand and found them correct; we did not collect systematic human labels. Neither behavior enters an outcome score, and fabrication is reported as prevalence only.

[W5] Outcome failure and mechanism failure lack a unified taxonomy. […] [W6, Q3] […] it cannot separate "information advantage" from "decision-structure advantage".

[W5; W6, Q3] Outcomes against mechanisms; information advantage against decision structure

We agree the two levels were mixed. The behavior analysis is exploratory, and Section 7 with Table 1 addresses the second point. Outcomes are the mechanical scores in each environment, the behaviors are mechanisms found by exploratory analysis (listed in Appendix Table 14), and the mitigations are the interventions that test them. Read as a mediation analysis, they separate the coordinator's two advantages, with each arm's caveats given in Appendix E. The shared-information arm, which we ran in the three benchmark environments for this revision, closes the gap in the personal assistant and for Sonnet 5 in the clinic, and all or most of it for Opus 5 in the API key environment. A single decision-maker without the information closes it only in the API key environment, at four and eight users.

Reviewer_4dTqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a case study of how multi-agent, multi-user systems can break down and be less performant than single agent systems which serve all users. It derives 4 different test environments: API Key management, merge queue, clinic, and personal assistant, in order to study how these different agentic architectures for multi-user, agentic systems behave. They find that most of the time these systems are less performant than single agent systems which serve everything, and introduce several environment-specific mitigations to each of the problems.

Critical strengths

The paper has a lot of documentation on the results, and is very thorough in testing each and every single model and architecture on each environment, and the environments themselves seem well suited for evaluating these distributed system phenomena.

Critical weaknesses

First weakness is that many of the insights and mitigations given in the paper are very well known problems in distributed computing and peer-to-peer communication. Many of these issues are very obvious when you look at the nature of heterogeneous distributed systems and none of these phenomena are very surprising. Furthermore, the mitigations are environment specific, and there really isn't anything tying up the paper to be one cohesive study on multi-agent systems. The paper is really just 3/4 small case studies stapled together back to back with no unifying analysis or taxonomy presented in the discussion.

Second weakness is the complete lack of discussion on the environment / benchmark presented in the paper. The experimental setup and following sections are extremely scant on any details on the dataset or how it was constructed, what distribution the data is in, etc. If the paper is to make the point that this environment artifact is valuable to the research community, then more discussion on the construction and properties of the dataset are required in the main paper.

Questions for authors

The mitigations for each of the environments is environment specific. Are the multi-agent architectures implemented in a cross-environment manner, or is it also environment specific? For instance, in the merge queue environment, it says that cross agent communication is implemented through PR comments, while in the clinic environment, communication is facilitated through patient notes, a special tool and the calendar.

Reviewer_dhzPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the idea is that each user is going to get their own agent. Each agent has to suggest one thing: the token budget, the runner calendar, and a good order. The paper also compares one agent serving everybody against one agent per user in a setting with or without the channel between them. Around 70 scenarios and 5 models are evaluated.

As a result, the observations are that the teams lose in all four settings. In the shared budget setting, where the token budget is shared by the teams, they reach 30% of the best possible outcome, compared to 64% for the single agent. In the group order setting, the single agent completes more user requests.

Critical strengths
  • Clear baseline with the single-agent setup
  • Shared benchmark environments
  • Transparent reporting of limitations and costs
  • Well-controlled test on information delivery
Critical weaknesses
  • Unfair information access favoring the single agent
  • Structural timing bias (team agents face real-time deadlines)
  • Important experimental setup differences buried in the appendix
Questions for authors
  • Can team agents be given requests upfront to ensure a fair test?
  • Was the LLM judge validated with human checks?
1 comment

Thank you for the review.

[W1] Unfair information access favoring the single agent

[Q1] Can team agents be given requests upfront to ensure a fair test?

[W1, Q1] Information access; giving team agents the requests upfront

Yes. For this revision we gave team agents the coordinator's information in the three benchmark environments (Section 7, Table 1, Appendix E).

  • Personal assistant. Every agent is given every user's note. The team then keeps the request in 91% (Opus 5) and 77% (Sonnet 5) of episodes, against 88% and 75% for the coordinator and 45% and 37% for a plain team run alongside.
  • API key. Every agent is given every other user's request. The Opus 5 team reaches 64%, 75% and 55% of the optimum at 4, 8 and 16 users (plain team 39%, 19%, 27%; coordinator 64%, 66%, 61%); the Sonnet 5 team reaches the coordinator's level only at four users. The gain comes from one informed agent rescaling the other users' jobs itself, so the arm shows an agent taking the coordinator's role, not peers coordinating.
  • Clinic. Every agent is shown the calls the coordinator has heard. The Sonnet 5 team then meets 86% of urgent deadlines against 69% (with scattered facts, 81% against 49%), above its coordinator's 70%. For Opus 5 the same information changes nothing (72% against 74%).

So information accounts for the gap in the personal assistant, for Sonnet 5 in the clinic, and for all or most of it for Opus 5 in the API key environment. We did not run this arm in the merge queue extension. Conversely, in the API key environment a lead that is given no user's request reaches the coordinator's level at four and eight users; its brief also tells it to put the budget where it does the most good (Appendix E).

[W2] Structural timing bias (team agents face real-time deadlines)

[W2] Timing

Both formations face the same deadline in every environment, and each environment is built to remove timing confounds (new Appendix F).

In the API key environment messages and reads cost no simulated time, so only the actions a coordinator would also take cost a team time. In the clinic every deadline is at the end of the day, so hearing a call late costs no deadline.

In the personal assistant environment both formations have the same ordering window, and deadlines are never hit.

Two confounds remain and the revision reports both: a short wrap-up in the clinic, worth about 4 and 3 points of urgent deadlines met for Opus 5 and Sonnet 5 (Appendix G.2), and turn order in the API key environment, where at sixteen users 94–98% of the agents that never act get no turn before the budget is spent (Section 4).

[W3] Important experimental setup differences buried in the appendix

[W3] Setup differences in the appendix

The setup differences are in one table that covers all four environments (Appendix G.1). The setup section points to it, to Appendix F for how each environment is built to avoid timing confounds, and to Appendix G.2 for the confound that remains in the clinic. With new Section 7, the table does not fit in the ten-page main text.

[Q2] Was the LLM judge validated with human checks?

[Q2] Validation of the LLM judge

Each behavior is now labeled by two LLM judges, and the main text counts only the cases on which both agree, and we spot checked a few samples to ensure soundness.

Comments on the paper

1 comment

We thank the reviewers for their careful and constructive reviews. Three concerns were shared:

  1. Is the coordinator's advantage information or decision structure? (dhzP, KyRp) New Section 7 of the main text reads the mitigations as a mediation analysis that separates the two (Table 1; arms in Appendix E). For this revision we ran an information arm, with no lead, in the three benchmark environments; it gives each agent the information the coordinator had. Lack of information is the limiting factor for both Claude models in the personal assistant environment. In the API key environment the results are mixed: information closes all or most of the gap for Opus 5, but for Sonnet 5 only at four users. In the clinic it lifts the Sonnet 5 team to the coordinator's level but not the Opus 5 team. Whether missing information explains the gap therefore depends on the environment and the model.

  2. How the benchmark was built (4dTq, KyRp) New Appendix F gives each environment's authoring pipeline and validity checks, a hand review by one author of a random sample of 23 scenarios, the spread of per-scenario scores, and a leave-one-scenario-out analysis.

  3. Validation of the LLM judges (dhzP, KyRp) Consent and fabrication are now each labeled by two LLM judges, and the main text counts only the cases on which both agree. The second judges confirm 93% of the consent cases and 86% of the fabrication cases; counting agreed cases only, cuts of a peer's job made without the owner's prior agreement go from 8.9–16.4 to 8.5–15.0 per episode for GPT and Qwen agents, and fabrication from 58% and 53% of Opus 5 and Sonnet 5 team episodes to 51% and 47%. One author also spot-checked a small sample of the labels by hand and found them correct.

We also adopt KyRp's framing: the abstract and introduction now present 74 scenarios in three environments, with the merge queue as a three-scenario extension of the API key environment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.