acceptodds
Under review as a conference paper at ICLR 2027

Does Multi-Agent Collaboration Pay When Agents Serve Users? Testing Agent Scaling in Dual-Control Settings

Abstract

Multi-agent systems (MAS) built from language models are increasingly deployed on the premise that collaboration among agents outperforms a single-agent system (SAS), yet when collaboration pays remains poorly understood. The most comprehensive study to date answers with a capability-saturation principle: coordination helps while the underlying SAS is weak and hurts once it is strong, with the crossing near 45% single-agent accuracy. That principle was derived on autonomous benchmarks, where no user takes part. We test it where agents increasingly operate, in dual-control settings, where the agent serves a user under a written policy and both act on a shared environment. On the airline, retail and telecom domains of -bench we compare a SAS with a centralized MAS built from the same model (an orchestrator, a database agent and a policy advisor) for six models from three families (four, from two families, in the primary analysis), over about 7,000 simulated conversations, with each system held to one completion-token ledger per episode. Under a fixed simulated user no cell shows a MAS benefit: the MAS is worse in 12 of 13 valid cells and tied in the thirteenth, a cell near 45% where the benchmark’s default simulator yields a 10-point MAS gain. Where the principle predicts a benefit, the MAS loses by 13 and 22 points, and removing the budget ceiling brings it at best level with the SAS at 5.5 times its median token spend. A capability index measured on held-out domains shows no detectable dependence of the gain on capability (slope , 95% CI ), although this interval is too wide to exclude a steep decline; the fitted gain is negative across the observed range, with an interval that excludes zero only from an index of 60% upward. The mechanism that would justify a MAS here is not detectable: the MAS shows no measurable reduction in wrongful database writes and converts other failures into budget exhaustion and inaction. For agents that serve people, these results make the SAS the default and place the burden of proof on multi-agent designs. We release all trajectories, budget ledgers, prompts and code.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent systems, LLM agents, agent architecture selection, preregistration, conversational agents, tool use, compute-matched evaluation, agent benchmarks, tau2-bench

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_bruqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether the benefits of Multi-Agent Systems (MAS) over Single-Agent Systems (SAS) also apply to dual-control settings, where agents need to interact with users throughout a task. The authors evaluate several models on the Airline, Retail, and Telecom domains of τ2\tau^2-bench, with around 7,000 simulated conversations and a shared completion-token budget for MAS. The results show that MAS generally performs worse than SAS under a fixed user simulator, even when the single agent is relatively weak. The paper also analyzes how model capability, token usage, and different types of failures affect MAS performance.

Critical strengths

S1. An important practical problem with good experiments.

I think the paper addresses an important question: whether MAS actually performs better than SAS when users are involved. The authors test multiple models across three domains and around 7,000 simulated conversations, comparing success rates, token costs, and failure types. I also like that they test different token budgets and user simulators, which makes the findings more convincing.

S2. Good analysis of results and failure cases.

The paper goes beyond success rates by analyzing failures like Wrongful Write, Budget Exhaustion, and Inaction. I also appreciate that the authors discuss the statistical coupling issue when predicting MAS gains from SAS accuracy and use a Leave-One-Domain-Out Capability Index to address it. The detailed experimental settings and released code also help with reproducibility.

Critical weaknesses

W1. Limited evaluation of MAS architectures.

The paper only tests one centralized MAS architecture, where the Orchestrator cannot directly access the Tool Schema and the Policy Advisor can only give suggestions. Appendix G shows that only about 28% of Write Orders are executed as requested. I think the results may depend on this specific architecture, and I would like to see comparisons with other MAS designs before generalizing the conclusions.

W2. Limited evidence for the main hypothesis.

The paper aims to test whether the 45% Capability-Saturation Principle holds in dual-control settings, but only two valid settings fall into the key low-accuracy range. Also, the statistical analysis does not clearly show whether MAS gains decrease as model capability increases. I think more experiments are needed to support a clear conclusion about this principle.

Questions for authors

Q1. Would MAS performance improve if the Orchestrator had direct access to the Tool Schema, or if the system used the M2 Checker architecture mentioned in the paper? A small ablation study on this would make the results more convincing.

Q2. Currently, only two valid settings have SAS accuracy clearly below 45%. Could the authors include more experiments with weaker models or harder tasks to better test whether the Capability-Saturation Principle applies to dual-control settings?

Reviewer_QAkpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tests whether the capability-saturation principle from prior work that reported a MAS helps while the single agent is weak and hurts above roughly 45% accuracy, transfers to dual-control settings, where the agent serves a simulated user under a written policy.

On the airline, retail and telecom domains of τ²-bench, the authors compare the stock single agent with a centralized MAS from the same model (orchestrator, database agent, policy advisor), all agents sharing one ledger. Over 7,000 episodes, the MAS is worse in 12 of 13 valid cells and tied in one, including in two cells where the capability-saturation principle predicts a gain. Tests on a held-out-domain capability index find no detectable dependence, and the MAS does not reduce wrongful writes, converting failures into budget exhaustion and inaction instead.

Critical strengths
  • The paper tackles the relevant question, of whether architecture rules derived without a user carry over to agents that serve users, with a careful compute-matching protocol and pre-stated reading rules.
  • The paper explicitly addresses the coupling introduced by regressing a performance gain on its own baseline by reporting a held-out-domain index alongside split-seed results.
Critical weaknesses
  • The conclusion that "these results make the SAS the default" rests on one MAS design that appears handicapped. In the evaluation, the orchestrator plans without any tool schemas, hence only 28% of write orders were executed as issued (Table 8), and 74% of the mean loss comes from budget-exhausted episodes (Table 10). Further, the S2 control used to attribute the loss to dividing the work sees the tools while planning (Appendix L). Since there is no control showing this MAS reproduces the original gains on an autonomous benchmark, it is unclear whether the result is about dual control or about this particular MAS implementation.
  • The wrongful-write analysis measures a narrower class of errors than the mechanism discussed in the paper. Appendix H counts a wrongful write only when the gold trajectory contains no write. However, incorrect arguments or additional unauthorized writes on tasks requiring a legitimate write are excluded from being counted in this category. Only 3 of 50 retail tasks can contribute due to this.
  • The paper is also hard to read in places. Section 3.4 introduces the regression, Lemma 1, H1/H2a/H2b, G1, R1/R2 and the power analysis before any result, and the many coined labels (WW, IN, BX, FP, S2) make the main text depend heavily on the appendices.
Questions for authors
  1. Could the authors run the MAS with tool schemas shared with the orchestrator, at least on the two telecom cells below 45%? Does the gap to S2 persist?
  2. Have the authors considered running the same MAS on one autonomous benchmark from prior work, to show it recovers the low-capability gain there?
  3. Can the authors report incorrect writes on tasks that do require a write, and discuss why the advisor is rarely consulted by the GPT-5 models?
1 comment

Thank you for the new ablation. Withdrawing the attribution to agent division and restricting claims to this implementation addresses my concern about overclaiming. However, even with schemas the MAS still falls below the 90% criterion, exhausting budget far more often, and the autonomous-benchmark control isn't implemented. Hence, it is still unclear whether the result reflects dual control or this paper's specific MAS design. The planned restructuring of Section 3.4 and the reference-action analysis are welcome. I will maintain my score.

Reviewer_3cYhOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether a rule linking single-agent capability to the benefits of multi-agent collaboration also holds in customer-service conversations. It compares a single agent with a centralized multi-agent system on three τ²-bench domains. The tested multi-agent system generally performs worse and costs more. The authors also examine budget exhaustion, simulator choice, and policy violations, but the experiments do not resolve how the benefit of collaboration changes with model capability.

Critical strengths
  1. This paper asks a useful practical question. It examines whether adding agents improves performance in customer-service conversations, where coordination takes time and resources. Reporting cost and budget exhaustion alongside task success makes the results more useful for deployment decisions.
  2. The authors are transparent about important limitations. They disclose the failed fidelity check and experimental deviations, and the released code and episode-level results make the analysis easier to inspect and reproduce.
Critical weaknesses
  1. The main comparison is difficult to interpret because the multi-agent planner lacks essential tool information. The orchestrator is expected to specify exact tool names and arguments without seeing their schemas, while the single-agent systems receive this information. Only about 25% of write orders were executed as issued, well below the paper’s own validity requirement. Sharing the schemas improved fidelity in a small smoke test, but the full experiments were not rerun. I therefore cannot tell how much of the performance loss comes from collaboration itself and how much comes from this avoidable limitation. The stronger single-agent control and uncapped experiments do not resolve this issue.
  2. The broader conclusion about the scaling rule is not sufficiently supported. The original validation considers the best of four multi-agent architectures, while this paper tests one centralized implementation that fails its own fidelity check. Its failure provides limited evidence that the original rule does not transfer. The capability analysis also has only 12 model-domain cells, with just two clearly below the proposed threshold, both in telecom. The uncertainty remains too large to determine whether the capability relationship holds. Although the authors acknowledge this limitation, describing the relationship as “disappearing” when the test is inconclusive goes beyond what the results establish.
Questions for authors

see weakness above

Reviewer_GghEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The aim of the paper is to study orchestration setups in LLMs. LLMs can be operated in a single-agent or a multi-agent setting. While intuition would suggest that multi-agent setups, where models can specialize for subtasks, would provide better performance, prior work shows that this is not necessarily the case. The paper builds on a recent study by Kim et al. who show that with a capable LLM, a single agent setup can provide better performance than a multi-agent ensemble. The main goal of the paper is to study a similar comparison in a setting where humans are the center of the orchestration. The idea being that having humans in the loop changes that characteristics of the problem at a fundamental level so the previous findings that were derived from synthetic regimes may not generalize.

Critical strengths
  1. The comparison of single vs multi-agent setups is an important problem. It would indeed be useful to know about settings in which single agent systems can suffice without relying to expensive multi-agent ones.
  2. The gap identified in the paper, lack of humans in the loop, is quite important for real world deployments.
Critical weaknesses
  1. Unfortunately, the paper constitutes experiments with tau-squared bench and no realistic dual-control setting, where a human is actively participating in the discussion, is considered. In the current state, the paper does not seem to do much beyond the prior work of Kim et al.
  2. The main premise of the paper, comparing single-agent vs. multi-agent LLM performance when humans are present in the loop, needs more justification. The paper does not convince the reader that inclusion of humans leads to fundamental changes. The intro seems to provide some justification in line 58 onwards. However, the arguments are not convincing. Namely, it is unclear what “one user-facing locus” and “no deployable analogue” mean. Similarly, the paper does not explain how the input tokens dominating or not dominating have an impact on the performance. The third argument, or users being simulated by models is also made at a high level, without describing exactly how the choices made in setting up this model affects the credibility of the previous studies’ findings.
  3. The writing can be improved in certain places. Clauses like “a MAS pays while the SAS is weak and costs once it is strong” are difficult to parse. The authors could consider simplifying. Similarly, one of the most important paragraphs (line 75) starts with “The principle does not survive the move”. It is not obvious to the reader what principle is being talked about here. It is also not clear what “cells” and “fixed simulated user” mean. This paragraph jumps too quickly into numbers like “45%” and “10-point gain” without describing the bigger picture result clearly and unequivocally.
  4. The contribution statement (line 85 onwards) and the intro text preceding it are disjointed. Contribution 1 mentions a “capability saturation principle” which is mentioned for the first time. Same for the protocol in Contribution 2 and the lemma in Contribution 3. The paper should consider rewriting the introduction to make it coherent, and more readable.
  5. Section 3 jumps straight into nitty-gritty details like the “final database state” and “required communication on airline”. The setup does not connect with the key aim of the paper (dual-control settings), hindering readability.
Questions for authors

Please see the weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.