- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether the benefits of Multi-Agent Systems (MAS) over Single-Agent Systems (SAS) also apply to dual-control settings, where agents need to interact with users throughout a task. The authors evaluate several models on the Airline, Retail, and Telecom domains of -bench, with around 7,000 simulated conversations and a shared completion-token budget for MAS. The results show that MAS generally performs worse than SAS under a fixed user simulator, even when the single agent is relatively weak. The paper also analyzes how model capability, token usage, and different types of failures affect MAS performance.
Critical strengths
S1. An important practical problem with good experiments.
I think the paper addresses an important question: whether MAS actually performs better than SAS when users are involved. The authors test multiple models across three domains and around 7,000 simulated conversations, comparing success rates, token costs, and failure types. I also like that they test different token budgets and user simulators, which makes the findings more convincing.
S2. Good analysis of results and failure cases.
The paper goes beyond success rates by analyzing failures like Wrongful Write, Budget Exhaustion, and Inaction. I also appreciate that the authors discuss the statistical coupling issue when predicting MAS gains from SAS accuracy and use a Leave-One-Domain-Out Capability Index to address it. The detailed experimental settings and released code also help with reproducibility.
Critical weaknesses
W1. Limited evaluation of MAS architectures.
The paper only tests one centralized MAS architecture, where the Orchestrator cannot directly access the Tool Schema and the Policy Advisor can only give suggestions. Appendix G shows that only about 28% of Write Orders are executed as requested. I think the results may depend on this specific architecture, and I would like to see comparisons with other MAS designs before generalizing the conclusions.
W2. Limited evidence for the main hypothesis.
The paper aims to test whether the 45% Capability-Saturation Principle holds in dual-control settings, but only two valid settings fall into the key low-accuracy range. Also, the statistical analysis does not clearly show whether MAS gains decrease as model capability increases. I think more experiments are needed to support a clear conclusion about this principle.
Questions for authors
Q1. Would MAS performance improve if the Orchestrator had direct access to the Tool Schema, or if the system used the M2 Checker architecture mentioned in the paper? A small ablation study on this would make the results more convincing.
Q2. Currently, only two valid settings have SAS accuracy clearly below 45%. Could the authors include more experiments with weaker models or harder tasks to better test whether the Capability-Saturation Principle applies to dual-control settings?
Thank you for the new ablation. Withdrawing the attribution to agent division and restricting claims to this implementation addresses my concern about overclaiming. However, even with schemas the MAS still falls below the 90% criterion, exhausting budget far more often, and the autonomous-benchmark control isn't implemented. Hence, it is still unclear whether the result reflects dual control or this paper's specific MAS design. The planned restructuring of Section 3.4 and the reference-action analysis are welcome. I will maintain my score.