- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ChatDev 3.0, a distributed multi-agent system designed to coordinate tasks across users and devices while respecting their permissions and local execution constraints. It separates user-level coordination from device-level execution and uses a unified communication protocol and asynchronous runtime to support collaboration and parallel execution. Evaluations on TeamBench, MDCBench-CLI, and MDCBench-GUI show improvements over several baselines in task success and execution efficiency, although the system remains slower than a single-agent baseline in some settings. The results are promising, but the limited GUI evaluation and reliability issues observed in a case study leave questions about the system’s robustness and generalizability.
Critical strengths
-
Clear problem formulation and practical relevance. The paper identifies execution asymmetry as an important challenge in multi-agent collaboration, highlighting the differences in user permissions, information access, and device-level execution capabilities. This provides a clear motivation for designing a distributed agent architecture.
-
Well-structured system design. ChatDev 3.0 separates user-level coordination from device-level execution and combines a unified communication protocol with an asynchronous event-driven runtime. This design allows agents to collaborate across users and devices while enabling independent tasks to run concurrently, potentially reducing unnecessary waiting and coordination overhead.
-
Extensive evaluation across different settings. The paper evaluates the proposed system on cross-user coordination, cross-device command-line tasks, and graphical user interface tasks. The reported results show improvements over several baselines in task success rates and execution efficiency, particularly on MDCBench-CLI. These experiments provide encouraging evidence for the practical value of the proposed architecture.
Critical weaknesses
- The framework combines role-based agent decomposition, message-based coordination, and asynchronous execution. However, the paper does not clearly isolate the contribution of each component. Without targeted ablation studies, it is difficult to determine whether the reported gains come from the proposed architecture itself or from the combined effect of these design choices.
- Limitations in experimental validation. The GUI evaluation includes only 20 tasks, with one trial per task and a single backbone model, making it difficult to assess the robustness and generalizability of the results. In addition, more ablation studies would help clarify whether the improvements come from the proposed architecture, the asynchronous runtime, or other implementation differences.
- Insufficient evidence of end-to-end execution reliability and reproducibility. The case study in Appendix E.2 reveals that the system reports successful task completion even though the restaurant-ranking procedure was not successfully executed or independently verified. This raises concerns about the reliability of its completion claims. Furthermore, the availability of the complete source code and experimental materials could not be confirmed from the provided manuscript, limiting independent reproducibility.
Questions for authors
1.Have the authors conducted ablation studies to isolate the contributions of the boundary-aligned architecture, unified communication protocol, and asynchronous runtime to the reported performance gains? 2.How does the system enforce and verify authorization boundaries during cross-user communication and delegated execution, and has it been evaluated against unauthorized access or information leakage?
3 comments
To weakness 2:
We acknowledge the limitations of the GUI evaluation in terms of the number of tasks and repeated runs. This is partly due to the substantial execution time and higher api cost required by each GUI task. And we would nevertheless like to clarify its role in the paper: it provides complementary evidence on whether the architecture extends from command-line execution to workflows requiring visual observation and interface interaction. Our conclusions about cross-device execution also draw on the broader MDCBench-CLI experiments, which show consistent performance advantages. We therefore view the GUI results as complementary evidence across interaction modalities, to be considered alongside the CLI evaluation.
And although the evaluation is limited in scale, GUI results in Table 3 shows substantial improvements on both metrics. ChatDev 3.0 achieves a Pass@1 of 75.00%, compared with 25.00% for TopoClaw and 35.00% for UFO³, and a Process Score of 79.35%, compared with 44.33% and 59.65%, respectively.
To weakness 3:
We thank the reviewer for examining the consistency between completion claims and execution evidence. We would like to clarify the distinction between the agent’s final self-report in the example and the paper’s assessment of the recorded trajectory. We do not equate an agent's self-reported completion with verified task completion. Appendix E.2 records the User Agent claiming completion, but immediately states: “This is the User Agent’s final claim; the recorded execution does not establish successful completion of the requested script-based ranking.” The concluding “Verification boundary” further explains that neither the ranking script nor the alternative analysis was confirmed to produce a valid ranking, and that the final text cannot serve as an independently computed result. Its conclusion explicitly states that the case represents “not verified completion of every requirement in the original task.” Thus, we do not present this example as verified end-to-end success. Benchmark evaluation likewise does not directly adopt the agent’s self-reported success: Appendix B.2.3 describes manual assessment based on final environment states, execution traces, and the task-specific scoring rubrics in Table 6. Appendix D.1.3 also distinguishes the evidentiary scope of archived case narratives from that of aggregate benchmark results.
And the supplementary materials, including the evaluation materials and source code, were submitted with the manuscript and are available in the supplementary submission. We hope this clarifies where the reproducibility materials can be accessed.
To question 2:
Device access is scoped by user at the execution layer. Each User Agent accesses Device Agents within its user's authorization scope through the corresponding User Server. Cross-user communication may convey information or collaboration requests, but does not automatically transfer device-control authority; device operations must still be initiated by the receiving side within its own authorization scope. Therefore, cross-user communication does not grant unauthorized access to another user's devices. We note that potential information leakage through agent communication is a separate security concern, which is beyond the scope of this work and remains an important direction for future investigation.
Does authors' response address your concerns?
Thank you for your review! Could you check the authors’ response and clarify whether it addresses your original concerns? Many thanks!
AC
Thank you for your valuable feedback and constructive comments. We sincerely appreciate your time and effort in reviewing our work. And we hope our response adequately addresses your concerns.
To weakness 1 and question 1:
We thank the reviewer for suggesting concrete comparisons to clarify the contributions of the architecture, communication mechanism, and runtime. These components address distinct aspects of distributed collaboration, and the existing evaluation provides complementary evidence supporting their utility. We will clarify the correspondence between these design choices and the reported evidence below.
User-level and device-level boundaries serve different purposes. In TeamBench, role-specific information and permissions are prescribed by the benchmark; preserving these boundaries is therefore part of the evaluation setting (section 3.2.1). For device-level organization, the evaluated TopoClaw configuration provides a relevant comparison: a single root agent directly invokes device tools and centrally maintains cross-device context, whereas ChatDev 3.0 delegates device-local state and execution to separate Device Agents. The comparisons align the backbone model, device environments, and available action capabilities (section 3.3.1), with Appendix D.1.3 documenting the functional correspondence between the underlying execution interfaces. ChatDev 3.0 achieves an average Pass@1 of 95.00% versus 90.74% for TopoClaw on MDCBench-CLI (Table 2). On MDCBench-GUI, Pass@1 increases from 25.00% to 75.00%, and Process Score from 44.33% to 79.35% (Table 3). These results support the utility of separating device-local execution and context, particularly in GUI environments.
TeamBench Full provides a relevant comparison for the coordination mechanism. Both methods retain the Planner–Executor–Verifier roles and their permission restrictions, with aligned backbone models, initial environments, and graders (section 3.2.1); Appendix D.1.3 further describes their functionally comparable execution capabilities. Within these conditions, ChatDev 3.0 allows agents to initiate communication according to local needs, rather than following the benchmark’s prescribed interaction flow. Average Solution Pass Rate increases from 53.41% to 57.10%, and Verification Pass Rate from 47.73% to 54.55% (Table 1). Figure 3 additionally documents autonomous reciprocal communication and varying interaction depths. The outcome comparisons and behavioral observations jointly support the utility of autonomous coordination.
The asynchronous runtime is an efficiency-oriented design: it reduces blocking and allows independent execution branches to progress concurrently under the existing task dependencies. Its scheduling objective is separate from the task-solving objectives of boundary-aligned execution and autonomous coordination; these contributions are therefore decoupled at the level of design objectives and evaluation criteria. Task effectiveness is evaluated through task-completion metrics, whereas runtime efficiency is evaluated directly through execution time, critical-path efficiency, and observed concurrency. Compared with UFO³, average execution time decreases from 321.85 s to 153.83 s, while critical-path efficiency increases from 41.98% to 53.49% (Table 2). Figure 4(b–c) and Appendix Figure 7 further document greater device concurrency and a larger share of concurrent execution time. Together, these measurements provide evidence directly corresponding to the runtime’s efficiency objective.
Taken together, we believe the existing comparisons and mechanism analyses provide appropriate support for these claims, and hope this clarification addresses the reviewer’s concern about their experimental basis.