- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies multi-developer agent systems, which they call MDAs. Unfortunately, I don't understand the novelty in this paper since multi-agent systems have been studied for years at this point.
Critical strengths
Unfortunately, I do not see how this paper is novel.
Critical weaknesses
Multi-agent systems are accessible to users today in every major harness to the best of my knowledge (e.g., Codex Ultra, Claude code, ...). Furthermore, there is an enormous literature on multi-agent systems that the related work does not cover. I do not understand the novelty in this paper nor what the contributions are in light of this reality.
Questions for authors
How does this different from the thousands of papers written on multi-agent systems?
2 comments
I disagree with this assessment, as techniques such as swarmed from OpenAI are exactly this kind of MAS.
Response to Reviewer XMSu
Thank you for the review. The distinction is who builds the system and decides which agents exist. Our paper studies how to scale agentic AI through contributions from independent developers, each with their own resources and incentives.
Why this is more than a larger MAS. The systems named in the review fall within our single-developer setting: one developer designs the agents and their orchestration. Such systems can grow very large, but remain bounded by the resources, expertise, data access, and integrations available to that developer. An MDA opens a different scaling axis by letting independent developers contribute capabilities to a shared system.
An MDA is an emergent system: its agents depend on what developers choose to contribute, and participation depends on whether doing so benefits them. This introduces second-order effects into system design. Alongside how existing agents work together, we must study how routing, evaluation, and compensation change what developers build in the first place.
What the paper contributes. Our experiments and theory provide arguments for this scaling direction and identify obstacles to realizing it. On GAIA, agents from 54 specialist developers reach 70% together although none exceeds 35% alone. The best single-developer MAS we evaluate reaches 67%, at more than twice the inference cost. An agent built to complement an existing MDA scores only 19% alone, yet raises the MDA from 40% to 53% (Fig. 4c). These results show how developers with limited individual capabilities can advance a stronger shared system. Please also see the general comment.
Related work. §2 discusses MAS frameworks, routing, interoperability protocols such as A2A, and open ecosystems such as AgentNet and IoA. These efforts reflect strong interest in extending MAS across developers. Our contribution is to explicitly study the developer population: why combining its contributions is useful, what developers have reason to contribute, and why a large-scale MDA may fail to emerge despite its potential. We would welcome references that address these questions and should be discussed alongside our work.
Soundness. The review raises novelty concerns but does not identify a specific issue with the experiments or theory. We would appreciate clarification of the soundness concerns so that we can address them directly.