- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ScaleMAS. This is a substrate-centric coordination framework for multi-agent collaboration (multiple asynchronous agents collaborate through a shared message data store, and can perform operations such as publishing, searching, and visiting); each retrieval uses a fixed top-k and is sorted based on relevance and citation weight. The paper conducted experiments on four benchmarks, and ScaleMAS outperformed parallel ReAct, MAD, DeLM, and AOrchestra overall, and demonstrated results for scaling up to 128 agents as well as robustness against agent failures.
Critical strengths
- The paper points out a scalability issue in current multi-agent systems and proposes a relatively simple design to reduce reading costs and enhance the robustness against failed agents. In particular, the experiments extending to 128 agents are quite valuable, as many multi-agent works only test smaller group sizes.
- The experimental analysis in the paper is also quite thorough, capable of differentiating the role of the shared substrate itself from simply increasing the number of agents, and the trace analysis also provides some evidence of the actual reuse of work results among agents.
Critical weaknesses
- I think the coordination ability demonstrated in the paper is a bit narrow. ScaleMAS does not have explicit task allocation, roles, or synchronization mechanisms. Figure 9 does indeed show that a certain degree of division of labor naturally forms, but the current experiments have not tested scenarios where multiple agents must complete different sub-tasks according to clear dependencies.
- I also have some reservations about the novelty of the method itself. From the overall structure, ScaleMAS is more similar to a retrieval-augmented shared blackboard: messages are stored in the shared datastore, retrieved using BM25, and sorted using references and citation signals. If it could be directly compared with an existing retrieval-based shared-memory system, it would be easier to see the novelty of the method itself.
- The current evaluation still does not answer a relatively practical system-level question. The three third-party benchmarks all use Pass@N, so only one of the N agents needs to get the correct answer for it to pass. The paper does not explain whether the entire group can stably produce a final answer (a single final answer). Additionally, top-k only ensures bounded input for a single read and does not guarantee that the total retrieval cost remains bounded as the number of agents increases.
Questions for authors
- Could an additional experiment be conducted to test ScaleMAS on a task where agents need to clearly divide and synchronously complete interdependent subtasks? I am particularly interested in knowing under what circumstances pure reliance on substrate-only coordination would start to fail.
- What is the key technical difference between ScaleMAS and a typical retrieval-based shared-memory/blackboard system? If such baselines could be directly compared under the same conditions, it would be easier to judge the novelty of the method.
- Could an experiment be added that requires the entire system to submit only one answer (single group answer), instead of using Pass@N? For Contradiction Audit, I also hope to see an indicator that considers precision, or a clear answer aggregation mechanism.
- How do datastore size, search/visit calls, retrieval latency, and accuracy change as the number of agents increases? Especially when expanded to 128 agents, is the fixed k=8 still sufficient, and how sensitive are the results to the choice of k?
Response to Authors
Thank you for the additional experiments you ran during the rebuttal—they’ve basically addressed all my concerns. I find the ScaleMAS framework you proposed very practical, and I look forward to seeing it open-sourced in the future. I can also tell that you put a lot of effort into the rebuttal, so I’ve decided to raise my score to 4.