acceptodds
Under review as a conference paper at ICLR 2027

Scaling Multi-Agent Intelligence with Substrate-Centric Coordination

Abstract

Existing large language model (LLM) based multi-agent systems have been shown to scale poorly, and we identify three problems: reading burden that grows with the group, useful information diluted by shared work of unknown quality, and fragility to failed agents. We argue that the problems are largely due to agent-centric designs: the coordination structure is wired into fixed workflows, central coordinators, or peer-to-peer messaging, and recent blackboard-style methods that move the coordination into a shared medium relieve only the last problem. To close the gaps, we build ScaleMAS, where asynchronous agents interact with a shared datastore of linked messages via three operations: publishing messages with optional references to earlier ones that they build on, searching for messages by content, and visiting messages by address. Search returns at most messages, so an agent's read stays bounded at any group size; results are ranked by relevance and by how often they are cited, which we find to be an effective proxy for quality; and messages persist after their producers leave, so the group carries on when individual agents fail. Across four agentic tasks, ScaleMAS outperforms agent-centric and blackboard-style baselines at lower inference compute, continues to scale up to 128 agents, and tolerates losing agents midway.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multi-agent Systems

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_WMnqOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ScaleMAS. This is a substrate-centric coordination framework for multi-agent collaboration (multiple asynchronous agents collaborate through a shared message data store, and can perform operations such as publishing, searching, and visiting); each retrieval uses a fixed top-k and is sorted based on relevance and citation weight. The paper conducted experiments on four benchmarks, and ScaleMAS outperformed parallel ReAct, MAD, DeLM, and AOrchestra overall, and demonstrated results for scaling up to 128 agents as well as robustness against agent failures.

Critical strengths
  1. The paper points out a scalability issue in current multi-agent systems and proposes a relatively simple design to reduce reading costs and enhance the robustness against failed agents. In particular, the experiments extending to 128 agents are quite valuable, as many multi-agent works only test smaller group sizes.
  2. The experimental analysis in the paper is also quite thorough, capable of differentiating the role of the shared substrate itself from simply increasing the number of agents, and the trace analysis also provides some evidence of the actual reuse of work results among agents.
Critical weaknesses
  1. I think the coordination ability demonstrated in the paper is a bit narrow. ScaleMAS does not have explicit task allocation, roles, or synchronization mechanisms. Figure 9 does indeed show that a certain degree of division of labor naturally forms, but the current experiments have not tested scenarios where multiple agents must complete different sub-tasks according to clear dependencies.
  2. I also have some reservations about the novelty of the method itself. From the overall structure, ScaleMAS is more similar to a retrieval-augmented shared blackboard: messages are stored in the shared datastore, retrieved using BM25, and sorted using references and citation signals. If it could be directly compared with an existing retrieval-based shared-memory system, it would be easier to see the novelty of the method itself.
  3. The current evaluation still does not answer a relatively practical system-level question. The three third-party benchmarks all use Pass@N, so only one of the N agents needs to get the correct answer for it to pass. The paper does not explain whether the entire group can stably produce a final answer (a single final answer). Additionally, top-k only ensures bounded input for a single read and does not guarantee that the total retrieval cost remains bounded as the number of agents increases.
Questions for authors
  1. Could an additional experiment be conducted to test ScaleMAS on a task where agents need to clearly divide and synchronously complete interdependent subtasks? I am particularly interested in knowing under what circumstances pure reliance on substrate-only coordination would start to fail.
  2. What is the key technical difference between ScaleMAS and a typical retrieval-based shared-memory/blackboard system? If such baselines could be directly compared under the same conditions, it would be easier to judge the novelty of the method.
  3. Could an experiment be added that requires the entire system to submit only one answer (single group answer), instead of using Pass@N? For Contradiction Audit, I also hope to see an indicator that considers precision, or a clear answer aggregation mechanism.
  4. How do datastore size, search/visit calls, retrieval latency, and accuracy change as the number of agents increases? Especially when expanded to 128 agents, is the fixed k=8 still sufficient, and how sensitive are the results to the choice of k?
1 comment
Reviewer_WMnqOpenReview ↗

Response to Authors

Thank you for the additional experiments you ran during the rebuttal—they’ve basically addressed all my concerns. I find the ScaleMAS framework you proposed very practical, and I look forward to seeing it open-sourced in the future. I can also tell that you put a lot of effort into the rebuttal, so I’ve decided to raise my score to 4.

Reviewer_3AY8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that LLM multi-agent systems scale poorly because coordination is wired into the agents, and proposes ScaleMAS, a substrate in which asynchronous agents coordinate only through a shared datastore of linked messages via publish, search (top-k, ranked by BM25 plus a citation weight) and visit. Across four agentic tasks and two backbones, the paper reports better performance than agent-centric and blackboard-style baselines at lower compute, scaling to 128 agents, and robustness to failing agents.

Critical strengths
  1. The motivation is clear: the paper identifies three concrete failure modes of scaling (reading burden, information dilution, fragility) and maps each to a design choice.
  2. The design is simple and general: coordination happens only through a shared datastore with bounded search and link-based ranking, so no roles or workflows need to be engineered.
  3. The evaluation spans four task types, two backbones, groups of up to 128 agents, and stress tests with failing and intermittently absent agents.
Critical weaknesses
  1. My biggest concern is that the evaluation is small. Each third-party benchmark only uses the first 100 examples, not a random subset, so many gaps could just be noise.
  2. The reported accuracy is actually Pass@N, so the gains with more agents partly come from more attempts, and the comparison with AOrchestra, which only gives one answer, is not fair.
  3. The paper says part of the gap to ReAct already exists with a single agent, which means it comes from the note-taking design rather than the coordination between agents. However, Table 2 only separates these two effects for four Qwen3.5-9B agents, so in Figs. 1 and 4 the comparisons with the other baselines still mix both effects.
  4. I think some claims in the abstract are a bit overclaimed. The abstract says "an agent's read stays bounded at any group size", but this is only true for a single search call, which returns at most kk messages, while agents can search as many times as they want (e.g., the searches per agent go up from 12 to 65 on Contradiction Audit x8). Besides, the abstract says ScaleMAS "continues to scale up to 128 agents", but on FATE-M the performance stops improving after 16 agents (Fig. 6), and only the custom Contradiction Audit keeps improving.
  5. There is no sensitivity analysis for kk, α\alpha and γ\gamma, which are all fixed in Table 4.
  6. Minor typos remain, e.g., "an SCALEMAS agent" (p.7) and "will be accepted iff." (p.5).
Questions for authors
  1. What are the results under a committed-answer metric, such as majority voting, instead of Pass@N?
  2. Why do the authors use the first 100 examples of each benchmark instead of a random subset, and what is the variance across seeds?
  3. How much of the gain over MAD, DeLM and AOrchestra remains if these baselines also get a private note-taking datastore?
  4. How does the total number of messages each agent reads grow with group size?
  5. Why does FATE-M stop improving after 16 agents while Contradiction Audit keeps improving?
  6. How sensitive are the results to kk, α\alpha and γ\gamma?
Reviewer_tWXmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ScaleMAS, a substrate-centric coordination framework designed to improve the scalability of LLM-based multi-agent systems. The method is inspired by the structure of the Web, which enables agents to coordinate through a shared, searchable datastore. Agents can publish messages, search for relevant information, and retrieve messages through references. The authors evaluate ScaleMAS on four benchmarks under two different backbones. Experimental results show that ScaleMAS generally achieves better performance against several baseline methods.

Critical strengths
  • The idea of drawing inspiration from the Web to design a multi-agent coordination framework is interesting. In particular, the shared, searchable information space and citation-based message chains offer a promising approach to addressing challenges in multi-agent systems.

  • The proposed benchmark provides a valuable testbed for evaluating how effectively multiple agents can share knowledge, reuse information, and collaborate to solve complex tasks.

  • The experimental results provide several interesting and valuable insights into multi-agent coordination, particularly regarding scalability, knowledge sharing, and fault tolerance.

Critical weaknesses
  • My biggest concern regarding the rule design is the potential error propogation risk, and the population bias. If, for example, an agent propose an incorrect conclusion, while providing what appears to be a plausible rationale, especially in an early stage, this could lead to potential problems such as the erroneous intermediate result is inherited by multiple agents. Also, this could be dangerous, as it appears as though multiple agents support a conclusion, but they do not constitute independent evidence; rather, they all stem from the same erroneous source. Even though figure 8 has somehow touched this concern, yet the chain of evidence is itself defined through citations. It's still not clear to be whether highly cited messages are more factually correct.

  • Following my first concern, the evaluation metric (Evidence recall) for the CONTRADICTION AUDIT seems problematic. As it doesn't involve the penalization of false positives. This tiggers my concern that if a system shares a large number of unreliable messages, this metric may not necessarily reflect the cost of error propagation. As long as these messages contain a sufficient number of target segments. Other metric (e.g. precision, F1) must also be reported and discussed.

  • The comparison with AOrchestra is not strictly fair. I'm not surprised of the results in Figure 7. To me, it seems AOrchestra and ScaleMAS, rather than ScaleMAS shows a clear advantage. ScaleMAS is evaluated across multiple agent counts, whereas AOrchestra is represented by a single configuration in which the coordinator determines the number of workers. The experiments do not appear to explore stronger coordinator models (see my fourth question). Also, the important question is not merely whether a multi-agent system can scale, but when scaling is beneficial, how far it should scale, and at what cost. Therefore, it remains unclear to me under what task conditions increasing the number of agents is actually beneficial, and when the additional inference and coordination costs outweigh the gains.

Questions for authors
  • According to my first concern, I think it's very dangerous to assume that the frequently cited information is euqal to what's useful. My question is: can the system identify dependencies between pieces of evidence when multiple agents reference the same erroneous source?

  • Line 133 states that ' N agents run the same harness under the same prompt.', will this cause a lot of repeated works when they work independently?

  • Why is GPT-5.6 used for both pair selection and LLM-based judging? Ideally, these two stages should be handled by different models to minimize potential model-specific biases and reduce the risk of correlated errors in the dataset construction and validation process.

  • AOrchestra relies on a stronger central coordinator for planning, task decomposition, etc. However, the experiments appear to use the same backbone model for both the coordinator and the worker agents. Is this comparison fair? Could the authors clarify the model configuration used for AOrchestra?

Comments on the paper

1 comment
~Elad_Sharony1OpenReview ↗

The SWE-bench Verified gain does not reproduce

We tried to reproduce the SWE-bench Verified result. We implemented it with Claude Opus 5.5 and GPT-6.1-Sol, both at xhigh reasoning effort. The two models audited the implementation against the paper independently and reviewed each other's findings.

We followed every setting the paper specifies. This covers mini-SWE-agent 2.4.6 with its SWE-bench configuration, the hyperparameter table, the appendix datastore prompt with its three tools, Qwen3.5-9B with thinking on and the first 100 test instances. We served the model with SGLang, ran three runs to match the paper's three seeds and graded with the official SWE-bench harness.

AgentsPaper, ReAct → sharedPaper gainOurs, ReAct → sharedOur gain (95% CI)
1≈56 → ≈66+1051.7 → 51.8+0.1 (−2.8, +2.8)
2≈62 → ≈76+1463.6 → 63.9+0.4 (−2.0, +2.8)
468.0 → 80.0+12.071.0 → 71.7+0.7 (−2.7, +4.0)

Both sides use the same 100 instances and the same pass@k metric. Our 1- and 2-agent values come from the 4-agent runs, which is valid here because no agent read the datastore.

Our ReAct baseline lands within a few points of the paper's, yet the shared substrate adds nothing measurable at any team size.

The agents almost never use the datastore. Across 1,176 agents and about 111,000 model calls, they searched twice, found nothing, and never published or visited a message. No agent could read another agent's work. Turning thinking off did not change this in a smaller probe.

The appendix prompt describes the datastore but not when to use it, and the case studies use labels such as CANDIDATE and LEMMA PROVED that the paper does not define. We may be missing benchmark-specific instructions. We would appreciate it if the authors could provide the following:

  1. Any SWE-bench prompt beyond the appendix prompt, in particular instructions on when and how to use the datastore, and whether it can be released.
  2. Whether the ReAct baseline received the same instructions. Without them, the comparison would not isolate the substrate.
  3. The average number of publish, search and visit calls a team makes on a SWE-bench task.
  4. The Qwen3.5-9B thinking mode and sampling settings beyond temperature.

We will make an effort to rerun with any details the authors share before the discussion closes and report the result here.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.