acceptodds
Under review as a conference paper at ICLR 2027

PANDA: A Decentralized Architecture with Flexible Orchestration for Scalable, Fault-Tolerant Multi-Agent Systems

Abstract

Existing architectures for LLM-based multi-agent systems (MAS) cannot reliably and efficiently solve multi-step tasks at scale: they struggle to support large numbers of agents and concurrent tasks, tolerate failures, govern agent interactions, and accommodate the diverse planning and execution patterns different tasks require. We present PANDA, a decentralized architecture that connects a large collective of heterogeneous, independently administered agents, letting them discover each other's capabilities and self-organize into small specialized teams per task. PANDA scales by decoupling collective communication from team communication, allowing agents to participate in multiple teams simultaneously, load-balancing tasks across the collective, and scheduling concurrent work within each agent. PANDA further separates the underlying architecture from the orchestration strategy, supporting three planning and execution patterns (star, chain, and mesh) that can be selected according to the structure and requirements of each task. PANDA detects infrastructure and orchestration failures and recovers affected tasks by dynamically replanning around failed components. Finally, to provide governance without a centralized service that would limit scalability, PANDA uses a web-of-trust model to constrain agent interactions to established trust relationships. We evaluate PANDA on the HotPotQA benchmark, demonstrating that it scales to thousands of agents, assembles teams in milliseconds, matches state-of-the-art accuracy at up to 8x the efficiency, and sustains 100% task completion under faults where existing systems fail.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Multi-Agent Systems, Decentralized, Fault Tolerant, Scalable

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_X7wEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work presents PANDA, a decentralized architecture for large-scale LLM multi-agent systems. PANDA maintains distributed agent registries, separates collective-scale communication from reliable within-team communication, supports load balancing and concurrent scheduling, and decouples the architecture from star, chain, and mesh orchestration strategies. It also provides infrastructure/orchestration failure detection and replanning, together with a web-of-trust mechanism for decentralized governance. Evaluation on HotPotQA studies team-assembly scalability, runtime and accuracy, behavior under injected faults, and trust propagation under agent churn.

Critical strengths

PANDA is a coherent end-to-end systems design. Scalability, orchestration, fault recovery, and governance are considered within a single decentralized architecture, and the implementation demonstrates substantial engineering effort.

Critical weaknesses
  1. The technical novelty appears to lie primarily in systems integration. Several central ingredients of PANDA have close precedents in recent LLM multi-agent systems. For example, Internet of Agents already studies heterogeneous third-party agents in a distributed setting with dynamic inter-agent communication [1]; Symphony proposes decentralized capability discovery and task allocation together with scalable and fault-tolerant orchestration [2]; and MultiAgentBench explicitly evaluates multiple communication topologies, including star, chain, tree, and graph structures [3]. Beyond the LLM-agent literature, PANDA's gossip-based dissemination, load balancing, failure detection, web-of-trust, and DAG-based recovery also largely instantiate established distributed-systems ideas. I therefore view the main contribution as a coherent integration of these mechanisms into one LLM-MAS architecture.

  2. The empirical scope is too narrow for several broad claims. Task-level evaluation is almost entirely based on HotPotQA, which does not establish that flexible orchestration is beneficial across structurally different agent tasks. Similarly, the fault experiment compares against systems without equivalent recovery mechanisms, while the governance experiment mainly tests liveness/trust propagation under churn rather than adversarial or malicious participants.

[1] Chen, Weize, et al. "Internet of agents: Weaving a web of heterogeneous agents for collaborative intelligence." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Wang, Ji, et al. "Symphony: A decentralized multi-agent framework for scalable collective intelligence." arXiv preprint arXiv:2508.20019 (2025).

[3] Zhu, Kunlun, et al. "Multiagentbench: Evaluating the collaboration and competition of llm agents." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Questions for authors
  1. Can the authors demonstrate the benefit of flexible topology selection across multiple qualitatively different tasks, ideally including an automatic mechanism for selecting star/chain/mesh rather than configuring the topology externally?

  2. How does PANDA compare against a baseline equipped with generic checkpointing/retry/failover mechanisms, rather than baselines that have no mechanism to recover from the injected crash?

  3. How does the web-of-trust mechanism behave under malicious certificates, Sybil participants, compromised trusted nodes, or strategically misadvertised capabilities?

2 comments

We thank the reviewer for their careful review and constructive feedback, and address their comments below. The submission has been updated accordingly, with all changes shown in blue\color{blue}{\text{blue}}.

Technical novelty Like other multi-agent systems, PANDA draws on established distributed-systems techniques. We deliberately leverage well-understood primitives for scalability and fault tolerance rather than inventing new ones, allowing us to focus on the architectural challenges specific to LLM agents. PANDA's contribution is the adaptation and coherent integration of these mechanisms into an architecture that jointly supports scalability, fault tolerance, flexible orchestration, and decentralized governance. As discussed in the Introduction and summarized in Table 1, existing systems address subsets of these requirements but do not satisfy all of them jointly. PANDA addresses the interactions among these requirements through architectural decisions specific to LLM-based multi-agent systems. To our knowledge, PANDA is the first system to satisfy all four requirements jointly.

Importance of flexible orchestration PANDA does not propose flexible topology selection as a performance optimization. Rather, different topologies accommodate different requirements concerning privacy, trust, coordination, and control, which may vary across users and applications. For example, mesh supports collaborative execution among agents, star centralizes coordination in a single orchestrator, and chain distributes decision-making along the execution path. These differences reflect architectural tradeoffs rather than a universally optimal topology. We support all three to accommodate diverse application requirements, not to demonstrate that selecting among them improves task performance. Accordingly, topology selection is driven primarily by external requirements and user preferences, which an automated selection mechanism would also need to consider alongside task characteristics. We add this discussion in Section 2.4.

Malicious participants PANDA focuses on scalability and fault tolerance under crash failures, rather than Byzantine resilience. We assume that participants in the web of trust are honest and that their identities and advertised capabilities have been verified at admission. Under these assumptions, certificates provide authenticated attestations that can be propagated through the trust graph. We acknowledge that malicious certificates, Sybil identities, compromised trusted nodes, and strategically misadvertised capabilities can undermine these guarantees if the assumptions are violated. Designing a Byzantine-resilient decentralized governance mechanism is outside the scope of this work and is the subject of ongoing complementary research. We clarify the threat model and its limitations in Section 2.3.

Comparison with a checkpointing baseline We conducted a semi-synthetic ablation measuring the effect of worker failures on task completion time and display the results in the paper in Figure 5. Over 5000 simulated trials across the three topologies, we sample the number of subtasks, and the duration of each subtask, uniformly from 1000 recorded runs of each topology. We then define the overhead added upon a failure in a centralized checkpointing case vs PANDA. With checkpointing the two main costs incurred are (1) the time it takes to recover and (2) how much work was lost reverting to the checkpoint. In PANDA, once we detect a failure, we immediately swap in a healthy agent, avoiding recovery latency and losing no completed work -- only the in-flight subtask is retried. Given PANDA’s failure detector, we detect a failure with probability 1−1e1 - \frac{1}{e} every TT seconds [1] and we model this timing in our experiment. We then evaluate how task completion time increases as the probability of a subtask failing increases and demonstrate that PANDA with a reasonable T=3T = 3, performs significantly better than any recovery baseline even assuming a relatively fast 15 second recovery and immediate checkpointing. We present the results of this experiment in Appendix C.

[1] Gupta, I., Chandra, T. D., and Goldszmidt, G. S. "On scalable and efficient distributed failure detectors." PODC 2001.

Dear reviewer X7wE,

Thank you again for your review. With the discussion period closing soon, we wanted to make sure our responses reached you and check if our clarifications address your concerns. We are happy to clarify further or run additional experiments if that would help your reviewing.

Best, The Authors

Reviewer_kcDsOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PANDA, a decentralized architecture for scalable and fault-tolerant LLM-based multi-agent systems. It enables dynamic team formation, flexible orchestration and decentralized governance through a web-of-trust mechanism. It also supports failure detection and task recovery through agent replacement and replanning. Experiments on HotPotQA show that PANDA scales to millions of agents, achieves up to 8× faster execution than existing systems, and maintains 100% task completion under simulated failures.

Critical strengths

PANDA separates orchestration strategies from the underlying agent architecture, supporting star, chain, and mesh execution without centralized coordination. It's a practical and flexible decentralized architecture.

Critical weaknesses

The web-of-trust mechanism is not clearly integrated with team formation and execution. It defines trust and different agents may have inconsistent trusted sets. However, team assembly appears to rely on the entrypoint's local registry, without explaining how conflicting trust relationships among team members are resolved.

Questions for authors
  1. Can independently administered agents actually form and execute a team under the proposed governance model? The governance experiments measure trust propagation but do not examine this interaction between trust and orchestration.

  2. How does PANDA reconcile asymmetric trust relationships with team-based orchestration? The web-of-trust mechanism allows each agent to maintain a different trusted set. Consider a team in which agent A trusts both B and C, but B does not trust C. Can these agents still participate in the same mesh-based execution?

  3. What exactly is being certified in the web-of-trust mechanism, and why should certification imply trust in an agent's capabilities? Equation (1) binds an agent's public key, identity, and advertised capabilities into a signed certificate, while Equation (2) extends trust through certificate chains. However, verifying an identity is different from verifying that an agent can reliably perform its advertised capabilities. If a legitimate agent certifies another agent with inaccurate capability claims, these claims can potentially propagate through the trust graph.

  4. How does PANDA detect orchestration failures without access to an external correctness oracle? In Section 3.2, faulty agents return unrelated search results, and PANDA replaces them through its swap protocol. However, an unrelated or incorrect result is not necessarily distinguishable from a valid intermediate result. What specific mechanism decides when to replace an agent? Does this decision rely on additional LLM-based verification, and if so, how are verification errors and their computational costs accounted for?

4 comments

We thank the reviewer for their careful review and constructive feedback, and address their comments below. The submission has been updated accordingly, with all changes shown in blue\color{blue}{\text{blue}}.

Trust and orchestration PANDA's governance and orchestration mechanisms are designed to work together. Agents administered by different entities can form teams provided that their certificates establish the required trust relationships. During team assembly, agents use the governance framework to identify trusted peers with the required capabilities, and orchestration proceeds among the selected agents. Thus, independent administration does not prevent team formation or execution; rather, the governance model determines which agents are eligible to collaborate.

We agree that the governance experiments focus on trust propagation and do not directly evaluate cross-administrative team execution. We clarify how governance interacts with team formation and orchestration in Section 2.4 as well as the scope of the current evaluation in Section 3.

Team formation under asymmetric trust Under PANDA's current implementation, team assembly does not enforce pairwise mutual trust. Thus, in the reviewer's example, A, B, and C may be selected for the same team even though B does not trust C. We acknowledge this limitation and propose an extension that enforces mutual trust during team assembly. For Tables 2–4, we assume mutual trust, so team formation can proceed while respecting governance. We clarify this assumption in Section 3.

Extending team assembly. During probing, the entrypoint sends the candidate set P=⋃c∈CPcP=\bigcup_{c\in C}P_c, where ∣Pc∣=d⋅r(c)|P_c|=d\cdot r(c). Along with its load, each probed agent ai∈Pa_i\in P returns a trust vector bi∈{0,1}∣P∣b_i\in\{0,1\}^{|P|}, where bi[j]=trust(ai,aj)b_i[j]=\texttt{trust}(a_i,a_j). The entrypoint extends the assembly objective for both consolidated and dedicated matchings with the mutual-trust constraint:

\forall i\ne j\in\bigcup_{c\in C}\phi(c).$$ This constraint can be incorporated into the greedy heuristic used for consolidated matching. Since $|P|$ depends on team size rather than collective size, local assembly computation remains independent of collective size. *Extending gossip search.* The broadcast includes the current team $A'$. A candidate $a_j$ responds only if $\texttt{trust}(a_j,a_i)=1$ for every $a_i\in A'$. Existing team members then verify the reverse relationships, accepting $a_j$ only if $\texttt{trust}(a_i,a_j)=1$ for every $a_i\in A'$. Asymmetric trust may increase candidate rejections and reduce the probability of successful team assembly. Potential mitigations include (1) increasing the oversampling factor $d$ and (2) preferring candidates that are nearby in the trust graph, where mutual trust may be more likely. These extensions preserve the independence of local assembly computation from collective size, although assembly success depends on the availability of mutually trusted agents. Finally, we note that large-scale P2P systems, from file sharing to blockchains, rely on baseline assumptions about honest participation or trusted peers. For example, EigenTrust uses pre-trusted peers to establish reputation, secure P2P routing assumes a bounded fraction of malicious participants, and Bitcoin assumes that honest nodes control a majority of computational power [1–3]. We formalize this extension in Appendix A. [1] Kamvar, S. D., Schlosser, M. T., and Garcia-Molina, H. "The EigenTrust Algorithm for Reputation Management in P2P Networks." WWW 2003. [2] Castro, M., et al. "Secure Routing for Structured Peer-to-Peer Overlay Networks." OSDI 2002. [3] Nakamoto, S. "Bitcoin: A Peer-to-Peer Electronic Cash System." 2008.

Certificate assurance We assume that members of the collective are trusted and that certificate issuers verify an agent's identity and advertised capabilities out of band. Certificates bind these claims to the agent's public key and identity, while certificate chains allow agents to extend trust through attestations made by trusted issuers. Importantly, certification does not independently guarantee that an agent can reliably perform its advertised capabilities. As the reviewer correctly observes, inaccurate capability claims certified by a trusted issuer may propagate through the trust graph.

Our threat model does not consider malicious agents or issuers, and we acknowledge that the proposed governance mechanism does not provide Byzantine resilience. Capability verification is orthogonal to our contribution; developing mechanisms to verify agent capabilities remains an active research area, and such mechanisms could be incorporated into PANDA [1]. Designing Byzantine-resilient distributed governance is outside the scope of this work and is the subject of ongoing parallel research. We clarify these assumptions and limitations in Section 2.3.

Orchestration failure detection PANDA uses LLM-based verification to assess whether an intermediate result is consistent with its assigned subtask. A failed verification triggers the swap protocol. We do not consider malicious agents deliberately attempting to evade the failure detector. Nevertheless, LLM-based verification is imperfect, and plausible but incorrect results may go undetected. Reliably vetting intermediate results remains an open problem: even with access to the full execution trace, state-of-the-art systems struggle to identify which agent step was erroneous [2]. Reliable online verification is outside the scope of this work. All verification calls are included in the reported runtimes and budgets. We have clarified these details in Section 3.

Ethics We noticed that the ethics flag was set and we were not sure if this was a mistake because there was no specific comment about this. Note that our experiments do not involve any attacks, we just inject failures. We would be happy to clarify any specific questions the reviewer has.

[1] South, T., et al. "Position: AI Agents Need Authenticated Delegation." ICML (Position Papers) 2025.

[2] Zhang, S., et al. "Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems." ICML 2025.

Dear reviewer kcDs,

Thank you again for your review. With the discussion period closing soon, we wanted to make sure our responses reached you and check if our clarifications address your concerns. We are happy to clarify further or run additional experiments if that would help your reviewing.

Best, The Authors

Thank you for the clarifications on the interactions between governance, team formation, and orchestration. I maintain my original rating.

Reviewer_vQiEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PANDA addresses a practical problem in running a shared network of LLM agents. A task may need only a few specialists, yet the pool from which to recruit them can be large and its membership can change during execution. The entrypoint uses its local registry to find trusted agents with the required capabilities. Once the team forms, detailed task communication stays within the team, which can execute the plan under different topology. If a worker fails, the system can reassign its unfinished step using state retained by the team.

Critical strengths
  1. PANDA separates network membership from task coordination. An agent can make its capabilities known to potential collaborators, while the details of an ongoing task stay within the team doing the work. A team can therefore give one orchestrator responsibility for its plan even when its members belong to independently managed organizations. This gives the rationale for supporting several coordination patterns on the same network.

  2. PANDA gives each agent a common interface for advertising its capabilities and accepting work from multiple teams. A retrieval agent can serve a central planner in one workflow and pass evidence directly to its successor in another, using the same retrieval code. This lets developers build a specialist before they know which teams will use it or how those teams will coordinate. I see this as a practical strength for independently maintained agents.

Critical weaknesses

My main concern focused on its actual effect.

  1. Figure 3 measures team assembly with fully connected peers whose capabilities are drawn from a fixed set of 100. As the network grows, most capabilities gain more providers, while PANDA probes a bounded number of candidates. This explains why assembly time levels off once the entrypoint knows enough providers. In deployment, an incomplete or stale registry can force the entrypoint to broadcast a request, so the cost of discovery depends on how often local lookup fails and how long those requests take. I remain skeptical about its end-to-end scalability.

  2. Tables 2–4 compare complete systems, so the reported gains reflect several design choices at once. In the crash experiment, PANDA returns an answer for every task, while all three baselines stop without one. This establishes the value of recovery in the tested setting, but the comparison leaves open how much of that benefit a checkpoint and a backup worker would provide. A centralized registry using PANDA’s own planner and recovery protocol would be suggested to isolate the contribution of decentralized discovery.

  3. PANDA preserves completed steps when it replans, which lets a replacement worker resume from earlier results. This works well when a worker crashes before returning an output. The orchestration fault experiment also gives the system a relatively clear signal because the faulty worker consistently returns unrelated search results. A relevant passage containing the wrong fact could be accepted and used downstream, and the experiment leaves open how PANDA would identify and revisit the affected steps.

Questions for authors
  1. What does each peer represent in the million-peer experiment, and which operations are included in Figure 3’s assembly time?

  2. What assurance does a certificate provide about an agent’s advertised capabilities, and how does that assurance relate to the online status measured by trusted precision?

4 comments

We thank the reviewer for their careful review and constructive feedback, and address their comments below. The submission has been updated accordingly, with all changes shown in blue\color{blue}{\text{blue}}.

Scalability Claims We acknowledge that Figure 3 may be confusing and will clarify both what it measures and the assumptions underlying our scalability claims. Figure 3 presents simulation results showing that the local computation required for team assembly remains constant as the collective grows. This computation consists of (1) selecting candidates to probe, with an oversampling factor of d=2d=2, and (2) selecting a team based on the probe responses.

Since all probes are issued in parallel, end-to-end assembly time additionally depends on the slowest probe's round-trip time (network latency and probe processing), bounded by a timeout for unresponsive peers. If no known provider offers a requested capability, assembly falls back to gossip search and incurs the full search timeout.

Under two assumptions, end-to-end assembly time remains constant with respect to collective size: (1) each node has sufficient bandwidth to handle its inbound and outbound probes, and (2) the ratio of agents requesting a capability to agents providing it remains fixed as the collective grows.

Our scalability argument relies on decomposing the system into team assembly and subsequent team communication. Figure 3 demonstrates constant local computation for assembly, while the established overlay enables communication within the formed team without introducing collective-size-dependent costs. Although we have not evaluated the complete system end-to-end with 10610^6 agents, our results support the claim that PANDA can scale without introducing costs that grow with collective size, subject to the stated assumptions.

If registries are stale, discovery may fail; however, our failure detector limits how long stale entries persist. The expected time to detect a failure is T⋅ee−1T\cdot \frac{e}{e-1}, where TT is a tunable parameter controlling the failure-probe rate. If needed, candidate oversampling (dd) can be increased to reduce the probability of not finding a live agent. We have clarified these questions in Section 3.1.

Comparison with a checkpointing baseline We conducted a semi-synthetic ablation measuring the effect of worker failures on task completion time and display the results in the paper in Figure 5. Over 5000 simulated trials across the three topologies, we sample the number of subtasks, and the duration of each subtask, uniformly from 1000 recorded runs of each topology. We then define the overhead added upon a failure in a centralized checkpointing case vs PANDA. With checkpointing the two main costs incurred are (1) the time it takes to recover and (2) how much work was lost reverting to the checkpoint. In PANDA, once we detect a failure, we immediately swap in a healthy agent, avoiding recovery latency and losing no completed work -- only the in-flight subtask is retried. Given PANDA’s failure detector, we detect a failure with probability 1−1e1 - \frac{1}{e} every TT seconds [1] and we model this timing in our experiment. We then evaluate how task completion time increases as the probability of a subtask failing increases and demonstrate that PANDA with a reasonable T=3T = 3, performs significantly better than any recovery baseline even assuming a relatively fast 15 second recovery and immediate checkpointing. We present the results of this experiment in Appendix C.

Why not a centralized registry A centralized registry using PANDA's recovery protocol would retain a single point of failure, as registry unavailability would prevent discovery of replacement agents. PANDA's decentralized discovery eliminates this dependency, allowing recovery to proceed without relying on a centralized service.

[1] Gupta, I., Chandra, T. D., and Goldszmidt, G. S. "On scalable and efficient distributed failure detectors." PODC 2001.

Detection and recovery from incorrect intermediate results The reviewer is correct that a plausible but factually incorrect intermediate result, if accepted, can corrupt downstream subtasks. PANDA's recovery mechanism addresses failures that can be detected, but does not guarantee detection of all semantic errors in agent outputs. If subsequent evidence reveals that an earlier result was incorrect, replanning can instruct subsequent subtasks to disregard that result. However, automatically identifying erroneous intermediate results and determining which downstream steps must be revisited remains an open problem. Even with access to the full execution trace, state-of-the-art systems struggle to reliably identify the erroneous agent step [1]. Reliable online detection and recovery from such errors are beyond the scope of this work. We have clarified this in Section 2.5.

Certificate assurance We assume that members of the collective are trusted and that certificate issuers verify an agent's identity and advertised capabilities out of band. Certificates bind these claims to the agent's public key and identity, while certificate chains allow agents to extend trust through attestations made by trusted issuers. Importantly, certification does not independently guarantee that an agent can reliably perform its advertised capabilities. Capability verification itself is orthogonal to our contribution; developing such mechanisms remains an active research area, and existing approaches could be incorporated into PANDA [2]. We clarify our assumptions in Section 2.3.

Trusted precision is a separate metric, defined as the fraction of trusted peers that are currently reachable. Certificates concern an agent's verified capabilities, whereas trusted precision measures the availability of trusted peers. An unreachable agent does not prevent team formation, as it will fail to respond to task dispatches and will be detected and evicted by the failure detector.

[1] Zhang, S., et al. "Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems." ICML 2025.

[2] South, T., et al. "Position: AI Agents Need Authenticated Delegation." ICML (Position Papers) 2025.

Dear reviewer vQiE,

Thank you again for your review. With the discussion period closing soon, we wanted to make sure our responses reached you and check if our clarifications address your concerns. We are happy to clarify further or run additional experiments if that would help your reviewing.

Best, The Authors

Thank you for your clarifications.

Your responses helped clarify several aspects of the paper, in particular, the scope boundary regarding subtle semantic errors in intermediate outputs, and the exact definitions of out-of-band capability assurance and trusted precision.

However, my core concerns regarding the empirical validation of end-to-end scalability and isolated architectural gains remain:

End-to-end Scalability: Thank you for clarifying that Figure 3 measures only local in-memory candidate selection and excludes network RTT, gossip state synchronization, and fallback broadcast searches. Given that the end-to-end HotPotQA experiments run with only 10 worker agents, the actual network-level cost of decentralized discovery (especially under partial/stale registries requiring gossip fallback) remains unverified at scale.

Isolation of Contributions & Checkpointing Baseline: While a centralized registry introduces a single point of failure conceptually, comparing against it was suggested to empirically isolate the overhead/benefit of PANDA's decentralized discovery from its DAG planner and local scheduling in the 10-worker experiments. Furthermore, in the new semi-synthetic ablation, assuming a 15-second recovery penalty and lost progress for a centralized checkpointing baseline appears overly pessimistic compared to lightweight task-state retention (similar to what PANDA-STAR's orchestrator already maintains) paired with simple worker retry.

I appreciate the improvements made to the clarity. I keep my original score.

Reviewer_4Db4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents PANDA, a decentralized architecture for LLM-based multi-agent systems that combines decentralized agent discovery, capability-based team assembly, web-of-trust governance, flexible star/chain/mesh orchestration, and failure recovery. It evaluates synthetic team-assembly scaling, HotPotQA performance and fault injection, and trust propagation under churn.

Critical strengths
  1. The architecture makes concrete, technically sensible separations between collective-level and team-level operations. Sections 2.2–2.5 distinguish gossip-based collective discovery from reliable task-team communication, and pair capability matching with load-aware team formation and local recovery. This is consequential because it identifies a reasonable systems abstraction for avoiding a global coordinator while retaining task-scoped reliable execution.

  2. PANDA demonstrates meaningful recovery in the tested infrastructure-failure setting. In Table 3, all three PANDA topologies complete every injected-crash task, whereas the reported baselines complete none under the stated protocol. This is direct evidence that the swap/recovery mechanisms can be effective under a severe fault model, even though its broader generalization remains unestablished.

Critical weaknesses
  1. The central scalability and fault-tolerance claims substantially exceed the scope of the evaluation. The abstract and Section 3 claim scaling to thousands or millions of agents under heavy workloads, with efficient task execution and robust recovery. However, the 10610^6-agent evidence in Figure 3 is only a synthetic team-assembly microbenchmark: peers advertise randomly sampled capabilities, and the measurement excludes HotPotQA execution, LLM inference, task concurrency at scale, and the actual star/chain/mesh workflows. In contrast, the end-to-end HotPotQA evaluation in Section 3.1 uses only 10 worker agents, 200 tasks, and concurrency five. Thus, Table 2 supports a speed-accuracy tradeoff in a small deployment, not the headline claim that the full system scales under large, heavy workloads.

    The fault claims are similarly overstated across fault types. Table 3 supports 100%100\% completion under the specific infrastructure-crash injection, but Table 4 does not report completion under orchestration failures. Its nonzero EM/F1 for all baselines indicates that they do complete some tasks under the injected faulty-agent condition, contrary to language suggesting that existing systems fail while PANDA maintains full completion. The paper should sharply separate the assembly microbenchmark from end-to-end scalability, narrow the fault-completion claim to the infrastructure-failure experiment, and report end-to-end experiments at materially larger agent counts and concurrency. At minimum, it should also provide repeated runs or uncertainty estimates and enough configuration detail to assess whether the runtime differences in Table 2 are robust and comparably tuned across systems.

Questions for authors
  1. [Clarification] In Figure 3 and Section 3.1, are the collectives up to 10610^6 agents instantiated as communicating peers, simulated registry entries, or another abstraction? Please specify the hardware/network deployment and the work performed per peer. This would determine how strongly Figure 3 supports the claimed million-agent scalability.

  2. [Clarification] For the HotPotQA results in Table 2, were all systems given equivalent agent models, tool access, prompts, task budgets, retries, and tuning budgets? Were the reported results aggregated over independent runs? Evidence that the speed and accuracy trends persist across comparable repeated runs would increase my confidence in the headline efficiency comparison.

  3. [Clarification] In Table 4, what were the completion rates for each system under orchestration failures? If all systems completed tasks but often returned incorrect answers, the paper should distinguish this result from the 100%100\% completion result under infrastructure failures in Table 3.

2 comments

We thank the reviewer for their careful review and constructive feedback, and address their comments below. The submission has been updated accordingly, with all changes shown in blue.\color{blue}{\text{blue}}.

Scalability Claims We acknowledge that Figure 3 may be confusing and will clarify both what it measures and the assumptions underlying our scalability claims. Figure 3 presents simulation results showing that the local computation required for team assembly remains constant as the collective grows. This computation consists of (1) selecting candidates to probe, with an oversampling factor of d=2d=2, and (2) selecting a team based on the probe responses.

Since all probes are issued in parallel, end-to-end assembly time additionally depends on the slowest probe's round-trip time (network latency and probe processing), bounded by a timeout for unresponsive peers. If no known provider offers a requested capability, assembly falls back to gossip search and incurs the full search timeout.

Under two assumptions, end-to-end assembly time remains constant with respect to collective size: (1) each node has sufficient bandwidth to handle its inbound and outbound probes, and (2) the ratio of agents requesting a capability to agents providing it remains fixed as the collective grows.

Our scalability argument relies on decomposing the system into team assembly and subsequent team communication. Figure 3 demonstrates constant local computation for assembly, while the established overlay enables communication within the formed team without introducing collective-size-dependent costs. Although we have not evaluated the complete system end-to-end with 10610^6 agents, our results support the claim that PANDA can scale without introducing costs that grow with collective size, subject to the stated assumptions. We have clarified this distinction and the scope of our evaluation in our revision in Section 3.1.

Repeatability of Results We ran each system in Table 2 for 5 independent trials and report the results below; overall trends from before hold, and the runtime deviation is small across trials, demonstrating robustness to agentic stochasticity. All systems use the same underlying model, tools, task budgets, and retry policies; the only per-system differences are the prompts needed to instruct each architecture's agents on how to operate in their environment. Baselines are used largely as originally implemented, with the same HotPotQA-specific prompting and answer format applied uniformly for consistency. No task-specific parameter tuning was applied to PANDA; our effort was focused on building a generalizable system rather than optimizing for HotPotQA. We have clarified the experimental design in Section 3, and updated Table 2 and the corresponding discussion with the updated experiments.

Table: Performance on HotPotQA benchmark without failures (mean ± std over 5 runs, 200 questions each).

Complete KnowledgePartial Knowledge
SystemEM (%)F1 (%)Total RT (s)EM (%)F1 (%)Total RT (s)
Magentic-One46.7 ± 1.554.9 ± 1.3927 ± 6022.0 ± 2.825.8 ± 3.81243 ± 36
AgentNet68.2 ± 1.477.2 ± 1.2922 ± 6440.9 ± 1.048.5 ± 1.31116 ± 74
Internet of Agents70.3 ± 1.878.4‾\underline{78.4} ± 1.41884 ± 9552.6‾\underline{52.6} ± 2.160.1‾\underline{60.1} ± 2.52977 ± 221
PANDA-star65.0 ± 1.474.9 ± 2.0528‾\underline{528} ± 2351.2 ± 1.459.2 ± 1.5676 ± 29
PANDA-chain59.2 ± 1.569.5 ± 1.5510 ± 1948.1 ± 1.856.5 ± 2.3687‾\underline{687} ± 32
PANDA-mesh69.8‾\underline{69.8} ± 1.478.6 ± 1.0815 ± 6457.1 ± 2.266.1 ± 2.5806 ± 49

Completion Rates In Table 4, the completion rates were 100% for all systems. We acknowledge that Table 4 demonstrates that existing systems can sometimes resolve orchestration failures -- either through chance or through redundant knowledge in the remaining agents. However, the results show that because they lack a dedicated failure-recovery mechanism, orchestration failures have a disproportionate impact on their output quality compared to PANDA. We have clarified this in Section 3.2.

Dear reviewer 4Db4

Thank you again for your review. With the discussion period closing soon, we wanted to make sure our responses reached you and check if our clarifications address your concerns. We are happy to clarify further or run additional experiments if that would help your reviewing.

Best, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.