- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes BR-MAS, a Byzantine-resilient multi-agent collaboration framework that integrates decentralized proposal verification, credibility-based voting weights, and adaptive quorum consensus. The authors derive fault-tolerance bounds beyond the classical one-third threshold and evaluate the framework across four benchmarks involving adversarial agents and misinformation attacks.
Critical strengths
-
The paper addresses an important problem in the security and reliability of LLM-based multi-agent systems.
-
The proposed framework integrates multiple mechanisms into a coherent protocol and demonstrates promising empirical improvements across several adversarial scenarios.
-
The evaluation covers multiple benchmarks and model configurations, and supplementary code is provided.
Critical weaknesses
-
Overstated theoretical contribution. The paper repeatedly claims to surpass the classical one-third Byzantine fault-tolerance bound. However, classical BFT concerns agreement among distributed nodes, whereas LLM-based multi-agent collaboration aims to produce correct conclusions through reasoning and discussion. These are different objectives, and agreement does not necessarily imply correctness. Moreover, the proposed bounds rely on stronger assumptions, including trustworthy weight assignment and reliable verification. Therefore, the claimed breakthrough over classical BFT appears overstated.
-
Unreliable credibility estimation. The proposed credibility scores are updated based on proposal acceptance and agreement with the final consensus rather than ground truth. Since consensus outcomes can themselves be manipulated, the mechanism may reward malicious agents and penalize honest dissenters, creating a self-reinforcing cycle of incorrect decisions. The paper does not adequately establish how this risk is prevented.
-
Limited novelty compared with prior work. Credibility-based weighted voting has already been explored in Credibility Governance (arXiv:2603.02640), including under misinformation and adversarial conditions. The paper does not sufficiently clarify how its credibility estimation and weighted voting mechanisms differ from existing approaches.
-
Inconsistent experimental results. Tables 6 and 19 report substantially different accuracies for apparently identical configurations. For example, at f=3 in the same-tier setting, the reported accuracy is 0.385 in Table 6 but 0.615 in Table 19. Such discrepancies raise concerns about the consistency of the experimental setup and the reliability of the ablation conclusions.
-
Insufficient experimental rigor. The MMLU-Pro experiments use only 13 questions per configuration, without adequate statistical analysis. Furthermore, the substantial token savings may primarily result from replacing lengthy discussions with shorter structured proposals, rather than from adaptive quorum consensus itself. More controlled comparisons are needed.
Questions for authors
How should the proposed fault-tolerance bounds be interpreted relative to classical Byzantine agreement, given the differences in objectives and assumptions? What exactly is guaranteed beyond the classical one-third bound?
How does BR-MAS prevent incorrect or manipulated consensus outcomes from corrupting credibility scores and reinforcing subsequent errors?
What are the essential technical differences between BR-MAS and Credibility Governance, particularly regarding credibility estimation and weighted voting?
Could the authors explain the discrepancies between Tables 6 and 19, and provide larger-scale evaluations with statistical analysis and efficiency comparisons under matched token budgets?