- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how communication topology influences the vulnerability of multi-agent LLM systems to adversarial answer propagation. The authors propose a local-to-global prediction framework that models individual agents' answer-switching behavior using a compact set of local features, including their previous answers, node degrees, and the number and identity of neighbors supporting the adversarial answer. These local switching probabilities are estimated through a hierarchical Beta-Binomial kernel and composed into a Markov chain to approximate group-level deliberation dynamics.
The framework is evaluated using three open-weight 8B language models, 600 MMLU-Pro questions, and three memory protocols. Experimental results show that the proposed kernel achieves a mean absolute prediction error of 0.012 on 52 unseen graph-placement configurations after calibration on eight configurations. Additional evaluations investigate larger agent groups, multiple adversaries, extended deliberation rounds, and distribution shifts involving new questions and adversarial messages.
The authors further apply the framework to rank all 112 connected six-agent graphs, revealing that topology safety depends on adversarial placement, model-specific response dynamics, and the initial distribution of agents' answers. The results suggest that centralized topologies are particularly vulnerable under worst-case adversarial placement, while the relative advantages of dense and sparse graphs depend on the initial correctness distribution and agents' conformity behavior.
Overall, the paper provides an empirical framework for predicting adversarial propagation and understanding topology-dependent robustness in multi-agent LLM systems.
Critical strengths
S1. A principled local-to-global framework for predicting adversarial propagation. The paper presents a systematic framework that connects individual agents' local switching behavior with group-level adversarial capture through a Markov-chain approximation. By decomposing complex multi-agent interactions into measurable local transition probabilities, the approach provides an interpretable alternative to repeatedly simulating every candidate communication topology. This is a meaningful methodological contribution because it enables systematic robustness evaluation while reducing the need for exhaustive multi-agent deliberation experiments.
S2. Strong predictive performance with relatively limited calibration data. The framework achieves a mean absolute prediction error of 0.012 on 52 unseen graph-placement configurations using only eight calibration configurations. The authors additionally evaluate alternative graph-level predictors, larger agent populations, multiple adversaries, and longer deliberation horizons. These results provide substantial empirical support for the proposed local-to-global decomposition and demonstrate its potential to reduce the computational cost of topology evaluation.
Critical weaknesses
W1. Limited generalizability to realistic and heterogeneous multi-agent systems. The experimental evaluation is primarily conducted on homogeneous groups of six agents using three instruction-tuned 8B language models. Although the authors provide additional results for larger groups, these experiments cover only a small number of regular graph structures and are conducted without memory. Consequently, it remains unclear whether the proposed framework can reliably characterize heterogeneous agent populations, larger and more irregular communication networks, or systems with specialized agent roles. This limitation reduces confidence in the applicability of the framework to realistic multi-agent deployments.
W2. Strong dependence on simplified state representations and conditional independence assumptions. The Markov approximation assumes that agents make independent decisions conditioned on their local states, while the primary state representation only records whether an agent holds the adversary's answer. Although the paper provides pairwise co-switching tests and a three-state extension, these analyses do not fully establish the validity of the assumptions under more complex interactions. In particular, semantic dependencies among messages, accumulated conversational context, and higher-order correlations between agents may introduce dependencies that are not adequately represented by the proposed kernel. This potentially limits the framework's predictive reliability beyond the controlled experimental setting.
Questions for authors
Q1. Generalization to heterogeneous agent populations. The current evaluation mainly considers homogeneous groups composed of instances of the same language model. How would the proposed kernel handle heterogeneous groups containing agents with different model architectures, capabilities, or communication roles? Would separate kernels be required for different agent types, and how would this affect the calibration cost and predictive accuracy?
Q2. Validity of the conditional independence assumption. The proposed Markov-chain approximation assumes conditional independence among agents given their local states. While the pairwise co-switching analysis provides supporting evidence, could the authors further clarify whether higher-order dependencies or shared semantic information may violate this assumption? In particular, how sensitive are the predictions to dependencies introduced by shared conversation histories or correlated reasoning errors?
Q3. Reliability of topology rankings under prediction uncertainty. The paper indicates that differences in capture rates below approximately 0.03 cannot be reliably distinguished. How many of the reported safest-topology rankings remain statistically distinguishable after accounting for prediction uncertainty and sampling variability? Could the authors provide confidence intervals for the performance gaps between the top-ranked graphs and demonstrate that the recommended topology offers meaningful improvements over competitive alternatives?
Q4. Transferability under stronger adaptive adversaries. The framework requires additional calibration when the question set or adversarial messages change. How robust are the predicted topology rankings when the adversary dynamically changes its target, message content, or attack strategy based on the evolving group state? Could the authors clarify whether the current adaptive-adversary evaluation adequately captures such strategic behavior, or whether the reported ranking stability primarily reflects the restricted attack protocols considered?
Q5. Scalability and calibration efficiency. The paper reports promising extrapolation results for eight- and twelve-agent groups. However, the state space of the exact Markov formulation grows exponentially with the number of agents, and larger groups may introduce local states not covered by the original calibration data. Could the authors provide a more explicit analysis of prediction accuracy, computational cost, and calibration coverage as the number of agents and their maximum degrees increase?
Q6. Applicability beyond multiple-choice deliberation. The proposed state representation relies on tracking whether agents adopt a designated adversarial answer. How would this framework extend to open-ended reasoning tasks, where agents may produce semantically equivalent but lexically different answers or progressively modify their conclusions? Is the discrete answer-state assumption essential for the reported predictive accuracy?