- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MAStrike, a Shapley-guided framework for collusive red-teaming in hierarchical multi-agent systems. It uses agent-level Shapley values and pairwise interaction indices to identify influential coalitions, followed by coordinated, role-aware adversarial manipulation and iterative refinement. The paper also introduces MABench, covering finance, software engineering, and CRM. Experiments across three frontier models demonstrate substantial improvements in attack success rates, supported by ablation studies and evaluation on TAMAS.
Critical strengths
Strong empirical evaluation. MAStrike achieves substantial ASR improvements across three domains and three frontier models. The authors also provide attribution validation, estimation reliability analysis, component ablations, and evaluation on an independently constructed benchmark.
Critical weaknesses
-
Restricted threat model and evaluation scope. The reported attacks modify compromised agents' system prompts, assuming substantial attacker control. Moreover, MABench consists of constructed hierarchical MAS with distributed safety gating. Although these settings are appropriate for studying collusion, the practical applicability of the findings to less privileged attacks and broader MAS deployments remains uncertain.
-
Incomplete assessment of Shapley estimation efficiency and transfer. While the paper evaluates estimation reliability and per-case attack costs, the total offline coalition-evaluation overhead is not fully reported. Furthermore, the TAMAS evaluation uses exact per-instance Shapley computation rather than the offline task-similarity-based transfer procedure used in MABench. This limits the evidence for the efficiency and generalizability of the complete selection pipeline under consistent evaluation conditions.
Questions for authors
-
What is the total computational overhead of offline Shapley estimation, and how does it affect the amortized cost of red-teaming compared with the baselines?
-
Can the authors evaluate TAMAS using the same offline Shapley estimation and task-transfer procedure adopted in MABench, rather than exact per-instance coalition enumeration?
We thank the reviewer for the positive assessment and the two questions.
The selection cost is paid once, offline, on synthetic reference tasks, and reused for every benchmark task, every target model, and every k. The baselines pay their generation cost on every target case with no reuse.
Thank you for the suggestion. We will add the Shapley estimation for TAMAS in the revision as well.