- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on “compositional failures” in multi-agent systems: although each individual agent behaves in accordance with its local rules, the agents’ combined actions may violate system-level safety, correctness, or privacy requirements. For example, one agent may legitimately generate a data-export handle, while another may legitimately publish that handle as a shared link. Neither action is problematic in isolation, but their combination may result in cross-tenant data leakage. The paper makes two main contributions: it introduces the MAS-Comp-Bench benchmark and proposes RedExplorer, an automated red-teaming framework.
Critical strengths
-
The paper introduces a relevant benchmark and an automated red-teaming framework.
-
The paper conducts meaningful experiments to evaluate the proposed benchmark and framework.
Critical weaknesses
-
The novelty over prior work is not sufficiently clear. The paper’s definition of compositional failure seems closely related to existing work on emergent failures, cascading failures, privacy leakage through composition, harmful tool/skill composition, and multi-agent collusion. The current differences, such as hidden dependencies and autonomous discovery, seem more like differences in the evaluation setting than a fundamentally new type of failure. The paper should clarify whether its main contribution is a new failure category or a new framework for discovering existing compositional risks, and compare more systematically with related work.
-
The experiments are too limited to support broad generalization claims. The benchmark contains only three synthetic environments, apparently with one fixed instance per domain and little variation in topology, permissions, tools, or system dynamics. More importantly, all four additional failures found by RedExplorer come from the multi-tenant-records environment, while neither method finds any failures in the other two environments. This makes it difficult to conclude that RedExplorer works broadly across MAS settings. More independent environments, held-out variants, and ideally a real or semi-realistic MAS should be evaluated.
-
The main results need repeated runs and uncertainty analysis. The headline result is an increase from one to five discoveries, reported as a 400% improvement, but it is unclear whether the experiments were repeated across independent runs. With such small absolute numbers, and with all additional discoveries coming from one environment, the percentage improvement can be misleading. The experiments should be repeated with multiple seeds, with confidence intervals or other statistical analysis reported. The paper should also emphasize the absolute recall improvement from 4% to 20%.
-
More baselines and ablations are needed to explain why RedExplorer works. The main comparison is against a solo-memory baseline, without stronger search or planning baselines such as random search, beam search, MCTS, evolutionary search, or coverage-guided approaches. The paper also lacks ablations of key components such as the memory agent, orchestration agent, multiple execution agents, and diversity constraint. As a result, it is unclear whether the gain comes from the multi-agent design itself, greater search diversity, additional candidate generation, or simply more computation. Budget-, model-call-, and token-matched baselines and component-level ablations would make the conclusions much stronger.
Questions for authors
-
Does the paper introduce a new type of failure, or does it propose a new evaluation framework for existing compositional risks? I suggest adding a systematic comparison with related work along dimensions such as the failure definition, information visible to the attacker, interaction setting, whether an attack plan is provided in advance, whether discovery is autonomous, and whether a complete failure catalog is available. The novelty claims in the title, abstract, and contribution statements should be adjusted accordingly.
-
Could the authors explain why the evaluation considers only the three scenarios of software engineering, e-commerce operations, and multi-tenant records?
-
In the no-feedback control reported in Table 5, why do both GPT-4o and Claude Opus 4.8 produce zero schema-valid plans, even though the environment descriptions and tool schemas are retained, as stated in Table 9?