- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces MASCSBench, a benchmark of 10,162 multi-agent conversations designed to study safety risks that emerge from combining individually benign responses. It shows that full-context evaluation detects substantially more risks than response-level evaluation (61.13% vs. 12.63%). The authors also introduce a Risk Propagation Graph (RPG) to analyze how risk-related information accumulates across agents and turns.
Critical strengths
The paper presents a relatively large benchmark with 10,162 multi-agent conversations, supported by human annotations, benign controls, and evaluations across multiple models and frameworks. The authors have also made an effort to document the dataset construction and validation process.
Critical weaknesses
-
The paper is poorly written, with excessive defensive language (e.g., “not as a competing deployable detector or as evidence that graph structure alone improves detection” and “not a deployable baseline and not a claim that role-edge structure alone explains the context gap”). Similar statements are repeated throughout the paper. Citation formatting is also consistently problematic (e.g., “content production Hong et al. (2024); Liu et al. (2024)”), making the paper harder to follow.
-
The paper does not establish why this is a distinct multi-agent safety problem. The benchmark is constructed to make individual responses appear safe, so the main performance gap is partly expected by design. Moreover, simple keyword filtering (60.50%) performs nearly as well as the full-context classifier (61.13%), raising doubts about whether the benchmark captures genuine semantic reconstruction.
Questions for authors
-
What makes this a distinct multi-agent safety problem rather than a standard multi-turn safety problem?
-
Given that keyword filtering (60.50%) performs nearly as well as the full-context classifier (61.13%), what evidence shows that the benchmark captures semantic reconstruction rather than keyword accumulation?
-
How do the authors validate that recovering harmful intent from individually benign responses represents a genuine safety failure?
Thank you again for your constructive feedback and for taking the time to consider our rebuttal.
As the discussion period is coming to an end, we would appreciate knowing whether our responses have addressed your main concerns. If so, we would be grateful if you could consider updating your score to reflect these clarifications.
Thank you for your time and consideration.