- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents MOSAIC, an outcome-aware security auditing framework for LLM-based Multi-Agent Systems. It introduces MOSAIC-Bench, a dataset of 43,844 execution trajectories covering diverse attacks and communication structures, with annotations for attack outcomes, component localization, and supporting evidence. The authors further develop MOSAIC-Auditor through supervised fine-tuning to generate structured audit reports. Experiments show that MOSAIC outperforms several baselines in attack outcome classification and localization.
Critical strengths
-
The research problem is practically meaningful. The paper clearly distinguishes attack exposure from realized compromise. This distinction is particularly important for multi-agent systems, as an individual agent receiving or following malicious instructions does not necessarily lead to a system-level security compromise. By focusing on actual execution outcomes rather than isolated anomalous behaviors, the paper addresses a well-motivated and practically relevant security auditing problem.
-
The benchmark is systematic and comprehensive. MOSAIC-Bench covers diverse attack objectives, injection surfaces, communication topologies, and structural placements, with over 40,000 execution trajectories. Beyond attack success labels, the benchmark also provides component localization and event-level evidence annotations, supporting more fine-grained security auditing research. This is a valuable contribution of the paper.
-
The experimental evaluation is relatively comprehensive. The authors compare against multiple baselines and further investigate the transfer of single-agent auditors to multi-agent settings and the effects of different input representations. The results demonstrate that MAS-specific training significantly improves attack outcome classification and component localization. The paper is also clearly organized, with methods and experimental settings that are easy to follow.
Critical weaknesses
-
The paper combines outcome assessment, attack profiling, localization, and evidence attribution into a single generation task. However, it is unclear whether this unified approach is better than handling these tasks separately. The paper lacks an ablation comparing these designs and does not discuss how conflicting predictions or errors across different audit outputs are handled.
-
The contribution of graph-based representation seems limited. In Table 2, Graph SFT only slightly outperforms Flat SFT, with improvements of less than one percentage point. While the flat representation still contains some structural information, the results do not clearly show how much the explicit graph representation contributes to auditing performance.
-
The attack construction relies on predefined injection surfaces and structural placements, with at most two injection sites per attack. While this design makes the experiments reproducible, it may not fully capture more complex attacks involving dynamic propagation across multiple agents. Moreover, localization mainly identifies where an attack was injected, rather than which components actually contributed to the final compromise. The paper could make better use of its controlled execution environment to investigate these relationships.
Questions for authors
-
Have the authors considered comparing unified auditing with independent prediction or a modular pipeline? How does the current approach handle inconsistencies between different audit outputs?
-
In which scenarios does the explicit graph representation provide the most benefit? Would the performance change significantly if some communication edges were removed or modified?
-
Could the authors evaluate more complex attack propagation scenarios? In particular, could interventions on intermediate agents or communication edges help verify whether the localized components actually contribute to the final security outcome?
4 comments
Follow-up on Our Response
Dear Reviewer VQsS,
Thank you again for your thoughtful feedback. We would appreciate knowing whether our response and revisions have adequately addressed your concerns. If any issues remain, we welcome further discussion and would greatly value your guidance. If your concerns have been resolved, we would be grateful if you could consider updating your evaluation accordingly.
Thanks for the rebuttal response. I have also read your summary of common concerns, which are also helpful. My concerns are addressed. I have raised my score accordingly.
Thank you very much for your thoughtful evaluation and positive recognition of our work. We sincerely appreciate your feedback and are happy to discuss any further questions or concerns.
Response to Reviewer VQsS
We thank Reviewer VQsS for recognizing the practical importance of the problem, the systematic design of MOSAIC-Bench, and the comprehensive evaluation. We address the concerns below.
W1 / Q1. Comparing unified auditing with independent prediction or a modular pipeline
We revisited a controlled MARBLE–AppWorld experiment (3,122 training; 406 validation trajectories). Unified Qwen SFT generated the full report directly; the modular pipeline froze the same Qwen representation, trained separate
verdict,scope, andcomponentheads, and used conditional SFT for report generation.Comparison between unified auditing and the modular pipeline (%)
Unified auditing gained 9.70 Outcome and 12.08 localization points, showing the benefit of joint cross-field modeling.
MOSAIC avoids post-hoc stitching through unified structured generation. A deterministic check flags any cross-field conflict after decoding without overwriting predictions.
W2 / Q2. Where does the explicit graph representation help most, and what is the effect of modifying communication edges?
We compared matched one-epoch Graph and Flat models across seeds 42–44 on the same 6,207-example test set.
Graph-over-Flat gains by scenario (percentage points)
Graph helped most in responsibility-tracing settings: Research gained +3.35 localization points; Tree and selector-group-chat topologies gained +3.18 and +2.65 points.
We then modified only the inference-time graph for the seed-44 auditor on a fixed 2,000-example subset.
Communication-edge intervention results (%)
Removing edges reduced localization F1 by 2.27 points (paired-bootstrap 95% CI: −4.25 to −0.36) but barely changed outcome prediction, indicating that graph structure mainly supports component attribution.
W3 / Q3. Predefined injection, dynamic propagation, and interventions on intermediate components
Injection surfaces and at most two injection points are predefined for clear exposure, provenance, and outcome labels; subsequent propagation is determined by the MAS runtime. We traced each attack's unique success marker (Appendix A.5) through existing logs.
Cross-agent propagation among successful attacks
Under a strict temporal, non-refusal matching criterion, 29.0% is a lower bound on dynamic cross-agent propagation. The other 71.0% were completed by the injected component. In a 12-run multi-hop pilot, relay agents forwarded the payload in 7 of 9 applicable cases.
Our localization labels capture provenance, not formal causal responsibility. Payload-free controls nevertheless support injection necessity: 0/2,835 produced consensus attack success (Appendix A.9). The revised manuscript now includes the propagation results in Appendix A.10, and the marker paths are released with the benchmark; causal re-execution interventions and more complex adaptive relays remain future work.
We thank Reviewer VQsS again for the careful and constructive feedback, which helped us improve the paper.