- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents ProMeTheUSE, a framework for checking permission violations in multi-agent LLM systems. The authors encodes agents, tools, resources, and delegation relationships as a graph, which then uses permission propagation, reachability checks, and cycle detection to flag security risks such as privilege and data-leakage problems. The authors also propose greedy edge removals to repair violations and add runtime pattern monitoring, verification-gated updates, and a neural classifier for more efficient/faster detection. The evaluation includes about 3,700 synthetic configurations, >1,300 graphs encoded from public agent-framework templates (mostly w/ injected violations, though).
The technical contribution seems rather limited to me. Much of the framework appears to combine existing graph-based security checks with relatively simple monitoring and classification techniques bolded up with complex naming which also lowers readability by a lot. I also don't think the experiments establish how useful the system would be for securing actual multi-agent applications.
Critical strengths
-
The authors use explicit permission checks rather than relying entirely on LLM-based detection. And the underlying graph checks are relatively easy to reason about and/or interpret.
-
There is a decent amount of evaluation across multiple graph corpora. The paper also reports per-class detection results (fig. 7 mainly), robustness to incomplete graphs, as well as verification latency.
Critical weaknesses
-
I don't see much novelty in the core method. Permission closure, graph reachability, SCC-based cycle detection, and greedy edge removal are well-established techniques. The memory and classifier components also seem fairly straightforward to me. It is unclear what new algorithmic insight the combination provides.
-
The evaluation mainly relies on injected violations (why do the authors think these are representative of real-world violations?) in constructed graphs, and assumes accurate permission graphs and contracts to begin with. It is unclear how realistic these assumptions are for actual multi-agent systems, where agent behavior and information flows may not match the specifications. I don't see much evidence that the approach would work reliably in such settings.
-
I am not convinced by the baselines. The classifier is mainly compared against a nearest-centroid model, which tells us relatively little about advantages over existing graph-policy analyzers.
-
The paper formulates minimum-cost mitigation but then apparently uses a greedy edge-removal procedure. No optimality argument and not much analysis of how these approaches affect useful functionality.
-
The figures are so low quality and cluttered and do not seem polished at all for a publication-ready manuscript.
Questions for authors
-
What can the proposed verifier detect that established graph-based permission or access-control analyses cannot? Is there a concrete example that requires the full framework?
-
How would permission graphs and contracts be obtained and validated in deployed multi-agent systems to begin with?? (including cases with undeclared interactions or changing tool behaviors or even where MCP remote tools are integrated into the system)
-
Could you evaluate actual attack scenarios on running agents and compare with stronger graph-analysis baselines?
-
What guarantees, if any, does the greedy repair procedure provide?
Last 24 hours of the reviewer discussion period
Dear Authors and Reviewers
Just a reminder we are soon to be entering the last 24 hours of the reviewer discussion period.
Authors: If you have not yet responded to the reviews and intend to please do so ASAP. Please note reviewers will most likely not have time to respond to any late rebuttal comments so please make sure any comments are clear and not in need of further clarification.
Reviewers: If the authors have responded to your reviews and you have not yet responded please do so, even if it is a one line response acknowledging you have read the rebuttal.
Thanks for all your time, and I hope you all learnt something interesting during the process.
Kind regards,
Your ICLR Area Chair