- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper challenges the assumption that applying classical "least privilege" (partitioning tools across specialized sub-agents) improves security in multi-agent LLM systems. Using SAFEMAS—an empirical benchmark spanning 4 domains (Banking, DevOps, Healthcare, Workspace), 240 tools, 24 multi-step tasks, 5 team topologies, and 3 LLM backbones—the authors evaluate 990 scenarios to show that tool partitioning merely relocates the attack surface. Under identical prompt injection attacks, orchestrated teams fail at rates similar to a single agent (39%–40% vs. 41%), but total attack success climbs to 53% once inter-agent channels are targeted. Inter-agent message tampering proves most lethal (76% success rate overall, up to 85% on GPT-5.6 Luna) because orchestrator delegation strips away original prompt provenance, causing sub-agents to trust tampered messages implicitly. Furthermore, partitioning severely degrades utility, dropping task completion from 90% (single agent) to 42%–49% while increasing LLM calls by 16× to 60×.
Critical strengths
Challenging Core Architectural Dogma: Demonstrates that rigid tool partitioning creates vulnerable communication channels rather than reducing exposure.
Deterministic Evaluation Grounding: Evaluates outcomes via post-run database state diffs rather than subjective LLM-as-a-judge scoring.
Comprehensive Vector Taxonomy: Disentangles instruction injection, indirect tool poisoning, and in-transit message tampering across multiple agent roles.
Shared-Read Counterfactual Control: Proves that the performance tax stems from data isolation rather than team coordination.
Critical weaknesses
Single-Run Sampling (N=1): Point estimates without variance bounds due to high multi-agent API costs. Suggestion: Run Monte Carlo repetitions on a 10% sub-sample to establish confidence intervals. (BUT, I understand that the entire array of experiments is vast, so that would be prohibitive, unless the authors choose a critical subset of the overall array in order to formally establish significance).
Unweighted Output Severity: Evaluates all unauthorized writes equally regardless of real-world impact. Suggestion: Categorize state changes by risk level (e.g., log edit vs. financial transfer).
Static Adversary Strategy: Uses fixed single-payload injections. Suggestion: Test adaptive, multi-turn prompt optimizations.
Questions for authors
How would message authentication protocols (e.g., cryptographic signatures) affect the success rate of the message tampering vector?
Did any orchestrator prompt-engineering patterns successfully preserve origin provenance during instruction delegation?
Did the expanded read scope in the shared-read counterfactual introduce new indirect tool-poisoning pathways?
Would you consider using the results of your large experimental array as an initial; exploration, and from there design a smaller, but more formal experiment with enough repetitions to establish 10% confidence intervals?
2 comments
Dear Reviewer drde,
Thank you again for your review. As the discussion period closes today, we wanted to check whether our response and the revised manuscript address your concerns, in particular the repeated runs with confidence intervals and the discussion of message authentication. If any point remains unclear, we would be glad to clarify it before the discussion closes.
Thank you for your time and consideration.
How would message authentication protocols affect the success rate of the message tampering vector?
Authenticating messages and protecting their integrity, for example with a MAC, removes this vector. We include it because prior work treats the channel as an attack surface (arXiv 2502.14847, 2511.05269, 2606.30602), which lets us compare it with the other two vectors. Two caveats: agent-to-agent authentication needs a key infrastructure that is not yet standardized (draft-klrc-aiagent-auth-02), and it does not stop an instruction injected at the orchestrator, which the orchestrator then sends itself in an authentic message. This is now in Section 6.
Did any orchestrator prompt-engineering patterns successfully preserve origin provenance during instruction delegation?
We did not vary the orchestrator's prompt. Rerunning the scenarios with other prompts would be informative, but reaching the same statistical confidence would need a comparable number of runs for each prompt, which we could not afford for this revision.
Did the expanded read scope in the shared-read counterfactual introduce new indirect tool-poisoning pathways?
No. With shared reads a poisoned tool return succeeds in 17% of runs against 24% with partitioned reads, and no position becomes easier to attack across the three models (Section 5.3, Table 7).
Would you consider a smaller, more formal experiment with enough repetitions to establish 10% confidence intervals?
We now run every scenario three times and report 95% intervals throughout (Section 5, Table 4). The headline comparisons are within ±7 percentage points. Single cells of Table 4 are wider (±7 to ±15 percentage points for attack success), because their width is set by the number of distinct attacks (42), which further repetitions do not change.
Unweighted output severity.
Every successful attack counts the same in the paper, and we now state this as a limitation (Section 6, limitation 4). If the reviewer finds it necessary, we can compute a severity-weighted analysis from the released traces. We did not plan severity levels before running the experiments, so the result would be post hoc and its conclusion weaker, which is why it is not in the current draft.
Static adversary strategy.
Our attacker places one fixed payload at one position, so every rate is a floor for an adaptive attacker (Section 6, limitation 1).