- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper shows that LLM multi-agent systems are more vulnerable when compromised agents can coordinate, particularly when those agents are densely connected to each other and to normal agents. They then propose CIV-Amplify, which picks a vulnerable group of attackers using a learned predictor and has an LLM planner assign them coordinated roles. Across three attack types and four topologies, this raises average ASR from about 26.58% to 36.27%.
Critical strengths
-
The distinction between attacker-to-attacker and attacker-to-normal connectivity is a useful lens on collaborative attacks in MAS. Beyond the density-group comparisons, Fig 5 varies one density while holding the other fixed, which gives some controlled evidence that both matters. The evaluation is broad, covering three attack types, four topologies, and several backbones. The average gain in Table 1 is 9.69 points, and it is largest for memory poisoning, which suggests the effect is not limited to a single setting.
-
Fig 6 shows that CIV-Amplify retains an advantage over the original attacks under all three evaluated safeguards. This makes the evaluation more informative than testing only undefended systems.
Critical weaknesses
-
The predictor’s effectiveness for subgraph selection is not sufficiently established. Table 2 reports aggregate metrics but does not directly evaluate ranking among alternative attacker sets within a fixed graph and budget. The predictor is also trained on basic-collaboration ASR, while selection precedes planner optimization, which may affect candidates differently. Fig 7 provides some downstream support, but within-graph selection results after planning and a trained structure-only baseline would better establish the predictor’s value.
-
The contribution of collaboration remains difficult to separate from attacker placement and generic role-based prompting. Tables 1 and 3 evaluate the combined framework, while section 3.2 specifies equal attacker counts without explaining Original’s placement rule. Fig 7’s “w/o Collab.” ablation partly addresses this, but its setting and whether identical attacker sets are retained are unclear; it also does not isolate the benefit of providing topology information to the planner. Matched comparisons with fixed attacker sets would clarify these contributions, while evaluation counts and uncertainty estimates would help assess the smaller gains.
Questions for authors
- How well does the predictor rank alternative attacker sets within a fixed graph and attacker budget, and how close is its selection to the best candidate?
- How does Original select its compromised agents and does Fig 7’s “w/o Collab.” comparison use identical attacker sets and tasks?
- How does the full predictor compare with a simple model trained only on the available structural features?
- What evaluation counts, independent repeat counts, and uncertainty estimates underlie Tables 1–3 and Fig 5–7?
- Do rankings predicted from basic-collaboration ASR remain consistent with the observed rankings after planner optimization?
- Does withholding edge information from the planner, while keeping the attacker set and execution graph fixed, reduce attack effectiveness?
7 comments
Response to Reviewer GLec (Part 2/3)
W1/Q5: It remains unclear whether rankings learned from basic-collaboration ASR remain consistent after planner optimization.
Thank you for this valuable comment. As described in our methodology, the predictor aims to identify structurally vulnerable attacker subgraphs based on basic-collaboration ASR, thereby selecting promising attacker sets before planner optimization. As shown in Figure 2, this design is motivated by the observation that structural factors underlying Collaboration-Induced Vulnerability (CIV) play an important role in determining attacker-subgraph vulnerability. Although planner optimization further enhances attack effectiveness through structure-conditioned instructions, it does not change the underlying attacker subgraphs or their communication topologies. Therefore, the vulnerability rankings learned under basic collaboration can still provide useful guidance for subsequent attacker-set selection. To verify whether the rankings remain consistent after planner optimization, we conducted additional experiments comparing the predicted rankings under basic collaboration with the observed rankings after planner optimization, using Spearman's rank correlation and Top-3 overlap.
Table 5: Ranking Consistency Between Basic-Collaboration Predictions and Post-Optimization Attack Outcomes.
| Attack Type | Spearman's (↑) | Top-3 Overlap (↑) |
|---|---|---|
| Tool Attack | 0.90 | 0.87 |
| Memory Poisoning | 0.97 | 0.93 |
| Overall Avg. | 0.94 | 0.90 |
As shown in Table 5, the rankings remain highly consistent across both attack types. For Tool Attack, Spearman's and Top-3 overlap reach 0.90 and 0.87, respectively, while for Memory Poisoning, they reach 0.97 and 0.93. The overall averages are 0.94 and 0.90, respectively. These results demonstrate that vulnerability rankings learned from basic-collaboration ASR largely remain consistent after planner optimization. This validates our design choice of training the predictor using basic-collaboration data, avoiding the need for additional planner optimization and its associated API costs and computational overhead during dataset construction, while maintaining reliable attacker-subgraph selection.
W2/Q2: Original’s attacker placement rule and whether Fig. 7’s “w/o Collab.” comparison uses identical attacker sets and tasks are unclear, making collaboration gains difficult to isolate.
Thank you for raising this important clarification. As stated in Lines 392–393 of the paper, the Original baseline uses the same number of compromised agents as CIV-Amplify (Ours). Specifically, Original randomly selects the compromised agents, whereas CIV-Amplify (Ours) selects attacker agents based on the predicted vulnerability of candidate subgraphs.
Furthermore, to better demonstrate the effectiveness of CIV-Amplify (Ours), the "w/o Collab." comparison in Figure 7 applies the Original attack strategy to the attacker subgraphs selected by CIV-Amplify (Ours), without introducing additional collaborative instructions. Specifically, it retains exactly the same attacker sets, execution graphs, and evaluation tasks as the Full Method, while disabling the collaboration mechanism. This controlled comparison allows us to isolate the contribution of collaboration from attacker-subgraph selection and placement.
Response to Reviewer GLec (Part 3/3)
W2/Q6: The benefit of providing edge information to the planner remains unclear without a comparison that holds the attacker set and execution graph fixed.
A: Thank you for this insightful comment. In CIV-Amplify (Ours), the planner uses internal communication edges to generate topology-conditioned collaborative instructions. To isolate the contribution of edge information, following your suggestion, we conducted additional controlled experiments comparing the full method with two variants: one without role-specific instructions and the other without structure-conditioned planning (i.e., without providing edge information to the planner). Importantly, all three configurations use identical attacker sets, execution graphs, and evaluation tasks, ensuring a fair comparison.
Table 6: Ablation Study of ASR (↑) for Role-Specific Instructions and Structure-Conditioned Planning under Fixed Attacker Sets and Execution Graphs.
| Attack Type | Full Method | w/o role-specific instruction | w/o structure-conditioned planning |
|---|---|---|---|
| Prompt Injection | 25.79 | 22.37 | 21.58 |
| Tool Attack | 46.32 | 42.11 | 39.21 |
| Memory Poisoning | 36.58 | 35.53 | 29.74 |
| Overall Avg. | 36.23 | 33.34 | 30.18 |
As shown in Table 6, removing structure-conditioned planning reduces the overall ASR from 36.23% to 30.18%, a decrease of 6.05, with consistent reductions across all three attack types. In particular, Tool Attack and Memory Poisoning decrease by 7.11 and 6.84, respectively. Removing role-specific instructions also reduces the overall ASR to 33.34%. These results demonstrate that both role-specific instructions and edge information contribute to attack effectiveness, with topology-conditioned planning providing substantial additional benefits beyond generic role-based collaboration.
W2/Q4: Evaluation counts, independent repeat counts, and uncertainty estimates for Tables 1–3 and Figs. 5–7 are needed to assess the reliability of the reported gains.
A: Thank you for this important comment. To clarify the evaluation scale, taking an 8-agent MAS as an example, each experimental setting involves 19 benchmark tasks with four interaction rounds, requiring agent accesses. Accordingly, a single run of Table 1 requires 58,368 agent accesses, Table 3 requires 72,960, Figure 5 requires 16,416, and Figure 6 requires 19,456 agent accesses. Due to the substantial API costs and execution time, we conducted two independent runs for the agent-based experiments.
For Table 2, predictor training uses the pre-generated dataset constructed as described in Section 2.2, without requiring additional agent interactions. We therefore evaluated the full predictor using five random seeds, obtaining MAE standard deviations of 0.0013, 0.0030, and 0.0020 for Prompt Injection, Tool Attack, and Memory Poisoning, respectively. Given the resource constraints and limited rebuttal timeline, we were unable to conduct additional repetitions. We sincerely hope for the reviewer's understanding and plan to perform more extensive evaluations with additional random seeds in future work.
Response to Reviewer GLec (Part 1/3)
ear Reviewer GLec,
Thanks very much for your thoughtful feedback. We sincerely appreciate your insightful comments on the effectiveness of our vulnerability predictor, the contribution of structure-conditioned collaboration, and the rigor of our experimental evaluation. In response, we have conducted additional experiments and clarified our methodology to better validate the effectiveness and reliability of CIV-Amplify (Ours). Our detailed responses to your comments are provided below:
A: Thank you for raising this important concern. In the paper, Table 2 demonstrates the predictor's effectiveness in estimating attack risks, achieving an MAE of 0.0403, an R2 of 0.8592, and a Spearman correlation (ρ) of 0.9469. Figure 7 further provides downstream evidence: the full method achieves an ASR of 56.84%, outperforming density-based selection (50.52%) and random selection (46.32%).
To directly evaluate the predictor's ranking ability within a fixed graph and attacker budget, following your suggestion, we conducted additional experiments comparing it with a density-based baseline under Top-1 and Top-3 selection settings. As shown in Table 3, our predictor achieves ranking accuracies of 87.50% and 94.44% under Top-1 and Top-3 settings, respectively, outperforming the density-based baseline (75.00% and 86.11%) by 12.50 and 8.33. These results demonstrate that our predictor more accurately ranks alternative attacker sets and identifies highly vulnerable candidates under fixed graph and budget constraints.
Table 3: Top-1 and Top-3 Ranking Accuracy of the Predictor and Density-Based Baseline for Attacker-Set Selection.
Together, these results provide complementary evidence of the predictor's effectiveness in risk estimation, attacker-set ranking, and downstream selection.
A: Thank you for this valuable suggestion. In the paper, Table 2 presents an ablation study demonstrating the contributions of Collaboration Risk, Individual Risk, and Count Risk to vulnerability prediction. Removing these components increases the MAE from 0.0403 to 0.0489, 0.0420, and 0.0422, respectively, confirming that they provide complementary information. However, these ablations do not directly establish the added value of the full predictor over a model using only structural features. To address this concern, following your suggestion, we conducted additional experiments comparing the full predictor with a structure-only baseline across three attack types.
Table 4: Vulnerability Prediction MAE (↓) of the Full Predictor and Structure-Only Baseline Across Attack Types.
As shown in Table 4, the full predictor consistently achieves lower MAE across all attack types, reducing the average MAE from 0.0769 to 0.0461 (40.05%). These results demonstrate that structural features alone are insufficient to fully capture attacker-subgraph vulnerability. By incorporating collaboration risk, individual risk, and count risk, our predictor captures complementary vulnerability information and provides more accurate attack-risk estimation.