- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RACG addresses attacks in LLM multi-agent workflows where individually plausible messages, tool outputs, or memory records contribute to an unsafe privileged action through their execution relationships. It represents the observed workflow as a typed event-dependency graph and, at a pre-commit gate, selects from executable message quarantine, memory restoration, and capability narrowing actions. An intervention-conditioned affected-cone model predicts post-intervention risk, task-utility loss, and uncertainty. It is trained with matched checkpoint replays and used with bounded coalition search and rollback-based regeneration. Across nine task benchmarks embedded in multi-agent workflows, the reported macro ASR and DTS are 13.81% and 70.32%, compared with 15.54% and 69.31% for the closest reported model baseline, Full-Graph Late Fusion. Further diagnostics examine typed-relation discrimination, joint intervention effects, operational stress, and transfer.
Critical strengths
- The event-level typed graph, action-valid rollback cut, and pre-commit gate provide a concrete way to act on relational risks instead of merely flagging suspicious text. The paper carefully states that logged dependencies are candidate paths rather than proven semantic causes.
- Relation-matched diagnostics and coalition ablations probe the central claims: composition-pair consistency is 76.41% versus 53.72% for a homogeneous graph, and pair, triple risk MAE is 0.086, 0.113 versus 0.104, 0.139 for Full-Graph Late Fusion. The operational and transfer studies also make some limitations visible.
Critical weaknesses
- The threat setting lists document injection, forged or delayed tool feedback, poisoned memory, and compromised low-privilege agents, but the manuscript does not clearly map each attacker capability to a specified attack construction, budget, and test population. Its main results use standard task benchmarks embedded in multi-agent workflows; the paper does not give enough consolidated detail about those workflows, attack generation, sample counts, repeated runs, and uncertainty intervals to judge how reliably the small aggregate gains generalize. The method also assumes an intact control plane, observable typed events, recoverable state, and pre-commit interception. These restrictions are legitimate, but they should bound the practical conclusion.
- Prior work already combines graph-based multi-agent defense with remediation (e.g., G-Safeguard), and action-level counterfactual replay has been studied for prompt-injection defense. RACG's potentially distinctive contribution is learning the joint safety–utility effects of multiple typed, executable interventions, but the related-work discussion does not compare the exact decision point, intervention space, replay use, and online cost against these approaches. The main improvement over Full-Graph Late Fusion is 1.73 ASR points and 1.01 DTS points, without reported confidence intervals, making the strength of the end-to-end advance hard to assess.
Questions for authors
- For each attacker-controlled channel in Section 3.1, what can the attacker read, replace, delay, or coordinate, and with what knowledge and budget? How many attack instances of each type appear in the nine-benchmark evaluation, and how are relation-matched and coalition cases generated and split across training, calibration, and testing?
- Can the authors report the number of independent task/attack instances and repeated runs, uncertainty intervals for the main ASR/DTS differences, and the exact agent backbones and workflow configurations? In particular, is the 1.01-point macro DTS improvement over Full-Graph Late Fusion stable across independent runs?
- What capability does RACG provide beyond G-Safeguard's graph-based remediation and AttriGuard's action-level counterfactual replay? A component-by-component comparison, and where feasible a matched executable-defense comparison, would materially affect my assessment of novelty.
- What are the absolute end-to-end latency and compute costs, including graph construction, candidate scoring, checkpoint restoration, and regenerated execution? How do they change with workflow length and candidate-set size, beyond the reported fraction of candidate sets scored?
3 comments
Thank you for the response.
My main remaining concern is the statistical support for the headline gain: the confidence intervals in Table 14 are reported only against the main-table Full-Graph Late Fusion configuration, whereas against the fairer LF baseline with matched calibration, budget, and suffix supervision (Table 25) the gain shrinks to 0.70 pp ASR and 0.43 pp DTS with point estimates only.
I will maintain my current assessment.
Response to Reviewer 8564
Thank you for this careful follow-up and for helping us sharpen the statistical interpretation of the headline result. We agree that the calibration-, budget-, and suffix-matched LF baseline is the more informative comparator for assessing RACG’s incremental contribution. The confidence intervals in Table 14 apply only to the original LF configuration and should not be extended to this matched comparison.
Changes in the manuscript. We have added the matched-comparison confidence intervals and the complete bootstrap protocol to Appendix E.1, immediately after the suffix-controlled end-to-end comparison.
We have now directly quantified uncertainty for RACG versus matched LF + Suffix:
| Metric | LF + Suffix | RACG | RACG gain (pp) | Paired 95% CI |
|---|---|---|---|---|
| ASR ↓ | 14.51 | 13.81 | +0.70 | [0.08, 1.32] |
| DTS ↑ | 69.89 | 70.32 | +0.43 | [−0.11, 0.97] |
We used a paired, benchmark-stratified cluster bootstrap with 10,000 resamples. Original task instances (task_id) were resampled within each benchmark, keeping all associated attack variants and repeated executions together. RACG and LF + Suffix remained paired by task and attack scenario. Each resample retained equal weighting across the nine benchmarks; intervals use the 2.5th–97.5th percentiles. These intervals quantify task-instance sampling uncertainty conditional on the fitted models and calibration settings, separately from training-run variability.
We appreciate the reviewer’s emphasis on this distinction. This direct analysis addresses the matched comparator rather than relying on the original LF intervals or treating mechanism-level ablations as substitutes for end-to-end statistical evidence.
Response to Reviewer 8564
We thank the reviewer for constructive suggestions and recognition of our operational design. LF: Full-Graph Late Fusion; pp: percentage points.
Weaknesses
W1. Evaluation transparency and practical scope
Thank you for requesting clearer operating conditions. Appendix C.1, Table 8, now maps nine benchmarks to role chains, attack entries, and protected effects. Security checks actual prohibited writes/calls separately from task correctness. Sections 3.1 and 5 specify trusted control, recoverable states, and pre-commit interception.
Table 29 tests observation faults: 10% missing edges changes ASR/DTS from 13.81/70.32 to 16.18/68.67; adding 10% version misbinding yields 18.76/66.58. Authorization/commit enforcement remains trusted. Q2 addresses variability.
W2. Relation to existing remediation
Thank you for encouraging precise positioning. The introduction now acknowledges G-Safeguard's topology repair and AttriGuard's online counterfactual checks. RACG instead learns joint safety/utility effects of bounded defensive action sets offline, models overlapping affected cones, and selects actions online without replaying every candidate. Action-valid rollback implements the selected actions.
Removing cone intersections increases pair/triple risk MAE from 0.086/0.113 to 0.112/0.151 and worsens ASR/DTS from 13.81/70.32 to 15.26/69.44 (Table 21). Q3 complements this ablation with native-system comparisons.
Questions
Q1. Attack permissions, populations, and diagnostics
Thank you for requesting specificity. Section 3.1 and Appendix C.1 distinguish attackable documents, tool feedback, memory, and low-privilege messages from trusted execution records and authorization. Table 8 identifies the applicable channels per task.
ASR/DTS use attacked workflows; BTS uses benign workflows; macro scores weight nine tasks equally. Blocked proposals alone are not successful attacks. Appendix D, Tables 17-18, reports orchestrator-constructed relation diagnostics separately from downstream defense. These test provenance, permissions, delegation, versions, tool bindings, and sinks.
Appendix A.2.1 groups related prefixes and continuations within splits and uses training-pool information for coalition acquisition. Table 6 reports 1,440/960/560 singleton/pair/triple action sets with 8/12/16 repetitions: 32,000 supplementary evaluation pairs, not training-pool size or independent benchmark attack counts. Table 22 separately varies reference budgets.
Q2. Stability, runs, and backbones
Thank you for examining the modest DTS gain. Against original LF, paired macro gains are 1.73 ASR pp (95% CI [1.11, 2.35]) and 1.01 DTS pp ([0.47, 1.55]). Three-run gains are 1.73/1.64/1.82 ASR pp and 0.90/0.94/1.19 DTS pp; sample SDs are 0.09/0.16 pp (Tables 14-15). DTS intervals for seven tasks include zero, so the claim is aggregate improvement, not significance on every task.
Appendix G.3 identifies GPT-5.5 as source and Qwen3-14B as target, with model identifiers. Tables 36-37 distinguish Frozen transfer from target-assisted calibration/training. Shift tests and repeated runs are separate analyses.
Q3. Native G-Safeguard and AttriGuard
Thank you for proposing direct comparisons. Appendix C.4, Table 16, preserves native mechanisms rather than replacing them with RACG's actuators. RACG selects defensive combinations by joint safety/utility effects, not merely multiple outputs.
On nine tasks, native G-Safeguard obtains ASR/DTS 17.60/69.04 versus RACG's 13.81/70.32: gains of 3.79/1.28 pp, with 180 more tokens per decision.
On the common tool-execution subset, AttriGuard has lower ASR (8.750 versus 10.125). RACG has higher DTS (74.25 versus 72.75), lower latency (3.30 versus 7.05 s), and fewer tokens (5,140 versus 8,540). This is a trade-off, not matched-ASR dominance; comparisons remain within populations.
Q4. Absolute costs and scaling
Thank you for requesting full-path measurements. Table 30 reports graph update/search/rollback/regeneration/commit times of 0.24/0.64/0.35/1.28/0.24 s: 2.75 s per decision, versus LF's 3.27 s. RACG p50/p95 is 2.12/5.41 s. This is decision-and-recovery time, not full-workflow time.
Table 31 varies events 64-512 at 12 candidates (2.02-3.68 s), and candidates 8-24 at 256 events (2.38-4.54 s). Appendix F.1 separates 4,280 tokens/decision from 3.2 calls and 2,580 tokens/triggered recovery. The 32,000 evaluation-replay pairs cost 204,800 calls and 116,480,000 tokens; these are replay-generation resources, not model-training GPU hours.
Summary for the Area Chair
We thank the AC and reviewer for improving the paper. The recognized operational design is now supported by clearer positioning (W2), direct native-defense comparisons (Q3), macro uncertainty estimates (Q2), and measured costs (Q4). These support learned joint intervention selection within the tested scope while preserving the observed AttriGuard trade-off and trusted-control-plane boundary.