acceptodds
Under review as a conference paper at ICLR 2027

Risk Lies in the Relations: Relation-Aware Collaboration Graph Defense for LLM Multi-Agent Systems

Abstract

Large language model (LLM) multi-agent systems are commonly defended by classifying messages or auditing agents in isolation. Safe components, however, do not guarantee safe collaboration: individually benign messages, retrieved documents, tool responses, and memory records can jointly trigger harmful actions after transformation, delegation, and downstream authority amplification. We introduce RACG, a relation-aware collaboration graph framework that treats multi-agent defense as intervention over an asynchronous execution process rather than anomaly detection over isolated text. RACG represents execution dependencies in a typed event-level partial order and learns the safety and utility effects of typed interventions from matched checkpoint replays. At deployment, affected-cone modeling and uncertainty-constrained search select executable intervention sets and regenerate the affected continuation from an action-valid rollback cut. Across nine benchmarks, RACG achieves 13.81% macro attack success rate and 70.32% defended task success, compared with 15.54% and 69.31% for Full-Graph Late Fusion. For joint interventions, it reduces pair/triple risk MAE from 0.104/0.139 to 0.086/0.113, while eMEIS attains 0.071 regret by evaluating 27.54% of candidate sets. These results support relation-aware intervention as a practical basis for secure multi-agent collaboration.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Multi-Agent Safety, Graph Neural Networks, Secure AI Systems, Intervention Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8564OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RACG addresses attacks in LLM multi-agent workflows where individually plausible messages, tool outputs, or memory records contribute to an unsafe privileged action through their execution relationships. It represents the observed workflow as a typed event-dependency graph and, at a pre-commit gate, selects from executable message quarantine, memory restoration, and capability narrowing actions. An intervention-conditioned affected-cone model predicts post-intervention risk, task-utility loss, and uncertainty. It is trained with matched checkpoint replays and used with bounded coalition search and rollback-based regeneration. Across nine task benchmarks embedded in multi-agent workflows, the reported macro ASR and DTS are 13.81% and 70.32%, compared with 15.54% and 69.31% for the closest reported model baseline, Full-Graph Late Fusion. Further diagnostics examine typed-relation discrimination, joint intervention effects, operational stress, and transfer.

Critical strengths
  1. The event-level typed graph, action-valid rollback cut, and pre-commit gate provide a concrete way to act on relational risks instead of merely flagging suspicious text. The paper carefully states that logged dependencies are candidate paths rather than proven semantic causes.
  2. Relation-matched diagnostics and coalition ablations probe the central claims: composition-pair consistency is 76.41% versus 53.72% for a homogeneous graph, and pair, triple risk MAE is 0.086, 0.113 versus 0.104, 0.139 for Full-Graph Late Fusion. The operational and transfer studies also make some limitations visible.
Critical weaknesses
  1. The threat setting lists document injection, forged or delayed tool feedback, poisoned memory, and compromised low-privilege agents, but the manuscript does not clearly map each attacker capability to a specified attack construction, budget, and test population. Its main results use standard task benchmarks embedded in multi-agent workflows; the paper does not give enough consolidated detail about those workflows, attack generation, sample counts, repeated runs, and uncertainty intervals to judge how reliably the small aggregate gains generalize. The method also assumes an intact control plane, observable typed events, recoverable state, and pre-commit interception. These restrictions are legitimate, but they should bound the practical conclusion.
  2. Prior work already combines graph-based multi-agent defense with remediation (e.g., G-Safeguard), and action-level counterfactual replay has been studied for prompt-injection defense. RACG's potentially distinctive contribution is learning the joint safety–utility effects of multiple typed, executable interventions, but the related-work discussion does not compare the exact decision point, intervention space, replay use, and online cost against these approaches. The main improvement over Full-Graph Late Fusion is 1.73 ASR points and 1.01 DTS points, without reported confidence intervals, making the strength of the end-to-end advance hard to assess.
Questions for authors
  1. For each attacker-controlled channel in Section 3.1, what can the attacker read, replace, delay, or coordinate, and with what knowledge and budget? How many attack instances of each type appear in the nine-benchmark evaluation, and how are relation-matched and coalition cases generated and split across training, calibration, and testing?
  2. Can the authors report the number of independent task/attack instances and repeated runs, uncertainty intervals for the main ASR/DTS differences, and the exact agent backbones and workflow configurations? In particular, is the 1.01-point macro DTS improvement over Full-Graph Late Fusion stable across independent runs?
  3. What capability does RACG provide beyond G-Safeguard's graph-based remediation and AttriGuard's action-level counterfactual replay? A component-by-component comparison, and where feasible a matched executable-defense comparison, would materially affect my assessment of novelty.
  4. What are the absolute end-to-end latency and compute costs, including graph construction, candidate scoring, checkpoint restoration, and regenerated execution? How do they change with workflow length and candidate-set size, beyond the reported fraction of candidate sets scored?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 8564

We thank the reviewer for constructive suggestions and recognition of our operational design. LF: Full-Graph Late Fusion; pp: percentage points.

Weaknesses

W1. Evaluation transparency and practical scope

Thank you for requesting clearer operating conditions. Appendix C.1, Table 8, now maps nine benchmarks to role chains, attack entries, and protected effects. Security checks actual prohibited writes/calls separately from task correctness. Sections 3.1 and 5 specify trusted control, recoverable states, and pre-commit interception.

Table 29 tests observation faults: 10% missing edges changes ASR/DTS from 13.81/70.32 to 16.18/68.67; adding 10% version misbinding yields 18.76/66.58. Authorization/commit enforcement remains trusted. Q2 addresses variability.

W2. Relation to existing remediation

Thank you for encouraging precise positioning. The introduction now acknowledges G-Safeguard's topology repair and AttriGuard's online counterfactual checks. RACG instead learns joint safety/utility effects of bounded defensive action sets offline, models overlapping affected cones, and selects actions online without replaying every candidate. Action-valid rollback implements the selected actions.

Removing cone intersections increases pair/triple risk MAE from 0.086/0.113 to 0.112/0.151 and worsens ASR/DTS from 13.81/70.32 to 15.26/69.44 (Table 21). Q3 complements this ablation with native-system comparisons.

Questions

Q1. Attack permissions, populations, and diagnostics

Thank you for requesting specificity. Section 3.1 and Appendix C.1 distinguish attackable documents, tool feedback, memory, and low-privilege messages from trusted execution records and authorization. Table 8 identifies the applicable channels per task.

ASR/DTS use attacked workflows; BTS uses benign workflows; macro scores weight nine tasks equally. Blocked proposals alone are not successful attacks. Appendix D, Tables 17-18, reports orchestrator-constructed relation diagnostics separately from downstream defense. These test provenance, permissions, delegation, versions, tool bindings, and sinks.

Appendix A.2.1 groups related prefixes and continuations within splits and uses training-pool information for coalition acquisition. Table 6 reports 1,440/960/560 singleton/pair/triple action sets with 8/12/16 repetitions: 32,000 supplementary evaluation pairs, not training-pool size or independent benchmark attack counts. Table 22 separately varies reference budgets.

Q2. Stability, runs, and backbones

Thank you for examining the modest DTS gain. Against original LF, paired macro gains are 1.73 ASR pp (95% CI [1.11, 2.35]) and 1.01 DTS pp ([0.47, 1.55]). Three-run gains are 1.73/1.64/1.82 ASR pp and 0.90/0.94/1.19 DTS pp; sample SDs are 0.09/0.16 pp (Tables 14-15). DTS intervals for seven tasks include zero, so the claim is aggregate improvement, not significance on every task.

Appendix G.3 identifies GPT-5.5 as source and Qwen3-14B as target, with model identifiers. Tables 36-37 distinguish Frozen transfer from target-assisted calibration/training. Shift tests and repeated runs are separate analyses.

Q3. Native G-Safeguard and AttriGuard

Thank you for proposing direct comparisons. Appendix C.4, Table 16, preserves native mechanisms rather than replacing them with RACG's actuators. RACG selects defensive combinations by joint safety/utility effects, not merely multiple outputs.

On nine tasks, native G-Safeguard obtains ASR/DTS 17.60/69.04 versus RACG's 13.81/70.32: gains of 3.79/1.28 pp, with 180 more tokens per decision.

On the common tool-execution subset, AttriGuard has lower ASR (8.750 versus 10.125). RACG has higher DTS (74.25 versus 72.75), lower latency (3.30 versus 7.05 s), and fewer tokens (5,140 versus 8,540). This is a trade-off, not matched-ASR dominance; comparisons remain within populations.

Q4. Absolute costs and scaling

Thank you for requesting full-path measurements. Table 30 reports graph update/search/rollback/regeneration/commit times of 0.24/0.64/0.35/1.28/0.24 s: 2.75 s per decision, versus LF's 3.27 s. RACG p50/p95 is 2.12/5.41 s. This is decision-and-recovery time, not full-workflow time.

Table 31 varies events 64-512 at 12 candidates (2.02-3.68 s), and candidates 8-24 at 256 events (2.38-4.54 s). Appendix F.1 separates 4,280 tokens/decision from 3.2 calls and 2,580 tokens/triggered recovery. The 32,000 evaluation-replay pairs cost 204,800 calls and 116,480,000 tokens; these are replay-generation resources, not model-training GPU hours.

Summary for the Area Chair

We thank the AC and reviewer for improving the paper. The recognized operational design is now supported by clearer positioning (W2), direct native-defense comparisons (Q3), macro uncertainty estimates (Q2), and measured costs (Q4). These support learned joint intervention selection within the tested scope while preserving the observed AttriGuard trade-off and trusted-control-plane boundary.

Thank you for the response.

My main remaining concern is the statistical support for the headline gain: the confidence intervals in Table 14 are reported only against the main-table Full-Graph Late Fusion configuration, whereas against the fairer LF baseline with matched calibration, budget, and suffix supervision (Table 25) the gain shrinks to 0.70 pp ASR and 0.43 pp DTS with point estimates only.

I will maintain my current assessment.

AuthorsOpenReview ↗

Response to Reviewer 8564

Thank you for this careful follow-up and for helping us sharpen the statistical interpretation of the headline result. We agree that the calibration-, budget-, and suffix-matched LF baseline is the more informative comparator for assessing RACG’s incremental contribution. The confidence intervals in Table 14 apply only to the original LF configuration and should not be extended to this matched comparison.

Changes in the manuscript. We have added the matched-comparison confidence intervals and the complete bootstrap protocol to Appendix E.1, immediately after the suffix-controlled end-to-end comparison.

We have now directly quantified uncertainty for RACG versus matched LF + Suffix:

MetricLF + SuffixRACGRACG gain (pp)Paired 95% CI
ASR ↓14.5113.81+0.70[0.08, 1.32]
DTS ↑69.8970.32+0.43[−0.11, 0.97]

We used a paired, benchmark-stratified cluster bootstrap with 10,000 resamples. Original task instances (task_id) were resampled within each benchmark, keeping all associated attack variants and repeated executions together. RACG and LF + Suffix remained paired by task and attack scenario. Each resample retained equal weighting across the nine benchmarks; intervals use the 2.5th–97.5th percentiles. These intervals quantify task-instance sampling uncertainty conditional on the fitted models and calibration settings, separately from training-run variability.

We appreciate the reviewer’s emphasis on this distinction. This direct analysis addresses the matched comparator rather than relying on the original LF intervals or treating mechanism-level ablations as substitutes for end-to-end statistical evidence.

Reviewer_w9cDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

RACG learns the safety and utility effects of executable intervention sets from checkpoint replays, then uses these predictions to select interventions over typed execution graphs. Its substantive contribution is intervention-conditioned modeling of joint defensive actions. This is a meaningful direction, and the coalition diagnostics provide encouraging evidence. However, the experimental specification and comparison controls are insufficiently clear to assess the strength of the empirical conclusions.

Critical strengths
  • Predicting intervention consequences addresses an important gap between identifying suspicious events and choosing effective defensive actions.
  • The joint-intervention results are aligned with the methodological motivation: cone-intersection modeling provides larger benefits for pairs and triples than for single interventions.
  • Comparing against Full-Graph Late Fusion, additive estimators, and intervention-selection alternatives is a sensible experimental design for evaluating the proposed mechanism.
Critical weaknesses
  • The security experiments are not specified sufficiently to interpret the main results. The paper lists nine task benchmarks, but does not adequately describe the concrete attacks, protected operations, and violation predicates used to construct their security evaluations. These choices determine what ASR measures and how difficult the defense problem is. The execution backbones are also unspecified, including the source and target models in the cross-backbone experiment. These omissions concern the definition of the experiments themselves, rather than peripheral reproducibility details.

  • The comparison does not clearly isolate the contribution of affected-cone modeling from the additional supervision. RACG combines its intervention-conditioned representation with privileged-suffix distillation and uncertainty calibration. The manuscript establishes a shared event representation and action interface across baselines, but does not clearly establish whether Full-Graph Late Fusion and HeteroGNN + SetEnc receive the same replay data and suffix supervision. Table 2 shows that suffix supervision materially affects prediction quality. The corresponding training conditions therefore need to be explicit before attributing the full performance gap to the proposed representation.

Questions for authors
  1. What exact attack procedures, violation predicates, execution models, and evaluation sample counts were used for each benchmark?
  2. Were the strongest learned baselines trained with identical replay data and privileged-suffix supervision? If so, where are these controls specified?
  3. What replay budget constructs the evaluation targets, and how stable are the main performance and prediction-error differences across repeated runs?
2 comments
AuthorsOpenReview ↗

Response to Reviewer w9cD

We thank the reviewer for constructive suggestions and recognition of executable-effect prediction. LF: Full-Graph Late Fusion; pp: percentage points.

Weaknesses

W1. Security evaluations and backbones

Thank you for emphasizing evaluation definitions. Appendix C.1, Table 8, now maps each benchmark to its role chain, attack entry, and protected operation. Restricted-write/call checks concern actual prohibited effects, not injected instructions or proposals blocked before execution. Task correctness is evaluated separately. Appendix G.3 identifies GPT-5.5 as the source execution model and Qwen3-14B as the target. Tables 36-37 separate backbone-only, attack-only, and joint shifts, distinguishing Frozen transfer from target-assisted calibration/training.

W2. Cone modeling versus suffix supervision

Thank you for highlighting this control. We added LF + Suffix and HeteroGNN + SetEnc + Suffix, sharing RACG's replay, training-only suffix targets, stop-gradient rule, and suffix-loss weight while retaining their encoders.

MethodPair risk MAETriple risk MAEECE
HeteroGNN + SetEnc + Suffix0.1090.1450.060
LF + Suffix0.0940.1250.051
RACG0.0860.1130.046
Relative to LF + Suffix, RACG lowers pair/triple risk MAE by 8.51%/9.60%. LF + Suffix has slightly better singleton MAE (0.061 versus 0.062), locating the advantage in joint interventions. Pair/triple utility MAE is 0.081/0.104 for RACG versus 0.085/0.115 for LF + Suffix.
Separately, removing cone intersections within RACG increases pair/triple risk MAE to 0.112/0.151 and changes ASR/DTS from 13.81/70.32 to 15.26/69.44. These controls provide complementary mechanism evidence.

Questions

Q1. Attack setup, checks, models, and counts

Thank you for seeking precise evaluation units. Table 8 specifies math workflows with restricted writes/submissions, coding workflows with sandbox/file/tool restrictions, and retrieval/browsing workflows with protected updates or commits. Attack entries include agent messages, retrieved evidence, tool feedback, and memory, as applicable to each workflow. Appendix C.1 distinguishes attacked evaluations (ASR/DTS), benign evaluations (BTS), and repeated executions; macro scores weight the nine benchmarks equally. Correct answers do not imply policy compliance. Table 6's replay counts concern checkpoint-conditioned intervention/control continuations, not independent main-benchmark attack instances. The source/target model identifiers and transfer conditions are in Appendix G.3, see W1.

Q2. Matched replay and suffix controls

Thank you for requesting a direct answer. Original LF and HeteroGNN + SetEnc shared RACG's replay pool, split, and primary outcome loss, but not its suffix loss. The added + Suffix variants match that additional supervision. No future continuation or suffix representation is provided online. Calibration is fitted independently on the same held-out set with matched eligible actions and online budgets.

Calibrated methodASRDTSCost
HeteroGNN + SetEnc + Suffix15.0969.530.56
LF + Suffix14.5169.890.57
RACG13.8170.320.49
Table 25 thus retains 0.70 pp lower ASR and 0.43 pp higher DTS than LF + Suffix. These are point differences. Original LF, supplementary calibrated LF, and calibrated LF + Suffix are distinct configurations; Table 14's confidence intervals apply only to original LF.

Q3. Replay budgets and repeated runs

Thank you for requesting stability evidence. Table 6 records 1,440/960/560 singleton/pair/triple action sets with 8/12/16 paired repetitions, totaling 32,000 supplementary evaluation pairs, not training-pool size. Table 22 separately varies reference budget R=2/4/8/16 on shared held-out sets. RACG has lower pair/triple MAE throughout; gaps at both R=8 and 16 are 0.018/0.026. Against original LF, Table 15 reports three-run ASR gains of 1.73/1.64/1.82 pp and DTS gains of 0.90/0.94/1.19 pp. Mean gains and sample SDs are 1.73 +/- 0.09 and 1.01 +/- 0.16 pp. Table 14 separately reports paired macro 95% CIs [1.11, 2.35] and [0.47, 1.55]. These distinguish reference-budget sensitivity, run dispersion, and paired uncertainty.

Summary for the Area Chair

We thank the AC and reviewer for helping strengthen the evidence. The recognized executable-effect contribution retains its joint-intervention advantage under matched suffix supervision (W2/Q2), complemented by a within-RACG ablation. Clarified workflow definitions (W1/Q1) and separate replay/run analyses (Q3) make the comparison more interpretable. The evidence supports this focused contribution, not superiority on every intervention cardinality.

Thank you for the clarifications. Your responses have addressed my concerns, and I will maintain my weak accept rating.

Reviewer_Yra7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RACG, a relation-aware graph-based defense framework for securing multi-agent LLM systems. It models dependencies among messages, tool calls, shared memory, and agent actions, then predicts how single or combined interventions can reduce security risks while preserving task utility. Across nine benchmarks, RACG achieves 13.81% attack success rate and 70.32% defended task success, outperforming a full-graph late-fusion baseline. While the approach is promising, questions remain about the realism of the security evaluation, the statistical significance of the gains, the novelty of individual components, and generalization to realistic deployments.

Critical strengths
  1. The paper addresses an important and underexplored security problem. It correctly distinguishes the security of individual messages or agents from the security of an entire workflow, where risk can emerge through information transformation, delegation, shared memory, and privileged actions. This is a meaningful problem formulation for increasingly tool-integrated multi-agent systems.

  2. The intervention-effect formulation goes beyond conventional detection. RACG attempts to predict how executable defensive actions change both violation risk and task utility, rather than simply classifying suspicious events. This is a useful distinction because detecting a risky event does not establish which intervention will mitigate the risk without unnecessarily disrupting the task.

  3. The method explicitly models interactions between defensive actions. The affected-cone representation injects action embeddings into relevant execution dependencies and models interactions where multiple intervention cones overlap. The reported improvements in pairwise and triple-intervention risk prediction support the value of modeling joint effects rather than assuming that individual action effects are additive.

  4. The matched-replay training design is conceptually well motivated. Comparing intervention and control trajectories from the same rollback state helps separate intervention effects from differences in execution history. The paper also distinguishes training-time access to realized continuations from deployment-time inputs, making the intended information boundary explicit.

Critical weaknesses
  1. The experimental evidence does not yet establish the generality of the proposed security framework. Although the evaluation spans nine benchmarks, the central claim concerns relational security in multi-agent workflows. The paper needs to make the construction of these workflows, attack scenarios, and privileged actions sufficiently transparent to determine how representative they are of realistic agent deployments. In particular, it should distinguish evidence from standard task correctness benchmarks from evidence that the framework handles realistic, multi-step adversarial execution.

  2. The end-to-end improvement over the strongest comparable learned baseline is relatively modest. RACG improves macro ASR from 15.54% to 13.81% and DTS from 69.31% to 70.32% over Full-Graph Late Fusion. These are encouraging gains, but the paper does not establish their statistical significance through reported confidence intervals or repeated-run variability in the main results. Given the complexity of RACG, the robustness of these gains needs to be demonstrated more clearly.

  3. The contribution relative to existing graph-based defenses and intervention methods needs sharper isolation. RACG combines typed graph modeling, affected-cone propagation, paired-effect supervision, privileged-suffix distillation, uncertainty calibration, and bounded coalition search. The ablations provide useful evidence, but they do not fully establish which elements are essential for end-to-end improvements or whether a simpler graph model with a well-calibrated intervention predictor could achieve comparable results. The novelty of the overall integration is plausible, but the distinct methodological contribution remains less clear than the system-level contribution.

  4. The threat model and trusted-control-plane assumptions substantially limit the security claims. RACG assumes reliable event identity, delivery, authorization, state-version, and external-effect records, as well as valid rollback and recovery semantics. These assumptions are understandable for a first implementation, but failures or manipulation of orchestration metadata could undermine the dependency graph itself. The current evaluation does not establish how gracefully the framework degrades when its structural observations are incomplete or incorrect.

  5. The intervention-effect evaluation requires stronger evidence about label reliability and sampling variability. The model is trained using repeated checkpoint replays, and the paper uses these outcomes to estimate safety and utility effects. However, the strength of this supervision depends on the number and diversity of replay continuations, the stability of the evaluators, and the representativeness of sampled intervention coalitions. The authors should report enough detail to establish that the observed pairwise and triple-wise improvements are not artifacts of limited replay sampling.

  6. The uncertainty mechanism does not establish a reliable safety guarantee under distribution shift. RACG uses calibrated upper-risk estimates to constrain intervention selection, but the paper itself reports nontrivial false-feasibility rates and reduced calibration coverage when calibration is removed. Moreover, calibration on held-out checkpoint-action pairs does not automatically ensure coverage after selecting actions based on their predicted risk. The evaluation should more clearly characterize the conditions under which the risk constraint remains dependable, especially on unseen workflows.

  7. The baseline comparisons need greater transparency about implementation and interface parity. Several detection and graph-defense baselines are augmented with adapters to connect their outputs to executable interventions, while RACG directly predicts intervention effects and selects actions. Although this can be a reasonable way to compare end-to-end systems, the results depend on the adapter design, access to graph information, action catalog, and tuning budget. A clearer accounting of these differences is needed to establish that the improvements arise from RACG's proposed modeling rather than unequal implementation or optimization choices.

  8. The practical scalability of the full pipeline remains insufficiently established. The search limits coalitions to at most three actions and uses a bounded beam, which is sensible for controlling computational cost. Nevertheless, candidate-pool expansion increases latency, and the overall system includes graph construction, uncertainty-aware scoring, rollback, and continuation regeneration. The paper should provide more comprehensive runtime and resource measurements across different workflow sizes and candidate counts before claiming practical applicability to large, asynchronous multi-agent deployments.

Questions for authors
  1. How are the nine benchmark tasks converted into multi-agent workflows, and how are the attacks, compromised inputs, and privileged sinks instantiated? Please clarify which scenarios are derived from existing security benchmarks and which are generated by the authors.

  2. Are the reported ASR and DTS improvements over Full-Graph Late Fusion consistent across independent random seeds and attack instances? Can the authors provide confidence intervals or paired significance tests for the macro results?

  3. How many independent checkpoint contexts and repeated rollouts are used to estimate singleton, pairwise, and triple-intervention effects? What evidence establishes that the reported improvements in effect prediction are stable under replay-sampling variability?

  4. Do all end-to-end baselines use the same underlying agent workflows, attack instances, executable intervention catalog, and task evaluators? How are their adapters trained and tuned, and do they receive equivalent information and computational budgets?

  5. How does the empirical violation rate of selected interventions compare with the calibrated upper-risk threshold, particularly under distribution shift? Does the reported coverage account for the fact that actions are selected based on their predicted risk?

  6. How much performance is retained when both the underlying LLM backbone and the attack construction change, without target-specific training? Which aspects of the learned intervention model transfer across these settings?

  7. What are the end-to-end latency and resource costs of RACG, including graph construction, candidate evaluation, rollback, and continuation regeneration? How do these costs scale as the number of workflow events and candidate interventions increases?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Yra7

We thank the reviewer for constructive suggestions and recognition of workflow security and joint interventions. LF: Full-Graph Late Fusion; pp: percentage points.

Weaknesses

W1. Security evaluation

Thank you for seeking concrete scenarios. Appendix C.1, Table 8, maps nine tasks to role chains, attack entries, and protected operations. Security checks actual unauthorized effects separately from task correctness.

W2. Stability

Thank you for emphasizing uncertainty. Against original LF, macro ASR improves by 1.73 pp (95% CI [1.11, 2.35]) and DTS by 1.01 pp ([0.47, 1.55]); both improve in all three runs (Tables 14-15). Seven task-level DTS intervals include zero.

W3. Joint-effect attribution

Thank you for requesting controls. Removing cone intersections changes ASR/DTS from 13.81/70.32 to 15.26/69.44 and pair/triple MAE from 0.086/0.113 to 0.112/0.151 (Table 21). Against calibrated LF + Suffix, RACG retains 0.70/0.43 pp ASR/DTS gains (Table 25). These point comparisons are distinct from original-LF CIs.

W4. Record integrity

Thank you for highlighting this assumption. With 10% missing edges, ASR/DTS becomes 16.18/68.67; adding 10% version misbinding yields 18.76/66.58 (Table 29). Authorization and commit enforcement remain trusted.

W5. Replay reliability

Thank you for suggesting this analysis. Table 22 varies paired repetitions R=2/4/8/16. RACG has lower pair/triple MAE throughout; gaps at both R=8 and 16 are 0.018/0.026. Matched checkpoints and grouped splits are specified in Appendix A.2.1.

W6. Post-selection coverage

Thank you for focusing on selected actions. Table 33 reports UCB coverage of 92.86%, 89.92%, and 85.61% on original, unseen-attack, and cross-backbone settings, respectively. This differs from 94.21% candidate coverage and supports empirical screening, not a distribution-free guarantee.

W7. Baseline adapters

Thank you for seeking clarity. Table 10 documents native observations, output-to-action mappings, validation, and budgets. Tables 11-12 separately report native detection/provenance. LF receives the full graph and the same action representations and observable state as RACG.

W8. Runtime

Thank you for requesting costs. Tables 30-31 report 2.75 s mean decision latency (p50/p95: 2.12/5.41), versus LF's 3.27 s. This covers graph update through recovery/commit, not a full workflow.

Questions

Q1. Workflow and attack construction

Thank you for requesting details. Table 8 maps math tasks to restricted writes/submissions, coding to unauthorized writes/calls, and retrieval/browsing to protected updates/commits. Inputs include messages, evidence, tool returns, or memory as applicable. An injection or blocked proposal alone is not attack success.

Q2. Repeated runs

Thank you for asking about variability. Table 15 gives ASR gains 1.73/1.64/1.82 pp and DTS gains 0.90/0.94/1.19 pp. Their sample SDs are 0.09/0.16 pp, distinct from paired CIs. Macro scores weight all nine tasks equally.

Q3. Replay counts

Thank you for requesting counts. Table 6 lists singleton/pair/triple workflows 180/160/140, checkpoints 360/320/280, and action sets 1,440/960/560, with 8/12/16 paired repetitions. The total is 32,000 evaluation pairs, not training-pool size. Table 22 separately sweeps reference budgets on shared held-out sets.

Q4. Tuning controls

Thank you for asking about fairness. Table 10 describes validation-tuned adapters. Appendix E.1 matches learned models' replay, splits, eligible actions, risk threshold, and search/execution budgets, with independent calibration. Native remediation in C.4 retains each method's original mechanism.

Q5. Accepted-action violations

Thank you for asking about execution outcomes. In Table 33's original/attack/backbone settings, accepted counts are 980/1,200, 744/1,000, and 681/1,000; violation rates are 10.84/13.18/14.37%. Selected-set threshold exceedance is 3.16/6.72/9.25%, a different diagnostic.

Q6. Joint shift

Thank you for suggesting this test. Tables 36-37 change both backbone and attack: Frozen RACG achieves ASR/DTS 27.18/60.94 versus Fixed Policy's 31.26/57.43. Calibration-only reaches 23.87/63.10 and Target-trained 18.62/68.34; these use target data, unlike Frozen.

Q7. Scaling and resources

Thank you for requesting accounting. Table 31 varies events 64-512 at 12 candidates (2.02-3.68 s), and candidates 8-24 at 256 events (2.38-4.54 s). Appendix F.1 separates 4,280 tokens/decision, 2,580 tokens/triggered recovery, and 116,480,000 tokens for evaluation-replay generation.

Summary for the Area Chair

We thank the AC and reviewer for improving this work. The recognized joint-intervention contribution is supported by controlled ablations (W3), positive macro CIs (W2), and frozen joint-shift gains (Q6). Added protocol, reliability, and cost analyses define the tested scope without claiming uniform gains or guaranteed safety.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.