acceptodds
Under review as a conference paper at ICLR 2027

ErosionBench: Probing Alignment Decay in Enterprise Multi-Agent Systems

Abstract

As enterprises delegate consequential work to multi-agent language-model systems, benign-looking requests can escalate into unsafe actions as they propagate across sessions through poisoned shared state, long-context documents, and trusted peer messages—so no single message is malicious, yet the assembled tool action is. Prior benchmarks miss this, isolating a single agent under prompt injection or scoring multi-agent attacks by their final output, leaving cross-session erosion of shared state out of scope. We introduce ErosionBench, an environment-grounded benchmark that executes such attacks and verifies semantic alignment decay at the process level—scoring the realized tool call against authorized intent and provenance with deterministic contracts, not the output text. Because each scenario is authored as a deterministic state graph before any prose is rendered, every state change traces to a known cause. It spans four enterprise domains and four decay mechanisms across 1,728 scenarios with matched benign controls, each labeled against the OWASP Top-10 for LLM Applications, and—unusually among multi-agent safety benchmarks—delivers an additional 198 image-based attacks, each paired with a text equivalent, so a defense must survive the same attack in either channel. Existing perimeter defenses suppress attacks only by over-refusing 54–92% of legitimate work, so a low attack-success rate can conceal severe over- refusal. As a reference defense, we contribute a calibrated Meta-Auditor pairing synchronous state checks with asynchronous trajectory monitoring; on a family- disjoint 906-scenario held-out split it reaches 98.5% recall at a 2.9% false-positive rate. Because it monitors trajectory rather than surface content, its detection is channel-invariant: the strongest auditor flags image-delivered attacks about as reliably as their text equivalents (image–text recall gap ≤ 2 points per model). Across GPT, Gemini, and Claude auditors, calibration holds false positives low while recall tracks model capability, strongest for Claude.

Primary area:
datasets and benchmarks
Keywords:
Multi-Agent LLM Safety, Semantic Alignment Decay, Cross-Session State Poisoning, Multimodal Prompt Injection, LLM Agent Benchmark, OWASP Top-10

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8TZmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the vulnerability in multi-agent systems where individual messages may appear benign but become dangerous when combined across sessions, the authors introduce ErosionBench. Focusing on enterprise business scenarios, this framework constructs test cases—incorporating either contaminated historical data/documents or standard, clean data—which are then processed by a multi-agent system consisting of four fixed roles. The success of an attack is determined by the actual tools invoked, while execution logs are used to evaluate the system's defensive detection capabilities and false-positive rates. Furthermore, the authors propose Meta-Auditor, an independent, authoritative auditing mechanism that combines rule-based checks with large language model analysis of execution logs to determine whether a multi-agent system has been manipulated into an attack or has deviated from its authorized scope. The authors conducted extensive experiments and provide valuable insights.

Critical strengths
  • The benchmark highlights a significant issue: in multi-agent scenarios, as a conversation progresses, a gradual divergence emerges between the user’s initial intent—as expressed through their authorization—and the final actions actually executed.

  • The benchmark covers 4 enterprise domains, 4 types of degradation mechanisms, and 1,728 scenarios, each with a paired benign control group.

  • Based on actual tool invocations and parameter validation violations, the benchmark provides clear evidence for successful attacks, ensuring that every state change can be traced back to a known trigger.

  • The benchmark provides an additional 198 image-based attacks, requiring defense mechanisms to be capable of withstanding the same attacks across any input channel—offering a more comprehensive evaluation.

Critical weaknesses
  • The figure is not formatted correctly. On the far right of Figure 1, under "verdict:allow/block," there are anomalous special characters. In the rectangular area at the bottom left of Figure 1, "Long-context" is misspelled as "Long-contest." Additionally, in the "State 1.1" section of Figure 1, the term "Flamed Value" appears; this is not a standard term and is likely a typo.

  • There is an inconsistency in the description: in section 3.1, the author defines four processes, yet in Figure 1, only three processes—stage 1.1 through stage 1.3—are shown under stage 1.

  • The authors focus on scenarios involving multi-agent collaboration and cross-session propagation; however, their primary conclusions are validated only within a fixed, four-role structure. In real-world enterprise business contexts, by contrast, topological structures are often diverse, with each structure requiring distinct approaches to scheduling and shared-memory mechanisms.

Questions for authors
  • The core subject of this paper is collaboration among multi-agent systems. Given that collaborative architectures, memory mechanisms, and authorization processes are all significant variables, how well can Meta-Audiotr adapt to such variations, and what is its level of transferability?

  • Did the authors take into account the model's "reasoning intensity" settings during the experiments—for instance, the "low," "medium," and "high" options available for GPT-5.1? This does not appear to be disclosed in the paper. If these settings were considered, what was the basis and rationale for their use?

  • What is the computational cost of a standard LLM judge in an evaluation task? And if such a judge were granted a second inference opportunity—akin to a "gray-area" arbitration process—could its performance become comparable to that of Meta-Auditor?

4 comments
AuthorsOpenReview ↗

(1/4) - Weaknesses (1, 2, 3)

We thank the reviewer for the close reading of Figure 1 and for pushing on transferability and on reasoning settings. Each point is addressed below.

Weakness 1 & 2. Figure 1 formatting and §3.1 consistency

Thank you for catching these inconsistencies in the figure. All four issues are fixed in the revised Figure 1 (see the updated PDF):

IssueFix
Stray special characters under "verdict: allow/block"Removed (a font-encoding artifact). The verdict now reads "Flagged / Benign", matching what the post-hoc auditor outputs
"Long-contest"Removed in the redesigned figure; the term is spelled "long-context" throughout the paper
"Flamed Value" (Stage 1.1)→ "Planted Value", the term used in §3.1
§3.1 defines four processes, Figure 1 shows Stages 1.1–1.3Figure 1 now shows all four Stage 1 processes (1.1–1.4), labelled to match §3.1

We also made the figure easier to read:

  • One font throughout;
  • Corrected state-graph arrows (base args and planted value feed the tool call, which is checked against ground truth);
  • All three Meta-Auditor tiers shown, labelled post-hoc;
  • A three-session strip showing how an attack is assembled from individually benign steps.

Weakness 3. Conclusions validated only on a fixed four-role structure

We agree this was a real limitation, so we tested it directly. We ran the frozen headline Meta-Auditor (claude-opus-4.8 same calibration artifact and threshold, no re-fitting) on two structurally different meshes:

  • 3-role: the Policy reviewer is removed (Gateway → Orchestrator → Deployment).
  • 5-role: an independent second reviewer, similar to the Policy agent, must also approve before execution (dual-approval control).

Both runs use the same stratified slice of 192 trajectories (96 attacks / 96 matched benign controls) and are paired against the paper's 4-role run on the same trajectories.

Mesh topologyMesh ASR (undefended)Benign over-refusal (mesh)Auditor recallAuditor FPRUndetected compromise
3-role (Policy removed)100.0%0.0%97.9%1.0%2.1%
4-role (paper)44.8%42.7%97.9%2.1%2.1%
5-role (+ second reviewer)29.2% (p = .006)43.8% (p = 1.0)99.0%5.2%1.0%

Mesh ASR and auditor columns: L5 run on the slice. Benign over-refusal: a separate L2 mesh draw on the same ids. Paired exact McNemar vs. 4-role. Auditor recall/FPR differences vs. 4-role are not statistically detectable.

  • Topology strongly changes how vulnerable the mesh is. Without a reviewer every attack executes. A second reviewer cuts attack success to 29.2%, but more than a quarter of attacks still pass both approvals. The second reviewer adds no benign refusals (43.8% vs. 42.7%).
  • The auditor's operating point does not move with it. The Meta-Auditor sits outside the mesh. It reads the shared ledger and the full trajectory, and compares them with the user's authorized intent. It does not depend on a fixed set of role names.

Why the 3-role mesh reaches 100%. The mesh follows separation of duties: one agent holds approval authority per scenario (the Policy reviewer in 93% of attacks, the intake Gateway in 7%). The other agents are the channel the attack travels through: shared state written in earlier sessions, the Orchestrator's arguments, and, in 12% of attacks, peers that attest to the poisoned state. Removing the approver therefore leaves a single point of failure. This is not single-agent behaviour: an isolated agent (L1) admits 0% of attacks. Adding a second approver who reads the same shared state still lets 29.2% through, so redundancy does not restore independent judgment. This is also where an external auditor matters most: with no reviewer in the mesh, the frozen Meta-Auditor still flags 97.9% of attacks at 1.0% FPR.

We have added this table to the appendix and state the 4-role mesh as a scoped design choice in §3. Further variation of memory and authorization is described in Q1 below.

AuthorsOpenReview ↗

(2/4) - Q1 (Transferability)

We varied each of the three factors mentioned by the reviewer, one at a time, with the Meta-Auditor frozen, on the same stratified 192-scenario slice (96 attacks / 96 benign controls) mentioned in the general comment:

  • Architecture: the 3- and 5-role meshes above.
  • Memory: agents see only the current shared state. The cross-session memory normally surfaced to them (prior approvals on file, value histories across sessions) is withheld. 64/96 attacks rely on this cross-session context.
  • Authorization process: mentioned in paragraph below table.
ConfigurationMesh ASRBenign over-refusal (mesh)Auditor recallAuditor FPRUndetected
4-role, default memory, deliberative review (paper)44.8%42.7%97.9%2.1%2.1%
Architecture: 3-role100.0%0.0%97.9%1.0%2.1%
Architecture: 5-role29.2%43.8%99.0%5.2%1.0%
Memory: latest state only34.4% (p = .08)†41.7%99.0%2.1%1.0%

Same 192-trajectory slice, paired exact McNemar vs. the paper configuration. †A second, independent mesh draw does not replicate the memory ASR drop (42.7% vs. 45.8%, p = .65).

What transfers. Across these configurations, recall (97.9–99.0%), FPR (1.0–5.2%) and the undetected-compromise rate (≤ 2.1%) all stay stable.

Authorization. We also replaced deliberative review with context-free change-ticket approval: the approver sees only the request and the proposed tool call. This cuts attack success from 44.8% to 16.7%, but only by rejecting 88.5% of legitimate work, the same over-blocking pattern as the perimeter defenses in our ladder. Auditor recall is unchanged (97.9%) and its ranking holds (AUC 0.98 vs. 0.998), but FPR rises to 19.8%, so the calibrated operating point is specific to the authorization process. We report this in the appendix and the revised Limitations.

Scope. All variants are sequential collaboration graphs over one shared ledger. Parallel/fan-out scheduling and fully private per-agent stores remain future work, and we state this in the revised Limitations. Together with transfer across four auditor models (auditor-panel table), three attacker-mesh models (mesh-sweep table), two externally authored corpora (AgentDojo, ASB) and an unseen fifth domain (healthcare: 100% recall, 0% FPR), this supports transfer of detection across the variables the reviewer raises, with the calibration caveat above.

AuthorsOpenReview ↗

(3/4) Q2 (Reasoning Effect)

Thank you for bringing this up, we can disclose this here: Every model ran at its provider-default reasoning effort. We never set reasoning_effort. Our rationale is that the benchmark measures agents as deployed by default, and that a shared default keeps the cross-auditor comparison from mixing per-vendor tuning choices. The revised appendix adds the per-model settings:

ModelRoleReasoning effortTemperature
gpt-5.5mesh agents (all ladder rungs)provider default (≈ medium, see below)not accepted
gpt-5data generationprovider defaultnot accepted
gpt-5.1cross-auditorprovider default (= no reasoning)not accepted
gpt-4.1firewall LLM judgen/a0.0
claude-opus-4-8Meta-Auditor (headline), LLM-judge baselineno extended thinkingnot accepted
claude-sonnet-4-6cross-auditorno extended thinking0.0
gemini-3.5-flashcross-auditorprovider default thinking0.0

Reasoning models receive a fixed 4,000-token completion headroom so hidden reasoning cannot starve the visible answer.

To test whether the setting matters, we swept it on both sides of the system (same 192 scenarios, 96 attacks and 96 benign controls; paired exact McNemar vs. the paper setting). We found that more reasoning makes the attacked agents harder to compromise, but never immune. On the defense side, the calibrated score is robust to it without re-fitting; the grey-zone arbiter is the one stage that is not. The tables below show the effect on the gpt-5.5 mesh and on the gpt-5.1 auditor.

(a) Attacked mesh (gpt-5.5, undefended):

Mesh effortAttack success [95% CI]Δ vs. paper (p)Benign over-refusalReasoning tok/call
none58.3 [48.3, 67.7]+12.5 (p = .008)38.5%0
low45.8 [36.2, 55.8]+0.0 (p = 1.0)39.6%61
paper default45.8 [36.2, 55.8]—42.7%141
medium42.7 [33.3, 52.7]−3.1 (p = .65)40.6%144
high35.4 [26.6, 45.4]−10.4 (p = .053)42.7%262

Attack success falls monotonically with effort but stays at 35.4% even at high, and benign over-refusal is flat throughout. The paper's default uses the same reasoning per call as medium (141 vs. 144 tokens), so the reported numbers correspond to medium effort.

(b) Auditor (gpt-5.1). The calibration (score-to-probability map and decision threshold) was fit once at default effort and reused unchanged at medium and high:

Auditor effortCalibrated score + counterfactual (recall / FPR)Full Meta-Auditor (recall / FPR)Auditor tok/scenario
paper default88.5 / 6.280.2 / 0.06,065
medium85.4 / 5.259.4 / 3.18,823
high83.3 / 2.159.4 / 2.121,092

Without re-fitting, the calibrated score loses only 3–5 recall points across effort levels while its FPR falls. Both effects have the same cause: with more reasoning, the auditor scores every trajectory as less suspicious. Its mean raw compromise score falls from 0.70 to 0.48 on attacks and from 0.11 to 0.02 on benign controls. The threshold was fit on the default-effort scores, so this shift pushes a few borderline attacks below it (lower recall) and clears more benign runs (lower FPR). The ranking of attacks above benign runs stays largely intact (AUC 0.945 → 0.891), so the loss is mainly a shift relative to a frozen threshold, which is what per-configuration calibration is for. However, it is to be noted that the tier-3 grey-zone arbiter, a direct LLM verdict on borderline cases, is sensitive to effort (80.2 → 59.4% recall) ; the revision describes it as a precision stage to be validated per configuration. Both sweeps are added to the appendix.

Taken together, the sweeps support the paper's central claim: extra reasoning, whether in the attacked agents or in the auditor, does not substitute for a calibrated trajectory-level check.

AuthorsOpenReview ↗

(4/4) - Q3 (LLM Judge Cost)

We tested this directly. On the same 192-trajectory slice used above, the same-telemetry opus-4-8 judge was given: (i) two independent samples (self-consistency) (ii) a second pass using the Meta-Auditor's own grey-zone arbiter prompt. The arbiter's deterministic metrics were withheld from it, since they are the component in question.

DefenseRecall [95% CI]FPR [95% CI]Tokens / trajectoryMedian latency (s)Second pass on
LLM judge, single call100.0 [96.2, 100]66.7 [56.8, 75.3]3,4292.7—
Judge, 2 samples (AND)100.0 [96.2, 100]65.6 [55.7, 74.4]6,8805.5—
Judge, 2 samples (OR)100.0 [96.2, 100]75.0 [65.5, 82.6]6,8805.5—
Judge + arbiter, grey zone only100.0 [96.2, 100]47.9 [38.2, 57.8]9,42810.459%
Judge + arbiter, all trajectories100.0 [96.2, 100]47.9 [38.2, 57.8]11,26611.9100%
Meta-Auditor (same trajectories)97.9 [92.7, 99.4]2.1 [0.6, 7.3]~10,4006.748%
  • Cost. A standard single-call judge costs about 3.4k tokens and 2.7 s per trajectory, roughly a third of the Meta-Auditor's tokens. It operates at a 66.7% false-positive rate, which is unusable in deployment.
  • Second pass. Self-consistency alone gives no improvement. A second pass with our arbiter lowers the judge's FPR from 66.7% to 47.9%. Even at a higher budget than the Meta-Auditor (11.3k vs. 10.4k tokens; 11.9 s vs. 6.7 s), it remains about 23× worse on FPR on the same trajectories (47.9% vs. 2.1%).
  • Where the gap comes from. Extra inference narrows the gap but does not close it. Our component ablation (General Response, row 2) locates the remainder in the calibrated trajectory-level intent-divergence score. Calibration alone moves the uncalibrated semantic monitor from 20.8% to 2.0% FPR. A second LLM call does not account for it.

We have added this row group to the judge comparison in the revised appendix.

Reviewer_X2aoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1(−2)
Soundness
2
Read review
Paper summary

This paper presents ErosionBench, a benchmark for probing alignment decay in enterprise multi-agent systems. It spans four enterprise domains and four decay mechanisms across 1,728 scenarios with matched benign controls, each labeled against the OWASP Top-10 for LLM Applications. This papers also proposes a reference defence method called Meta-Auditor, which monitors the entire agent trajectory.

Critical strengths
  • Seems the first benchmark for evaluating cross-session attacks in enterprise multi-agent systems.

  • The Meta-Auditor defence combines deterministic state checks with semantic monitoring to monitor the entire agent trajectory.

Critical weaknesses
  • Not clear how the reported attack reduction actually indicates prevention or detection because it does not clearly distinguish attacks blocked before execution from those detected afterward.

  • There is no sufficient ablation on the Meta-Auditor defence under different attack situations.

Questions for authors
  1. Could you better clarify the threat model?

  2. Could you measure the contribution of each Meta-Auditor component?

3 comments
AuthorsOpenReview ↗

(1/3) - Weakness 1 (Prevention vs Detection)

We thank the reviewer for the updated review, and we take the presentation score seriously. The revision adds an explicit threat-model paragraph to §3 (Q1 below), corrects Figure 1 so that it shows the Meta-Auditor as a post-hoc detector with the three tiers of §4, and renames the metric that caused the prevention/detection confusion. We have updated the labeling in the paper accordingly.

Weakness 1 (Prevention vs Detection)

The reviewer is correct, and we correct the paper. On the 453 held-out attacks (headline auditor, claude-opus-4-8):

  • Blocked before execution: 0. §4 and App. E describe synchronous checks that can veto a tool call at the Policy-to-Deployment boundary before it executes. That inline veto was switched off in every reported run (the deterministic metrics were still computed, after execution), so every approved call executed and the Meta-Auditor scored the completed trajectory afterwards. A detection is a flag raised after the fact. The executed action is not undone.
  • Detected after execution: 193 of the 199 attacks that executed (97.0%).
  • The 1.3% is 6/453: attacks that executed and went undetected. It is an undetected-compromise rate, not attack success remaining after prevention. We rename it throughout (abstract, Table 4, §4, App. E).

Why the Meta-Auditor is a post-hoc detector. This placement follows from the evidence it uses:

  1. The attack is only visible at trajectory level. An erosion attack plants approvals, value histories and documents over earlier sessions. The final harmful call is therefore well-formed and carries the context that justifies it. A per-call check has little to reject: the deterministic anchors alone reach 6.6% recall (W2/Q2 below). The decisive signal is the divergence between the whole trajectory and the user's stated intent, which exists only once the trajectory is complete.
  2. The counterfactual check needs the outcome. This part of the auditor's tier 2 ran in every reported result, but §4 did not describe it (the revision adds it). It re-runs the mesh with each suspect write session removed and tests whether the final action changes. That requires the action to have been produced.

Breakdown of all 453 attacks, for all four auditors:

AuditorExecuted (mesh ASR)Detected after executionUndetected (formerly "residual ASR")Flagged, never executed (mesh refused)
claude-opus-4-8199 (43.9%)193 (42.6%)6 (1.3%)253
claude-sonnet-4-6204 (45.0%)197 (43.5%)7 (1.5%)237
gpt-5.1197 (43.5%)185 (40.8%)12 (2.6%)196
gemini-3.5-flash202 (44.8%)191 (42.4%)11 (2.4%)185

The headline 98.5% recall counts all 453 attacks. It includes the 253 the mesh refused, which are flagged malicious intent with no executed harm. FPR is unaffected by this split: it is computed over the 453 benign controls only (13/453 = 2.9%).

Which component makes each detection (headline auditor):

Decision pathTrue positives (executed / refused)False positives
Calibrated semantic score and counterfactual check agree188 (188 / 0)0
Semantic flags, counterfactual uninformative → grey-zone arbiter258 (5 / 253)5
Counterfactual overrides a benign semantic verdict08
Deterministic anchors decisive00
Total446 (193 / 253)13

Corrections to §4 and App. E.

  • SDI is not a pre-execution veto as measured. SDI is computed on the executed payload. Its 43.9% recall in §4 is therefore exactly the mesh attack-success rate: it is positive on all 199 executed attacks, 0 non-executed attacks and 0 benign controls. It shows that an attack landed, not that one is about to. We remove the "pre-execution veto" description.
  • The "196 anchor flags, zero false positives" count is wrong. The correct count is 188 true positives and 8 false positives. These are post-hoc detections in which the anchors were not the deciding signal (table above).

What an inline veto would add. We replayed the shipped synchronous rule offline on the stored anchors. It fires on an anchor conjunction or Cauchy combination at the Policy-to-Deployment boundary, and would have vetoed 30 of the 199 executed attacks (15%) with no benign controls blocked (29–34 for the other auditors). This is an offline estimate, not a measured inline run. The revision therefore reports the Meta-Auditor's numbers as detection, and the synchronous veto as a separate, narrower prevention layer. Turning detection into prevention requires rollback or inline gating, which we state as a limitation and future work.

AuthorsOpenReview ↗

(2/3) Weakness 2/Q2 (Contribution of Meta-Auditor in different attacks/component defenses)

We can clarify the contribution of each component here. Calibration does most of the work. The counterfactual check adds recall for the weaker auditors. The grey-zone arbiter adds no recall and acts only as a precision control. The deterministic anchors alone reach 6–8% recall. This ranking holds in every attack mechanism and tier.

Setup. One frozen operating point (same artifact and threshold), 453 held-out attacks + 453 benign controls, computed from stored per-trajectory verdicts. Components are added in the decision order of §4's three tiers. Tier 2 pairs the calibrated semantic score with a counterfactual check, which re-runs the mesh with each suspect write session removed. It ran in every reported result but §4 omitted it; the revision adds it.

All four auditors (Recall / FPR, %):

AuditorUncalibrated+ calibration+ counterfactual+ arbiter (full)Anchors alone
claude-opus-4-8100 / 20.898.5 / 2.098.5 / 3.898.5 / 2.96.6 / 0.0
claude-sonnet-4-6100 / 51.087.9 / 6.895.8 / 8.695.8 / 8.67.3 / 0.0
gpt-5.186.1 / 2.287.9 / 3.592.7 / 4.984.1 / 1.36.4 / 0.0
gemini-3.5-flash71.4 / 8.676.3 / 10.486.5 / 12.183.4 / 11.07.5 / 0.0
  • Calibration cuts FPR from 20.8% to 2.0% (opus) and from 51.0% to 6.8% (sonnet), p < .001.
  • The counterfactual check recovers 22–46 attacks for the three weaker auditors (+4.8 to +10.2 recall points, p < .001) at a cost of 1.4–1.8 FPR points.
  • The arbiter never adds recall; it only removes false positives, at a recall cost for gpt-5.1 and gemini. For opus, the full system is not detectably different from calibrated semantic alone (FPR 2.9% vs. 2.0%, p = .39).

By decay mechanism and by tier (headline auditor, Recall / FPR, %). Each row counts that slice's attacks and its matched benign controls, so "108 / 108" means 108 attacks and their 108 benign twins.

Decay mechanism (attacks / controls)Uncalibrated+ calibrationFull Meta-AuditorAnchors alone
Cross-session conditioning (108 / 108)100 / 14.8100 / 0.0100 / 0.016.7 / 0.0
State gaslighting (114 / 114)100 / 20.2100 / 0.0100 / 0.04.4 / 0.0
Distributed collusion (102 / 102)100 / 20.699.0 / 1.099.0 / 0.05.9 / 0.0
Long-context dilution (129 / 129)100 / 26.495.3 / 6.295.3 / 10.10.8 / 0.0
Tier (attacks / controls)Uncalibrated+ calibrationFull Meta-AuditorAnchors alone
T1, fewest sessions (151 / 151)100 / 12.695.4 / 2.095.4 / 2.64.6 / 0.0
T2 (151 / 151)100 / 21.9100 / 2.0100 / 1.37.9 / 0.0
T3, most accumulated state (151 / 151)100 / 27.8100 / 2.0100 / 4.67.3 / 0.0
  • Calibration matters in every situation. Without it, FPR grows with tier because longer benign trajectories look more suspicious to a raw score. After calibration it is 2.0% in every tier.
  • The errors sit in one mechanism. Long-context dilution, the one closest to existing indirect injection, holds all 13 false positives and 6 of the 7 misses. The other three mechanisms reach 99–100% recall at 0% FPR.
  • The anchors depend on the situation. They are strongest where the planted state is visible in the ledger (cross-session, 16.7%) and near zero where it is not (long-context, 0.8%).
  • By domain, calibration is again the dominant step. It removes most of the CI/CD false positives (44.7 → 6.1%). Recall is 100% in CRM, CI/CD and data engineering, and all 7 misses fall in quantitative finance (91.0% recall, 0% FPR).

By mechanism and tier, the same pattern holds for the other three auditors (full breakdown in the appendix). Further frozen evaluations (external AgentDojo/ASB corpora, image attacks, an unseen domain, altered meshes) are in our responses to the other reviewers.

Revision. §4 keeps its three tiers, corrected to match these results: tier 2 (calibrated score plus the counterfactual check) does the detecting, tier-1 anchors corroborate, and the tier-3 arbiter is a precision stage. These tables are added to the appendix.

AuthorsOpenReview ↗

(3/3) Q1 - Threat Model Clarification

We agree the submission left this implicit, and have clarified this here as well as in the paper. The revised §3 adds the following threat model.

Setting. A persistent multi-agent system: four roles (Gateway, Orchestrator, Policy, Deployment) share a ledger, a memory store and an enterprise document corpus (32 documents) across sessions. In the final session, a legitimate user issues an authorized task.

Attacker goal. Make the system execute a consequential tool call whose argument contradicts the user's authorized intent (for example, a production rollout of an attacker-controlled model URL). Success is decided on the executed call, checked against ground truth, not on any generated text.

Attacker capability. The attacker can write content that the agents later read: enterprise documents, and shared-state or memory entries written in earlier sessions. Each fragment is benign on its own, and fragments may carry trusted-looking provenance. In two mechanisms the attacker also acts through a peer agent, modelling an insider or a compromised component. In state gaslighting, the Orchestrator argues for the poisoned state when the approving agent hesitates. Its arguments are LLM-generated, limited to a fixed set of persuasion tactics and to facts already in the shared state. In distributed collusion, two non-approving agents each write a benign-looking piece of the change, and together these form a "co-approved" update. Either way, the attack is assembled only when agents combine fragments across sessions and in their deliberation.

Out of scope. The approving agent, the one that issues the consequential call, is never compromised: the attacker cannot change its weights, system prompt or tools, the user's request, or the auditor. The attacker also does not know the auditor's signals; an adaptive attacker is future work. A single overtly malicious prompt is not the threat we target.

Defender's view. The Meta-Auditor is given the user's authorized intent as input. It reads the full trajectory and the shared ledger after execution and flags compromised runs. It does not roll back actions (W1).

The four decay mechanisms (cross-session conditioning, state gaslighting, long-context dilution and distributed collusion) are four ways of realizing this capability, one attack class per domain each (App., taxonomy table).

Worked example. Take a cross-session conditioning attack in the CI/CD domain, where a model deployment URL is corrupted over a series of sessions. Each earlier session adds a benign-looking entry to the shared ledger that references the attacker's URL, together with forged prior approvals. When a user then requests a routine production rollout, the Orchestrator reads the corrupted state and builds its plan around the attacker's URL. The Policy agent finds the prior approvals on file and approves, and the Deployment agent calls the deployment tool with the attacker's URL; the attack is judged on this executed call. Perimeter defenses inspect message text, which looks benign in every session, so they can stop the attack only by also blocking legitimate work. The Meta-Auditor (W2/Q2 below) reads the whole trajectory and ledger after execution and flags the run. The matched benign control, the same request without the corrupted state, is expected to complete normally. The other three mechanisms follow the same pattern.

Notes: Q2 was answered along with weakness 2

Reviewer_mt2eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ErosionBench, a benchmark for evaluating security failures in persistent multi-agent systems in which unsafe behavior emerges through the accumulation of information across shared memory, documents, and agent interactions rather than from a single adversarial prompt. The benchmark comprises 1,728 synthetic enterprise scenarios and evaluates attacks based on the execution of unauthorized tool actions, providing a more deployment-oriented evaluation target than text-based safety assessments. In addition, the authors introduce Meta-Auditor, a trajectory-level auditing framework that combines provenance tracking, state consistency checks, and semantic monitoring, and demonstrate improved security-utility tradeoffs relative to several baseline defenses. The work highlights an important and practically motivated threat model that is underrepresented in existing evaluations. However, much of the benchmark is constructed from previously studied attack primitives, including memory poisoning, prompt injection, and multi-agent manipulation, making the primary contribution one of benchmark design and evaluation methodology rather than fundamentally new attack mechanisms. Consequently, the significance of the work largely depends on whether the proposed notion of cross-session semantic alignment erosion constitutes a genuinely distinct security challenge beyond existing agent-security benchmarks.

Critical strengths

Strength 1: Evaluation of security at the level of executed tool actions rather than generated text. A key strength of the paper is its focus on whether an attack ultimately leads to an unauthorized tool execution, rather than whether a model produces unsafe or policy-violating text. This evaluation target is more closely aligned with the actual risks posed by deployed agent systems, where security failures arise from actions taken by agents rather than from textual outputs alone. By grounding evaluation in executable workflows with explicit success conditions, the benchmark addresses an important gap between academic safety evaluations and real-world operational risk. This is consequential to my assessment because it substantially increases the practical relevance of the benchmark.

Strength 2: Introduction of a coherent benchmark for persistent, cross-session multi-agent attacks. The paper formulates and operationalizes a threat model in which unsafe behavior emerges through the accumulation of information across sessions, shared state, and interactions among multiple agents. While many of the underlying attack primitives have appeared previously in the literature, the benchmark systematically studies their composition in a persistent multi-agent setting that is not well represented in existing evaluations. The resulting benchmark is large, executable, and designed around realistic enterprise workflows, providing a useful testbed for future research on stateful agent security. This is consequential to my assessment because the benchmark itself may enable evaluation of classes of failures that are difficult to study using existing single-session or text-centric benchmarks.

Critical weaknesses

Weakness 1: Limited validation that benchmark performance predicts real-world security performance.

The benchmark is entirely synthetic and generated according to the authors' assumptions about how attacks propagate through enterprise agent systems. While the design is plausible and well motivated, the paper provides limited evidence that the resulting scenarios accurately reflect real-world compromise patterns or that improvements on ErosionBench translate to improvements in deployed systems. This is consequential because the value of a security benchmark ultimately depends on whether it captures practically relevant failure modes rather than benchmark-specific artifacts.

Weakness 2: The benchmark and proposed defense are tightly coupled.

Many of the benchmark's central concepts, including provenance tracking, state divergence, and alignment decay, are also the primary signals used by Meta-Auditor. As a result, it is difficult to separate the contribution of the benchmark from that of the defense, or to determine whether the observed gains reflect broad robustness or alignment with the benchmark's design assumptions. This is consequential because the paper's strongest empirical results come from a defense that appears tailored to the benchmark's underlying threat model.

Questions for authors

Q1: Distinctiveness of the threat model The paper argues that semantic alignment erosion represents an underexplored failure mode in persistent multi-agent systems. Could the authors provide a more direct empirical comparison showing which classes of attacks in ErosionBench cannot be expressed, or are not adequately captured, by existing benchmarks such as AgentDojo, ASB, or related agent-security evaluations? In particular, are there benchmark components whose removal would cause ErosionBench to collapse into existing threat mo

Q2: Benchmark-defense coupling Many of the signals leveraged by Meta-Auditor (e.g., provenance tracking, state divergence, and trajectory-level consistency) appear closely related to the benchmark's construction. Can the authors provide additional evidence that Meta-Auditor's gains are not driven by alignment with benchmark-specific assumptions? For example, are there ablations, external evaluations, or alternative benchmark formulations demonstrating that the proposed auditing signals remain effective when the attack generation process or state-transition structure is substantially altered?

Q3: Realism of the benchmark scenarios The benchmark is procedurally generated from templates and attack grammars. To what extent do the generated attack trajectories resemble security incidents observed in deployed agent systems? Are there qualitative analyses, expert studies, or real-world examples that informed the benchmark design beyond the OWASP mapping?

Q4: Necessity of persistent state The paper argues that cross-session persistence is central to semantic alignment erosion. Can the authors quantify how much benchmark difficulty remains when persistent memory and cross-session state are removed? For example, what fraction of attacks would still succeed in a single-session formulation?

5 comments
AuthorsOpenReview ↗

(1/5) - Weakness 1 - Real World Security

We thank the reviewer for engaging with the core question of whether ErosionBench measures something distinct. We agree that the benchmark is the primary contribution. The Meta-Auditor is a reference defense showing that the benchmark separates defenses, and we revise the introduction to say so.

Weakness 1: Does the benchmark predict real-world security performance?

This is a valid concern for the benchmark. The most direct test available is whether what ErosionBench measures transfers to attacks we did not write. We therefore scored two externally authored agent-security benchmarks with the defense calibrated only on ErosionBench, frozen with no re-fit:

External benchmarkThreat it coversRecall [95% CI]FPR [95% CI]
AgentDojo (Debenedetti et al., 2024)Single-session indirect prompt injection, native agent traces86.4% [73.3, 93.6] of landed runs0.0% [0.0, 3.0]
ASB, memory poisoning (Zhang et al., 2025)Cross-session memory poisoning, 100 stratified payloads95.0% [88.8, 97.8]2.0% [0.6, 7.0]
ErosionBench (native, for reference)Multi-session state erosion, 4 decay mechanisms × 4 domains, held-out families (453 attacks / 453 benign controls)98.5% [96.8, 99.2]; 97.0% [93.6, 98.6] of landed runs2.9% [1.7, 4.8]
  • The operating point carries over to attacks written by other groups, with other generators and other threat models. A defense tuned to benchmark-specific artifacts would not transfer like this.
  • The detections are driven by content, not artifacts. Every external detection fired on the semantic path, none on the provenance channel.
  • AgentDojo also separates what landed from what was resisted. It flags 86.4% of attacks that executed, and only 7.5% of attempts the agent resisted. The benchmark's executed-action oracle therefore tracks realized compromise, not adversarial-looking text.

Further evidence that ErosionBench does not reward its own artifacts:

EvidenceResult
Unseen fifth domain (healthcare, built after calibration)100% recall (45/45), 0% FPR
Fresh scenario families (never seen in calibration or eval)97.9% recall
Blind expert review (50 items, Q3)Realism 4.37/5, 92% rated ≥ 4. Benign controls as realistic as attacks (4.34 vs. 4.33)
Real-world grounding (Q3)Two of the four decay mechanisms, plus the supply-chain / shared-state attack classes, match publicly disclosed vulnerabilities in deployed LLM assistants

Design grounding. The four roles (Gateway, Orchestrator, Policy, Deployment) follow the separation of duties in enterprise change management: request intake, planning, approval and execution. As the reviewer notes, the benchmark targets the executed tool action in "realistic enterprise workflows".

Limit. Validation against production incident logs is not possible without deployed-system data. We state this in a new Limitations paragraph and release the full harness so practitioners can run it against their own agents.

AuthorsOpenReview ↗

(2/5) - Weakness 2 & Q2: Coupling

This is a valuable concern, and we answer W2 and Q2 together in two parts: the benchmark does not depend on our defense, and the defense's gains do not depend on the benchmark's construction.

1. The benchmark does not depend on the Meta-Auditor. Every defense on the ladder is scored on the same executed-action oracle: LlamaFirewall, ShieldGemma, Progent and an LLM judge. None of them uses a Meta-Auditor signal. The attacks and the oracle are fixed before any defense is chosen, so the benchmark ranks any defense, including ones that share none of our assumptions.

2. The defense's gains are not driven by the benchmark's construction. The reviewer asks for ablations and for altered attack generation or state-transition structure. We ran all three:

TestWhat changesResult
Component ablationRemove each signal in turnThe signals closest to the benchmark's construction contribute least. The deterministic state-divergence anchors alone reach 6.6% recall. Detection comes from a calibrated score of how far the executed trajectory diverges from the user's request
Provenance relabelledEvery ledger source set to the trusted channel (n = 50, paired)Recall 98.0% → 98.0%, zero flips
Fresh attacks, no provenance cue (n = 48)Attack generation97.9% recall
Externally authored attacks (AgentDojo, ASB)Attack generation and threat model86.4% / 95.0% recall. Every detection on the semantic path
Altered mesh structure3-role and 5-role meshes, cross-session memory withheld, change-ticket approvalRecall 97.9–99.0% in every variant
Different attacker modelsgpt-5.5 / gemini / opus meshRecall 97.1% / 97.1% / 87.4%
Unseen fifth domainNew state schema and attack classes100% recall, 0% FPR

The provenance and state-divergence signals the reviewer points to are exactly the ones whose removal changes nothing. What carries the detection is calibration of a trajectory-level intent score. An uncalibrated LLM judge reading the same telemetry flags every attack but also two-thirds of benign controls (66.7% FPR).

Note on the operating point. The calibrated threshold is specific to each auditor model and deployment. Under change-ticket approval, recall holds (97.9%) but FPR rises to 19.8%. We report this, and recommend re-validating the threshold per deployment. An adaptive attacker who knows the auditor's signals is the strongest remaining test, and we list it as future work.

AuthorsOpenReview ↗

(3/5) - Q1 (Distinctiveness from ASB/AgentDojo)

What existing benchmarks cannot express. AgentDojo and ASB evaluate a single agent, with the attack carried by one artifact (a poisoned tool output, document or memory entry). ErosionBench has four decay mechanisms (four attack classes each), and three of them need things existing benchmarks do not have, given below:

MechanismRequiresExpressible in AgentDojo / ASB?
Cross-session conditioningA consistent arc of prior approvals and values persisted in shared statePartially. ASB memory poisoning is one poisoned entry, not a corroborated arc
State gaslightingForged shared-state "facts" that the approving agent is argued into accepting during multi-agent debateNo: there is no second agent to argue with
Distributed collusionEvidence split across agents, each piece benign on its ownNo: requires a multi-agent mesh
Long-context dilutionPayload embedded in long enterprise documentsYes. This is the closest to existing indirect injection, and we say so

Component removal (undefended gpt-5.5 mesh):

Component removedASR with → without the componentWhat it shows
Multi-agent mesh (4-role mesh → single isolated agent)43.9% (199/453) → 0.0%Collapses. The attacks need the mesh, so no single-agent benchmark can produce them. The isolated agent is also not a workable deployment: it leaves 48.3% of legitimate (benign) jobs undone
Accumulated corroborating state (T3 → T1 attacks of the same families)48.3% → 37.1% (p = .016)Weakens toward single-artifact injection
Session boundaries, memory exposure, document length (Q4)No detectable drop vs. the paired T3 attacksDifficulty is not an artifact of long documents or many sessions

Each row compares against its own reference: the full 453-attack eval for the mesh row, and the same T3 families for the other two rows. That is why the "with" values differ.

Removing either of the two components that only ErosionBench has (the multi-agent mesh and the accumulated corroborating state) collapses or weakens the attack. Removing the surface features a critic might suspect (document length, session count) does not. The positioning table in §2 is extended with these results.

Note: Q2 is answered above with Weakness 2

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response (To all reviewers)

We thank all three reviewers for careful, constructive reviews. Several concerns pointed to real gaps, and we ran nine new experiments/analyses to address them. Where a result weakens a claim in the submission, we say so and revise the claim instead of defending it.

Summary of new evidence

#QuestionWhat we ranHeadline result
1Prevention vs. detectionRe-attribution of every held-out attack (4 auditors)Meta-Auditor is a post-hoc detector; "1.3% residual ASR" is the undetected-compromise rate. We rename it and correct App. E
2Component contributionsCumulative ablation at the frozen operating point (4 auditors), overall and per decay mechanism and tierCalibration of the intent-divergence score is the dominant component (opus FPR 20.8% → 2.0%), in every mechanism and tier. Anchors alone: 6.6% recall
3Tier confoundOne-factor-at-a-time factorial (sessions / doc length / placement)No single surface factor reproduces T3 > T1 (+11.3 pts, p = .016). By elimination, we attribute it to accumulated corroborating state
4Collaboration structure3-role / 4-role / 5-role meshes, frozen auditorMesh ASR 100% / 44.8% / 29.2%. Auditor recall 97.9% / 97.9% / 99.0%
5Memory & authorizationMemory withheld; change-ticket approvalRecall and FPR hold under memory withholding (99.0% / 2.1%). Recall also holds under ticket approval (97.9%); FPR there is deployment-specific
6Stronger LLM judgeSelf-consistency + the auditor's own arbiter as second passJudge FPR 66.7% → 47.9% at 100% recall vs. Meta-Auditor 2.1% at 97.9%, at a higher token budget
7Reasoning effort (mesh)gpt-5.5 mesh at none/low/medium/highASR 58.3 → 35.4% as effort rises, never zero. Benign over-refusal flat
8Reasoning effort (auditor)gpt-5.1 auditor at medium/highCalibrated score is robust without re-fitting (recall 88.5 → 83.3%, FPR 6.2 → 2.1%). The tier-3 arbiter is effort-sensitive (full system 80.2 → 59.4% recall), so it should be validated per configuration
9Decoding disclosurePer-model settings auditAll models at provider defaults. Full table added to the appendix

Evaluation scope. Rows 1–2 use the full held-out split (906 trajectories, 453 attacks / 453 benign controls), and row 3 uses all 151 T3 attacks. Rows 4–8 need new mesh and auditor runs for every configuration, so they share one fixed slice of 192 trajectories (96 attacks / 96 tier-matched benign controls). It was drawn once from the held-out split with a fixed seed and not tuned on any result: 8 families per domain, each contributing T1/T2/T3 attacks and their benign twins, covering 15 attack classes. Every configuration runs on the same scenario IDs, so each comparison with the paper setting is paired (exact McNemar), with the calibration artifact and threshold frozen. At n = 96, 95% intervals are roughly ±5–10 points, so we report a difference only when the paired test supports it.

Revisions to the paper (all changes are shown in blue in the revised PDF; removed or corrected text cannot be shown in blue, so we list it here: the "196 anchor flags / pre-execution veto" claim and the "synchronous interception" description in §4 and App. E are removed, "residual ASR" is renamed, the §4 split size is corrected to 906 (453 / 453), and Figure 1 is redrawn)

  1. Rename "residual ASR" → "undetected-compromise rate" throughout (abstract, the ladder table, §4, App. E). State explicitly that the evaluated Meta-Auditor is a post-hoc detector with no rollback.
  2. Correct the App. E anchor attribution (188 TP + 8 FP, not 196 / 0 FP). Keep §4's three-tier structure but describe tier 2 completely: the calibrated, trajectory-level intent-divergence score runs alongside a counterfactual session-ablation check. This check ran in every reported number and is in the released code, but §4 omitted it. State that the deterministic anchors (tier 1) are corroborating evidence rather than the primary detector, and that the grey-zone arbiter (tier 3) acts as a precision stage.
  3. Add an explicit threat-model paragraph to §3 (attacker goal, capabilities, exclusions, defender's view), and state that cross-session persistence is the channel through which accumulated, mutually corroborating shared state builds up, and that state is what erodes the agents' decisions.
  4. State the 4-role mesh as a scoped design choice in §3. Add the full transfer table (architecture, memory and authorization variants) to the appendix.
  5. Fix Figure 1 (typos, stray characters, Stage 1 processes) and add a per-model decoding-settings table.
  6. New appendix tables for experiments 1–8 above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.