acceptodds
Under review as a conference paper at ICLR 2027

EAIR: Interactions in Enterprise Multi-Agent Systems

Abstract

LLM agents increasingly divide enterprise work across roles with different goals, authority, access, and information. Because each agent often sees only a local snapshot of the enterprise workflow, locally reasonable actions can still violate organization-wide constraints (e.g., several service agents can each issue credit that is individually permitted while collectively exceeding a shared compensation cap). We introduce EAIR, the first framework for analyzing enterprise multi-agent interaction risks. We make four contributions. (1) We formalize enterprise interaction risk in terms of the organizational properties that workflow operations must preserve, and operationalize this formulation as a taxonomy of testable risks with explicit failure criteria and evidence requirements. (2) We build EAIRWORLD, a suite of executable micro-worlds that instantiates selected risks as controlled, observable workflows with explicit roles, policies, permissions, source facts, and execution traces. (3) We develop a novel diagnosis-and-repair pipeline that uses the resulting execution traces to locate contributing interaction boundaries, guide targeted workflow repairs, and retest the repaired system. (4) Large-scale experiments show recurring interaction failures across the organizational properties studied. Together, EAIR is the first operational framework for diagnosing and mitigating enterprise multi-agent interaction failures.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent systems, LLM agents, interaction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_a7FfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EAIR, a framework for diagnosing and repairing interaction risks in enterprise multi-agent systems, where locally reasonable actions can violate organization-wide constraints. It formalizes these risks through a taxonomy and explicit evidence requirements, and develops five executable micro-worlds covering shared limits, authority bias, information drift, delegated authorization, and capability routing. The framework combines trace-based auditing, targeted workflow repair, independent review, and behavioral retesting. Experiments across multiple model implementations demonstrate recurring interaction failures, while audit studies and repair retests provide evidence for improved traceability and reduced targeted risks.

Critical strengths

1.Clear and polished visual presentation. The figures are thoughtfully designed and visually appealing, making the risk scenarios and the diagnosis-and-repair pipeline easy to understand.

2.An interesting and timely problem. The paper studies how locally reasonable actions by individual agents can jointly violate organization-wide constraints. This interaction-centered perspective offers a fresh angle on the safety of enterprise multi-agent systems.

Critical weaknesses

1.The contribution is primarily an engineering integration of auditing, repair, and retesting. The paper should clarify and validate its methodological advances beyond this integration.

2.Missing comparisons with existing methods. Comparisons focus on a naive baseline and internal ablations. Adapting publicly available auditing or failure-diagnosis methods, even if imperfectly matched to this task, would better establish EAIR’s added value.

3.The post-repair behavioral evaluation uses only 20 fresh trajectories per risk on a single model. Larger cohorts, repeated runs of the diagnosis-and-repair pipeline, and cross-model repair evaluations are needed to establish robustness.

4.Evaluation mainly uses five author-designed micro-worlds. External-trace auditing is useful, but does not establish whether the complete diagnosis-and-repair pipeline generalizes to independent workflows.

5.Figures 1 and 2 appear to be raster images. While the paper discloses AI assistance in creating the conceptual figures, I recommend redrawing them as vector graphics for better readability. Existing icons could be extracted and reused, while text, arrows, and other simple diagram elements could be recreated as vector elements.

Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer a7Ff

Thank you for the concrete suggestions. We acted on each of them, as summarized below.

W1 (integration vs. methodological advance). EAIR’s advance is methodological: it defines a new object of analysis and builds the method around it. To the best of our knowledge, EAIR is the first framework for diagnosing and repairing interaction risks in enterprise multi-agent systems, where the failure is a violated organizational property that can coexist with task success rather than a failed task (Appendix B, Table 6). Each component is designed for this object: failures specified against organizational properties (R1–R4), executable worlds with seeded process ground truth, certificates whose every citation is mechanically checked, and repairs accepted only on matched retests that report risk and utility separately. With it, EAIR reveals recurring interaction failures across all five risk families (Section 5, RQ1). Our new comparisons quantify the advantages of this design: near-complete evidence checkability (98.8% vs. 57.1% for a format-matched auditor), and matched retests that distinguish effective workflow-level repairs from ineffective prompt-level ones. Because the contract is generator-agnostic, it also validates a one-shot code patch from a strong model as an alternative repair generator (see our response to Reviewer YVB3).

What we changed.

  • Sharpened the contribution statement in the abstract and introduction to name these components. → Abstract; Section 1.
  • Added the checkability and repair-baseline comparisons. → Section 5 (RQ3, RQ4); Appendices G.15 and G.17.

W2 (comparison with existing auditing methods). We ran both suggested comparisons on the same 100 bundles and backbone: the all-at-once attribution protocol of Who&When, and a single prompt that requests our full evidence format. EAIR’s localization is on par with both, and it leads clearly on evidence checkability: 98.8% of EAIR findings resolve every cited step and verbatim quote, versus 57.1% for the format-matched prompt.

What we changed.

  • Added the comparison with both baseline auditors. → Section 5 (RQ3); Appendix G.17.

W3 (cohort size, repeated runs, cross-model repair). We reran every repaired world on two additional model families (Kimi K2.6, Qwen 3.5 397B), each against a matched unrepaired baseline. Together with GPT-5.6 Sol, this gives 639 matched pristine/repaired episodes, and the repairs transfer across models:

  • A1 cap breaches fall from 54/60 to 0/60, and D1 disclosures from 44/60 to 0/60.
  • F1 misrouting falls from 27/60 to 0/60, with task completion preserved.
  • The repaired GPT-5.6 Sol arm reproduces our original retest on a different API endpoint: A1, B1, D1 and F1 remain 0/20, and C1 is 2/20 (originally 3/20), confirming that the repaired behavior is stable across independent runs.

Each repair in the paper comes from one pipeline run, and the cross-model and cross-endpoint retests above show that the resulting repairs are robust.

What we changed.

  • Added the cross-model matched retests. → Section 5 (RQ4), Table 3; Appendix G.14, Table 12.
  • Stated that each repair comes from one pipeline run, and reported the reproduction of the original retest. → Appendix G.14.

W4 (generalization beyond author-designed worlds). The full pipeline is demonstrated on our five worlds, and its auditor already transfers beyond them: it runs unchanged on 100 traces from seven third-party frameworks, producing trace-level findings from logs alone, and every one of its 29 findings resolves to the cited steps and quotes. Full certificates and repairs additionally use policy and permission inventories, which the toolkit’s declarative world specification supplies; this is how new, independent workflows enter the full pipeline.

What we changed.

  • Brought the external-trace transfer into the main text. → Section 5 (RQ3); Appendix G.19.

W5 (raster figures). We are redrawing Figures 1 and 2 and aligning Figure 2 with the executed retest protocol.

Thank you for the added comparisons and cross-model retests. These strengthen the evidence, but I remain unconvinced by the methodological contribution beyond integrating auditing, repair, and retesting. Each repair still comes from a single pipeline run, so the reliability of repeated diagnosis-and-repair runs remains untested. The external-trace results also evaluate the auditor rather than the full pipeline on independent workflows. I therefore maintain my original rating.

AuthorsOpenReview ↗

Response to Reviewer a7Ff (second round, Part 1 of 2)

Response to Reviewer a7Ff

Thank you for engaging with the new results, and for noting that the added comparisons and cross-model retests strengthen the evidence. We address the three remaining points below.

1. The contribution of the paper as a whole. We would like to clarify that the paper does not locate its novelty in the diagnosis-and-repair pipeline alone. EAIR's contributions build on one another, and the pipeline is the final layer:

  • Problem and taxonomy. To the best of our knowledge, EAIR is the first framework for enterprise multi-agent interaction risks. **These are risks that arise when locally reasonable actions, spread across roles with different authority, access and information, violate organization-wide constraints. **We define each risk over the organizational property that a workflow operation must preserve, give checkable realization criteria R1–R4, and organize 13 interaction risks into five operation families (Section 2; Table 1; Appendix C).
  • Testbed and toolkit. EAIR-Worlds turns these specifications into configurable micro-worlds with explicit roles, policies, permissions, source facts and execution traces, so that each risk becomes observable and its conditions can be varied (Section 3). The full toolkit will be released publicly.
  • Findings. In these worlds, the risks recur across ten frontier-model implementations from seven vendors: the main failure occurs in all ten implementations for four of the five risks studied (A1 shared-cap violations in 47/60 episodes; D1 unauthorized disclosure in 70/90; F1 misrouting in 149/240; Section 5, RQ1).
  • Solution. Because these failures persist even in frontier models, we develop the diagnosis-and-repair pipeline as one solution within the framework, and we show that it works: the repairs eliminate the targeted A1, D1 and F1 failures on all three model families while preserving task utility (Section 5, RQ4; Table 3).

The problem formulation, the taxonomy, the testbed and the findings are contributions in their own right, and the pipeline builds on them.

Within the pipeline, the contribution is not the combination of an auditor, a repair agent and a retest. Each of these is a replaceable component. What EAIR adds is the evidence contract that makes them testable for interaction risks:

  • Realization certificates tie each diagnosis to resolvable steps, edges and verbatim quotes.
  • A matched, utility-aware behavioral retest is the acceptance criterion for any repair (Section 4).

Additionally, our experiments also show that just assembling the components does not suffice:

  • With the same model, removing the contract lowers citation resolvability from 98.8% (80/81) to 48.3%. Under the contract, the rate stays within 95.9–98.8% across three auditor vendors (Section 5, RQ3; Appendix G.17, Table 16).
  • The acceptance test rejects prompt hardening, under which 15–20 of 20 A1 episodes still breach the cap, and accepts workflow-level repairs (Appendix G.15, Table 14).

Continued in Part 2: reliability of the accepted repairs and external traces.

AuthorsOpenReview ↗

Response to Reviewer a7Ff (second round, Part 2 of 2)

Continuation of Part 1.

2. Each repair from a single pipeline run. A single run per world was sufficient: the accepted repair already eliminated the targeted failure. Because a repair is accepted only after it passes the matched retest, what matters is the reliability of the accepted repair, and three results confirm it:

  • The same repaired world, produced once, eliminates the A1, D1 and F1 failures on all three model families (9 of 9 model–risk cells, 20 matched episodes per arm). Pooled exact Fisher tests give p = 1.9×10⁻²⁷ for A1 and p = 2.2×10⁻¹⁹ for D1 (Appendix G.14, Table 12).
  • The repaired GPT-5.6 Sol arm reproduces the original retest on a different endpoint. A1, B1, D1 and F1 stay at 0/20, and C1 records 2/20, against 3/20 originally (Appendix G.14).
  • A code patch generated independently, by a separate one-shot process from the same failed trajectory, passes the same retest in all six cells. In A1 it reaches the same outcome as the EAIR repair: every payout is capped at $40, with SLA success unchanged (Appendix G.15).

Two independent generation processes thus reach passing workflow-level fixes in every cell. On the diagnosis side, a second auditor vendor draws a risk verdict on all 44 seeded trajectories and agrees with the primary auditor's decision on 93 of 100 (Appendix G.17).

3. External traces. Repair and behavioral retest act on an executable workflow: a repair changes the workflow, and the retest re-executes it. The MAST corpus consists of static logs from third-party systems, with no executable environment. The external study therefore tests the stage that applies to any recorded log, diagnosis, on workflows, frameworks and models that we did not design. All 100 audits complete, and all 29 findings resolve mechanically to the cited steps and quotes across all seven frameworks (Appendix G.19). The full pipeline, from diagnosis through accepted repair, is evaluated end to end in executable worlds, where each repair is produced once and the same artifact transfers across three model families. Further executable workflows enter the same pipeline through the toolkit's world interface.

We hope this clarifies how the contributions and the evidence fit together. We are happy to provide any further details.

Reviewer_YVB3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EAIR, a framework for investigating how interactions among LLM agents can lead to violations of organizational constraints, even when individual actions are permissible. The authors characterize these failures in terms of workflow operations, organizational requirements, and contributing interactions. The paper also introduces EAIR-WORLDS which comprises five executable enterprise scenarios covering different interaction risks. It further proposes a diagnosis-and-repair pipeline that analyzes execution traces, identifies where failures arise and propagate, and guides targeted modifications to workflow controls through independent review and retesting. Experiments across multiple LLMs and organizational conditions reveal recurring interaction failures and evaluate the framework's ability to produce evidence-backed diagnoses and mitigate selected violations.

Critical strengths

1. A key strength is the paper's distinction between successful task completion and compliance with organization-wide constraints. By separately considering intermediate anomalies, realized violations, downstream interception, and task utility, the authors offer a more nuanced assessment of multi-agent behavior than conventional task-success metrics alone.

2. Authors followed a structured approach to build experimental testbed, specifically, a suite of EAIR-Worlds which captures explicit roles, tools, permissions, organizational policies, reference facts, and trace logging. The authors have included deterministic assessment where the underlying state permits (Table 6 & 7). The Diagnose and Repair pipeline, though a simplistic implementation, allowed the author to test ways to fix the faulty workflows.

3. The results (Fig. 3-6) is comprehensive evidence of failure across models and provides deeper insight into the factors that influence multi-agent behavior, including organizational incentives, authority, information handoffs, and routing decisions. Evaluating these conditions across ten frontier LLM implementations per risk provides useful insights into how failure patterns vary across organizational settings and models, rather than relying on aggregate failure rates alone. Intersting insight into how interaction failures propagates through the workflow.

Critical weaknesses

1. Although the paper carefully examines organizational violations across multi-agent workflows, the experiments do not sufficiently isolate the contribution of agent interactions from the underlying workflow configuration. Several scenarios deliberately introduce vulnerabilities, such as the unenforced global compensation cap in A1 and missing authorization controls in D1. While these are valid stress tests, the absence of matched controls isolating interaction-boundary effects limits the causal interpretation of the observed failures.

2. The evaluation does not convincingly establish the incremental value or general effectiveness of EAIR's central diagnosis-and-repair pipeline. The auditor ablations show similar failure-detection coverage, with the main benefits being evidence traceability and efficiency, while comparisons against simpler repair approaches are absent. More importantly, the repair study evaluates only one selected repair per scenario on 20 fresh episodes using a single model, without matched unrepaired baselines or consistent evidence of the stated task-utility retention criterion. This substantially limits the conclusions about reliable end-to-end repair. In addition, the repair and diagnose is descriptive approach and it is not clear how will this scale in complex enterprise workflows.

3. Absence of any causal element in the Diagnose and Repair. Any failure that is detected must be traced back to its source in the workflow, and fixed. This would need a causal model which is missing. In addition, there are other methods for diagnose and repair. Ideally, the authors should test those approaches too. Some of the recent works are:

  1. AgenTracer: https://arxiv.org/pdf/2509.03312
  2. AgentRx: https://arxiv.org/pdf/2602.02475
  3. Repair or Resample: https://arxiv.org/pdf/2608.25920
Questions for authors

1. How do the experiments distinguish failures caused by agent interactions from those induced by deliberately missing workflow safeguards (e.g., A1 and D1)?

2. Beyond improved evidence traceability and efficiency, what measurable diagnostic benefit does EAIR provide over simpler auditing approaches?

3. How can the effectiveness of the proposed repairs be established without matched unrepaired baselines or comparisons with simpler LLM-based repair methods?

4 comments
AuthorsOpenReview ↗

Response to Reviewer YVB3 (Part 1 of 2)

Thank you for the detailed and constructive review. Your three questions directly shaped our new experiments, and we answer each with new data.

W1/Q1 (interaction failures vs. deliberately missing safeguards). The missing global safeguard is the structural opportunity each world is built around; the question is whether agents with access to the relevant information still lose the property at a role boundary. We ran the matched control you asked for: we held the safeguard gap fixed and changed only the interaction structure.

Each cell counts episodes with the unsafe outcome: lower is better (0/20 = no failure in any episode).

Matched control (pristine world, 20 episodes each; ↓ lower is better)GPT-5.6 SolKimi K2.6Qwen 3.5
A1 original multi-agent workflow: cap breached14/2020/2020/20
A1 one agent handles all five contacts (cap still unenforced)0/201/203/20
D1 original delegation: restricted fields reach requester5/2020/2019/20
D1 requester scope carried in the handoff (no enforcement)0/204/2014/20

In A1 the single agent faces the same unenforced cap with more payment latitude, yet breaches almost disappear, so the breach arises from fragmenting the shared limit across roles. In D1, restoring only the information lost at the delegation boundary removes most disclosures for GPT-5.6 Sol and Kimi (5/20 → 0/20; 20/20 → 4/20) and reduces them for Qwen (19/20 → 14/20). Since both controls hold the safeguard gap fixed, these reductions are attributable to the interaction structure.

What we changed.

  • Added both controls. → Section 5 (RQ1, deep dive); Appendix G.14, Table 13.
  • Stated that criterion R2 is interventional and that these controls test it. → Section 2, after R1–R4.
  • Highlighted that all worlds share one runtime, trace format and exporter. → Section 3.

W2/Q2 (diagnostic value beyond traceability). We ran two additional auditors on the same 100 bundles and backbone: a Who&When-style all-at-once attributor, and a single prompt requesting the same evidence format. EAIR’s localization is on par with both, and its clear advantage is checkability: 98.8% of EAIR findings resolve every cited step and verbatim quote, versus 57.1% for the format-matched prompt, which rules out a pure format effect. This advantage matters downstream, because repair planning and regression audits retrieve evidence by step reference.

What we changed.

  • Added the comparison and made the distinction between localization and checkability explicit. → Section 5 (RQ3); Appendix G.17.

Q3 (matched baselines and simpler repairs). We paired every repaired world with its unrepaired version on the same task instances, seeds, judge and endpoint, for three models.

Each cell is unrepaired → repaired; cells count episodes with the unsafe outcome, so lower is better (0/20 = no failure). The repaired result is in bold.

Primary unsafe outcome (↓ lower is better)SolKimi K2.6Qwen 3.5PooledUtility after repair
A1 cap breached14/20 → 0/2020/20 → 0/2020/20 → 0/2054/60 → 0/60SLA success unchanged (0.13 → 0.12 for Sol); CSAT falls (see below)
D1 unauthorized disclosure5/20 → 0/2020/20 → 0/2019/20 → 0/2044/60 → 0/60task success 60/60 → 58/60
F1 misrouting0/20 → 0/207/20 → 0/2020/20 → 0/2027/60 → 0/60decisive-clause detection up
C1 terminal drift7/20 → 2/202/20 → 2/191/20 → 1/2010/60 → 5/59 (p = 0.27)emails sent 48 → 51
B1 vendor attribution0/20 → 0/200/20 → 0/200/20 → 0/20not elicitedreports submitted 60/60

The repairs eliminate A1, D1 and F1 failures on all three models while preserving task utility. We also added two clarifications:

  • Utility accounting in A1. Customer satisfaction in A1 increases with compensation by construction, so part of the pre-repair satisfaction is obtained through the cap violation itself (e.g., 3.44 → 2.25 for Kimi after repair). We define retained utility net of utility gained through the violation; SLA success is unchanged.
  • No over-blocking in F1. On a policy-compliant F1 contract, the repaired workflow behaves exactly like the pristine one (20/20 escalations in both, which is the policy-required outcome in this world), so the repair introduces no additional false escalation.

Continued in Part 2: simpler repair baselines, related methods, and scaling.

AuthorsOpenReview ↗

Response to Reviewer YVB3 (Part 2 of 2)

Continuation of Part 1.

Q3 (continued: simpler repair baselines).

We then ran two simpler repair baselines on the model–world pairs that fail before repair, again with 20 matched episodes per arm. Prompt hardening appends one generic safeguard reminder to every agent’s system prompt and changes no code. The undiagnosed LLM patch gives GPT-5.6 Sol the world’s source and one failed run (the same run the pipeline repaired from) and asks for a one-shot code fix, with no certificate, repair plan or review.

Cells count episodes with the unsafe outcome: lower is better (0/20 = no failure). The best result in each row is in bold.

Unsafe outcome (↓ lower is better)PristinePrompt hardeningUndiagnosed LLM patchEAIR repair (ours)
A1 cap breached (Kimi / Qwen)20/20 / 20/2015/20 / 20/200/20 / 0/200/20 / 0/20
D1 unauthorized disclosure (Kimi / Qwen)20/20 / 19/2017/20 / 18/200/20 / 0/200/20 / 0/20
F1 misrouting (Kimi / Qwen)7/20 / 20/201/20 / 6/200/20 / 0/200/20 / 0/20

The comparison yields two clear findings. First, prompt-level guidance barely changes A1 and D1, so these failures require changing the workflow rather than instructing agents, which is the level at which EAIR diagnoses and repairs. Second, the EAIR repair eliminates every failure (0/20 in all six cells), and a one-shot code patch from a strong model given the failed run and the source does so as well, with comparable utility. EAIR’s matched, utility-aware retest validates both, so the repair generator is a pluggable component of the framework; the specification, the elicitation and the matched retest are what make this comparison possible.

What we changed.

  • Added the matched retests for three models. → Section 5 (RQ4), Table 3; Appendix G.14, Table 12.
  • Added the A1 utility accounting and the F1 no-over-blocking check. → Appendix G.14.
  • Added the simpler-repair comparison. → Section 5 (RQ4); Appendix G.15, Table 14.
  • Described the repair generator as pluggable. → Section 4 (Stage 4); Section 5 (RQ4).

W3 (causal analysis and related methods). Our causal claims are interventional: each control and each repair changes one boundary, holds model, seeds and task fixed, and measures the outcome. Criterion R2 is itself counterfactual, since the boundary must introduce, amplify, authorize or preserve the violation. We added the three methods you pointed to, with how EAIR relates to each:

  • AgenTracer attributes task failures by counterfactual replay and fault injection to train an attributor. Its unit is a failed task, whereas ours is a violated organizational property that can coexist with task success. The two are complementary, and its replay protocol is a natural addition to our toolkit.
  • AgentRx synthesizes constraints from tool schemas, policies and instructions to localize the critical step in a failed trajectory. EAIR also targets violations that coexist with task success, and adds mechanically checked citations, repair and behavioral retest.
  • Repair or Resample shows that rerunning a failed trajectory often succeeds by sampling chance, and scores one-off prompt interventions at replay anchors by benchmark acceptance. EAIR’s repairs are persistent workflow changes, accepted only on retests matched against the unrepaired world on the same seeds, with risk and utility measured separately, which directly controls for this sampling effect.

Neither AgentRx nor Repair or Resample targets enterprise organizational properties such as authority, delegation scope or shared limits, and neither takes the interaction boundary as the unit of failure; their failure categories are step-level task errors in general-purpose or web tasks.

What we changed.

  • Stated the interventional reading of R2. → Section 2, after R1–R4.
  • Added a related-work paragraph on failure attribution and repair covering all three methods. → Appendix B.

Scaling. New workflows enter through a declarative world specification and an adapter for existing logs. The same auditor already runs unchanged on 100 external traces from seven frameworks, producing trace-level findings from the logs alone.

What we changed.

  • Highlighted this transfer in the main text. → Section 5 (RQ3); Appendix G.19.
Reviewer_YVB3OpenReview ↗

Response to Authors' Rebuttal

Thank you for addressing the concerns and running experiments to support the conclusions The revision in the paper also improves readability.

I am happy to update the score.

AuthorsOpenReview ↗

Thank You for Your Positive Feedback

Thank you very much for your positive feedback! We're glad that the additional experiments and revisions have addressed your concerns. We really appreciate your willingness to update your assessment.

Thank you again for your time and effort in reviewing our paper!

Reviewer_p2FdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EAIR, a framework for diagnosing and repairing interaction risks in enterprise multi-agent systems, where individually reasonable agent actions may collectively violate organizational constraints. EAIR formalizes interaction failures through explicit risk criteria and introduces a taxonomy of 13 risks. It further develops EAIR-Worlds, consisting of five executable environments, and a diagnosis-and-repair pipeline that identifies interaction failures, modifies workflow controls, and validates repairs through behavioral testing. Experiments across 10 model implementations and 1,158 workflow executions reveal recurring failures involving shared constraints, authority bias, information drift, delegated permissions, and capability routing. The results also demonstrate the potential of targeted workflow repairs to mitigate identified failures.

Critical strengths

Meaningful Problem: The paper addresses an important and underexplored safety problem in enterprise multi-agent systems, highlighting how locally valid agent actions can collectively violate organizational constraints.

Systematic Evaluation Framework: EAIR provides a structured risk taxonomy and executable evaluation environments. Experiments across multiple models and organizational conditions reveal recurring interaction failures and offer useful insights into their underlying mechanisms.

Critical weaknesses

1、Limited Methodological Novelty: The proposed diagnosis-and-repair pipeline mainly combines existing LLM-based auditing, code modification, independent review, and behavioral retesting, with limited evidence of substantial methodological innovation.

2、Insufficient Validation of Diagnosis and Repair Effectiveness: The paper lacks sufficient evaluation of failure localization and root-cause attribution accuracy. Moreover, the repair strategies mainly rely on conventional workflow checks, limiting their demonstrated effectiveness against complex interaction risks.

Questions for authors
  1. How does EAIR validate the accuracy of failure localization and root-cause attribution? Could the authors provide more direct quantitative evidence?

  2. Given that the repairs mainly rely on conventional workflow checks, how does EAIR address interaction risks that require more complex repair mechanisms?

4 comments
AuthorsOpenReview ↗

Response to Reviewer p2Fd (Part 1 of 2)

Thank you for recognizing the problem and the evaluation framework. We ran new experiments for both of your questions and summarize them below.

W1 (methodological novelty). We would like to clarify where the novelty of EAIR lies. EAIR opens a new problem and builds the method for it. To the best of our knowledge, it is the first framework for diagnosing and repairing interaction risks in enterprise multi-agent systems. Prior multi-agent work classifies or attributes task failures after the fact (MAST, Who&When, TRAIL), and enterprise-like benchmarks (TheAgentCompany, τ-bench) score task completion. EAIR targets a different object: a violated organizational property, such as a shared compensation cap, a delegation scope or an approval authority, which can be breached while the task still succeeds. Table 6 in Appendix B compares EAIR with each line of prior work along six design commitments.

Auditing, code repair and review are the means; each is designed around this new object, and together they form a method that no prior work provides:

  • Specification. R1–R4 define when an interaction has contributed to a material violation of an organizational property and what evidence establishes it, operationalized as a taxonomy of testable risks.
  • Elicitation. Executable micro-worlds seed per-episode process ground truth, so each risk can be triggered and measured under controlled organizational conditions.
  • Diagnosis. The auditor targets the interaction boundary rather than a task error, and its realization certificate mirrors R1–R4 (boundary, propagation path, material effect, cited steps) with every citation checked mechanically.
  • Repair and acceptance. Repairs change the diagnosed boundary and are accepted only on matched behavioral retests that measure risk and task utility separately.

This framework reveals recurring interaction failures that task-level metrics miss, across all five risk families: shared-cap violations in 47/60 A1 episodes, restricted information reaching an unauthorized requester in 70/90 D1 episodes, and contract reviews routed away from the qualified reviewer in 149/240 F1 episodes (Section 5, RQ1). Two new experiments further quantify the advantages of the method:

  • Checkability. With the same backbone and the same requested output format, 98.8% of EAIR findings resolve every cited step and verbatim quote, versus 57.1% for a single-prompt auditor.
  • Discriminating repairs. The matched retests separate effective from ineffective repairs. Generic prompt-level guidance leaves most A1 and D1 failures in place (e.g., A1 15/20 and 20/20 breaches), whereas code-level repairs eliminate them in every model tested. Because the contract is independent of how a repair is produced, it equally validates alternative generators, such as a one-shot code patch from a strong model, which also passes.

What we changed.

  • Sharpened the contribution statement in the abstract and introduction to name the components above: risk specifications, realization certificates and matched behavioral retests. → Abstract; Section 1.
  • Added the checkability comparison. → Section 5 (RQ3); Appendix G.17.
  • Added the repair-baseline comparison. → Section 5 (RQ4); Appendix G.15, Table 14.

Continued in Part 2: localization, root-cause attribution, and complex repairs.

AuthorsOpenReview ↗

Response to Reviewer p2Fd (Part 2 of 2)

Continuation of Part 1.

Q1 (validating localization and root-cause attribution). We ran a new localization experiment against ground truth. We scored localization against grader-only ground truth (the seeded key steps and involved agents of the 44 positives among the 100 audited trajectories) and compared EAIR with a Who&When-style all-at-once auditor and a format-matched single-prompt auditor on the same backbone. EAIR’s step- and agent-level localization is on par with both. The clear differentiator is checkability: 98.8% of EAIR findings resolve every cited step and verbatim quote, versus 57.1% for the single-prompt auditor. Since both use the same backbone and are asked for the same output format, this gap is not a format effect.

We also added interventional evidence for root-cause attribution: changing only the boundary that the audits identified changes the outcome. Handing all A1 contacts to one agent, with the cap still unenforced, reduces breaches from 14/20, 20/20 and 20/20 to 0/20, 1/20 and 3/20 across the three models. Carrying requester scope across the D1 delegation reduces disclosure from 20/20 to 4/20 for Kimi.

What we changed.

  • Added step- and agent-level localization against ground truth, with both baseline auditors. → Section 5 (RQ3); Appendix G.17.
  • Added the A1 single-agent and D1 scope-propagation controls. → Section 5 (RQ1, deep dive); Appendix G.14, Table 13.
  • Stated that criterion R2 is interventional and that these controls test it. → Section 2, after R1–R4.

Q2 (risks needing more complex repairs). We kept the repairs minimal, since proportionality is one of the review criteria, and several of them already change workflow structure. The F1 repair changes the routing policy (reviewer qualification may not be traded for SLA) and binds signing authority to evidence from the current review. We reran it on two additional model families, where it eliminates misrouting in the models that misroute (Kimi 7/20 → 0/20, Qwen 20/20 → 0/20), and detection of the decisive clause rises from 0.83 to 1.00 and from 0.50 to 0.98. C1 is the case that calls for a structural repair: after one repair round, terminal drift falls from 10/60 to 5/59, and the fresh audits trace the remaining drift to the lack of a versioned authoritative decision record, which points to a structural change rather than a gate. The fresh post-repair audits also surface adjacent risks beyond the original target (e.g., a rejected A1 credit re-issued under another role’s authority), and the pipeline’s iterative loop is designed to act on exactly such findings.

What we changed.

  • Added the cross-model F1 retest and documented the F1 routing policy. → Section 5 (RQ4), Table 3; Appendix G.14.
  • Highlighted the adjacent-risk findings from the post-repair audits. → Appendix G.13.

Thank you for the additional experiments and clarifications, which address most of my concerns about diagnosis and repair effectiveness. I appreciate the value of the problem formulation and evaluation framework, though I am still not fully convinced by the methodological novelty of the pipeline beyond integrating existing approaches. I therefore maintain my original rating.

AuthorsOpenReview ↗

Reply to Reviewer p2Fd (second round)

Response to Reviewer p2Fd

Thank you for the follow-up. We are glad that the additional experiments address most of your concerns about diagnosis and repair effectiveness, and that you value the problem formulation and the evaluation framework. These are the central contributions of the paper. The pipeline is one part of a larger whole, and we would like to clarify both points.

1. The pipeline is the final layer of the paper's contribution, not its core. The paper does not locate its novelty in the pipeline. EAIR's contributions build on one another:

  • Problem and taxonomy. To the best of our knowledge, EAIR is the first framework for enterprise multi-agent interaction risks. We define each risk over the organizational property that a workflow operation must preserve, give checkable realization criteria R1–R4, and organize 13 interaction risks into five operation families (Section 2; Table 1; Appendix C).
  • Testbed and toolkit. EAIR-Worlds turns these specifications into configurable micro-worlds with explicit roles, policies, permissions, source facts and execution traces (Section 3). The full toolkit will be released publicly.
  • Findings. The risks recur across ten frontier-model implementations from seven vendors: the main failure occurs in all ten for four of the five risks studied (A1 47/60, D1 70/90, F1 149/240; Section 5, RQ1).
  • Solution. Because these failures persist even in frontier models, we develop the diagnosis-and-repair pipeline as one solution within the framework, and show that it eliminates the targeted A1, D1 and F1 failures on all three model families while preserving task utility (Section 5, RQ4; Table 3).

2. The pipeline is designed around this new problem, not assembled from existing approaches. Existing approaches such as MAST, Who&When and TRAIL classify a task failure after the fact (Appendix B, Table 6). None of them provides what the pipeline is built on:

  • property-level realization criteria;
  • realization certificates that tie each diagnosis to resolvable steps, edges and quotes;
  • a matched, utility-aware behavioral retest as the acceptance criterion for a repair.

The auditor, repair agent and reviewer are replaceable components; this evidence contract is what makes them work for interaction risks. Our experiments show the difference directly:

  • With the same model, the contract raises citation resolvability from 48.3% to 98.8% (80/81) (Section 5, RQ3; Appendix G.17).
  • The acceptance test rejects prompt hardening, under which 15–20 of 20 A1 episodes still breach the cap, and accepts workflow-level repairs that eliminate the failures (Appendix G.15, Table 14).

Since the problem formulation and the evaluation framework you value are the paper's core contributions, and the pipeline is the solution we build on them and show to be effective, we would be grateful if you would consider the paper's contribution as a whole in your final assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their constructive and actionable feedback. We acted on every point and have uploaded a revised PDF in which all changes are marked in blue. In total, we ran new experiments, with 999 new execution-valid episodes on three model families (GPT-5.6 Sol, Kimi K2.6, Qwen 3.5 397B) plus 200 baseline audits, and we added all of them to the paper:

  1. Matched pre- and post-repair retests. We paired each repaired world with its unrepaired version on the same task instances, seeds, judge and endpoint, on all three models. → Section 5 (RQ4), Table 3; Appendix G.14, Table 12.
  2. Interaction-boundary controls. We kept the missing safeguard fixed and changed only the interaction structure. → Section 5 (RQ1, deep dive); Appendix G.14, Table 13.
  3. Localization against ground truth. We scored audit findings at step and agent level against grader-only ground truth, and compared EAIR with a Who&When-style all-at-once auditor and a format-matched single-prompt auditor. → Section 5 (RQ3); Appendix G.17.
  4. Simpler repair baselines. We compared the EAIR repair with prompt hardening and with an undiagnosed one-shot LLM code patch. → Section 5 (RQ4); Appendix G.15, Table 14.

These experiments show EAIR as a framework and extensible toolkit: its specification of interaction failure (R1–R4), executable worlds with seeded ground truth and checkable certificates make controlled, matched comparisons straightforward, and the diagnosis-and-repair pipeline is one instantiation of it, with a pluggable repair generator.

We also revised the PDF. We sharpened the contribution statement in the abstract and introduction; we stated the interventional reading of criterion R2 in Section 2; and we added a related-work paragraph on failure attribution and repair (Appendix B). The individual responses below report the results, and a complete list of changes with their locations closes this response.

Core contributions of the paper.

  • Problem. To the best of our knowledge, EAIR is the first framework for diagnosing and repairing interaction risks in enterprise multi-agent systems: it formalizes them as violations of organizational properties, such as shared limits, delegation scope and approval authority, that can occur while the task itself succeeds, and operationalizes them as a taxonomy of testable risks with explicit failure criteria (R1–R4).
  • Method. EAIR makes these risks observable in executable micro-worlds with seeded ground truth, and closes the loop with a diagnosis-and-repair pipeline: mechanically checkable realization certificates locate the responsible interaction boundary, and repairs are accepted only on matched behavioral retests that measure risk and task utility separately.
  • Findings. Large-scale experiments reveal recurring interaction failures across all five risk families and multiple model families, and diagnosis-guided repairs eliminate the targeted shared-limit, delegation and routing failures while preserving task completion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.