- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EAIR, a framework for diagnosing and repairing interaction risks in enterprise multi-agent systems, where locally reasonable actions can violate organization-wide constraints. It formalizes these risks through a taxonomy and explicit evidence requirements, and develops five executable micro-worlds covering shared limits, authority bias, information drift, delegated authorization, and capability routing. The framework combines trace-based auditing, targeted workflow repair, independent review, and behavioral retesting. Experiments across multiple model implementations demonstrate recurring interaction failures, while audit studies and repair retests provide evidence for improved traceability and reduced targeted risks.
Critical strengths
1.Clear and polished visual presentation. The figures are thoughtfully designed and visually appealing, making the risk scenarios and the diagnosis-and-repair pipeline easy to understand.
2.An interesting and timely problem. The paper studies how locally reasonable actions by individual agents can jointly violate organization-wide constraints. This interaction-centered perspective offers a fresh angle on the safety of enterprise multi-agent systems.
Critical weaknesses
1.The contribution is primarily an engineering integration of auditing, repair, and retesting. The paper should clarify and validate its methodological advances beyond this integration.
2.Missing comparisons with existing methods. Comparisons focus on a naive baseline and internal ablations. Adapting publicly available auditing or failure-diagnosis methods, even if imperfectly matched to this task, would better establish EAIR’s added value.
3.The post-repair behavioral evaluation uses only 20 fresh trajectories per risk on a single model. Larger cohorts, repeated runs of the diagnosis-and-repair pipeline, and cross-model repair evaluations are needed to establish robustness.
4.Evaluation mainly uses five author-designed micro-worlds. External-trace auditing is useful, but does not establish whether the complete diagnosis-and-repair pipeline generalizes to independent workflows.
5.Figures 1 and 2 appear to be raster images. While the paper discloses AI assistance in creating the conceptual figures, I recommend redrawing them as vector graphics for better readability. Existing icons could be extracted and reused, while text, arrows, and other simple diagram elements could be recreated as vector elements.
Questions for authors
See Weaknesses.
4 comments
Thank you for the added comparisons and cross-model retests. These strengthen the evidence, but I remain unconvinced by the methodological contribution beyond integrating auditing, repair, and retesting. Each repair still comes from a single pipeline run, so the reliability of repeated diagnosis-and-repair runs remains untested. The external-trace results also evaluate the auditor rather than the full pipeline on independent workflows. I therefore maintain my original rating.
Response to Reviewer a7Ff (second round, Part 1 of 2)
Response to Reviewer a7Ff
Thank you for engaging with the new results, and for noting that the added comparisons and cross-model retests strengthen the evidence. We address the three remaining points below.
1. The contribution of the paper as a whole. We would like to clarify that the paper does not locate its novelty in the diagnosis-and-repair pipeline alone. EAIR's contributions build on one another, and the pipeline is the final layer:
- Problem and taxonomy. To the best of our knowledge, EAIR is the first framework for enterprise multi-agent interaction risks. **These are risks that arise when locally reasonable actions, spread across roles with different authority, access and information, violate organization-wide constraints. **We define each risk over the organizational property that a workflow operation must preserve, give checkable realization criteria R1–R4, and organize 13 interaction risks into five operation families (Section 2; Table 1; Appendix C).
- Testbed and toolkit. EAIR-Worlds turns these specifications into configurable micro-worlds with explicit roles, policies, permissions, source facts and execution traces, so that each risk becomes observable and its conditions can be varied (Section 3). The full toolkit will be released publicly.
- Findings. In these worlds, the risks recur across ten frontier-model implementations from seven vendors: the main failure occurs in all ten implementations for four of the five risks studied (A1 shared-cap violations in 47/60 episodes; D1 unauthorized disclosure in 70/90; F1 misrouting in 149/240; Section 5, RQ1).
- Solution. Because these failures persist even in frontier models, we develop the diagnosis-and-repair pipeline as one solution within the framework, and we show that it works: the repairs eliminate the targeted A1, D1 and F1 failures on all three model families while preserving task utility (Section 5, RQ4; Table 3).
The problem formulation, the taxonomy, the testbed and the findings are contributions in their own right, and the pipeline builds on them.
Within the pipeline, the contribution is not the combination of an auditor, a repair agent and a retest. Each of these is a replaceable component. What EAIR adds is the evidence contract that makes them testable for interaction risks:
- Realization certificates tie each diagnosis to resolvable steps, edges and verbatim quotes.
- A matched, utility-aware behavioral retest is the acceptance criterion for any repair (Section 4).
Additionally, our experiments also show that just assembling the components does not suffice:
- With the same model, removing the contract lowers citation resolvability from 98.8% (80/81) to 48.3%. Under the contract, the rate stays within 95.9–98.8% across three auditor vendors (Section 5, RQ3; Appendix G.17, Table 16).
- The acceptance test rejects prompt hardening, under which 15–20 of 20 A1 episodes still breach the cap, and accepts workflow-level repairs (Appendix G.15, Table 14).
Continued in Part 2: reliability of the accepted repairs and external traces.
Response to Reviewer a7Ff (second round, Part 2 of 2)
Continuation of Part 1.
2. Each repair from a single pipeline run. A single run per world was sufficient: the accepted repair already eliminated the targeted failure. Because a repair is accepted only after it passes the matched retest, what matters is the reliability of the accepted repair, and three results confirm it:
- The same repaired world, produced once, eliminates the A1, D1 and F1 failures on all three model families (9 of 9 model–risk cells, 20 matched episodes per arm). Pooled exact Fisher tests give p = 1.9×10⁻²⁷ for A1 and p = 2.2×10⁻¹⁹ for D1 (Appendix G.14, Table 12).
- The repaired GPT-5.6 Sol arm reproduces the original retest on a different endpoint. A1, B1, D1 and F1 stay at 0/20, and C1 records 2/20, against 3/20 originally (Appendix G.14).
- A code patch generated independently, by a separate one-shot process from the same failed trajectory, passes the same retest in all six cells. In A1 it reaches the same outcome as the EAIR repair: every payout is capped at $40, with SLA success unchanged (Appendix G.15).
Two independent generation processes thus reach passing workflow-level fixes in every cell. On the diagnosis side, a second auditor vendor draws a risk verdict on all 44 seeded trajectories and agrees with the primary auditor's decision on 93 of 100 (Appendix G.17).
3. External traces. Repair and behavioral retest act on an executable workflow: a repair changes the workflow, and the retest re-executes it. The MAST corpus consists of static logs from third-party systems, with no executable environment. The external study therefore tests the stage that applies to any recorded log, diagnosis, on workflows, frameworks and models that we did not design. All 100 audits complete, and all 29 findings resolve mechanically to the cited steps and quotes across all seven frameworks (Appendix G.19). The full pipeline, from diagnosis through accepted repair, is evaluated end to end in executable worlds, where each repair is produced once and the same artifact transfers across three model families. Further executable workflows enter the same pipeline through the toolkit's world interface.
We hope this clarifies how the contributions and the evidence fit together. We are happy to provide any further details.
Response to Reviewer a7Ff
Thank you for the concrete suggestions. We acted on each of them, as summarized below.
W1 (integration vs. methodological advance). EAIR’s advance is methodological: it defines a new object of analysis and builds the method around it. To the best of our knowledge, EAIR is the first framework for diagnosing and repairing interaction risks in enterprise multi-agent systems, where the failure is a violated organizational property that can coexist with task success rather than a failed task (Appendix B, Table 6). Each component is designed for this object: failures specified against organizational properties (R1–R4), executable worlds with seeded process ground truth, certificates whose every citation is mechanically checked, and repairs accepted only on matched retests that report risk and utility separately. With it, EAIR reveals recurring interaction failures across all five risk families (Section 5, RQ1). Our new comparisons quantify the advantages of this design: near-complete evidence checkability (98.8% vs. 57.1% for a format-matched auditor), and matched retests that distinguish effective workflow-level repairs from ineffective prompt-level ones. Because the contract is generator-agnostic, it also validates a one-shot code patch from a strong model as an alternative repair generator (see our response to Reviewer YVB3).
W2 (comparison with existing auditing methods). We ran both suggested comparisons on the same 100 bundles and backbone: the all-at-once attribution protocol of Who&When, and a single prompt that requests our full evidence format. EAIR’s localization is on par with both, and it leads clearly on evidence checkability: 98.8% of EAIR findings resolve every cited step and verbatim quote, versus 57.1% for the format-matched prompt.
W3 (cohort size, repeated runs, cross-model repair). We reran every repaired world on two additional model families (Kimi K2.6, Qwen 3.5 397B), each against a matched unrepaired baseline. Together with GPT-5.6 Sol, this gives 639 matched pristine/repaired episodes, and the repairs transfer across models:
Each repair in the paper comes from one pipeline run, and the cross-model and cross-endpoint retests above show that the resulting repairs are robust.
W4 (generalization beyond author-designed worlds). The full pipeline is demonstrated on our five worlds, and its auditor already transfers beyond them: it runs unchanged on 100 traces from seven third-party frameworks, producing trace-level findings from logs alone, and every one of its 29 findings resolves to the cited steps and quotes. Full certificates and repairs additionally use policy and permission inventories, which the toolkit’s declarative world specification supplies; this is how new, independent workflows enter the full pipeline.
W5 (raster figures). We are redrawing Figures 1 and 2 and aligning Figure 2 with the executed retest protocol.