- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents OpenAgentFlow, a shared pre-execution governance framework for heterogeneous AI agents. It normalizes GUI, API, and tool actions into a common AgentEvent representation, with centralized policy enforcement, session provenance tracking, and dynamic rule updates. The framework combines deterministic rules with semantic adjudication. Evaluation includes controlled safety suites, 1,220 AgentDojo-Traj cases, and Android execution, reporting 94.00%, 97.62%, and 92.86% accuracy, respectively.
Critical strengths
-
Practical system-level architecture. The shared action-commit boundary and centralized provenance management offer a reasonable approach to enforcing consistent policies across heterogeneous agents. Separating governance from individual agent implementations is particularly useful for real deployments.
-
Encouraging empirical results. The evaluation includes an external benchmark and live Android execution. The 97.62% accuracy and 1.96% false-intervention rate on AgentDojo-Traj are promising, particularly compared with direct classification using the same Qwen backbone.
Critical weaknesses
-
Limited validation of the central system-wide claim. AgentDojo-Traj primarily evaluates individual pending actions with independent sessions, rather than persistent cross-agent interactions. The evidence for shared provenance across agents largely comes from three constructed chains comprising only 14 stages. This is insufficient to establish robustness in realistic heterogeneous agent fleets.
-
The incremental benefit and robustness of the full architecture remain unclear. On the controlled suite, the full system improves accuracy only from 93.33% (static T1+T2) to 94.00%, while the deterministic matcher misses all ten obfuscated-leak cases. Meanwhile, the external benchmark primarily exercises semantic adjudication rather than the proposed provenance mechanism. The paper also lacks end-to-end latency measurements for the expensive semantic stages.
Questions for authors
-
Could the authors evaluate longer, independently generated multi-agent workflows that require cross-agent provenance tracking, including transformed or obfuscated sensitive information?
-
Can the authors isolate the contribution of shared provenance and session state under matched policy and model configurations, and report end-to-end latency and computational overhead for the full T1–T4 pipeline?
2 comments
Further Discussion on System-Wide Governance for Heterogeneous Agents
Further Discussion on System-Wide Governance for Heterogeneous Agents (Submission 20441)
Dear Reviewer,
Thank you again for your continued engagement with our work and for the specific follow-up questions. We appreciate that you have taken the additional experiments into account, and we would like to highlight three further results that directly address the remaining questions you raised.
1. Additional value of typed provenance under bounded history
We agree that the identical results between Full Shared and Shared History under complete traces do not, by themselves, establish the additional value of typed provenance.
To investigate this more directly, we conducted a new experiment with bounded session history, retaining only the most recent three events and the last 16,000 characters of serialized history.
We compared Full History, Bounded History, and Bounded + Typed Provenance across 64 controlled workflows.
The results show a clear difference once source observations leave the history window:
| History length | Full History | Bounded History | Bounded + Typed |
|---|---|---|---|
| ~8K | 16/16 | 16/16 | 16/16 |
| ~18K | 16/16 | 0/16 | 16/16 |
| ~32K | 16/16 | 0/16 | 16/16 |
| ~48K | 16/16 | 0/16 | 16/16 |
Across the 48 longer-history cases, Bounded + Typed Provenance retains all 48 correct source attributions, matching Full History, while Bounded History alone fails to recover the corresponding source information.
Importantly, all configurations prevent execution of the unauthorized actions. However, Bounded History escalates all 48 longer-history cases, including 24 with valid authorization, whereas Full History and Bounded + Typed Provenance correctly distinguish authorized from unauthorized actions.
This additional experiment demonstrates the specific value of typed provenance: retaining authorization-relevant source–value relationships even after the original observations have left the text-history window.
2. Execution-grounded reconstruction of transformation provenance
Regarding how transformation bindings are obtained, they are not simply taken from pre-annotated expected labels.
We conducted an execution-grounded provenance audit over 90 non-identity transformation edges. For each edge, the trusted tool adapter reconstructed lineage from the observed source value and recorded operation, re-executed the registered transformation, and checked the result against the actual recorded tool output.
The audit successfully recovered 90/90 correct transformation bindings and correctly attributed 87/87 explicitly bound final output fields.
Thus, the provenance evidence is grounded in observed inputs, actual tool-call arguments, and verified execution outputs.
3. Complete PEP replay and end-to-end classification
We also re-executed all 1,220 actions through the complete OpenAgentFlowPEP.evaluate() pipeline, allowing T1–T3 to determine early termination and T4 invocation dynamically.
The complete replay achieves:
- Accuracy: 97.30%.
- Unsafe-action recall: 97.73%.
- Safe false-intervention rate: 2.88%.
- T4 invocations: 496/1,220.
For comparison, the fixed-route composite evaluation achieves 97.62% accuracy, 96.59% unsafe-action recall, and 1.96% safe false-intervention rate.
We have now reported both protocols explicitly to distinguish the fixed-cohort comparison used for backbone evaluation from the complete execution of the implemented PEP routing logic.
We also measured the full PEP runtime separately, obtaining a P50 latency of 229.066 ms, P95 of 3.301 s, and P99 of 3.646 s under the serial-action latency protocol.
We hope these additional experiments directly address your remaining questions and provide a more complete picture of the role of typed provenance, its execution-level integrity, and the behavior of the complete enforcement pipeline.
We would greatly appreciate any further thoughts you may have as the discussion deadline approaches. If there are still aspects you would like us to clarify, we would be very happy to continue the discussion while there is still time.
Thank you again for your thoughtful feedback and consideration!
Best regards,
The Authors
Response to Reviewer iSpp
We thank the reviewer for emphasizing the need to evaluate shared provenance in realistic multi-agent workflows and to quantify runtime overhead. We expanded our evaluation to 64 multi-agent workflows, added matched provenance ablations and a provenance integrity audit, and measured the complete T1–T4 enforcement pipeline.
1. Multi-Agent Workflow Evaluation
We expanded the evaluation from three constructed chains (14 stages) to 64 multi-agent workflows, covering 2–5 agent roles, five topologies, and eight data transformations, including spacing, masking, Unicode digits, Base64, hexadecimal encoding, and reversal.
The 64 workflows form 32 matched pairs. Within each pair, source data, workflow structure, communication channel, recipient, and authorization context are held constant, while field selection or revision varies. This design tests whether policy-relevant information is tracked across agents and execution stages under controlled changes.
We compared four configurations on the same workflow inputs:
Full Shared improves accuracy by 28.13 percentage points over Per-Agent and 48.44 points over No Provenance, achieving 100% unsafe-action recall with zero false interventions on determinate benign cases.
Full Shared and Shared History produce identical decisions. This equivalence is expected by construction: Shared History receives the complete serialized session trace, including source observations and downstream actions. This control therefore holds cross-agent information constant and isolates the contribution of provenance representation—typed bindings versus serialized history—rather than information availability. Typed, field-level bindings expose source–value relationships directly for deterministic source–sink matching and evidence inspection; serialized history requires these relationships to be reconstructed from the trace.
We also audited provenance integrity. OAF retained lineage across 90/90 executed non-identity transformation edges and correctly attributed 87/87 explicitly bound final output fields. This mechanism-level result addresses a different question from classification accuracy: it verifies that the lineage used for source–sink enforcement and audit is preserved across transformations, independent of whether Shared History yields the same decision.
2. Shared Provenance and End-to-End Runtime Overhead
The ablations use matched workflow inputs, policies, authorization context, and tool-effect definitions. Per-Agent state limits each decision to locally available information, whereas Full Shared makes provenance accumulated across the workflow available to the PEP. The results establish the classification benefit of shared context. The Shared History comparison isolates a second contribution: typed provenance provides an explicit, auditable representation for deterministic field-level matching, whereas serialized history requires reconstruction from raw traces.
End-to-end latency. We measured the complete
OpenAgentFlowPEP.evaluate()execution on all 1,220 AgentDojo actions, including actual T1–T4 control flow and model-service requests.The P95 from action arrival to decision completion is 3.352 s. T3 uses Qwen3.5-0.8B; T4 uses Qwen3.8-27B and is invoked for 471/1,220 actions, reserving the more expensive semantic stage for escalated actions.
Model invocation and token overhead. Across the 1,220 actions, the pipeline completed 1,220 T3 calls and 471 T4 calls: 1,691 successful action-level calls. The request log records 1,693 attempts, including two retries. Recorded usage totals 4,372,333 input tokens and 67,293 output tokens, averaging 1.386 successful model calls per action. T1/T2 remains millisecond-level, while semantic stages account for most end-to-end latency.
3.Manuscript revisions.
We will add the 64-workflow evaluation, provenance ablations, integrity audit, and full-pipeline latency/token measurements, and clarify how shared session context and typed provenance jointly support cross-agent governance.
We thank the reviewer. These experiments show that a common enforcement boundary and shared governance state let policy decisions account for cross-agent information flows rather than isolated actions.