- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper identifies a critical safety gap in LLM agent frameworks: accepting a developer's policy configuration does not guarantee runtime enforcement. Using CHAINAUDIT—an auditing protocol tracking policy activation, authorization decisions, and effects across a four-position authorization chain—the authors audit 18 agent frameworks. They reveal that 5 frameworks pass rules merely as advisory LLM prompts, 21 of 65 policy clauses across adjudicating frameworks load silently without governing decisions, and alternative tool paths bypass checks in 4 of 5 frameworks. To fix these flaws, the authors formalize an Enforcement Contract and implement native runtime primitives (waiting reservations for concurrency and explicit child-hook propagation for sub-agents) in the OpenAI Agents SDK. Across 150 test runs per SDK version, these repairs eliminate all policy violations (75 to 0) and lost legal work (15 to 0) with only 1.1–1.4 ms median overhead.
Critical strengths
Conceptual Originality: Shifts agent safety from vulnerable, model-level prompt compliance to deterministic, harness-level complete mediation. Formal Foundation: Grounded in a 5-obligation Enforcement Contract (activation, route coverage, referent binding, value binding, and admission serialization). Broad Census & Controls: Evaluates 18 frameworks with fixed models and realistic adversarial proposals to isolate framework mechanics. Practical Repairs: Native runtime primitives eliminate violations without sacrificing operational utility or introducing heavy performance overhead. Dual-Axis Metrics: Simultaneously measures policy violations and lost legal work, setting a high standard for benchmark evaluations.
Critical weaknesses
Native primitives were evaluated solely within the OpenAI Agents SDK. Suggestion: Validate the repairs on a second, distinct framework architecture (e.g., LangGraph or AgentGuard) to demonstrate broader applicability.
Route coverage assumes static tool graphs. Suggestion: Discuss enforcement guarantees when tools are loaded dynamically at runtime (e.g., via Model Context Protocol).
Some tools rely on executor-reported logs rather than independent out-of-process sink witnesses. Suggestion: Clarify behavior under real-world network latency or API failures.
Questions for authors
- How easily can waiting reservations and child-hook propagation be retrofitted into distributed microservice architectures?
- Did you observe any deadlock risks or thread contention under heavy multi-agent concurrency?
- What static validation tools do you recommend adopting at the loader layer to prevent silent activation failures?
Thank you again for your review. As the discussion period has passed its midpoint, we would like to ask whether our response and the revised paper address your concerns. We would be glad to clarify any point or provide additional analysis while there is still time for discussion :).