acceptodds
Under review as a conference paper at ICLR 2027

Accepted but Not Enforced: Auditing Tool-Call Policy in LLM Agent Frameworks

Abstract

Agent frameworks expose policy loaders, guardrails and approval hooks to control tool execution, but accepting a policy does not ensure that it governs the resulting effects. We study this gap through a proposal–decision–effect authorization chain that records policy activation, authorization decisions and committed effects as separate observations, and we state the enforcement contract a deployment must meet before committing a protected effect. Auditing four pinned deployments, a five-framework route study and an eighteen-framework census, we find that contract broken at every position it names: in two native policy languages, 10 of 13 clause instances that loaded without governing a decision produced no diagnostic, and an alternative tool path escaped the configured check in four of five frameworks, the fifth covering both tested paths with one kernel-level filter. We then name the primitive each obligation needs and build them natively in two versions of OpenAI Agents SDK. Waiting reservations preserve feasible legal effects that immediate refusal loses, and explicitly propagated child hooks enforce policy on nested calls. Across 150 runs per configuration per version, the integrated layer takes violating runs from 75 to zero and runs that lose feasible legal effects from 15 to zero, restoring enforcement without blocking the work a deployer wants done.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Agent Security, Prompt Injection, Tool Use, Guardrails, Policy Enforcement, Access Control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aeA4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a critical safety gap in LLM agent frameworks: accepting a developer's policy configuration does not guarantee runtime enforcement. Using CHAINAUDIT—an auditing protocol tracking policy activation, authorization decisions, and effects across a four-position authorization chain—the authors audit 18 agent frameworks. They reveal that 5 frameworks pass rules merely as advisory LLM prompts, 21 of 65 policy clauses across adjudicating frameworks load silently without governing decisions, and alternative tool paths bypass checks in 4 of 5 frameworks. To fix these flaws, the authors formalize an Enforcement Contract and implement native runtime primitives (waiting reservations for concurrency and explicit child-hook propagation for sub-agents) in the OpenAI Agents SDK. Across 150 test runs per SDK version, these repairs eliminate all policy violations (75 to 0) and lost legal work (15 to 0) with only 1.1–1.4 ms median overhead.

Critical strengths

Conceptual Originality: Shifts agent safety from vulnerable, model-level prompt compliance to deterministic, harness-level complete mediation. Formal Foundation: Grounded in a 5-obligation Enforcement Contract (activation, route coverage, referent binding, value binding, and admission serialization). Broad Census & Controls: Evaluates 18 frameworks with fixed models and realistic adversarial proposals to isolate framework mechanics. Practical Repairs: Native runtime primitives eliminate violations without sacrificing operational utility or introducing heavy performance overhead. Dual-Axis Metrics: Simultaneously measures policy violations and lost legal work, setting a high standard for benchmark evaluations.

Critical weaknesses

Native primitives were evaluated solely within the OpenAI Agents SDK. Suggestion: Validate the repairs on a second, distinct framework architecture (e.g., LangGraph or AgentGuard) to demonstrate broader applicability.

Route coverage assumes static tool graphs. Suggestion: Discuss enforcement guarantees when tools are loaded dynamically at runtime (e.g., via Model Context Protocol).

Some tools rely on executor-reported logs rather than independent out-of-process sink witnesses. Suggestion: Clarify behavior under real-world network latency or API failures.

Questions for authors
  1. How easily can waiting reservations and child-hook propagation be retrofitted into distributed microservice architectures?
  2. Did you observe any deadlock risks or thread contention under heavy multi-agent concurrency?
  3. What static validation tools do you recommend adopting at the loader layer to prevent silent activation failures?
Reviewer_eXM8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether policies accepted by LLM agent frameworks are actually enforced during tool execution. The authors propose CHAINAUDIT, which separates proposal, decision, and effect, and defines an enforcement contract with seven failure classes (C1–C7). They survey 18 agent frameworks and test policy activation, route coverage, concurrent admission, and predicate inputs. They also implement runtime mechanisms in two versions of OpenAI Agents SDK to prevent policy violations while preserving legal work.

Critical strengths
  1. The agent harness is worth studying. The harness controls tool routing, policy decisions, and execution. Studying it separately can help clarify the security boundaries of agent systems.

  2. The experiments cover multiple frameworks and use careful controls. The authors survey 18 agent frameworks and use fixed proposals and matched benign and forbidden calls to isolate harness behavior. They also distinguish different levels of effect evidence and validate their scoring method.

  3. The evaluation considers both policy violations and lost legal work. This helps distinguish effective enforcement from simply blocking tool calls.

Critical weaknesses

W1. The practical security importance of the problem is not sufficiently established. The paper shows that accepted policies do not always govern committed effects. However, it does not clearly connect these policy violations to real security risks. In particular, it groups together documented advisory behavior, deployer configuration errors, and bypasses of policy checks. These cases have different security implications. The paper does not clearly establish which findings represent failures of actual security guarantees.

W2. The systematic nature and novelty of the analysis framework are unclear. The authors introduce the proposal–decision–effect chain, an enforcement contract, and seven failure classes (C1–C7). However, it is unclear how these classes are systematically derived from a common set of principles. They cover different issues, including policy expression, activation, predicate inputs, route coverage, and concurrent admission. Their boundaries and relationships are not clearly justified.

W3. The experiments provide limited support for broader claims. The 18-framework census mainly tests one argument constraint rather than systematically auditing all enforcement properties. The main experiments use fixed, authored adversarial proposals. This helps isolate harness behavior, but does not establish whether the failures are reachable or important in real agent workflows. The held-out experiments also provide limited evidence of generalization. In addition, the integrated repair is compared mainly against an authored local baseline and does not address policy activation.

Questions for authors
  1. How do the authors distinguish documented advisory behavior, deployer configuration errors, and actual failures of security guarantees?

  2. What common principles are used to derive C1–C7? How would CHAINAUDIT help identify previously unknown enforcement failures in a new agent framework, rather than only test predefined cases?

  3. What new analysis capabilities does CHAINAUDIT provide compared with existing policy conformance testing, reference monitor analysis, and checks that compare authorization decisions with execution effects?

2 comments

Thank you again for your review. As the discussion period has passed its midpoint, we would like to ask whether our response and the revised paper address your concerns. We would be glad to clarify any point or provide additional analysis while there is still time for discussion :).

Thank you for the detailed rebuttal. The revisions make the paper clearer and strengthen the experimental evidence. I appreciate these improvements. However, some concerns remain, and they still affect my assessment of the paper's research significance and contribution.

W1. The authors have better explained what CHAINAUDIT measures, but I am still not fully convinced of the practical security importance of the measured failures. The new evidence is helpful, but the connection between the paper's main findings, realistic deployments, and concrete security consequences remains insufficiently established.

W2. The authors now provide a clear common principle for deriving C1–C7, which addresses my concern about the taxonomy. However, I am still not fully convinced of the generalizability of the analysis method.

Q2. The authors explain how the framework can be applied to new systems, and the held-out experiments provide some evidence. However, how does CHAINAUDIT guide the selection and construction of tests for a previously unseen framework, beyond applying a predefined set of probes?

Overall, the rebuttal has addressed several of my concerns. However, my main concern about the practical security significance of the problem remains. I am therefore not yet convinced that the paper provides sufficient evidence to support acceptance at this moment.

Reviewer_Uj47OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper raises the issue that, in an LLM agent framework, just because a policy is accepted does not mean it governs the effects committed by a deployment. Therefore, this paper proposes CHAINAUDIT to separately record policy activations, authorization decisions, and committed effects.

Critical strengths

I like the paper’s central observation that accepting a policy is not the same as enforcing it: a framework can load a deployer’s rule without error while the rule has no effect on which tool calls are executed. This distinction is easy to overlook and matters to anyone who relies on these frameworks to constrain an agent.

Critical weaknesses

The presentation makes the contribution hard to evaluate from the main text. The necessary information exists in the paper, but it is spread across sections. After reading the paper, I believe it would be clearer if presented in the following order.

  1. The paper proposes CHAINAUDIT to record policy activation, authorization decisions, and committed effects as separate observations. After reading the abstract and introduction, however, it is unclear to me what happens once these observations are recorded. Are they used to determine that an accepted policy is not enforced? If so, what criterion is this determination based on? And once a policy is found not to be enforced, what follows? These should be clearly stated in the introduction.

  2. Following that, the natural next question is how CHAINAUDIT is designed. The section after the introduction should present this directly: what CHAINAUDIT takes as input, what it outputs for each tool call (for example, whether the call was correctly blocked, escaped the policy, or was wrongly refused), and the criterion by which that output is assigned. In the current version, these elements are spread across Section 2 (setting and threat model), Section 3 (the enforcement contract), Section 4 (the measurement protocol), so the reader has to assemble the design of CHAINAUDIT from several places.

  3. The design of CHAINAUDIT should be followed by an experimental setup section that states which agent frameworks are evaluated and how they were selected, which policies are applied to them, and how these policies and the test cases were constructed. This information exists in the paper but is scattered: the selection rule for the census appears in Section 5.1 and Table 12, the pinned versions in Table 5 (Appendix A.3), the policies are described in Section 2 only as “policies we wrote” and reproduced in Appendix E.4, and the generation of test cases is described in Section 4 and Appendix A.2. In addition, different experiments use different sets of targets (18 frameworks for the census, two policy languages for activation, five frameworks for route coverage, one SDK for the repairs), and the main text does not state which experiment uses which.

  4. Finally, the results should be organized so that each one visibly supports a specific aspect of the central claim. In the current Section 5, the results that most directly show that an accepted policy is not enforced (the census in Table 11 and the activation outcomes in Tables 10 and 13) appear only in the appendix, while the only figure in Section 5 (Figure 2) reports the repair experiments rather than evidence for the central claim. The subsections also do not state which aspect of the claim each result supports: Section 5.1 shows that an accepted policy may never take effect, and Section 5.2 shows that a policy which does take effect may fail to cover every route, input or point in time, while Section 5.3 addresses a separate question, whether these failures can be repaired. Each subsection should make its role explicit.

  5. Beyond presentation, the experiments support a narrower claim than the paper makes. Apart from the census, which has a defined population (Table 12), the results are existence demonstrations on tool calls, policies and deployer mistakes written by the authors, and no real model is run.

Questions for authors
  1. How does the enforcement contract relate to the central claim? Does a policy count as “not enforced” whenever any of the five obligations fails, so that route escape and concurrency failures are instances of the central claim, or are they separate findings?
  2. What exactly does CHAINAUDIT output for each tool call, and what criterion turns these outputs into the conclusion that a policy is not enforced?
  3. Without running a model, how do the authors establish that these failures matter in practice, particularly for advisory surfaces, where the risk depends on whether the model follows the schema it is shown?
1 comment

Thank you again for your review. As the discussion period has passed its midpoint, we would like to ask whether our response and the revised paper address your concerns. We would be glad to clarify any point or provide additional analysis while there is still time for discussion :).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.