- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PACT, a framework for auditing the “contract checkers” used to keep LLM agents safe. These checkers flag an agent action, such as sending a message or making a payment, when its destination doesn’t appear word-for-word in the task text. The paper argues that a clean checker result is often misread as “the agent did only what the task authorized,” while in fact the checker only verifies string matching. The paper identifies several ways the checker’s verdict and the true authorization disagree. The authors contribute a blinded LLM-judge audit protocol, a provenance-aware resolver that authorizes destinations by where they came from, and exact risk certificates for the checker’s literal label.
Critical strengths
The paper asks what a “clean” checker result actually guarantees, and it shows clearly that checker verdicts and true authorization can disagree. This matters because developers increasingly rely on these checkers to judge whether agents are safe.
The audit uses predeclared rubrics, blinded adjudication, exact statistical bounds, and an honest separation of what is certified from what is only measured. Because of this rigor, the findings are trustworthy and reproducible.
The paper finds that the checker is mostly sound on attacks but over-strict on benign tasks, and that a provenance-aware resolver reduces both kinds of error. This gives practitioners actionable guidance on how to design better checkers.
Critical weaknesses
The paper mixes several research questions. It combines (1) analyzing how and why current checkers and verifiers are wrong with (2) designing a new auditing system (the resolver and the risk certificates), and it also brings in prompt injection, which is a separate, well-studied research line. Each of these already has a substantial body of work, so putting them together makes the paper unfocused and makes it hard to tell what the core contribution is or how it improves on prior work in any single line.
The related work is not positioned in recent LLM-auditing research. The related work relies mainly on classical statistics references from the 1930s–1960s (e.g., Clopper–Pearson, Horvitz–Thompson, Hoeffding) and engages little with recent work on LLM-based verifiers and agent evaluation (e.g., AgentRewardBench, Agent-as-a-Judge, guardrail models like Llama Guard). Without this comparison, it is unclear how PACT differs from or improves on existing methods for auditing LLM agents, which weakens the paper’s claim of novelty.
I would suggest that:
- Split the paper’s message more clearly: either focus on auditing checker validity, or on the resolver and certificate design.
- Evaluate on at least one more agent benchmark beyond AgentDojo to show the error patterns generalize.
- Cover a wider range of errors, such as content errors where the destination is right but the payload is wrong, and multi-step leaks, which the destination-only contract can’t see.
Questions for authors
The paper does three things: analyzes checker errors, designs a resolver and certificates, and analyzes prompt-injection defenses. If you had to pick one main contribution, which is it, and how does it advance beyond prior work in that specific line?
How do the literal checker and the provenance-aware resolver compare with recent LLM-based judges or guardrail models (e.g., AgentRewardBench judges, Llama Guard) on the same 400 adjudicated pairs? If an LLM judge already catches most of these errors, what extra value do PACT’s certificates provide?
The rubric was written by the same team that built the checkers, and most adjudication is done by LLM sessions from a single model stack. How confident are you that the 57.5% over-strictness rate is not shaped by the rubric itself? Human validation covers only the 135 checkable units, out of 615 boundary units.
The resolver has hand-written bindings for only four task families, and its prospective test on unseen tasks was MARGINAL. What would it take to extend it to new families or other benchmarks, and how much manual effort would each new family need?