- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces an approach to check that, when an agent delegates a task to another agent, a number of rules (encoded in a contract) are adhered to, together with a check intended to reduce the substitution of permitted but unrequested operations. The framework assumes that actions are performed by a trusted executor. The so-called contract is encoded in ProcPact, a language that allows to specify the execution steps, together with rules that should be followed at each step. An additional name-then-check mechanism asks the model which tool the request requires and checks whether that tool is reachable under the contract, without guaranteeing task completion. The proposal is evaluated against three LLMs on a simulated procurement use case, which is also used as a running example.
Critical strengths
- The paper addresses a timely, relevant topic. The idea that performing authorized tasks does not correspond to correctness is also a key insight.
- Table 1 is very useful in identifying assumptions and limitations.
Critical weaknesses
-
There is a strong underlying assumption in the paper: every action covered by the guarantee must occur inside the trusted executor, and the caller must trust that is execution boundary is respected. While Table 1 clearly identifies assumptions and limitations, there seems to be a contradiction with the claims in line 111-118 that LLMs are not trusted and that caller and provided do not trust each other. The limited evaluation on a synthetic procurement use case does not help clarifying this issue: a real model experiment with delegated procedures is required. The authors should frame their contribution in terms of what are the additional benefits over a mutually-trusted monitoring mechanism.
-
The presentation is dense in some parts, and critical sections are only available in the very long appendices. For instance, Algorithm 1 line 3417 should be in main body. Additional symbols are introduced and not defined in the main text (\tau and \pi on line 135, the policy language, \varsigma on line 244, etc.)
Questions for authors
How does the framework scale with the number of participating agents and the depth of delegation, given its reliance on a trusted executor, procedure inlining, and caller-specific checks?