acceptodds
Withdrawn from ICLR 2027

Check The Task, Not Just The Rules, When One LLM Agent Hires Another

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When one LLM agent hires another, the caller’s rules can be made binding by an executor that offers the model only the moves the rules allow. But keeping the rules does not show that the provider did the task it was asked to do. In this paper, we first show that a model at such a menu often substitutes a permitted operation for the one the request needs: asked to change an existing order, it tries to place a new one in 19 of 36 episodes, while every rule holds. Asked separately which tool the request needs, it names one that does not buy in 35 of 36 episodes, yet, shown that answer at the menu, it still tries to buy. Then, we propose name-then- check: the model names one tool, and the executor removes every branch that acts if the signed contract cannot reach that tool from the current decision. We build it on PROCPACT, a procedural contract whose rules we prove bind every action, including inside a procedure the provider copies in from a third agent; the check keeps this guarantee because it only removes moves. In one procurement world with three models, attempts fall to at most 1 of 36, while at least 35 of 36 purchases go through. The check tests whether the named tool is available from the current decision, not whether the task is done, and it declines some requests it reads literally.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent delegation, procedural contracts, compositional verification, runtime policy enforcement, execution auditing, agent safety.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qXenOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces an approach to check that, when an agent delegates a task to another agent, a number of rules (encoded in a contract) are adhered to, together with a check intended to reduce the substitution of permitted but unrequested operations. The framework assumes that actions are performed by a trusted executor. The so-called contract is encoded in ProcPact, a language that allows to specify the execution steps, together with rules that should be followed at each step. An additional name-then-check mechanism asks the model which tool the request requires and checks whether that tool is reachable under the contract, without guaranteeing task completion. The proposal is evaluated against three LLMs on a simulated procurement use case, which is also used as a running example.

Critical strengths
  • The paper addresses a timely, relevant topic. The idea that performing authorized tasks does not correspond to correctness is also a key insight.
  • Table 1 is very useful in identifying assumptions and limitations.
Critical weaknesses
  • There is a strong underlying assumption in the paper: every action covered by the guarantee must occur inside the trusted executor, and the caller must trust that is execution boundary is respected. While Table 1 clearly identifies assumptions and limitations, there seems to be a contradiction with the claims in line 111-118 that LLMs are not trusted and that caller and provided do not trust each other. The limited evaluation on a synthetic procurement use case does not help clarifying this issue: a real model experiment with delegated procedures is required. The authors should frame their contribution in terms of what are the additional benefits over a mutually-trusted monitoring mechanism.

  • The presentation is dense in some parts, and critical sections are only available in the very long appendices. For instance, Algorithm 1 line 3417 should be in main body. Additional symbols are introduced and not defined in the main text (\tau and \pi on line 135, the policy language, \varsigma on line 244, etc.)

Questions for authors

How does the framework scale with the number of participating agents and the depth of delegation, given its reliance on a trusted executor, procedure inlining, and caller-specific checks?

Reviewer_ubKaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors point out that an agent’s adherence to predefined rules does not preclude it from performing additional, unintended actions. To address this, they introduce PROCRAFT—a framework featuring a "name-then-check" mechanism designed to audit an agent’s tool calls. This approach yields strong results within predefined scenarios and offers valuable insights.

Critical strengths
  • Modular design is clear and easy to understand.
  • The issue is time-sensitive and closely related to security.
  • The author puts forward an important point: following all the rules does not equate to carrying out the right tasks.
Critical weaknesses
  • The number of experiments is too small, and the scope is very narrow. The experiments were conducted entirely within a pre-defined environment, and the datasets used are quite small. Consequently, the approach lacks generalizability and is not sufficiently convincing.
  • This method does not guarantee the final task completion rate, a point the author does not discuss.
  • The author notes that "the executing LLM is not trusted: it may say anything." However, given this premise, if the model makes an error at the very first step—selecting a tool—the entire subsequent process becomes invalid.
  • The paper is framed around the concept of an "LLM agent hires another agent"; however, the subsequent content focuses primarily on the behavior of a single agent—the "provider"—thereby departing from the multi-agent context.
Questions for authors
  • If the caller needs to modify an order by adding two keyboards, why did they also establish a rule stating that "existing orders must never be modified"? Isn't this contradictory?
  • In complex, real-world business scenarios, a composite task often requires invoking multiple tools. In such cases, if the model is constrained to output only a single tool name, how should the executor determine the reachability of the required tool sequence?
  • The authors do not disclose whether the models used in the experiments had "thinking" modes enabled, nor do they specify the intensity of the reasoning processes. If a more powerful model were used, could the "menu-only" mechanism simply opt to issue a refusal? Furthermore, would the performance gains offered by the authors' proposed method remain as significant?
  • In practical deployment, does this approach rely on manually constructed graphs? What is the cost of building these graphs? Furthermore, does each distinct scenario require a purpose-built design?
Reviewer_6eoAOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to solve the problem where an LLM agent may not follow explicit rules while performing a task but the end goal is still reached. The paper proposes a solution which first enforces a workflow contract and then uses a name-then-check strategy where it uses an LLM to see which tools should be used for a request and then for each tool to be used checks if the signed workflow reaches that tool.

Critical strengths

The problem is described clearly with a concrete example.

Critical weaknesses

As such the method is pretty simple and I do not understand what is the primary challenge in solving the problem.

the proposed task check reduces to LLM classification plus reachability, which is the baseline in many existing research

It seems like the problem boils down to propositional logic check. Which has many solutions including well established SMT solvers. The paper needs to establish what its formulation and composition guarantees add beyond standard constraint checking and graph analysis.

Also whether the task was performed correctly or not is never checked. What is checked is whether a tool is being correctly used or not.

Questions for authors

What is novel beyond a constraint checker on a graph analysis engine?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.