- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents ACORN, a contract-based runtime control framework for LLM agents executing standard operating procedures (SOPs) through tool calls. To address the limitations of prompt-based guidance and passive guardrails, ACORN formalizes procedural requirements as contracts in finite-trace temporal logic (LTLf) and compiles them into deterministic finite automata (DFAs). Based on the resulting control states, ACORN introduces three mechanisms: dynamic tool masking, which restricts the tools exposed to the LLM; contract-based fast-forward, which executes tool calls without invoking the LLM when the tool and its arguments are determined; and runtime verification, which blocks actions that violate contracts or make future contract satisfaction impossible. Experiments demonstrate improvements in task success and efficiency.
Critical strengths
A key strength of this paper is its integration of formal contract enforcement with proactive agent execution control. ACORN leverages temporal contract library to dynamically restrict tool availability and directly execute contract-determined tool calls without invoking the LLM. This provides a mechanism for reducing unnecessary model decisions while retaining the flexibility of LLM-based reasoning when the next action is not fully determined.
Critical weaknesses
A major concern is whether the use of LLM agents is sufficiently justified in the framework. ACORN formalizes SOPs into symbolic contracts and directly executes tool calls when the next action and its arguments can be determined by these contracts. This raises a fundamental question: if procedural rules already determine much of the execution, why is an LLM agent needed instead of a rule-based engine? For example, Figure 3 shows that eight of nine steps can bypass the LLM, while Table 1 reports that 89% of tool calls in the Know Your Business domain are executed without LLM involvement. In these cases a rule-based engine may be used to process these requests instead of an LLM agent. Therefore, it remains unclear what additional value the LLM agent provides.
Another concern is that the τ^2-bench comparison about baseline parity and attribution. The authors’ empty-library shell already increases pass4 from 31.6% (official LLM Agent) to 37.7% and reduces the reported violation rate from 37.3% to 0%, despite having no policy contracts. ACORN then improves pass4 only marginally over this shell (39.5% vs 37.7%). This suggests that differences in the agent wrapper, tool handling, or evaluation interface may account for a substantial portion of the reported gains. The authors are encouraged to investigate this discrepancy, provide implementation-level comparisons, and emphasize shell-relative improvements with statistical uncertainty when assessing the contribution of contract-based control.
Questions for authors
Q1. Figure 3 shows that ACORN executes eight of nine tool calls without the LLM, and Table 1 reports an 89% fast-forward rate for Know Your Business. Could the authors clarify which remaining decisions require capabilities that a purely rule-based workflow engine cannot provide? A comparison against a rule-based controller using the same contracts, or a breakdown of success on tasks requiring LLM reasoning versus fully specified workflows, would help establish the incremental value of the LLM. Q2. In Table 2, the empty-library shell has a 0.0% violation rate on τ²-retail, compared with 37.3% for the official LLMAgent, despite having no policy contracts; the corresponding shell violation rate on τ²-airline is 21.0%. Could the authors explain this discrepancy and confirm that the same policy-grounded audit was applied to all conditions? In particular, does the retail result reflect differences in agent behavior, in the evaluation wrapper, or in how violations are counted? Q3. In Table 2, the empty-library shell already improves pass4 on τ^2-retail from 31.6% to 37.7%, while ACORN achieves 39.5%. Could the authors clarify what accounts for the improvement introduced by the empty-library shell and to what extent the additional gains can be attributed to contract-based control? A more detailed comparison of the implementation differences and shell-relative improvements would help isolate the contribution of the proposed method.
2 comments
W2, Q2 and Q3: The agent shell on τ²-bench and attributing the gain
We agree that on retail the gain over the official agent comes mostly from the shell, which is our agent loop without any contract, and the contracts account for the gain on airline. The revision reports the numbers relative to the shell (Section 4.2).
Q2. The same policy-grounded audit is applied to all conditions, by replaying the same policy contracts over the saved conversations. The retail difference comes from the agent loop. The shell uses the official prompt, temperature and model, and it differs from the official agent only in that each turn is either a message to the user or a set of tool calls. The official agent's retail violations break a turn-taking rule of the policy, which forbids replying to the user and calling a tool in the same turn, so the shell never breaks it. On airline, rules such as cancellation eligibility depend on the reservation, and the shell cannot check them. It violates a contract in 21.0% of simulations and ACORN in 0.0% (Table 3 in the revision, Table 2 in the submission).
Q3. On retail, the shell and the official agent tie on pass^1 (62.5 vs 63.6), and the shell's higher pass^4 (37.7 vs 31.6) reflects more consistent behavior across trials. Relative to the shell, ACORN adds a small +1.8 points of pass^4, because the retail API already rejects the calls its policy forbids. In the gpt-4.1-mini trajectories released with τ²-bench it rejects 134 such calls, so few violations are left for contracts to block.
The airline cancellation API performs no eligibility check. In the same trajectories, 53 of the 99 failed airline simulations make a database change that the reference solution does not contain, mostly cancellations the policy forbids (Appendix A, Table 4). The contracts block these calls, and under a paired bootstrap over the 50 airline tasks ACORN gains +18.5 points of pass^1 over the shell, [8.0, 29.5], and +20.0 points of pass^4, [6.0, 34.0]. Within this gain, verification alone adds 11.0 points of pass^1 over the shell, and the mask adds 7.5 more, [1.5, 14.0] (Table 3).
More generally, ACORN is designed for agentic workflows in which the agent follows a written procedure, as in SOP-Bench. We include τ²-bench to test whether the control transfers to a third-party conversational benchmark without lowering task success. In such an interactive setting, the next step depends on what the user says, so the contracts determine fewer steps. Fast-forward never fires there (Table 3), and the gain comes from the mask and verification alone. We therefore expect smaller gains in interactive settings than on SOP-Bench, which is what the two benchmarks show.
W1 and Q1: Why the LLM is still needed
Thank you for the careful reading and the precise questions. We have revised the submission PDF, and the changes are marked in blue.
A rule-based engine can execute only the steps whose tool and arguments the contracts determine, and the results in the paper already show that these are a minority. Figure 3 and Know Your Business are the most fast-forwarded cases, and across the ten SOP-Bench domains 31% of tool calls run without the model (Table 1). At every other step several tools are admissible, or an argument must be read from the task text or the dialogue, such as an account ID or an inspection note, and the model chooses the call (Section 1). Know Your Business shows this as well. It has the lowest TSR (55.6%) despite 89% fast-forwarded calls, because its last step is the verdict on the business (approve, escalate, or await information), which the SOP leaves to human experience (Section 4.1). Fast-forward itself is a prediction backed by verification. The predicted call is still verified before execution, and if verification rejects it, the model decides that step (Section 3.4).
The revision adds the comparison you suggest in Q1. We removed the model and kept the same contracts, argument maps and controller (Section 4.1, Table 2). This rule-only controller completes 5 of 1,474 SOP-Bench tasks (0.2%), against 94.5% for ACORN. In nine domains it stops at the first call, whose arguments come from the task text, and in Know Your Business it executes the 8 verification calls and stops at this verdict. Reading a value from free text and making a judgment that the SOP leaves open are the steps a rule engine cannot take, where LLM is still needed.