- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
When building automated workflows with LLM agents based on an organization’s procedural manuals, methods for enabling LLMs to identify compliance with manuals written in natural language have been advancing. However, existing methods are limited because they block tool calls locally: 1) when tool calls are blocked, the LLM is given no information about which actions it should take next to complete the task, and 2) tool calls can be blocked excessively, which leads the agent to behave overly conservatively. Against this backdrop, this paper proposes MENTOR, a guardrail designed to produce procedure-compliant progress and to make that compliance context-aware. MENTOR operates in two stages: offline and online. In the offline stage, run once per manual, an autoformalization loop translates temporal requirements, such as the ordering of tool calls and obligations, into LTL, and constraints on argument values into SMT. It then computes a game graph and a strategy template from the LTL, and this strategy template marks unsafe and progressing calls at each state. In the online stage, a labeler LLM converts the natural-language task and tool outputs into formal events. An evaluator then judges each tool call using the template and SMT (argument repair/checking), and the resulting allow, block, or advice decision is delivered to the agent through either a prompt shield (textual feedback) or a token shield adapted from STARS (token masking and boosting). The authors tested both open-source and closed-source models on 4 domains of MANTRA and 2 domains of AgentOrca, and report improvements over the baselines and the unshielded setting in nearly all configurations. They also report that the labeler costs roughly 0.3–1.2 cents per MANTRA case with GPT-4.1, and that a fine-tuned open-source model incurs no cost. In addition, they report that the prompt shield adds almost no latency overhead. The paper also explicitly states two limitations: autoformalization requires a strong model, and the method cannot verify whether the agent’s natural-language outputs comply with the procedures.
Critical strengths
- To address the two existing approaches proposed for procedural-manual compliance in LLM agents, namely the instability of natural-language processing with LLMs and the difficulty of writing deterministic rules, the paper proposes a hybrid methodology in which the manual is automatically formalized so that an evaluator, both deterministic and able to understand context, judges tool calls and provides guidance to the agent. Using this harness, the paper reports superior performance over existing guards in Table 1.
- In particular, the novelty lies in bringing STARS, previously used in RL methods, into the LLM agent setting and resolving the new problems that arise there through manual formalization, argument constraints, and token-level guidance. To preserve the agent's flexibility, the paper proposes several components: it blocks and advises only on violating calls using a strategy template synthesized from LTL, tracks open duty values using the game graph, and uses SMT to inspect and revise arguments. Some or all of this system could be applied to similar problems that require compliance.
- Another strength is that the paper uses a single running example consistently throughout (the example in Fig. 1) and provides many examples of failure cases in the appendix, which makes the paper clear to follow and easy to read.
Critical weaknesses
- W1. As the authors themselves acknowledge in the main text, the biggest limitation is that MENTOR operates only on tool calls and cannot verify compliance when the agent must produce natural-language output. Many agents frequently ask the user follow-up questions between tool calls, and the user's requirements are often revised in the process; in such cases, the agent's output cannot be guaranteed. The paper does not state how the conversations in the τ2 domains were handled during evaluation. If they were excluded, please clarify whether this affects the evaluation.
- W2. The main text does not specify which model sizes were tested and instead refers the reader to Appendix D, but upon checking Table 6, I found inconsistencies in the models used. The prompt shield and None settings used Ollama 4-bit weights, while the token shield used vLLM bf16 weights, and I also found that the models differ between the prompt shield and the token shield (Qwen3.6-35B-A3B (MoE) vs. Qwen3.6-27B). It seems unreasonable to compare the prompt and token shields on equal footing. Please provide a comparison under identical conditions. In Table 1, the results for the two shields are presented side by side under the same Qwen3.6 name, which is potentially misleading. In the Results on p. 8, the paper claims that "MENTOR consistently improves compliance for every agent, proprietary and open-weights, in comparison to unshielded agents, and outperforms the baselines." However, for MANTRA, AGENTSPEC reported better performance with Q3.6, and for Q3 on τ2-retail, AGENTSPEC also reported better performance. This claim should be toned down.
Minor
- As LTL specifications become more complex, the complexity of the game graph is likely to grow, yet the paper does not report statistics such as the number of tools or the number of LTL rules covered in the benchmarks. Please provide these statistics.
- In the Cost and latency paragraph on p. 8, the authors state only that "we used a 100 USD per month Claude subscription." A calculation is needed showing how many input/output tokens and how much time were used, how many loops were run, and what this would amount to when converted to API pricing. Please specify this.
Note to authors and AC: The weaknesses have been condensed and grouped for clarity. This is an editorial change only — the content and my overall evaluation remain the same as in the original version.
Questions for authors
- Q1. The autoformalization loop was not compared against writing the specification in a single pass. I am curious what difference this makes. In addition, the main text does not specify criteria such as how many iterations the loop runs. Furthermore, the results of autoformalization were not measured. If you have any measurement results, please share them.
- Q2. Regarding the labeler, GPT-4.1 was used for the open models and GPT-4.1 mini for the closed models. Please clarify where this difference comes from.
- Q3. The authors only mention that the unshielded scores are inflated because the SOP-Bench grader does not check arguments. Since similar cases seem likely to be quite common, I believe the paper should present a metric that can measure this.
- Q4. In the STARS paper, I believe the policy was to discard tokens below a threshold and decrease or increase the reward, whereas this paper applies a hard mask that sets token masks to 0. It would be helpful to explain why this differs.
- Minor:
- The Pass^3 and Pass^5 metrics are used interchangeably; I am curious what the criterion is for using each.
- Typos: p. 4, "finte"; Abstract, "over-resticting"; p. 22, "addtion"
- p. 19: Chen et al. (?) — no reference
6 comments
[Q1] In our specification language, the first generation of the specification in the loop contained conflicts in all domains (between 5-30% of examples), and between 10-90% of generated rules differed between the first and last iteration of our loop depending on the domain. We used 6 iterations maximum in our autoformalization loop, but the AGENTORCA and SOP-Bench domains terminated early, at 3 and 2 iterations respectively. We will add this information to the revised version of the paper.
We evaluate autoformalization indirectly through downstream procedural compliance, because there do not exist ground-truth formal specifications to compare against directly. Instead, we perform ablations in Appendix D.4 comparing different autoformalization models and generation techniques (including AgentSpec’s single-pass generation with in-context examples), measuring how these choices affect the resulting shield’s effectiveness. Finally, we compared the graph sizes resulting from autoformalization with different models from our ablations, and found that worse-performing models may produce much larger graphs (up to 600k states from GPT-5.6-Luna vs up to 5.5k from Claude Opus 5). We will add these results to the revised version of the paper.
[Q2] We also used GPT-4.1-mini as a labeler for the open-weights models, see our ablations in Appendix D.4. We chose to present the results with the stronger labeler in the main paper.
[Q3] Quantifying how often the SOP-Bench grader accepts executions with invalid arguments would require a manual audit of the execution traces, since the grader itself does not check argument compliance. Such benchmark validation falls outside the scope of this work, which focuses on procedural shielding. Instead, we qualitatively report a few such examples in Appendix D.3 to inform future evaluations and because it affects the interpretation of our results.
[Q4] STARs sets the probability of unsafe actions to zero (Eq. 1b), as we do for unsafe tokens. STARs decreases the probability of co-live actions, which do not exist in our setting. STARs additionally uses probability thresholding to enforce liveness over infinite executions, but MENTOR operates on finite tool-call traces with a bounded execution budget.
[MinorQ] Pass^3 was used for the ablations due to resource constraints. Thank you for catching the typos and missing reference.
Thank you for the thorough response and for rerunning the experiments. The consistent open-weight reruns resolve my main soundness concern (W2), the blocking-only ablation supports the value of progress guidance, and the clarifications answer my other questions.
On reflection, I consider the cost reporting more important than I indicated in my original review. Since the paper’s practical argument relies on amortizing the one-time autoformalization cost, that cost should be verifiable for the main setting. The GPT ablation figures are helpful, and I understand the Claude counts are still being compiled. Could you report, per domain for the Claude Opus 5 setting, token usage, loop iterations, wall-clock time, and equivalent API cost? Re-measured figures are fine if labeled as such.
For the revision: (1) some cells changed noticeably in the rerun (e.g., Qwen3/Operations/AgentSpec 19→55; Gemma 4/Sneaky Sasquatch/no shield 42→23), so “numerically similar” may understate this, and the p. 8 claim that MENTOR consistently outperforms the baselines should be revisited accordingly; (2) the p. 8 statement about the prompt shield and Llama 3.1 may need updating given the new 35–41 results on three MANTRA domains.
I see clear practical value in an external, reusable procedural shield, and I will maintain my score of 4. I look forward to the Claude cost figures.
We thank the reviewer for the thoughtful comments and suggestions. Below, we provide point-by-point responses.
[W1] MENTOR focuses on procedural compliance of tool calls, consistent with a recent line of work (AgentSpec, Progent, Agent-C). Tool calls are important to verify because they can change external states, eg by making bookings. As we acknowledge in Section 5, verifying agent-generated natural language is outside our scope. Regarding changing user requirements, the labeler can pbe expanded to update the requirements mid-run by grounding followup user messages and updating task requirements during execution. We consider this a promising extension, whose implementation and evaluation we leave to future work. Finally, we clarify that we evaluate the tau2-derived domains provided by MANTRA, which do not involve multi-turn user conversations. All necessary user-provided information is included in the initial prompt, so we did not remove conversational turns during evaluation. We do not evaluate on the original tau2-bench because verifying agent-generated natural language is outside our scope, as mentioned in Section 5. We will make this clearer in the revised version.
[W2] We thank the reviewer for detecting the inconsistency in our configurations for our two shields. We have reran all open-weights model experiments with the same configuration, using vLLM bf16 and replaced Qwen3.6-35B-A3B (MoE) with Qwen3.6-27B everywhere for consistency. The updated results are numerically similar and preserve our conclusions. MENTOR improves the compliance of unshielded agents and outperforms the baselines in almost all cases, and we will revise our presentation of results to note the few cases where AGENTSPEC outperforms our shields. See our official comment Updated Table 1 in our response to reviewer eA4Q for the updated main results table and will replace all relevant Appendix tables in the revised version.
[MinorW1] The number of tools in each benchmark domain ranges between 14-26 in AgentOrca and MANTRA, and 6-10 in SOP-Bench. The number of LTL rules depends on how the specification is written, as multiple rules can be combined into a single conjunction, so the number of rules does not directly correspond to complexity. Instead, we will report our main game graphs’ sizes, which varied between 360 and 5500 across domains. To give an idea of the scaling of solving such graphs, in STARs (Anand et al., 2026b), Figure 8 in Appendix G.3 analyses the time required to solve game graphs with varying sizes, from less than 100K to over 2M states, needing fewer than 90 minutes for the largest cases. We would like to point out though that this construction incurs a one-time cost per fixed specification, after which the resulting shield can be reused across tasks and agent runs.
[MinorW2] We are still working on collecting the Claude token usage and will include it in the revised paper. Meanwhile, we report the available costs from our autoformalization ablations in Appendix D.4 across two MANTRA domains with GPT models. Specifically, GPT-5.5 used around 30-36 million tokens per domain and GPT-5.6-Luna used 40-50 million tokens, of which 95% were cached input, and 1% were output, corresponding to an equivalent price of around 30 USD per domain for GPT-5.5 and 2 USD for GPT-5.6-Luna according to current API prices.