acceptodds
Under review as a conference paper at ICLR 2027

MENTOR: Procedural Shielding of Tool-Using LLM Agents

Abstract

Tool-using LLM agents are increasingly used to automate workflows governed by procedural manuals, such as company rules for booking travel or procuring hardware. These manuals are written in natural language for humans and describe general requirements over such complex workflows. Ensuring compliance of LLM agents with these manuals is difficult, and state-of-the-art techniques use either LLM judges or manually written rules to forbid problematic tool-calls locally at runtime. This is insufficient because (i) only blocking tool calls keeps the agent clueless about how to make correct progress towards fulfilling its task, and (ii) can lead to overly conservative behavior of the agent, as tool calls might only be problematic in certain contexts. To overcome these challenges, we introduce MENTOR, which (i) additionally provides guidance for the agent to make procedural-compliant progress on the task, and (ii) ensures that this compliance is context-aware. We achieve these goals by building a novel history-dependent runtime shield that is obtained from the procedural manual via a multi-step autoformalization pipeline. At its core, MENTOR utilizes a game-based technique to ensure robustness against runtime uncertainty without over-restricting agents' behavior. This yields a formally assisted framework for procedural shielding of LLM agents that improves compliance well beyond existing methods. In particular, MENTOR lifts the compliance of shielded light-weight open-source LLMs above that of non-shielded proprietary models. Since a shield is synthesized once per manual and then serves every task, a MENTOR-shielded agent becomes a viable choice for repeatedly executed workflows, potentially reducing deployment costs when the offline construction cost is amortized over repeated tasks.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
LLM Agents, Runtime Shielding, Policy Compliance

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UjJ7OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

When building automated workflows with LLM agents based on an organization’s procedural manuals, methods for enabling LLMs to identify compliance with manuals written in natural language have been advancing. However, existing methods are limited because they block tool calls locally: 1) when tool calls are blocked, the LLM is given no information about which actions it should take next to complete the task, and 2) tool calls can be blocked excessively, which leads the agent to behave overly conservatively. Against this backdrop, this paper proposes MENTOR, a guardrail designed to produce procedure-compliant progress and to make that compliance context-aware. MENTOR operates in two stages: offline and online. In the offline stage, run once per manual, an autoformalization loop translates temporal requirements, such as the ordering of tool calls and obligations, into LTL, and constraints on argument values into SMT. It then computes a game graph and a strategy template from the LTL, and this strategy template marks unsafe and progressing calls at each state. In the online stage, a labeler LLM converts the natural-language task and tool outputs into formal events. An evaluator then judges each tool call using the template and SMT (argument repair/checking), and the resulting allow, block, or advice decision is delivered to the agent through either a prompt shield (textual feedback) or a token shield adapted from STARS (token masking and boosting). The authors tested both open-source and closed-source models on 4 domains of MANTRA and 2 domains of AgentOrca, and report improvements over the baselines and the unshielded setting in nearly all configurations. They also report that the labeler costs roughly 0.3–1.2 cents per MANTRA case with GPT-4.1, and that a fine-tuned open-source model incurs no cost. In addition, they report that the prompt shield adds almost no latency overhead. The paper also explicitly states two limitations: autoformalization requires a strong model, and the method cannot verify whether the agent’s natural-language outputs comply with the procedures.

Critical strengths
  • To address the two existing approaches proposed for procedural-manual compliance in LLM agents, namely the instability of natural-language processing with LLMs and the difficulty of writing deterministic rules, the paper proposes a hybrid methodology in which the manual is automatically formalized so that an evaluator, both deterministic and able to understand context, judges tool calls and provides guidance to the agent. Using this harness, the paper reports superior performance over existing guards in Table 1.
  • In particular, the novelty lies in bringing STARS, previously used in RL methods, into the LLM agent setting and resolving the new problems that arise there through manual formalization, argument constraints, and token-level guidance. To preserve the agent's flexibility, the paper proposes several components: it blocks and advises only on violating calls using a strategy template synthesized from LTL, tracks open duty values using the game graph, and uses SMT to inspect and revise arguments. Some or all of this system could be applied to similar problems that require compliance.
  • Another strength is that the paper uses a single running example consistently throughout (the example in Fig. 1) and provides many examples of failure cases in the appendix, which makes the paper clear to follow and easy to read.
Critical weaknesses
  • W1. As the authors themselves acknowledge in the main text, the biggest limitation is that MENTOR operates only on tool calls and cannot verify compliance when the agent must produce natural-language output. Many agents frequently ask the user follow-up questions between tool calls, and the user's requirements are often revised in the process; in such cases, the agent's output cannot be guaranteed. The paper does not state how the conversations in the τ2 domains were handled during evaluation. If they were excluded, please clarify whether this affects the evaluation.
  • W2. The main text does not specify which model sizes were tested and instead refers the reader to Appendix D, but upon checking Table 6, I found inconsistencies in the models used. The prompt shield and None settings used Ollama 4-bit weights, while the token shield used vLLM bf16 weights, and I also found that the models differ between the prompt shield and the token shield (Qwen3.6-35B-A3B (MoE) vs. Qwen3.6-27B). It seems unreasonable to compare the prompt and token shields on equal footing. Please provide a comparison under identical conditions. In Table 1, the results for the two shields are presented side by side under the same Qwen3.6 name, which is potentially misleading. In the Results on p. 8, the paper claims that "MENTOR consistently improves compliance for every agent, proprietary and open-weights, in comparison to unshielded agents, and outperforms the baselines." However, for MANTRA, AGENTSPEC reported better performance with Q3.6, and for Q3 on τ2-retail, AGENTSPEC also reported better performance. This claim should be toned down.

Minor

  • As LTL specifications become more complex, the complexity of the game graph is likely to grow, yet the paper does not report statistics such as the number of tools or the number of LTL rules covered in the benchmarks. Please provide these statistics.
  • In the Cost and latency paragraph on p. 8, the authors state only that "we used a 100 USD per month Claude subscription." A calculation is needed showing how many input/output tokens and how much time were used, how many loops were run, and what this would amount to when converted to API pricing. Please specify this.

Note to authors and AC: The weaknesses have been condensed and grouped for clarity. This is an editorial change only — the content and my overall evaluation remain the same as in the original version.

Questions for authors
  • Q1. The autoformalization loop was not compared against writing the specification in a single pass. I am curious what difference this makes. In addition, the main text does not specify criteria such as how many iterations the loop runs. Furthermore, the results of autoformalization were not measured. If you have any measurement results, please share them.
  • Q2. Regarding the labeler, GPT-4.1 was used for the open models and GPT-4.1 mini for the closed models. Please clarify where this difference comes from.
  • Q3. The authors only mention that the unshielded scores are inflated because the SOP-Bench grader does not check arguments. Since similar cases seem likely to be quite common, I believe the paper should present a metric that can measure this.
  • Q4. In the STARS paper, I believe the policy was to discard tokens below a threshold and decrease or increase the reward, whereas this paper applies a hard mask that sets token masks to 0. It would be helpful to explain why this differs.
  • Minor:
    • The Pass^3 and Pass^5 metrics are used interchangeably; I am curious what the criterion is for using each.
    • Typos: p. 4, "finte"; Abstract, "over-resticting"; p. 22, "addtion"
    • p. 19: Chen et al. (?) — no reference
6 comments

We thank the reviewer for the thoughtful comments and suggestions. Below, we provide point-by-point responses.

[W1] MENTOR focuses on procedural compliance of tool calls, consistent with a recent line of work (AgentSpec, Progent, Agent-C). Tool calls are important to verify because they can change external states, eg by making bookings. As we acknowledge in Section 5, verifying agent-generated natural language is outside our scope. Regarding changing user requirements, the labeler can pbe expanded to update the requirements mid-run by grounding followup user messages and updating task requirements during execution. We consider this a promising extension, whose implementation and evaluation we leave to future work. Finally, we clarify that we evaluate the tau2-derived domains provided by MANTRA, which do not involve multi-turn user conversations. All necessary user-provided information is included in the initial prompt, so we did not remove conversational turns during evaluation. We do not evaluate on the original tau2-bench because verifying agent-generated natural language is outside our scope, as mentioned in Section 5. We will make this clearer in the revised version.

[W2] We thank the reviewer for detecting the inconsistency in our configurations for our two shields. We have reran all open-weights model experiments with the same configuration, using vLLM bf16 and replaced Qwen3.6-35B-A3B (MoE) with Qwen3.6-27B everywhere for consistency. The updated results are numerically similar and preserve our conclusions. MENTOR improves the compliance of unshielded agents and outperforms the baselines in almost all cases, and we will revise our presentation of results to note the few cases where AGENTSPEC outperforms our shields. See our official comment Updated Table 1 in our response to reviewer eA4Q for the updated main results table and will replace all relevant Appendix tables in the revised version.

[MinorW1] The number of tools in each benchmark domain ranges between 14-26 in AgentOrca and MANTRA, and 6-10 in SOP-Bench. The number of LTL rules depends on how the specification is written, as multiple rules can be combined into a single conjunction, so the number of rules does not directly correspond to complexity. Instead, we will report our main game graphs’ sizes, which varied between 360 and 5500 across domains. To give an idea of the scaling of solving such graphs, in STARs (Anand et al., 2026b), Figure 8 in Appendix G.3 analyses the time required to solve game graphs with varying sizes, from less than 100K to over 2M states, needing fewer than 90 minutes for the largest cases. We would like to point out though that this construction incurs a one-time cost per fixed specification, after which the resulting shield can be reused across tasks and agent runs.

[MinorW2] We are still working on collecting the Claude token usage and will include it in the revised paper. Meanwhile, we report the available costs from our autoformalization ablations in Appendix D.4 across two MANTRA domains with GPT models. Specifically, GPT-5.5 used around 30-36 million tokens per domain and GPT-5.6-Luna used 40-50 million tokens, of which 95% were cached input, and 1% were output, corresponding to an equivalent price of around 30 USD per domain for GPT-5.5 and 2 USD for GPT-5.6-Luna according to current API prices.

[Q1] In our specification language, the first generation of the specification in the loop contained conflicts in all domains (between 5-30% of examples), and between 10-90% of generated rules differed between the first and last iteration of our loop depending on the domain. We used 6 iterations maximum in our autoformalization loop, but the AGENTORCA and SOP-Bench domains terminated early, at 3 and 2 iterations respectively. We will add this information to the revised version of the paper.

We evaluate autoformalization indirectly through downstream procedural compliance, because there do not exist ground-truth formal specifications to compare against directly. Instead, we perform ablations in Appendix D.4 comparing different autoformalization models and generation techniques (including AgentSpec’s single-pass generation with in-context examples), measuring how these choices affect the resulting shield’s effectiveness. Finally, we compared the graph sizes resulting from autoformalization with different models from our ablations, and found that worse-performing models may produce much larger graphs (up to 600k states from GPT-5.6-Luna vs up to 5.5k from Claude Opus 5). We will add these results to the revised version of the paper.

[Q2] We also used GPT-4.1-mini as a labeler for the open-weights models, see our ablations in Appendix D.4. We chose to present the results with the stronger labeler in the main paper.

[Q3] Quantifying how often the SOP-Bench grader accepts executions with invalid arguments would require a manual audit of the execution traces, since the grader itself does not check argument compliance. Such benchmark validation falls outside the scope of this work, which focuses on procedural shielding. Instead, we qualitatively report a few such examples in Appendix D.3 to inform future evaluations and because it affects the interpretation of our results.

[Q4] STARs sets the probability of unsafe actions to zero (Eq. 1b), as we do for unsafe tokens. STARs decreases the probability of co-live actions, which do not exist in our setting. STARs additionally uses probability thresholding to enforce liveness over infinite executions, but MENTOR operates on finite tool-call traces with a bounded execution budget.

[MinorQ] Pass^3 was used for the ablations due to resource constraints. Thank you for catching the typos and missing reference.

Thank you for the thorough response and for rerunning the experiments. The consistent open-weight reruns resolve my main soundness concern (W2), the blocking-only ablation supports the value of progress guidance, and the clarifications answer my other questions.

On reflection, I consider the cost reporting more important than I indicated in my original review. Since the paper’s practical argument relies on amortizing the one-time autoformalization cost, that cost should be verifiable for the main setting. The GPT ablation figures are helpful, and I understand the Claude counts are still being compiled. Could you report, per domain for the Claude Opus 5 setting, token usage, loop iterations, wall-clock time, and equivalent API cost? Re-measured figures are fine if labeled as such.

For the revision: (1) some cells changed noticeably in the rerun (e.g., Qwen3/Operations/AgentSpec 19→55; Gemma 4/Sneaky Sasquatch/no shield 42→23), so “numerically similar” may understate this, and the p. 8 claim that MENTOR consistently outperforms the baselines should be revisited accordingly; (2) the p. 8 statement about the prompt shield and Llama 3.1 may need updating given the new 35–41 results on three MANTRA domains.

I see clear practical value in an external, reusable procedural shield, and I will maintain my score of 4. I look forward to the Claude cost figures.

Reviewer_eA4QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MENTOR auto-formalizes each manual and its tool schema into LTL temporal requirements and SMT-based argument constraints, compiles the temporal specification into a history-dependent game strategy, and uses a runtime labeler together with a deterministic evaluator to block unsafe calls and recommend prerequisite or repaired calls. The authors implement prompt-level and token-level shields and evaluate them across multiple domains from MANTRA and AGENT ORCA using open-weight and proprietary agents. The results show substantial compliance gains in several settings, particularly for lightweight agents, while also revealing dependence on formalization quality, labeling accuracy, and deployment configuration.

Critical strengths
  1. The problem and solution are well aligned. MENTOR goes beyond block-only guardrails by modeling context-dependent obligations and suggesting actions that help the agent make compliant progress. The combination of LTL strategy templates and SMT-based argument checking is coherent and practically motivated.

  2. The evaluation demonstrates meaningful system-level value. The experiments cover multiple domains, open-weight and proprietary agents, prompt- and token-level enforcement, latency, cost, and labeler/autoformalization choices. The results indicate that a reusable procedural shield can improve lightweight agents and, in some settings, outperform unshielded proprietary agents.

Critical weaknesses
  1. The reliability claim is conditional on two error-prone LLM translation steps. The deterministic evaluator is only as reliable as the manual-and-tool-schema formalization and the runtime labeler. Ambiguous or changing manuals, missing or contradictory tool outputs, unfamiliar language, and adversarial outputs are not evaluated and could lead to incorrect constraints or labels. The paper does not quantify end-to-end specification and labeling errors or provide an uncertainty-aware fallback.

  2. The empirical evidence does not cleanly isolate the source of the gains or establish scalability. Prompt and token shields use different model configurations and serving stacks, and the paper does not separately ablate blocking, progress advice, SMT checking, and token masking. The evaluation uses five runs per task, and SOP-Bench is excluded from the main results because its evaluator ignores tool arguments. The paper also does not report the time, memory, or game-state growth of autoformalization, leaving behavior on longer manuals, longer traces, more entities, complex argument domains, concurrent calls, and side-effectful tools unclear.

Questions for authors
  1. I suggest providing an end-to-end error analysis covering formalization errors, labeler errors, ambiguous or missing tool outputs, and adversarial or out-of-distribution inputs.

  2. I suggest comparing prompt and token shielding under the same model checkpoint, precision, serving stack, tokenizer, sampling parameters, and tool-calling harness, while also adding ablations for blocking, progress advice, SMT checking, and token masking with paired uncertainty estimates.

  3. I suggest reporting the offline formalization cost, strategy-graph size, memory usage, and runtime scaling, clarifying which constraint types and tool behaviors are unsupported, and independently auditing the evaluators used by MANTRA and AGENT ORCA?

4 comments

We thank the reviewer for their thoughtful comments and suggestions. Below, we provide our point-by-point responses.

[Reliance on LLM components] We acknowledge that our implementation relies on two LLM-based components. Attributing errors to one particular component would require nontrivial manual review of the generated specifications and traces, and any conclusions from such an analysis would primarily concern the specific LLMs we used for those components and not the framework itself. As these components are replaceable, our framework does not depend on the specific LLMs or techniques that we tried, therefore assessing them in detail individually is not our goal. That said, we did examine our framework’s performance and sensitivity to different choices of current LLMs and techniques for autoformalization and labeling in our ablation studies in Appendix D.4, and found that they suffice to outperform existing shielding methods.

[Ambiguous manuals, missing tool outputs, unfamiliar language, adversarial inputs] Our evaluation focuses on compliance under the conditions provided by the benchmarks we consider, which involve fixed manuals without adversarial inputs or outputs. Such cases would primarily challenge the translation of requirements into specifications and tool outputs into labels, while MENTOR’s core shielding mechanism operates on the specifications and labels independently of how they are produced. Addressing these challenges therefore requires improving or extending those components. For instance, conflicts that remain unresolved after our autoformalization loop may indicate ambiguous requirements, which can be then deferred to human review.

[Experiment configurations] We thank the reviewer for detecting the inconsistency in our configurations for our two shields. We have reran all open-weights model experiments with the same configuration, using vLLM bf16 and replaced Qwen3.6-35B-A3B (MoE) with Qwen3.6-27B everywhere for consistency. The updated results are numerically similar and preserve our conclusions. MENTOR improves the compliance of unshielded agents and outperforms the baselines in almost all cases. We write our updated Table 1 in an official comment below and will replace all relevant Appendix tables in the revised version.

[Blocking, progress, SMT and token shield] Following the reviewer’s suggestion, we performed additional ablations with 2 LLMs on 2 domains from MANTRA to examine the effect of progress guidance vs just blocking. Specifically, we ran a version of the prompt shield that only blocks non-compliant tool calls and does not give any progress guidance, and a version of the token shield that only masks non-compliant tokens and does not boost progressing ones. The results are in the tables in an official comment below, and demonstrate that progress guidance is important to the shields’ performance improvements. We will add these ablations to the revised version.

SMT checking is a core part of the shield’s evaluator that determines whether a tool call is compliant or progressing, so removing it will not allow correct monitoring of tool calls. We also point to Appendix A.2, where we show concrete examples of failures of prior shielding methods that result from lacking (some of) our shields’ features, such as stalling caused by lack of progress advice or violations of context-dependent argument constraints.

[Autoformalization details] The manuals we considered varied in length (between 500 and 3700 words) and the main autoformalization took less than 1h and resulted in game graphs with 360-5500 states. To give an idea of the scaling of solving such graphs, in STARs (Anand et al., 2026b) Figure 8 in Appendix G.3 analyses the time required to solve game graphs with varying sizes, from less than 100K to over 2M states, needing fewer than 90 minutes for the largest cases. Table 5 reports the maximum cap we set on tool calls per trace (40 on MANTRA, 200 on AGENTORCA and SOP-Bench), though in practice traces were mostly between 3 and 20 tool calls long.

[Constraint types and tool behaviors] MENTOR expresses temporal requirements using standard linear temporal logic (LTL) syntax with finite-trace semantics. Argument constraints are expressed in our DSL, whose SMT encoding uses quantifier-free constraints over strings and linear integer arithmetic. Concurrent tool calls are not supported, as our shield monitors tool calls one at a time, but side-effectful tools are supported as long as their effects are described in the tool schema and correctly captured by the specification.

[Benchmark audits] Our goal is to improve procedural compliance through shielding, not an independent audit of existing benchmarks. We report the SOP-Bench evaluation issue because we identified it during our experiments and considered it important for interpreting the results and informing future evaluations. We did not find any similar issues on MANTRA or AGENTORCA.

AuthorsOpenReview ↗

Blocking-only ablations

Pass^5 ±\pm standard error in percentages over 5 runs.

DomainModelUnshieldedBlocking onlyFull prompt shield
operationsQwen344.68 ± 7.3348.94 ± 7.3772.34 ± 6.60
operationsLlama3.16.38 ± 3.6017.02 ± 5.5436.17 ± 7.08
tau2-retailQwen310.53 ± 4.1015.79 ± 4.8733.33 ± 6.30
tau2-retailLlama3.15.26 ± 2.985.26 ± 2.9815.79 ± 4.87
DomainModelUnshieldedMasking onlyFull token shield
operationsQwen344.7 ± 7.340.4 ± 7.255.3 ± 7.3
operationsLlama3.16.4 ± 3.619.1 ± 5.838.3 ± 7.2
tau2-retailQwen310.5 ± 4.128.1 ± 6.040.4 ± 6.6
tau2-retailLlama3.15.3 ± 3.015.8 ± 4.929.8 ± 6.1
AuthorsOpenReview ↗

Updated Table 1

Pass^5 ±\pm standard error (percentages) over 5 runs with consistent configurations.

(a) Open-weight agents on MANTRA

DomainAgentNoneAgentSpecProgentDynaGuardMENTOR (prompt)MENTOR (token)
Operations (N=47)Llama 3.16 ± 413 ± 54 ± 311 ± 536 ± 738 ± 7
Operations (N=47)Qwen345 ± 755 ± 747 ± 721 ± 672 ± 755 ± 7
Operations (N=47)Qwen3.660 ± 772 ± 768 ± 740 ± 766 ± 766 ± 7
Operations (N=47)Gemma 451 ± 762 ± 747 ± 728 ± 772 ± 757 ± 7
Sneaky Sasquatch (N=43)Llama 3.121 ± 614 ± 521 ± 619 ± 635 ± 742 ± 8
Sneaky Sasquatch (N=43)Qwen333 ± 728 ± 730 ± 730 ± 763 ± 737 ± 7
Sneaky Sasquatch (N=43)Qwen3.653 ± 863 ± 751 ± 849 ± 870 ± 758 ± 8
Sneaky Sasquatch (N=43)Gemma 423 ± 716 ± 623 ± 719 ± 665 ± 765 ± 7
τ²-retail (N=57)Llama 3.15 ± 39 ± 49 ± 412 ± 416 ± 530 ± 6
τ²-retail (N=57)Qwen311 ± 425 ± 612 ± 423 ± 633 ± 640 ± 7
τ²-retail (N=57)Qwen3.621 ± 526 ± 625 ± 619 ± 533 ± 618 ± 5
τ²-retail (N=57)Gemma 416 ± 519 ± 521 ± 519 ± 540 ± 725 ± 6
τ²-airline (N=44)Llama 3.120 ± 630 ± 727 ± 734 ± 741 ± 757 ± 8
τ²-airline (N=44)Qwen336 ± 743 ± 839 ± 734 ± 764 ± 761 ± 7
τ²-airline (N=44)Qwen3.652 ± 850 ± 845 ± 839 ± 764 ± 755 ± 8
τ²-airline (N=44)Gemma 434 ± 743 ± 839 ± 727 ± 750 ± 850 ± 8

(b) Proprietary agents on MANTRA Unchanged.

(c) Open-weight agents on AgentOrca

DomainAgentNoneMENTOR (prompt)MENTOR (token)
Bank (N=51)Llama 3.10 ± 076 ± 643 ± 7
Bank (N=51)Qwen337 ± 780 ± 663 ± 7
Bank (N=51)Qwen3.682 ± 590 ± 484 ± 5
Bank (N=51)Gemma 433 ± 780 ± 667 ± 7
Healthcare (N=39)Llama 3.10 ± 085 ± 659 ± 8
Healthcare (N=39)Qwen323 ± 792 ± 479 ± 7
Healthcare (N=39)Qwen3.687 ± 592 ± 490 ± 5
Healthcare (N=39)Gemma 415 ± 682 ± 674 ± 7

Thanks for these thoughtful and detailed responses. I have no further concerns and will keep my rating as weak acceptance.

Reviewer_zfKROpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the challenges of tool-using LLM agents in following complex procedural rules and making progress toward task completion after their actions are blocked. The proposed MENTOR framework leverages LLMs to construct formal constraints. Specifically, Linear Temporal Logic (LTL) and Satisfiability Modulo Theories (SMT) are used to check temporal constraints and argument validity, respectively. The authors further employ formal algorithms to construct an offline state-decision graph that identifies rule-violating actions and actions that promote task completion under different states. Experiments across multiple tool-use benchmarks demonstrate its effectiveness.

Critical strengths

The paper evaluates MENTOR across multiple LLM backbones and datasets, demonstrating its effectiveness. The results show that guiding agent behavior with MENTOR substantially improves task success rates.

Critical weaknesses
  1. Insufficient experimental evaluation. The main paper presents only one major results table, lacking component-wise ablations and separate evaluations of rule compliance and task success. Baselines such as Plan-and-Act and ReAct are necessary but missing here. Claims about the limitations of manually defined rules and the importance of context-aware constraints lack experimental support.

  2. Limited methodological novelty. The approach mainly relies on LLM-based trajectory planning and offline evaluation of action safety and task progress. Overall, it appears to combine existing formal verification, rule checking, and workflow guidance techniques into an elaborate rule-engineering system rather than introduce a novel agent decision-making or reasoning mechanism.

Questions for authors

The paper lacks sufficient experiments to demonstrate the effectiveness and necessity of the proposed framework and its individual components. Could the authors address the experimental concerns raised in the weaknesses above?

3 comments

We thank the reviewer for their thoughtful comments and suggestions. Below, we provide our point-by-point responses.

[Component-wise ablations] We point out that Appendix D.4 reports ablations comparing different autoformalization models, specifications, and generation techniques, as well as different labeler models. Further, during the discussion period, we have performed additional ablations removing the progress guidance from both our shields and keeping only the blocking (see Blocking-only ablations in response to reviewer eA4Q for details). In the revised version of the paper, we will use the extra page to include some of these results in the main paper.

[Rule compliance vs task success] We intentionally selected benchmarks that evaluate procedural compliance as part of task success. The evaluation metrics in these benchmarks require both task completion and procedural compliance, so completing a task while violating the procedure does not count as successful.

[Other baselines] Plan-and-Act and ReAct concern the agent’s planning and reasoning about actions, while MENTOR provides an external shield that checks and constrains tool calls, and gives guidance according to a procedural specification. That is why our experiments compare against existing shielding methods, to focus on the effect of shielding while holding the underlying agent fixed. Approaches such as Plan-and-Act and ReAct are complementary to shielding, and MENTOR can be combined with either approach to provide an additional layer of checks. This is an interesting extension that we will discuss in the revised version of the paper.

[Manually defined rules and context-aware constraints] Our concern about manual rule construction is the effort required to do so across different domains and manuals, as it is a difficult and time-consuming task for humans (Czepa et al., 2019; Greenman et al., 2023). We do not claim that manually written rules necessarily result in worse compliance. Separately, in Appendix A.2 we provide examples to demonstrate the importance of context-dependent constraints. Specifically, in a software-installation task, whether install_software is permitted depends on the outcomes of earlier vulnerability scans, which for example Progent often failed to account for.

[Novelty] First, we would like to clarify that the goal and contribution of MENTOR is a shield for tool-using LLM agents, rather than a new planning or reasoning mechanism. Shielding (as discussed in Related Work) is a separate and complementary research direction that can be combined with planning and reasoning approaches by checking and constraining the actions they generate. In this context, we advance prior shielding methods by addressing history-dependent temporal obligations, argument constraints, and guidance towards task completion. Although we build on a strategy-template algorithm developed for shielding RL policies, adapting it to LLM agents requires handling arguments over potentially unbounded domains, interpreting natural-language observations, and constraining calls generated token by token, all of which MENTOR addresses. Further, our experiments show that MENTOR improves compliance over prior shielding methods on several domains, and we explicitly showcase failures of those methods that our shield is designed to address.

Finally, we clarify that MENTOR does not use an LLM to plan the agent’s trajectory, and the evaluation of the agents’ actions happens online. Offline, LLMs are used to translate the manual into a formal specification, from which we deterministically synthesize a reusable strategy template. At runtime, MENTOR checks proposed tool calls and their arguments and provides guidance based on the strategy template. The only LLM-based online component of our shield is the labeler, that grounds task descriptions and tool outputs to our specification.

Thank you for the rebuttal and your efforts. The clarifications resolve some concerns, and I acknowledge MENTOR’s overall gains. My remaining concern is that it remains unclear whether the unshielded baselines explicitly incorporate CoT reasoning and iterative correction based on tool feedback, making MENTOR’s additional benefit over such agent setups difficult to assess. The described blocking-only ablation assesses the contribution of progress guidance, but does not by itself establish an advantage over LLM-generated guidance using the same checker. I therefore raise my rating to weak rejection.

We are glad our rebuttal addressed some concerns and thank the reviewer for raising their score.

We would like to clarify that all agents in our experiments, including unshielded ones, receive tool outputs and use it to reason about their future actions, and the Qwen agents had reasoning enabled. Our baseline guards also provide feedback to the agent for iterative correction. Specifically, DynaGuard returns an LLM-generated explanation of a violation, AgentSpec returns the triggered rule when blocking a call, and Progent returns a rejection reason. Further, our blocking-only ablation uses MENTOR’s checker while removing progress advice, but the agent still receives blocking feedback and can use it to propose alternative actions in the next rounds. Finally, in MENTOR, the strategy template supports both checking and deterministic progress guidance, so the evaluator can directly identify progressing actions and prerequisites. Equipping an LLM guide with our evaluator to infer this same information would therefore use extra inference for an already-existing functionality.

Comments on the paper

1 comment

We thank the reviewers for their suggestions and discussions and are glad our shield was appreciated. We have posted a revised pdf, with all changes marked in violet. Specifically, we have made the following changes:

  • We have updated our tables with the new results with consistent experiment configurations (main evaluations on all benchmarks, labeler ablations and autoformalization ablations). We have also updated our setup description in Appendix D accordingly and clarified that some Appendix example traces were from preliminary runs. Overall, MENTOR’s compliance gains remain supported. As a result of the new configuration, we have edited our claims interpreting the results to tone down our claim of consistently outperforming the baselines and about Llama struggling with the prompt shield, as suggested by the reviewers. Further, the performance of non-finetuned open-weights labellers increased, so we adjusted a claim in section 4 to reflect this change.
  • We added our re-measured autoformalization token usage, loop iterations, wall-clock time and API-equivalent costs, together with information about graph sizes, in Appendix D.5.
  • We have added our ablation regarding the progress guidance vs just blocking in Table 2 in the main paper.
  • We specified that our setting assumes sequential and not concurrent tool calls, and that it is compatible with planning and reasoning methods such as Plan-and-Act and ReAct in section 2.
  • We discussed adversarial, unfamiliar, ambiguous inputs and changing user requirements in the Discussions section.
  • Fixed typos.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.