acceptodds
Under review as a conference paper at ICLR 2027

MANTRA: Synthesizing SMT-Validated Compliance Benchmarks for Tool-Using LLM Agents

Abstract

Tool-using large language model (LLM) agents are increasingly deployed in settings where their reliable behavior is governed by strict procedural manuals. Ensuring that such agents comply with the rules from these manuals is challenging, as they are typically written for humans in natural language while agent behavior manifests as an execution trace of tool calls. Existing evaluations of LLM agents rely on manually constructed benchmarks or LLM-based judges, which either do not scale or lack reliability for complex, long-horizon manuals. To overcome these limitations, we present MANTRA, a framework that automatically synthesizes machine-checkable compliance benchmarks from a natural-language manual and a tool schema. Since no ground truth exists for translating natural language into formal checks, and even humans are known to be unreliable at such tasks, MANTRA replaces ground truth by redundancy: it independently generates (i) a symbolic world model capturing procedural dependencies, and (ii) a set of trace-level compliance checks for a given task, and validates their consistency using SMT solving. A structured, counterexample-based repair loop resolves inconsistencies, with human review only as a fallback. Importantly, MANTRA supports arbitrary domains and long procedural manuals, and provides a tunable notion of task complexity, so harder tasks can be derived from the same document when agents capabilities improve. Using MANTRA, we build a new benchmark suite with 285 tasks across 6 domains scaling to 50+ page documents with minimal human effort. Empirically, we show that our compliance checks are richer with stronger constraint enforcement compared to existing benchmarks. Additionally, the granularity of the checks can be used for debugging the agents' failure modes. These results demonstrate that combining automated benchmark generation with formally grounded validation methods enables scalable and reliable benchmarking of tool-using agents.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
LLM Agents, Benchmark Generation, SMT, Policy Compliance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M4NPOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

MANTRA automatically generates compliance benchmarks for tool-using LLM agents from procedural manuals and tool schemas. It separately generates trace-level checks and a symbolic world model, uses bounded SMT queries to identify disagreements, and repairs them using counterexamples. The resulting benchmark contains 285 tasks across six domains.

Critical strengths
  1. An important and practically relevant problem.

I particularly appreciate the focus on whether agents follow prescribed procedures throughout execution. Checking only the final outcome can overlook consequential violations, making reliable trace-level evaluation valuable.

  1. A promising technical design and useful artifact.

Combining complementary formal representations with counterexample-guided repair is an appealing approach. Deterministic, fine-grained checks enable reproducible grading and interpretable failure analysis, while the six-domain benchmark demonstrates substantial implementation effort.

Critical weaknesses
  1. Agreement between two generated representations does not establish correctness.

Both artifacts and their repairs use the same underlying LLM, so they may share the same misunderstanding or omission while remaining mutually consistent. Although the paper acknowledges this limitation, it provides insufficient independent quantitative evidence that the checks faithfully capture the original manuals. This is central to the reliability of the resulting benchmark.

  1. Reliability beyond short validation horizons remains unclear.

All experiments use a validation bound of (h=16), justified mainly by average task lengths. Average lengths do not establish coverage of longer workflows or conflicts that require longer traces. Without a trace-length distribution or horizon-sensitivity analysis, the practical impact of this bound—and the support for broader claims about complex procedural workflows—remains uncertain.

Questions for authors
  1. Why does the radiation-poisoning example on p. 35 require bed_rest when the cited manual specifies anti-radiation medication?

  2. What independent evidence shows that the checks and world model do not share the same misinterpretations of the manual?

  3. Does increasing the validation horizon beyond (h=16) reveal additional inconsistencies in currently validated cases?

  4. Can you provide the missing Qwen3.6 results in Table 7 that support the claim that it is the strongest model?

I find the research problem important and the approach promising. I would consider raising my score if these concerns are satisfactorily addressed.

3 comments

[Consistency vs correctness] We explicitly acknowledge that our cross-validation does not guarantee the checks’ correctness with respect to the manual, but we would like to point out that establishing that a formal specification faithfully captures an natural-language manual remains an open problem in the field. With MANTRA, we propose a principled approach towards making progress on this problem using redundancy and counterexample-guided refinement, as disagreements between the world model and checks expose potential errors in at least one of the two. Further, we use generated compliant and non-compliant traces to provide more partial evidence about intended behavior, much like unit tests do in software development, and our manual inspection did not find any cases of shared hallucinations between the two artifacts. Finally, although our experiments use GPT-5.4 throughout, we would like to clarify that the pipeline supports using different LLMs for different stages.

[Trace bound] The trace bound is configurable and can be increased to accommodate longer workflows. We chose h=16 in our experiments because we observed shorter average trace lengths of 3-8. To assess sensitivity to this choice, we reran the solver with h=100 and found no additional inconsistencies. All observed conflict witnesses had lengths between 1 and 7.

[Radiation-poisoning example] We thank the reviewer for catching this error in the appendix example. We verified in our code that both the submitted testcase and the checks used in our evaluation runs correctly require Anti-Radiation Medication after inspecting the patient. The incorrect bed_rest argument appears only in the manuscript example and does not affect the benchmark or reported results. We will correct the example to match the actual testcase.

[Qwen3.6 results] We thank the reviewer for pointing out the incomplete Table. Below, we provide the Qwen3.6:36B results. Qwen3.6 achieves the highest domain-averaged Pass@1 and Pass@5 and leads the most domain–metric comparisons (7 of 18), while Claude Haiku achieves the highest domain-averaged Pass(^5).

DomainPass@1Pass@5Pass^5
Cabin57.8 ± 10.076.2 ± 10.634.9 ± 11.9
Operations51.9 ± 13.757.4 ± 14.344.7 ± 14.4
Sasquatch55.3 ± 12.967.4 ± 14.234.9 ± 14.4
Airline54.5 ± 13.565.9 ± 14.243.2 ± 14.8
Telecom49.7 ± 15.764.5 ± 17.138.7 ± 17.4
Retail46.7 ± 11.357.9 ± 12.926.3 ± 11.5

Thank you for the clarifications and additional results. My concerns have been partially addressed, but the response on consistency versus correctness does not yet resolve my main concern. I am asking for independent empirical evidence that the generated checks and world model faithfully capture the manuals, rather than a formal correctness guarantee. Could you provide details of the manual inspection, including how many cases were examined, who performed the inspection, and how shared misinterpretations and omissions were checked?

We are glad that our response addressed some of your concerns. We have described our manual inspection in a comment in response to reviewer uwpY above, see [Manual Inspection].

Reviewer_uwpYOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MANTRA, a pipeline for generating deterministic, trace-level compliance benchmarks for tool-using LLM agents from natural-language procedural manual, tool schema, and a database. MANTRA first builds a document-dependence graph and samples regions of the manual to generate concrete scenarios. For each sampled region, an LLM independently produces (i) scenario-specific checks over tool-call traces and (ii) a symbolic world model in a typed DSL. Both artifacts are compiled into bounded SMT formulas. Forward conflict search identifies traces accepted by the checks but disallowed by the world model, while a backward audit identifies potentially over-restrictive checks. Solver counterexamples are passed to an LLM-based, typed repair loop, with structured human review available as a fallback. The authors use MANTRA to construct 285 benchmark cases across six domains. Six LLM agents are evaluated with five runs per task. The resulting check-level analysis finds that most observed failures involve omitted required calls or missing anchors, and it identifies a recurring "write before read" behavior.

Critical strengths

A useful two artifact formulation for deterministic trace level grading. MANTRA separately generates a symbolic world model and scenario-specific compliance checks, cross-validates them using an SMT solver, and refines inconsistencies through solver-generated counterexamples and typed repairs. This design supports deterministic grading of required and forbidden tool calls and their ordering without relying on an LLM judge at evaluation time.

The benchmark is accompanied by unusually actionable trace level analysis. The paper demonstrates generation across manuals and tool schemas of different sizes, while the appendix provides a detailed analysis of 10,895 failed checks, showing that missing required calls and anchors account for 76% of failures. This highlights a meaningful advantage of deterministic trace checks: they support diagnosis beyond a task-level success bit.

Critical weaknesses

SMT cross-validation establishes bounded consistency rather than fidelity to the source manual. Because the compliance checks and world model are generated from the same sampled text, shared omissions or misinterpretations may remain undetected. The appendix confirms that some semantic issues passed automated validation, including unjustified call restrictions, brittle argument pinning, convention mismatches, and missing alternatives. Although the final tasks were manually inspected and filtered, the paper does not report a systematic audit protocol. The results therefore support internal consistency and deterministic grading more strongly than reliable translation of the original manual.

The claimed advantage over existing benchmarks is not sufficiently supported. The comparison is primarily qualitative, and the paper does not evaluate MANTRA against related benchmarks using matched tasks or a common constraint taxonomy. The results demonstrate that MANTRA supports detailed trace level checks, but do not establish that its checks are consistently richer or enforce stronger constraints than existing approaches.

Questions for authors

What is the individual contribution of each component in the validation pipeline? Table 2 reports only the sequential yield after forward validation and backward auditing. An ablation removing each stage would help clarify the value of the two validation directions.

Appendix D.1 states that six models were evaluated, but Table 7 reports results for only five and omits Qwen3.6:36B. The surrounding discussion nevertheless identifies Qwen3.6 as the strongest overall model, and later analyses include it. I could not find its per-domain Pass@1, Pass@5, or Pass(^5) results elsewhere in the paper. Please provide the missing results and clarify whether Table 7 and the aggregate failure analyses were computed from the same six-model evaluation runs.

Among the 297 cases that passed backward validation, how many were accepted unchanged, manually edited and retained, or removed? The paper describes several issues found during manual inspection, but does not report the review protocol or quantify the extent of manual intervention. Clarifying this process would help assess the reliability of the final benchmark.

3 comments

[Consistency vs correctness] We explicitly acknowledge that our cross-validation does not guarantee the checks’ correctness with respect to the manual, but we would like to point out that establishing that a formal specification faithfully captures an natural-language manual remains an open problem in the field. With MANTRA, we propose a principled approach towards making progress on this problem in order to reduce (not completely replace) the manual effort involved in constructing a benchmark. While consistency of the generated artifacts does not guarantee the lack of shared misinterpretations or omissions, the artifacts are generated independently using different DSLs to reduce the likelihood of this happening. Further, our manual inspection did not find any cases of shared hallucinations between the two artifacts. Removed or edited testcases, as described in Appendix E, were either trivial (0 checks) or were problematic due to engineering issues rather than conceptual errors in the framework. For instance, brittle argument pinning and convention mismatches come up due to the ambiguous nature of natural-language manuals and can be partially solved with stricter tool schema design (outside our framework). However, human review and correction are part of the benchmark construction process that MANTRA supports, and the human review interface we provide allowed our human reviewer to identify and easily fix the reported recurring issues in the generated benchmark.

[Comparison with prior benchmarks] We clarify that our claims of richer evaluation concern the granularity of our trace-level checks, the failure analysis that it enables beyond simple accuracy metrics, and the ability to tune the difficulty of the generated testcases. MANTRA’s checks can distinguish missing required calls, forbidden calls, argument constraints, and ordering violations, as demonstrated in the detailed analysis in Appendix D.2. Further, since prior benchmarks involve different domains, manuals, tool schemas, and evaluation mechanisms, comparing types of constraints (wherever applicable) across them would confuse differences in evaluation capability with differences in what the tasks require. Such comparisons become even more complicated on benchmarks that evaluate task success through final-state checks or LLM-based judgments, without exposing constraints comparable to MANTRA’s checks.

[Contribution of the validation directions] The two validation directions detect different, complementary issues in the generated artifacts, and removing one of them allows these issues to persist in the final checks. Specifically, forward validation searches for traces that satisfy the checks but violate the world model, and detects potentially missing constraints in the checks or excessive restrictions in the world model. On the other hand, the backward audit searches for traces that satisfy the world model and all but one check, and detects potentially unjustified restrictions in that particular check or missing constraints in the world model. We perform forward refinement of the checks first and then the backward audit of all remaining checks, because the backward audit examines each of these checks individually. Removing a validation stage would increase the number of accepted cases simply because fewer problems are detected. In particular, without the backward audit, overly restrictive checks can pass forward validation and exclude otherwise admissible traces. Table 2 already reports the outcome before backward auditing, corresponding to stopping validation after the forward stage. On the other hand, removing the forward validation will allow overly permissive checks to survive because allowing invalid traces doesn’t necessarily create a backward conflict. See also Appendix B.5 for an example of this cross-validation and repair process.

[Qwen3.6 results] We thank the reviewer for pointing out the incomplete Table. We have confirmed that the results were computed from the same six-model evaluation runs, but had omitted Qwen3.6:36B unintentionally. Below, we provide the Qwen3.6:36B results and we will add them to the revised version:

DomainPass@1Pass@5Pass^5
Cabin57.8 ± 10.076.2 ± 10.634.9 ± 11.9
Operations51.9 ± 13.757.4 ± 14.344.7 ± 14.4
Sasquatch55.3 ± 12.967.4 ± 14.234.9 ± 14.4
Airline54.5 ± 13.565.9 ± 14.243.2 ± 14.8
Telecom49.7 ± 15.764.5 ± 17.138.7 ± 17.4
Retail46.7 ± 11.357.9 ± 12.926.3 ± 11.5

[Manual inspection] We thank the reviewer for requesting this clarification to strengthen the reliability judgement of MANTRA. Out of the 297 cases, only 12 were removed and 9 required the addition of checks which were missing due to an incomplete sampling of the document. The remaining 276 cases were logically correct. We choose to perform minor edits on these correct cases to improve the permissiveness of the shield, i.e. to lower the rate of false negatives. These edits affected 51 cases, where specific argument pinning was removed or convention mismatches were resolved. Further, we removed unnecessary checks from 157 cases, to allow for repeated tool calls which are not restricted by the manual. We emphasize that using these cases unchanged would have also led to a sound shield, i.e., all shielded executions would be correct.

The review protocol used to perform these checks and edits was as follows. One author first performed spot checks on randomly selected test cases using MANTRA’s interface to inspect the sampled manual passages, scenarios, and generated checks. Through these spot checks they identified the recurring issue categories described in Appendix E, including the minor issues which do not impact correctness but only reduce permissiveness. The author then performed an LLM-assisted sweep of all checks across the 297 cases that passed backward validation, specifically looking for these concrete recurring issues, and examined all suggestions. This process highlights the need for the targeted human review that MANTRA supports to find and later address issues that LLM generation and cross-validation cannot deal with. We will clarify this in the revised version.

Thank you for your response. I recognize the value of this framework, but the experimental evaluation is not sufficiently comprehensive. More importantly, the broader applications enabled by this scalable approach deserve greater attention, yet remain underexplored in the paper.

Reviewer_34srOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MANTRA generates benchmarks for testing whether tool-using agents follow procedures in natural-language manuals. Given a manual and tool schema, it generates scenarios, trace-level compliance checks, and a symbolic world model. It uses an SMT solver to find disagreements between the checks and model, then uses counterexamples in a repair loop. The authors apply this pipeline to 285 tasks across six domains and evaluate several LLM agents.

The paper’s main contribution is a structured way to generate and deterministically grade procedural-compliance benchmarks at scale. Its validation supports consistency between two generated formalizations, but does not establish that either one fully captures the source manual. The experiments show that agents often skip required tool calls; they also show that ordering violations are a small share of observed failures.

Critical strengths
  • A concrete pipeline for generating trace-level checks. The combination of scenario generation, a symbolic world model, SMT counterexamples, and structured repairs is a thoughtful approach to scaling procedural evaluations.

  • A broad demonstration and diagnostic analysis. The authors test manuals of different lengths across six domains, report agent results, and categorize failure modes. The finding that missing required calls account for most failed checks helps clarify what these tasks measure.

Critical weaknesses
  • SMT cross-validation establishes consistency between generated artifacts, not correctness with respect to the manual. The world model and checks are generated separately, but the paper uses GPT-5.4 for all pipeline stages. Shared omissions or misinterpretations can therefore survive the solver checks. The authors report inspecting automatically validated items, but give no independent annotation protocol or measured error rate. Likewise, generated compliant and non-compliant traces can test whether the checks behave as written without establishing that those checks capture the manual’s intent. This leaves the paper’s central reliability claim insufficiently validated.

  • The evidence for stronger or richer benchmarks is limited. The comparison with prior benchmarks is primarily a qualitative feature table, and the agent experiments show that models fail the generated checks—not that the checks accurately represent the manuals or improve on existing evaluations. The paper also does not report rule-level coverage of the source manuals or an independent quality comparison against human-authored checks, making it difficult to assess the final suite’s coverage and validity.

Questions for authors
  • How was the manual inspection of automatically validated items conducted? In particular, were reviewers independent of the benchmark generation, did they assess alignment with the source manual, and were disagreements or error rates recorded?

  • Since GPT-5.4 is used across pipeline stages, what makes the world-model and check generation independent in practice? Is there evidence that correlated omissions or misinterpretations are rare?

1 comment

[Consistency vs correctness] We explicitly acknowledge that our cross-validation does not guarantee the checks’ correctness with respect to the manual, but we would like to point out that establishing that a formal specification faithfully captures a natural-language manual remains an open problem in the field. With MANTRA, we propose a principled approach towards making progress on this problem in order to reduce (not completely replace) the manual effort involved in constructing a benchmark. We use redundancy and counterexample-guided refinement, as disagreements between the world model and checks expose potential errors in at least one of the two. Although our experiments use GPT-5.4 throughout, the pipeline supports using different LLMs for different stages, and the two formal artifacts are generated independently using different DSLs to reduce correlation. Further, compliant and non-compliant traces do provide partial evidence about intended behavior, much like unit tests in software development.

Finally, we note that MANTRA supports targeted human review through an interface that presents unresolved conflicts, concrete counterexamples, relevant manual passages, checks and world models, helping domain experts identify and resolve specific specification issues. Human review and correction are explicit parts of the benchmark construction process that MANTRA supports. We used our interface for our manual inspection, and it allowed us to easily identify and fix recurring issues documented in Appendix E. See [Manual inspection] in response to reviewer uwpY for details on our review.

[Comparison with prior benchmarks] We clarify that our claims of richer evaluation concern the granularity of our trace-level checks and the failure analysis that it enables beyond simple accuracy metrics, and the ability to tune the difficulty of the generated testcases. MANTRA’s checks can distinguish missing required calls, forbidden calls, argument constraints, and ordering violations, as demonstrated in the detailed analysis in Appendix D.2.

[Human-authored checks] While a comparison with human-authored checks would be insightful, it requires annotators to understand the manual, tool schema, scenarios, and specification language, and to encode requirements while preserving valid alternative behaviors. Such a study requires targeted recruitment of domain experts, who can be difficult to reach through standard academic study recruitment protocols, together with training in the specification language. The resulting recruitment, training, and annotation effort is substantial, and this domain understanding and formalization effort is precisely what MANTRA aims to help with and reduce.

[Rule coverage] Measuring rule-level coverage of a manual requires a list of its rules, which is not available for the manuals we considered. If it was available, such a list could also be used by MANTRA to aid the generation pipeline. That said, MANTRA’s document region sampling strategy, described in Appendix C.6, encourages broad coverage of the manual by prioritizing under-covered regions of the manual’s dependence graph (Appendix C.6).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.