- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MANTRA automatically generates compliance benchmarks for tool-using LLM agents from procedural manuals and tool schemas. It separately generates trace-level checks and a symbolic world model, uses bounded SMT queries to identify disagreements, and repairs them using counterexamples. The resulting benchmark contains 285 tasks across six domains.
Critical strengths
- An important and practically relevant problem.
I particularly appreciate the focus on whether agents follow prescribed procedures throughout execution. Checking only the final outcome can overlook consequential violations, making reliable trace-level evaluation valuable.
- A promising technical design and useful artifact.
Combining complementary formal representations with counterexample-guided repair is an appealing approach. Deterministic, fine-grained checks enable reproducible grading and interpretable failure analysis, while the six-domain benchmark demonstrates substantial implementation effort.
Critical weaknesses
- Agreement between two generated representations does not establish correctness.
Both artifacts and their repairs use the same underlying LLM, so they may share the same misunderstanding or omission while remaining mutually consistent. Although the paper acknowledges this limitation, it provides insufficient independent quantitative evidence that the checks faithfully capture the original manuals. This is central to the reliability of the resulting benchmark.
- Reliability beyond short validation horizons remains unclear.
All experiments use a validation bound of (h=16), justified mainly by average task lengths. Average lengths do not establish coverage of longer workflows or conflicts that require longer traces. Without a trace-length distribution or horizon-sensitivity analysis, the practical impact of this bound—and the support for broader claims about complex procedural workflows—remains uncertain.
Questions for authors
-
Why does the radiation-poisoning example on p. 35 require bed_rest when the cited manual specifies anti-radiation medication?
-
What independent evidence shows that the checks and world model do not share the same misinterpretations of the manual?
-
Does increasing the validation horizon beyond (h=16) reveal additional inconsistencies in currently validated cases?
-
Can you provide the missing Qwen3.6 results in Table 7 that support the claim that it is the strongest model?
I find the research problem important and the approach promising. I would consider raising my score if these concerns are satisfactorily addressed.
3 comments
Thank you for the clarifications and additional results. My concerns have been partially addressed, but the response on consistency versus correctness does not yet resolve my main concern. I am asking for independent empirical evidence that the generated checks and world model faithfully capture the manuals, rather than a formal correctness guarantee. Could you provide details of the manual inspection, including how many cases were examined, who performed the inspection, and how shared misinterpretations and omissions were checked?
We are glad that our response addressed some of your concerns. We have described our manual inspection in a comment in response to reviewer uwpY above, see [Manual Inspection].
[Consistency vs correctness] We explicitly acknowledge that our cross-validation does not guarantee the checks’ correctness with respect to the manual, but we would like to point out that establishing that a formal specification faithfully captures an natural-language manual remains an open problem in the field. With MANTRA, we propose a principled approach towards making progress on this problem using redundancy and counterexample-guided refinement, as disagreements between the world model and checks expose potential errors in at least one of the two. Further, we use generated compliant and non-compliant traces to provide more partial evidence about intended behavior, much like unit tests do in software development, and our manual inspection did not find any cases of shared hallucinations between the two artifacts. Finally, although our experiments use GPT-5.4 throughout, we would like to clarify that the pipeline supports using different LLMs for different stages.
[Trace bound] The trace bound is configurable and can be increased to accommodate longer workflows. We chose h=16 in our experiments because we observed shorter average trace lengths of 3-8. To assess sensitivity to this choice, we reran the solver with h=100 and found no additional inconsistencies. All observed conflict witnesses had lengths between 1 and 7.
[Radiation-poisoning example] We thank the reviewer for catching this error in the appendix example. We verified in our code that both the submitted testcase and the checks used in our evaluation runs correctly require Anti-Radiation Medication after inspecting the patient. The incorrect bed_rest argument appears only in the manuscript example and does not affect the benchmark or reported results. We will correct the example to match the actual testcase.
[Qwen3.6 results] We thank the reviewer for pointing out the incomplete Table. Below, we provide the Qwen3.6:36B results. Qwen3.6 achieves the highest domain-averaged Pass@1 and Pass@5 and leads the most domain–metric comparisons (7 of 18), while Claude Haiku achieves the highest domain-averaged Pass(^5).