- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work presents a benchmark for measuring how well LLMs generate symbolic policies from enterprise agent instructions written in prose, so that other systems can readily interpret them. The authors point out that existing benchmarks either evaluate only trajectories with the guardrail fixed or absent, or compare against a hand-crafted policy in a single domain, which has made comparison across synthesis systems difficult. The benchmark consists of 32 LLM-synthesized workflows across 8 domains, comprising system prompts, trust models, tool descriptions, 543 golden intents, and 907 labelled test cases. Generated policies are frozen and then evaluated from two perspectives, static fidelity and behavioural consequence: the former is measured by strict coverage and rule precision, and the latter by the false-allow rate (FAR) and false-block rate (FBR). The authors evaluate 7 models and point to a fidelity–utility trade-off, noting that models with high static fidelity also show high FBR. They further analyze that high FAR is mostly due to requirements missing from the policy, whereas high FBR stems less from rule generation itself than from when the rules are applied.
Critical strengths
- By clearly identifying what existing benchmarks do not address, this paper targets LLMs' policy-generation capability itself, and its contribution lies in providing, for the first time, a public multi-domain shared resource that enables comparison across systems.
- It measures LLM capability separately in terms of static fidelity and behavioural consequence, reporting missing rules as false allows and rule application as false blocks. In other words, it splits LLMs' policy-generation capability into two axes, coverage and utility, and reports that there is a trade-off between the two.
- It explicitly states that a change of judge can affect behavioural consequence, and honestly reports, metric by metric, which are reliable (FAR) and which can fluctuate (FBR, BA).
Critical weaknesses
W1. I checked the supplementary material for the trust model, but could not find any information about it anywhere in the paper. Even if it is a commonly used term in the security field, please define its meaning in the paper. It also needs to be explained how it differs from the prompt and why the two are separated. The submitted policy in the figure is presented only as an abbreviated checklist, so the actual rule structure defined in Section 3 (governed target, description, and the four constraint kinds) cannot be seen anywhere in the main text. Including one rule example each for a structured constraint and a natural-language constraint, taken from an actual submission, would greatly help readers understand the contract.
Related to the above, there is no example of the actual reference prompt, so the explanation of how policies are generated is also insufficient. Overall, it is difficult to picture what the benchmark looks like. Examples of the model’s input and output are also lacking. Some content appears to be summarized only in Table 2 without being introduced in the main text, such as the concepts of system spread and run shift. To be precise, the numbers and concepts are described in the main text, but their names (system spread, etc.) and how they are computed appear only in the caption.
W2. The benchmark is described as fully synthetic, but the model used to synthesize it is not specified. The entire dataset is synthetic and there seems to be no human verification at all, which concerns me. Without an end-to-end evaluation, I wonder whether it is difficult to determine if these policies actually affect the safety of the system.
W3. For FBR, the results vary too much depending on the judge (especially for Opus). Since models tend to rate their own outputs more highly (self-preference bias), I think a non-overlapping judge should be used, or this issue should otherwise be addressed.
Minor:
The paper states that ST-WebAgentBench is the closest work in agent safety evaluation, but the explanation of why it is the closest is too brief. It should clarify in what respects the two are similar. Beyond the harmfulness-oriented works mentioned earlier, the paper should include more related work that measures policies. Figure 1 contains both the generation process and the two evaluation views in a single figure, making its information density excessively high. Although the evaluation boundary is marked, the generation stage and the evaluation stage are not visually separated enough, which makes it hard for first-time readers to follow the flow.
Note to authors and AC: The weaknesses have been condensed and grouped for clarity. This is an editorial change only — the content and my overall evaluation remain the same as in the original version.
Questions for authors
Q1. Are there no other reference benchmarks that measure policy guardrails? Is ST-WebAgentBench the only related work? There are also works such as SHIELDAGENT-BENCH; are they not related to this paper?
Q2. Is the trust model a concept that is commonly used explicitly? If so, is there a need to set up such a trust model under realistic conditions? I also wonder whether the trust model makes the problem easier, since it directly provides the constraints in natural language.
Q3. Are there any research results or measurements showing that creating policies actually makes agents safer?
Q4. Even if the system policy is wrong, are there cases where the model fills in the content with its own reasoning power?
Q5. If the benchmark was synthesized with one particular model, wouldn't that model have an advantage in scoring because of its similar distribution? I am also unsure whether the benchmark will remain applicable when next-generation models come out.
Q6. Regarding generation, the paper states that "temperature zero is not a determinism guarantee." I wonder why the authors did not run generation multiple times and average the results. Does the fact that the generator's outputs differed even at temperature 0 mean that reasoning was used? If so, the quality could vary depending on the reasoning effort used. Please also specify whether the judge used reasoning and what its reasoning effort was.
Minor
M1. Please make sure that no text is cut off in Figure 2.
M2. In particular, the paper should specify what kind of correlation coefficient ρ is. It appears to be Spearman's ρ; please confirm.