- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces in-context policy guardrailing: given a conversation and an application-specific policy, a guardrail must predict the exact set of violated rules. To evaluate this, the authors build SafePyramid, with 1,000 LLM-generated conversations across 10 domains and 3,000 policies containing 61,641 rules. The benchmark has three levels: independent rules (L0), interacting exception and conditional rules (L1), and novel fictional policy frameworks (L2). Across 16 systems, performance drops sharply from L0 to L2. The best model, GPT-5.5, achieves exact match on only 54.0%, 35.3% and 12.9% of cases at L0, L1 and L2. Per-rule decomposition, agentic harnesses and higher reasoning effort all improve results.
Critical strengths
- The paper provides useful task formulation and diagnostic structure.
- The paper provides comprehensive evaluations and results.
Critical weaknesses
- Low data diversity. Each data type comes from a single generator: Grok writes all conversations, and GPT-5.4, Gemini and Claude write all L0, L1 and L2 policies respectively. This limits diversity and confounds difficulty level with generator, so part of the drop from L1 to L2 may reflect Claude’s writing style rather than framework adaptation.
- Unrealistic, conversation-dependent policies. Policies are generated after and conditioned on each conversation. Real policies are written before traffic is observed and must generalize, which weakens the benchmark’s realism claim.
- Unclear per-rule protocol. The paper never explains how the “relevant policy context” is selected. Results with the full policy and conversation are needed.
Questions for authors
- In per-rule evaluation, how is the “relevant policy context” selected? How do per-rule results change if each rule is judged with the full, unorganized policy and the full conversation?
6 comments
Weakness 2: Unrealistic, Conversation-dependent Policies
W2. Unrealistic, conversation-dependent policies. Policies are generated after and conditioned on each conversation. Real policies are written before traffic is observed and must generalize, which weakens the benchmark’s realism claim.
A2: We thank the reviewer for this insightful comment. We agree that real-world policies should be general and applicable across different conversations. In fact, We carefully considered this issue when designing SafePyramid and would like to clarify our design rationale:
-
Our design choice is driven by the goal of the benchmark.
- We carefully considered generating general policies first and then evaluating them against conversations. However, a general policy may contain many rules that are not relevant to a given conversation, making it difficult to ensure that each conversation covers the specific rules, exceptions, and conditions we want to evaluate. As a result, some challenges may rarely appear in the benchmark, while others may appear together in ways that are difficult to separate. This would limit our ability to understand which in-context policy guardrailing capabilities those models struggle with and why they fail.
- We therefore chose to generate conversations first and then construct policies based on them. This allows us to deliberately design cases that test specific reasoning requirements, such as understanding individual rules (L0), handling exceptions and conditions (L1), and adapting to new policy frameworks (L2). It also allows us to control which rules are violated and which are not, providing clear ground-truth labels for evaluating model predictions. Our goal is not simply to rank models by their overall performance, but to help researchers better understand their strengths, weaknesses, and failure modes across different policy reasoning tasks. We believe this makes SafePyramid more useful for studying and improving in-context policy guardrails.
-
Our policies still reflect realistic policy structures. Although the policies are generated based on conversations, their structures are not arbitrarily designed around those conversations. As discussed in the paper, our policy schemas are inspired by real-world regulatory policies, including the GDPR, which contains exceptions, conditional obligations, and alternative safeguards. We use these real-world patterns to design the rules and reasoning requirements in SafePyramid. In addition, using synthetic policies helps reduce potential contamination from existing policies that models may have encountered during training, allowing us to better evaluate whether models can understand and apply policies provided in context.
-
Additional Experiment 1: Policies contain substantial content beyond the given conversation. Although each policy is generated with a particular conversation in mind, it also contains many rules that do not apply to that conversation. Our additional analysis in Table 1 below shows that distractor rules account for 25.2–52.0% of rules across the three levels.
Table 1. Policy composition across difficulty levels.
Level Rules per case Distractor rules Exception rules Conditional rules L0 16.1 52.0% 0.0% 0.0% L1 28.7 29.2% 19.4% 24.5% L2 32.9 25.2% 20.6% 30.4%
These results show that models must distinguish relevant from irrelevant policy rules, rather than simply matching all rules to the given conversation.
-
Additional Experiment 2: Policies are not simple lexical reproductions of conversations. We further measured the proportion of content words in each rule that also appear in its corresponding conversation. As shown in Table 2, decisive rules share only 38–39% of their content words with the corresponding conversation, while other rule types show even lower overlap (i.e., 23–26%). This suggests that rule violation identification requires more than simple lexical matching.
Table 2. Lexical overlap between policy rules and conversations.
Rule type Number of rules Mean lexical overlap Decisive, violated 15,931 38% Decisive, not violated 7,371 39% Distractor 25,066 26% Conditional 17,034 25% Exception 12,353 23%
Overall, our conversation-first approach is intended to make SafePyramid more useful for understanding models' in-context policy guardrailing capabilities and failure modes, rather than simply measuring overall performance. While our policies are constructed for specific conversations, they follow realistic policy structures and include substantial content beyond those conversations. We believe this provides a meaningful benchmark for studying in-context policy guardrailing, and we acknowledge that evaluating fixed policies across unseen conversations is an important direction for future work.
Weakness 3 & Question 1: Unclear Per-rule Protocol
W3 & Q1. Unclear per-rule protocol. The paper never explains how the “relevant policy context” is selected. Results with the full policy and conversation are needed. In per-rule evaluation, how is the “relevant policy context” selected? How do per-rule results change if each rule is judged with the full, unorganized policy and the full conversation?
A3: We thank the reviewer for pointing out that the per-rule evaluation protocol needs further clarification.
-
Why per-rule evaluation? The per-rule protocol is designed for guardrail models that cannot directly output a list of violated rules, as required by our per-policy evaluation. These models typically perform binary violation classification and often have limited context windows. Therefore, we evaluate each rule separately, asking the model whether the given conversation violates the target rule, and aggregate the predictions to compute the evaluation metrics.
-
How is relevant policy context selected? The relevant context depends on the design of each level:
Level Relevant policy context Evaluation L0 Target rule only Determine whether the full conversation violates the rule L1 Target rule + associated exception and conditional rules Determine whether the target rule is violated after considering its dependencies L2 Target rule + associated exception and conditional rules + new policy framework Determine whether the target rule is violated under the new framework In all three levels, the model is given the full conversation. For L1 and L2, the additional context ensures that the model has the information needed to correctly interpret the target rule. The context is selected based on policy structure and rule dependencies, not on ground-truth violation labels.
-
Per-rule evaluation with full, unorganized policy and full conversation. We thank the reviewer for this insightful question, which we had not previously considered. We agree that this makes for an interesting and valuable ablation study, and we have conducted additional experiments to investigate it. As shown in Table 1 below, we additionally evaluated Jev-1.13 using the full, unorganized policy and full conversation.
Table 1. Jev-1.13 performance comparison between relevant-context and full-policy evaluation (%).
Setting RMR L0 RMR L1 RMR L2 RMR Avg. [email protected] L0 [email protected] L1 [email protected] L2 [email protected] Avg. RDR L0 RDR L1 RDR L2 RDR Avg. Original (Relevant Context) 67.30 64.00 32.90 54.80 48.70 40.70 12.00 33.80 12.30 13.50 28.80 19.10 Full Policy 66.17 24.47 17.40 36.01 46.70 3.90 3.20 17.93 12.51 31.29 38.96 27.59 Difference (pp) -1.13 -39.53 -15.50 -18.79 -2.00 -36.80 -8.80 -15.87 +0.21 +17.79 +10.16 +8.49 -
Why does Jev's performance drop under full-policy evaluation? Our additional analysis suggests that the main difficulty lies in identifying the relevant exception and conditional rules from the full policy, rather than applying these rules once they are provided.
- We compared Jev's predictions when the relevant dependencies are explicitly provided versus when the full policy is given. Table 2 below show that Jev can often apply the relevant dependencies when they are explicitly provided, particularly for conditions (87.0% on L1 and 85.9% on L2). However, when given the full policy, performance drops substantially on cases where dependencies change the correct answer. For example, L2 conditional-rule accuracy drops from 85.9% to 28.3%. In contrast, cases where modifiers do not change the outcome show no comparable degradation.
- A likely explanation is the difficulty of retrieving dependencies from the full policy. In SafePyramid, exception and conditional rules reference the base rules they modify, but base rules do not reference their modifiers. Therefore, when evaluating a target rule, the model must search the full policy to identify which other rules modify it. The relevant-context setting removes this retrieval step by providing these dependencies directly.
Table 2. Jev's predictions on rules affected by exceptions and conditions (violation prediction rate, %).
| Rule category | Ground truth | Relevant context | Full policy |
|---|---|---|---|
| L1: Exception changes the outcome | Not violated | 33.6 | 59.6 |
| L1: Condition changes the outcome | Violated | 87.0 | 68.1 |
| L2: Exception changes the outcome | Not violated | 70.6 | 88.8 |
| L2: Condition changes the outcome | Violated | 85.9 | 28.3 |
| Control: Modifier does not change the outcome (L1/L2) | Not violated | 9.8 / 32.1 | 10.6 / 12.7 |
Weakness 1: Low Data Diversity
A1: We thank the reviewer for the insightful comments! We would like to clarify a few points:
Data diversity is not solely determined by the choice of generator. SafePyramid's conversation and policy diversity are primarily shaped by human expert-designed generation schemes, which define conversation scope, topics, scenarios (see Appendix C), policy structures, and reasoning requirements at each level. In particular, these requirements are explicitly defined by human experts rather than left to the generators to decide. The generators mainly serve to produce data following these predefined schemes. We deliberately use different generators across levels to reduce potential biases and artifacts associated with relying on a single generator.
Data diversity and quality are carefully reviewed by human experts. Human experts review the generated data against our predefined requirements, including whether the policies follow the intended reasoning requirements of each level. Samples that fail to meet these standards are excluded from the benchmark. This helps ensure that the generated policies meet our expectations for each difficulty level, particularly that L2 policies introduce meaningful new policy frameworks rather than arbitrary complexity.
Additional Experiment 1: Potential generator-family bias. To examine whether Claude's writing style favors models from the same family, we analyzed Claude-Opus-4.7's performance on L2 using a two-way additive model that accounts for overall model performance and average level difficulty. Claude-Opus-4.7 shows almost no relative advantage on L2. While this does not fully rule out generator-specific effects, we find no evidence of a Claude-family advantage in this analysis.
Table 1. Claude's performance on L2 policies generated by the Claude family.
Here Relative residual = observed RMR − expected RMR, where expected RMR is estimated from the model's overall performance and the average difficulty of each level. A positive residual indicates a relative advantage on that level.
In conclusion, this suggests that Claude's writing style alone is unlikely to explain the substantial performance drop from L1 to L2.