acceptodds
Withdrawn from ICLR 2027

PAIRBench: Complete Response-Function Coverage for Auditing Policy-Conditioned Safety Guards

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Policy-conditioned safety guards decide whether a request violates a policy supplied at runtime. Policy revisions require correct updates on affected requests and stable decisions elsewhere. High cell-level scores can coexist with incomplete recovery of joint policy–scenario patterns, while selected relational tests leave other patterns unmeasured. We introduce PAIRBENCH, a benchmark covering all 16 deterministic binary response functions formed by two policies and two scenarios. Its human-reviewed panel balances 160 quartets; averages exact recovery equally across functions, and matched policy interventions test updates and stability with scenarios fixed. DynaGuard and SingGuard achieve cell accuracies of 0.969 and 0.986 but scores of 0.888 and 0.944. Complete stratification yields lower root-mean-square error for population-level exact recovery than the tested reduced panels at a matched 80-quartet budget. Four guard families meet prospectively frozen selective-update criteria, and a separate policy-replacement control confirms behavioral dependence on the supplied policy in three tested systems. On a harder document-rich panel, LPG and SingGuard show lower selective updating and transition recovery. The benchmark connects complete local function coverage with controlled tests of policy response, complementing per-decision scores with an explicit population audit.

Primary area:
datasets and benchmarks
Keywords:
policy-conditioned safety guards, LLM safety evaluation, response-function coverage, exact function recovery, counterfactual evaluation, selective policy updating

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SrLuOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PAIRBench, a benchmark for evaluating policy-conditioned safety guards beyond per-instance accuracy. It constructs quartets of two policies and two scenarios, exhaustively covering all 16 possible binary response functions, and proposes Exact Function Recovery (EFR16) to measure joint decision correctness, macro-averaged across these functions. Controlled policy interventions further assess whether guards correctly update affected decisions while preserving unaffected ones. The evaluation includes policy replacement, protocol comparisons, independent confirmation, and document-rich scenarios. Experiments on DynaGuard, SingGuard, LPG, and GPT-OSS reveal substantial gaps between cell accuracy and exact function recovery. For instance, DynaGuard achieves approximately 96% accuracy but only 85.5–86.7% EFR16 on 1,024 independent quartets. Larger gaps emerge in document-rich settings, highlighting the importance of evaluating joint decision correctness and policy-change responsiveness alongside conventional per-instance accuracy.

Critical strengths
  1. The paper addresses an important limitation of existing policy-conditioned guard evaluations. Conventional cell-level metrics may overlook failures in jointly satisfying related policy–scenario decisions. The paper highlights the importance of complete response-function recovery and selective responses to policy changes, providing a useful perspective beyond evaluating individual classification decisions.
  2. The paper proposes a systematic and well-defined evaluation framework. By exhaustively covering all 16 binary response functions, PAIRBench establishes an explicit and balanced evaluation population. The proposed EFR16 metric measures joint decision correctness, while the intervention metrics distinguish intended decision changes from unintended ones. Together, these metrics provide a more comprehensive assessment of guard behavior than conventional accuracy or selected relational tests alone.
  3. The experimental evaluation is thorough and includes multiple validation procedures. The authors employ independent human annotation, deterministic label verification, controlled policy interventions, policy replacement, and comparisons across native inference protocols. The independent evaluation on 1,024 quartets confirms that the accuracy–recovery gap persists within the controlled data distribution. The document-rich experiments and cellwise solvability controls further strengthen the empirical findings.
Critical weaknesses
  1. Limited methodological novelty. The proposed framework primarily combines established evaluation techniques, including grouped exact-match evaluation, Boolean function enumeration, counterfactual interventions, and stratified sampling. EFR16 is essentially a macro-averaged exact-match metric over four binary decisions. While applying these techniques to policy-conditioned guards is useful, the paper does not clearly establish its methodological novelty relative to existing behavioral, relational, and metamorphic testing approaches. A more systematic comparison with these methods would strengthen the contribution.
  2. Limited scope of complete coverage. The claimed completeness is restricted to the 16 binary response functions defined by two policies and two scenarios. While this exhaustively covers possible output patterns, it does not guarantee coverage of policy semantics, decision boundaries, rule interactions, or safety risks. Different quartets representing the same function may require substantially different reasoning capabilities. Furthermore, complex policy structures involving exceptions, conflicting rules, and hierarchical priorities are not systematically evaluated. It remains unclear whether complete response-function coverage adequately reflects policy-following capabilities in realistic settings.
  3. Limited realism of the benchmark. The evaluation relies primarily on synthetic administrative policies with explicit, deterministic rules and carefully constructed ground truths. Real-world safety policies often involve ambiguity, overlapping constraints, contextual exceptions, and domain-specific interpretations. Although the document-rich extension introduces longer contexts and irrelevant information, it remains synthetic and varies multiple difficulty factors simultaneously. Experiments involving real organizational policies or established policy-compliance benchmarks would strengthen the evidence of practical applicability.
  4. Insufficient justification of the practical value of EFR16. Since exact function recovery requires all four decisions to be correct, a lower EFR16 than cell accuracy is mathematically expected. The observed accuracy–recovery gaps therefore do not, by themselves, establish the superiority or practical utility of the proposed metric. Moreover, uniformly weighting all 16 functions does not reflect their prevalence in real-world deployments. The paper would benefit from demonstrating whether EFR16 provides additional diagnostic or predictive value beyond conventional accuracy and consistency metrics, particularly for deployment-relevant failures.
  5. Limited generalizability and diagnostic utility. The independent confirmation follows the same controlled data-generation distribution, leaving generalization to unseen policy structures and domains insufficiently explored. The document-rich experiments also compare models with substantially different scales and training, making it difficult to isolate the causes of performance differences. Furthermore, prioritizing low-recovery functions identified during discovery does not improve error detection on the independent confirmation set. This finding raises questions about the practical utility of function-level diagnostics for guiding subsequent testing.
Questions for authors
  1. What is the key methodological novelty of PAIRBench compared with existing behavioral, relational, and metamorphic testing approaches?
  2. How does coverage of all 16 binary response functions translate into meaningful coverage of real-world policy semantics, rule interactions, and safety risks?
  3. How well do the findings generalize to real-world safety policies involving ambiguity, exceptions, and conflicting rules?
  4. Beyond the mathematically expected gap between cell accuracy and exact recovery, what additional practical or diagnostic value does EFR16 provide?
Reviewer_rihLOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a new evaluation framework and benchmark, PAIRBench, that evaluates policy-conditioned guard models. They target a situation where a a judge model evaluates two policies across two requests/scenarios. This gives the guard model 2^4 = 16 possible responses of compliance of violation decision tuples against the two policies in the two scenarios. They discuss single cell accuracy , define function matching (does the model get all decisions in the 2x2 grid of situations x policies correct) and then a macro averaged metric (EFR16) that measures the recall of a guard model separately over all 16 response tuples from all possible decisions over the paired policy and situation decisions, and weights each of these tuples equally for a more balanced and fair evaluation. They then propose PairBench, a synthetic dataset designed to evaluated existing guard models using their metrics and use their metrics to evaluate judge decision quality when changing a policy is revised while scenarios are held fixed.

Critical strengths

The strengths of the paper include:

  • Defining the problem of fairly evaluating guard models making binary decisions of compliance or violation conditioned on two policies in which to base decision on across two scenarios.
  • Defining and introducing new metrics for evaluating guard models in such scenarios, and discussing the limitations of simple metrics such as the cell accuracy or function recovery.
  • Introducing PairBench, a new dataset that allows evaluation of judge models conditioned on policies.
  • They use their metrics and datasets to investigate the performance of a judge if a policy is revised/changed for the same fixed set of scenarios.
Critical weaknesses

The weaknesses of the paper include the following:

  • The paper is extremely difficult to parse and understand, which is not justified given the simplicity of the topic and ideas being discussed. The prose is extremely verbose, convoluted making even the most simple of ideas difficult to parse.
  • Strange language usage and writing, to me, strongly indicates AI use beyond just "polishing". Frequent use of words like "arms" are a well known claude-ism as well as the overly abstract and convoluted prose presented here too. Other examples of words and phrasing pointing to this include: the frequent use of "quartet" , several terms in the abstract not defined making this on a first read extremely difficulty to understand what the paper is even about, and "enveloping missingness bounds where needed" which while loosely connected to statistics terminology is extremely strangely phrased. There are many other instances of such problems throughout the paper which I don't think meet acceptable writing standards for a top tier conference like ICLR.
  • The setup is rather limited and seems quite disconnected from generally what one uses policy-conditioned models for e.g. in the safety evaluating of say toxic language etc.
Questions for authors
  • Why consider only two policies and two scenarios, why not more scenarios exactly? I understand that the number of possible binary strings that can be formed will grow exponentially but at the moment this seems rather arbitrary and quite limiting.
  • Can you please describe the exact policies and scenarios and motivation for these for your proposed benchmark and why you focussed on these exactly? They look rather specific and artificial.
  • In simple language, please could you summarise what the key takeaways are from the paper in your own words?
Reviewer_37cGOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes PAIRBench, a benchmark for evaluating policy-conditioned safety guards. It systematically covers all 16 binary response functions arising from two policies and two scenarios and introduces Exact Function Recovery (EFR16) to measure whether models correctly recover all four decisions. Experiments on several guard models demonstrate that high per-decision accuracy does not necessarily translate into complete function recovery, particularly under complex policy documents and policy changes.

Critical strengths
  1. Evaluating whether safety guards correctly follow dynamically changing policies is an important problem with practical implications for trustworthy LLM deployment.

  2. The paper provides systematic evaluations across multiple guard models, including controlled policy interventions, independent confirmation experiments, and more challenging document-rich scenarios.

Critical weaknesses
  1. The proposed EFR16 metric seems essentially to be a group-level exact-match accuracy over four binary predictions. Similar ideas have been explored in consistency evaluation and relational testing. The paper should better clarify its methodological novelty beyond enumerating all 16 response functions.

  2. The observed gap between cell accuracy and EFR16 is largely expected because exact function recovery requires all four predictions to be correct. The paper does not sufficiently demonstrate whether this gap reveals additional relational inconsistencies beyond those expected from individual prediction errors.

  3. The benchmark primarily relies on synthetically generated policies and scenarios with relatively simple deterministic rules. It remains unclear whether the findings generalize to real-world safety policies involving ambiguous requirements, conflicting constraints, and complex contextual dependencies.

Questions for authors

See Weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.