- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces PAIRBench, a benchmark for evaluating policy-conditioned safety guards beyond per-instance accuracy. It constructs quartets of two policies and two scenarios, exhaustively covering all 16 possible binary response functions, and proposes Exact Function Recovery (EFR16) to measure joint decision correctness, macro-averaged across these functions. Controlled policy interventions further assess whether guards correctly update affected decisions while preserving unaffected ones. The evaluation includes policy replacement, protocol comparisons, independent confirmation, and document-rich scenarios. Experiments on DynaGuard, SingGuard, LPG, and GPT-OSS reveal substantial gaps between cell accuracy and exact function recovery. For instance, DynaGuard achieves approximately 96% accuracy but only 85.5–86.7% EFR16 on 1,024 independent quartets. Larger gaps emerge in document-rich settings, highlighting the importance of evaluating joint decision correctness and policy-change responsiveness alongside conventional per-instance accuracy.
Critical strengths
- The paper addresses an important limitation of existing policy-conditioned guard evaluations. Conventional cell-level metrics may overlook failures in jointly satisfying related policy–scenario decisions. The paper highlights the importance of complete response-function recovery and selective responses to policy changes, providing a useful perspective beyond evaluating individual classification decisions.
- The paper proposes a systematic and well-defined evaluation framework. By exhaustively covering all 16 binary response functions, PAIRBench establishes an explicit and balanced evaluation population. The proposed EFR16 metric measures joint decision correctness, while the intervention metrics distinguish intended decision changes from unintended ones. Together, these metrics provide a more comprehensive assessment of guard behavior than conventional accuracy or selected relational tests alone.
- The experimental evaluation is thorough and includes multiple validation procedures. The authors employ independent human annotation, deterministic label verification, controlled policy interventions, policy replacement, and comparisons across native inference protocols. The independent evaluation on 1,024 quartets confirms that the accuracy–recovery gap persists within the controlled data distribution. The document-rich experiments and cellwise solvability controls further strengthen the empirical findings.
Critical weaknesses
- Limited methodological novelty. The proposed framework primarily combines established evaluation techniques, including grouped exact-match evaluation, Boolean function enumeration, counterfactual interventions, and stratified sampling. EFR16 is essentially a macro-averaged exact-match metric over four binary decisions. While applying these techniques to policy-conditioned guards is useful, the paper does not clearly establish its methodological novelty relative to existing behavioral, relational, and metamorphic testing approaches. A more systematic comparison with these methods would strengthen the contribution.
- Limited scope of complete coverage. The claimed completeness is restricted to the 16 binary response functions defined by two policies and two scenarios. While this exhaustively covers possible output patterns, it does not guarantee coverage of policy semantics, decision boundaries, rule interactions, or safety risks. Different quartets representing the same function may require substantially different reasoning capabilities. Furthermore, complex policy structures involving exceptions, conflicting rules, and hierarchical priorities are not systematically evaluated. It remains unclear whether complete response-function coverage adequately reflects policy-following capabilities in realistic settings.
- Limited realism of the benchmark. The evaluation relies primarily on synthetic administrative policies with explicit, deterministic rules and carefully constructed ground truths. Real-world safety policies often involve ambiguity, overlapping constraints, contextual exceptions, and domain-specific interpretations. Although the document-rich extension introduces longer contexts and irrelevant information, it remains synthetic and varies multiple difficulty factors simultaneously. Experiments involving real organizational policies or established policy-compliance benchmarks would strengthen the evidence of practical applicability.
- Insufficient justification of the practical value of EFR16. Since exact function recovery requires all four decisions to be correct, a lower EFR16 than cell accuracy is mathematically expected. The observed accuracy–recovery gaps therefore do not, by themselves, establish the superiority or practical utility of the proposed metric. Moreover, uniformly weighting all 16 functions does not reflect their prevalence in real-world deployments. The paper would benefit from demonstrating whether EFR16 provides additional diagnostic or predictive value beyond conventional accuracy and consistency metrics, particularly for deployment-relevant failures.
- Limited generalizability and diagnostic utility. The independent confirmation follows the same controlled data-generation distribution, leaving generalization to unseen policy structures and domains insufficiently explored. The document-rich experiments also compare models with substantially different scales and training, making it difficult to isolate the causes of performance differences. Furthermore, prioritizing low-recovery functions identified during discovery does not improve error detection on the independent confirmation set. This finding raises questions about the practical utility of function-level diagnostics for guiding subsequent testing.
Questions for authors
- What is the key methodological novelty of PAIRBench compared with existing behavioral, relational, and metamorphic testing approaches?
- How does coverage of all 16 binary response functions translate into meaningful coverage of real-world policy semantics, rule interactions, and safety risks?
- How well do the findings generalize to real-world safety policies involving ambiguity, exceptions, and conflicting rules?
- Beyond the mathematically expected gap between cell accuracy and exact recovery, what additional practical or diagnostic value does EFR16 provide?