- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces PolicyGround for selecting one of five policy-conditioned actions: ALLOW, SAFE_PARTIAL, CLARIFY, REFUSE, or ESCALATE, together with supporting rule IDs. It contains 2,400 Controlled instances in counterfactual families, 1,026 public-seeded Diverse instances, and a separate balanced Crossed binding diagnostic. Experiments show substantial action errors among outputs that pass a coarse refusal test or cite exactly the required rules. Lexical audits expose strong single-input shortcuts in the original splits, while strong models nearly solve the explicit Crossed mappings. Family-disjoint LoRA training improves action accuracy but increases the action-level unsafe-compliance proxy on Diverse. The evaluation explicitly stops short of judging generated-response safety.
Critical strengths
-
The five-way action space captures operational distinctions that binary refusal metrics miss. The joint action/citation analysis is especially useful: checking the right rule IDs does not ensure selection of the appropriate response route. The conditional error rates and oracle-evidence controls provide a concrete diagnostic rather than relying only on aggregate accuracy. This is a useful extension of policy-compliance evaluation, even though the broader ideas of policy sensitivity and crossed rule–scenario testing have precedents.
-
The empirical work is auditable and appropriately qualified. The supplement includes frozen data, predictions, exact supervised split memberships, prompts, metric code, and human-audit lineage. The authors also report shortcut strength, cluster-aware uncertainty, unmatched prompt conditions, and the LoRA unsafe-compliance trade-off, which makes the conclusions more trustworthy and useful.
Critical weaknesses
-
The benchmark does not yet bridge realistic action-boundary decisions and a clean test of joint policy–request interpretation. Policy-only lexical accuracy reaches 0.971 on Controlled and 0.913 on Diverse, so high scores on those splits need not require combining the inputs. Crossed removes the marginal signal, but its explicit access-category mapping is simpler than the natural-language exceptions and applicability conditions motivating the paper, and the strongest models saturate it. The paper acknowledges this correctly, but it limits how specifically the original-split errors or LoRA gains diagnose policy grounding. A balanced natural-language subset would strengthen the benchmark substantially. For the Qwen3/Gemma diagnostics, retaining the same action definitions and boundary instructions across full-JSON and label-only conditions would also help separate action interpretation from prompting differences.
-
Validation of the fine action boundaries is less independent than the high agreement number might suggest. Both audit reviewers saw the candidate action, evidence, scopes, and rationale; agreement therefore measures verification of proposed labels, not blind reconstruction of the intended action. This is disclosed, but leaves uncertainty about the genuinely ambiguous ALLOW/SAFE_PARTIAL/CLARIFY boundaries that drive the contribution. Only 76 retained Diverse examples received the primary human audit, including all 21 ESCALATE cases, so agreement on that sample gives limited coverage of the remaining Diverse boundaries. A blinded, boundary-focused annotation subset would make the error analysis more convincing.
Questions for authors
-
Can you evaluate a subset with natural-language policies and counterfactual request–policy pairings balanced against single-input cues, while keeping action definitions fixed? Even a small test would clarify whether the original-split failures persist beyond the explicit mapping task.
-
Do you have blind reannotation results for disputed or model-error cases near the ALLOW/SAFE_PARTIAL/CLARIFY boundaries? How often do independent annotators select a different but defensible route when candidate gold fields are withheld?
-
Appendix B.3 removes duplicate-prompt groups with conflicting candidate labels. Did this check also consider policy identity? Identical requests can legitimately require different actions under different governing policies. Please distinguish inconsistent labels for the same request–policy pair from valid policy-conditioned counterfactuals, and quantify any of the latter removed.
2 comments
Thank you for the additional analyses. The correction sensitivity analysis and clarified duplicate filtering are helpful. Lexical shortcuts and low complete-family consistency remain limitations, so the narrower interpretation of the results is appropriate. Overall, the rebuttal strengthens the diagnostic contribution, and I maintain my weak-accept rating.
Naturalistic Binding, Blind Validation, and Duplicate Filtering
Thank you for identifying the gap between benchmark performance and evidence of genuine request-policy binding. We have added direct diagnostics and narrowed the associated claims.
[W1] Binding gap and matched prompt control
We agree that Controlled/Diverse retain marginal cues, while Crossed uses simplified mapping syntax. We have added a Matched label-only condition that retains the Direct prompt's five action definitions and boundary rules and changes only the requested output from JSON to one label. Relative to the submitted Simple label-only prompt, Qwen3 improves by 13.54/8.28 points on Controlled/Diverse and Gemma by 12.50/7.41 points (all McNemar p <= 7.3e-13). This is a prompt-specification control, not a same-content schema isolation, so we have withdrawn any causal claim that output format alone explains the original gap (App. C.2, Table 19).
[Q1] Naturalistic policy pairing
We have added Naturalistic-Paired, a controlled diagnostic with 25 fixed requests and four natural-language policy conditions per request (100 instances). Two gold-blind annotators agree on 94.00% of labels (kappa = 0.925), and a third gold-blind reviewer adjudicates all six disagreements before model evaluation.
The exact-request oracle reaches 0.28 and the fitted request-only baseline 0.16. Gemma reaches 0.52 (95% CI [0.45, 0.59]), a paired gain of 36.00 points over the fitted request-only baseline (95% CI [28.00, 44.00]; exact McNemar p = 4.04e-7); Qwen3 reaches 0.49. However, Gemma solves 0/25 complete four-condition families and Qwen3 only 2/25. We therefore interpret this as nontrivial but brittle binding, not saturation. The complete table and protocol are in App. C.3, Table 20; a separate 21-row pilot artifact is not pooled with the primary panel.
[W2] Blind validation and Diverse coverage
The submitted 99.2% figure was candidate-label verification, not blind reconstruction. The revised manuscript reports a separate 200-instance study in which annotators see only the request, policy, and action definitions. Candidate fields and model outputs are hidden.
On the 100-instance representative panel, adjudicated exact match is 93.00% (95% CI [88.00%, 97.00%]), macro-F1 is 0.940, and pre-adjudication kappa is 0.832. The representative panel contains 50 Diverse cases and the boundary-enriched panel 60, with no repeated source seed, giving 110 blind Diverse examples across both panels.
[Q2] Boundary disagreements and defensible alternatives
The primary action differs from released gold in 7/100 representative and 23/100 enriched cases, with disagreement concentrated at ALLOW versus SAFE_PARTIAL. At least one annotator proposes a defensible alternative in 7/100 representative and 14/100 enriched cases; both propose an alternative in 2/100 in each panel, and both independently name the same alternative in 1/100 in each panel. A frozen 30-row correction overlay leaves all six Direct rankings unchanged; the largest accuracy change is 1.37 points, and full-output LoRA gains remain 14.60-33.50 points. Appendix I reports the full protocol and sensitivity analysis (App. I.1, Tables 27-28).
[Q3] Repeated requests
All 26 rows remain outside the main scored artifact, so the Controlled/Diverse scores are unchanged. Twenty-one are restored only as a separate annotated pilot forming eight exact-request counterfactual families; the other five remain excluded from that pilot as normalization-only collisions whose exact requests differ. None is a conflicting identical request-policy pair. The corrected duplicate validator keys the complete request-policy pair (App. B.3).