- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors ask whether policy-conditioned safety guards actually condition their verdicts on the enabled policy. Holding the prompt fixed and varying which taxonomy categories are enabled, they find that verdicts on five open guards barely change when the target policy is removed. Linear probes recover the input–policy relation from hidden states, but the verdict's gradient and decision boundary follow an estimated harmfulness direction. They propose policy-gated harmfulness routing, which adjusts the harmfulness coordinate with a relation-based gate and improves paired accuracy without updating weights.
Critical strengths
-
The counterfactual design is clean. PAIRACC penalizes both "never flags" and "always flags," and the separate reporting of TPR and FPR is appropriate.
-
The representation analysis uses several complementary methods (probes, gradient alignment, boundary geometry, subspace removal, attention blocking) that point the same way.
Critical weaknesses
-
Single-policy and empty-block conditions are confounded with out-of-distribution inputs. These guards were trained on full taxonomies. Shrinking the policy block to one category, or emptying it (Figure 3), is a distribution shift. The fact that 62–88% of prompts still get flagged with an empty block may reflect default guard behavior rather than policy-ignoring. A control with a valid but unrelated policy block of the same format would separate these explanations.
-
gpt-oss-safeguard is evaluated outside its intended mode. The authors bypass its reasoning and supply the LlamaGuard 3 taxonomy (Appendix B.2). The model is designed to reason over the policy, so this setup does not test its intended capability. The title and abstract should be narrowed, or the model should be evaluated in its reasoning mode.
Questions for authors
- What are the unsafe rates on benign or unrelated prompts under a valid but unrelated policy block?
- How does gpt-oss-safeguard perform in its reasoning mode?
3 comments
Dear Reviewer YpUP,
As the discussion period is nearing its end, we would like to kindly follow up on our responses. We hope our clarifications have adequately addressed your concerns, and we would greatly appreciate it if they could be reflected in your final assessment, if you find them satisfactory. Thank you again for your time and consideration.
I have read authors' response and kept my score.
[w1] Single-policy and empty blocks as distribution shift
[re1] We ran the suggested control, a valid but unrelated policy block of the same format, added to the revision as Appendix B.4 and summarized in Section 4.1. Prompts and prompt templates are those of Table 3, and only the policy block changes. The block holds one policy line whose topic lies outside every guard's taxonomy, in the same slot as the target line and matched in token length (unsafe rates on harmful prompts, %, native guards).
[w2, q2] gpt-oss-safeguard in its reasoning mode
[re2] We evaluated gpt-oss-safeguard in its reasoning mode with both the one-line category and the policy-document format of its model card. Reasoning raises paired accuracy from 21.5% to 61–72%, so the model follows the enabled policy in its intended mode. The revision adds these results as Appendix D.6 and narrows Section 3.1 to verdicts produced without reasoning, the mode of the other four guards. Routing without reasoning matches low-effort reasoning (64.9% vs. 62.7%) while generating only the verdict, against 91 tokens on average for reasoning.
[q1] Unsafe rates on benign prompts under an unrelated policy
[re3] Under the unrelated block of our response to W1, the 700 benign on-topic prompts are flagged at the following rates (%, native guards, Appendix B.4).
For every guard, the unrelated block flags benign prompts no more often than their own policy, while 63–88% of harmful prompts are still flagged under the same block. Guards do not fall back to flagging everything when the block is unfamiliar. They separate harmful from benign prompts as usual and ignore only which policy is enabled.