acceptodds
Under review as a conference paper at ICLR 2027

Safety Guards Judge Harmfulness, Not Policy Violation

Abstract

Policy-conditioned safety guards are designed to judge a prompt against the policies enabled for them, not merely by whether it is harmful. Across the evaluated open guards, verdicts on harmful prompts rarely flip when the enabled policy stops covering them, even when the guard correctly identifies the policy they violate. Linear probes show that hidden states encode this input–policy relation, yet verdicts mainly follow an estimated harmfulness direction, and in several guards intervening on the relation leaves them largely unchanged. To reconnect this relation to the verdict, we introduce *policy-gated harmfulness routing*, which uses it to gate an adjustment along the harmfulness direction without updating the guard's weights. Routing improves *paired accuracy* on the evaluated guards, with some loss in target-policy detection and no increase in benign on-topic false-positive rates.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
safety guard models, LLM guardrails, content moderation, interpretability, representation analysis, activation steering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YpUPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors ask whether policy-conditioned safety guards actually condition their verdicts on the enabled policy. Holding the prompt fixed and varying which taxonomy categories are enabled, they find that verdicts on five open guards barely change when the target policy is removed. Linear probes recover the input–policy relation from hidden states, but the verdict's gradient and decision boundary follow an estimated harmfulness direction. They propose policy-gated harmfulness routing, which adjusts the harmfulness coordinate with a relation-based gate and improves paired accuracy without updating weights.

Critical strengths
  • The counterfactual design is clean. PAIRACC penalizes both "never flags" and "always flags," and the separate reporting of TPR and FPR is appropriate.

  • The representation analysis uses several complementary methods (probes, gradient alignment, boundary geometry, subspace removal, attention blocking) that point the same way.

Critical weaknesses
  • Single-policy and empty-block conditions are confounded with out-of-distribution inputs. These guards were trained on full taxonomies. Shrinking the policy block to one category, or emptying it (Figure 3), is a distribution shift. The fact that 62–88% of prompts still get flagged with an empty block may reflect default guard behavior rather than policy-ignoring. A control with a valid but unrelated policy block of the same format would separate these explanations.

  • gpt-oss-safeguard is evaluated outside its intended mode. The authors bypass its reasoning and supply the LlamaGuard 3 taxonomy (Appendix B.2). The model is designed to reason over the policy, so this setup does not test its intended capability. The title and abstract should be narrowed, or the model should be evaluated in its reasoning mode.

Questions for authors
  • What are the unsafe rates on benign or unrelated prompts under a valid but unrelated policy block?
  • How does gpt-oss-safeguard perform in its reasoning mode?
3 comments

[w1] Single-policy and empty blocks as distribution shift

[re1] We ran the suggested control, a valid but unrelated policy block of the same format, added to the revision as Appendix B.4 and summarized in Section 4.1. Prompts and prompt templates are those of Table 3, and only the policy block changes. The block holds one policy line whose topic lies outside every guard's taxonomy, in the same slot as the target line and matched in token length (unsafe rates on harmful prompts, %, native guards).

GuardTargetUnrelatedFull −- targetTarget swapped2 / 4 / 8 random others
LlamaGuard 3-8B79.663.172.171.964.4 / 67.9 / 71.3
LlamaGuard 3-1B79.775.676.976.875.9 / 76.5 / 76.6
NemoGuard94.187.883.784.083.8 / 84.3 / 83.9
gpt-oss-safeguard95.975.794.495.382.8 / 87.8 / 92.5
Qwen3Guard-8B84.682.582.582.682.6 / 82.2 / 82.5
  • Harmful prompts stay flagged under a valid unrelated policy, at 63–88%, matching the 62–88% of the empty block in Figure 3. Blocks closest to training, the full taxonomy with the target deleted or swapped, keep the rate at 72–95%, so the effect does not come from a shortened block. Category subsetting is also a documented use, as the Llama Guard 3 model card shows how to exclude categories.
  • This is not a default response to unfamiliar blocks. Under the same unrelated policy, benign prompts are flagged at only 0.7–17.6%, no more often than under their own policy (Q1).
  • The policy information is present but unused. Routing lowers the unsafe rate under the unrelated policy from 63.1% to 0.2% on LlamaGuard 3-8B and from 75.7% to 2.1% on gpt-oss-safeguard.

[w2, q2] gpt-oss-safeguard in its reasoning mode

[re2] We evaluated gpt-oss-safeguard in its reasoning mode with both the one-line category and the policy-document format of its model card. Reasoning raises paired accuracy from 21.5% to 61–72%, so the model follows the enabled policy in its intended mode. The revision adds these results as Appendix D.6 and narrows Section 3.1 to verdicts produced without reasoning, the mode of the other four guards. Routing without reasoning matches low-effort reasoning (64.9% vs. 62.7%) while generating only the verdict, against 91 tokens on average for reasoning.

[q1] Unsafe rates on benign prompts under an unrelated policy

[re3] Under the unrelated block of our response to W1, the 700 benign on-topic prompts are flagged at the following rates (%, native guards, Appendix B.4).

GuardUnrelated policyOwn on-topic policyFull taxonomy
LlamaGuard 3-8B0.74.63.4
LlamaGuard 3-1B9.111.312.1
NemoGuard17.628.116.0
gpt-oss-safeguard7.725.028.0
Qwen3Guard-8B6.36.76.4

For every guard, the unrelated block flags benign prompts no more often than their own policy, while 63–88% of harmful prompts are still flagged under the same block. Guards do not fall back to flagging everything when the block is unfamiliar. They separate harmful from benign prompts as usual and ignore only which policy is enabled.

Dear Reviewer YpUP,

As the discussion period is nearing its end, we would like to kindly follow up on our responses. We hope our clarifications have adequately addressed your concerns, and we would greatly appreciate it if they could be reflected in your final assessment, if you find them satisfactory. Thank you again for your time and consideration.

I have read authors' response and kept my score.

Reviewer_f1aFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper's central question is an examining whether policy-conditioned safety guardrail models base their violation verdicts solely on which policy is enabled, or rather, the prompt's harmfulness. This still bases the premise of the policy domains only to safety categories rather than a larger corpus of policies. To address this, a counterfactual evaluation is proposed where each harmful prompt is fixed while only the enabled policy block is modified. The expected verdict is naturally "unsafe" under its target policy and "safe" otherwise. An important metric introduced is "paired accuracy" which inly credits a prompt when flagged under its target policy but not a non-target one.

Five open-source guardrail models are investigated in this study (LlamaGuard, NemoGuard, Aegis, Qwen3Guard, and GPT-OSS-SafetGuard) on single-category harmful prompts as well as benign on-topic examples across five safety categories. The top-level observation from this study shows that verdicts respond weakly to policy changes. Also, importantly, prompt category identification does not predict policy following.

The different probing questions in each section are accompanied by analyses that present new aspects to better present this phenomenon. Linear probes of internal representations recover the input-policy relation, but it is shown that the verdict's margin gradient aligns far more with the harmfulness direction than with the orthogonal relation direction, and that the figged verdict boundary is almost flat along the relation axis. Also, blocking policy-block attention or removing the relation subspace do not substantially affect the verdicts on most guards. The authors then introduce policy-gated harmfulness routing, which is an intervention that uses a relation-score gate to route the harmfulness coordinate toward a benign one when the policy enabled is not the prompt's target. Thus, routing achieves enhanced paired accuracy.

Critical strengths
  1. The paper presents a foundational question addressed with a well-motivated counterfactual design. By keeping the enabled policies as the only variable (the prompt is fixed) and by a correct flip of a prompt given its target policy, the authors isolate the two effects: policy following vs. harmfulness detection/category identification. This is a relatively new take not done in prior policy adaptation work that study guardrails' generalization to new policies.

  2. The authors provide additional layered representation analysis that separates policy being encoded from policy being used to derive the final verdict. Also, the training-free routing intervention turns their diagnosis into a causal demonstration, showing that reconnecting the already encoded relation to the model's verdict enhances the paired accuracy on all guardrail models. This renders the paper's premise beyond a correlation observation.

Critical weaknesses
  1. One main weakness is that the core metric's viability rests on single-category labeling assumption, which is rather under-verified (so far as I reviewed the different sections). In cases where harmful inputs violate more than one policy, then positive verdicts under "non-target" policies can be deemed correct and therefore, the headline effect is overstated at best. The only check I could find was in Appendix A.4 that covers roughly 41% of prompts and agrees 83.7%, and uses a weak keyword pattern. Since this is not quantified on the full PolyGuard set, it leaves the magnitude of the main findings uncertain.

  2. The proposed routing method is more a diagnostic measure rather an a usable solution, which affects the practical significance of the third contribution. It requires white-box activation access, per-category labelled direction estimation, and layer selection. Also, residual non-target FPR is relatively high on NemoGuard while Qwen3Guard barely moves. Moreover, the target-detection costs are real on every guardrail model, however, authors are honest about this. But it means the intervention primarily corroborates the analysis rather than offering a deployable fix.

  3. The general organization of the paper is logical, from the central question to evaluation to representation analysis to intervention. However, several points must be raised to improve the structure and presentation of the paper affecting its readability. First, the text itself does not read well throughout the paper. Some explanations are short and readers are left to their devices to understand authors' points. Second, the analyses juggle many interrelated quantities (harmfulness direction, the raw relation direction, the orthogonal relation direction, the relation score, and two different relation-direction estimators. The notations are not well-established or consistent, jump from one to the other, not well-explained for readers to understand what each term represents, and the concepts represented by the expressions are not grounded and explained (in some cases, at all). Personally, I had to make multiple assumptions and guess what authors were trying to achieve to make better sense of the results. These need to be significantly improved from their current form.

  4. Results are spread across tables on different pages sometimes not close to the relevant discussions, each use a different population and therefore, report different baselines for the same quantity.

Questions for authors
  1. The multi-category violation needs to be more rigorously quantified than the 41% coverage lexical rule, and by reporting what percentage of non-target flags happen to coincide with a second category the guardrail itself names. This is very important, and directly bounds the extend to which the measured FPRirr is a genuine error or a spurious effect that arose from this labeling noise.

  2. The harmfulness direction is estimated from harmful-vs-benign labels (hence by definition, near the verdict direction). Could you report the gradient-alignment and boundary-slope analyses using a harmfulness direction derived independent of verdict labels (possible routes: from a held-out harmfulness probe or a different model), to show the "follows harmfulness not relation" result is not mainly definitional?

  3. For GPT-OSS-Safeguard, how would the evaluation and routing results change if its native reasoning-enabled classification pipeline is used instead of the prefilled direct final-channel verdict, given that is the regime it is trained for?

4 comments

[w1] Single-category labeling assumption

[re1] The revision replaces the keyword check of Appendix A.4 with a multi-label annotation of all 936 harmful prompts. GPT-4o assigns more than one LlamaGuard 3 category to 23.9% of prompts.

  • Non-target flags are mostly errors. Only 5.6–7.7% of native non-target flags fall on a policy the prompt also violates, and at most 16% if any of four annotators suffices, so FPRirr\text{FPR}_\text{irr} stays at 51–72% (Table 8).
  • The headline gap is unchanged. Counting such flags as correct raises native paired accuracy by at most 5 points and changes the routing gain by at most 3.4 points (Table 9).
  • Robust to the annotator. With three open-weight annotators (Jaccard 0.80 with GPT-4o), the gain changes by at most 9.7 points.

[w2] Practicality of routing

[re2] Routing is a practical intervention for operators who host an open-weight guard, not only a diagnostic.

  • Light requirements. Most guard models, including all five here, release their weights. Policy following otherwise takes retraining, as DynaGuard fine-tunes on about 80k policy and safety examples followed by GRPO, or reasoning, which costs 70–91 tokens per verdict even at low effort (Appendix D.6). Routing updates no weights, needs only 5 harmful and 5 benign prompts per concept, adds at most 0.25 ms per request and picks its layer by a training-fold rule (Appendix C.5).
  • Better than training. On the prompts of Table 3, DynaGuard reaches 23.9–34.5% paired accuracy at 49.7–74.3% TPR, whereas routing reaches 64.3% on LlamaGuard 3-8B and 67.7% on gpt-oss-safeguard at 71.4% and 86.9% TPR.
  • Low detection cost. At the same non-target false-positive rate on LlamaGuard 3-8B, a uniform margin shift keeps 38.7% TPR and routing 71.4% (Figure 6). Restoring native TPR still leaves 70.3% paired accuracy, against 19.0% natively.
  • Predictable gains. Forced contraction shows before routing whether the verdict reads the harmfulness coordinate (Appendix D.2), so an operator knows in advance where routing helps.
GuardPairAcc, 5 labels / conceptPairAcc, 100 labels / conceptAdded latency
LlamaGuard 3-8B62.864.1+0.25 ms
LlamaGuard 3-1B35.040.1+0.13 ms
NemoGuard30.629.4+0.0 ms
gpt-oss-safeguard66.066.9+0.09 ms
Qwen3Guard-8B3.63.5+0.18 ms

[w3] Readability and notation

[re3] The revision defines each quantity once and reuses it.

  • L313–316 define zHz_H and zRz_R once, and Sections 5.2 and 6.1 reuse them (L336–337, L427).
  • The gate uses zRz_R instead of a separate sRs_R, and L418–419 state how Eq. 5 differs from Eq. 4.
  • The Figure 4 caption (L285–290) explains v^H\hat v_H, v^R\hat v_R and r^R\hat r_R in words.

[w4] Table placement and populations

[re4] In the revision, Figure 4 appears in Section 5, and Table 3 sits with its discussion in Section 6.2 and merges single- and multi-policy results under one native baseline. The captions of Figure 2, Table 2 and Table 3 name their populations.

[q1] Second-category overlap

[re5] The revision adds to Appendix A.4 a comparison with each guard's top-ranked category other than the target. A flagged non-target policy coincides with it in only 7.9–19.8% of cases, within 0.1–3.3 points of the rate over all non-target rows, so flagged rows are not enriched for the guard's second category (flagged / all, %).

LG3-8BLG3-1BNemoGuardgpt-oss-sgQwen3Guard
2nd category14.1 / 10.819.8 / 18.97.9 / 6.917.3 / 15.417.4 / 17.3

[q2] Harmfulness direction may be definitional

[re6] We estimate two harmfulness directions that use neither our evaluation prompts nor the guard's verdicts, a probe on WildGuardMix-train prompts and the mean difference over 200 XSTest contrast pairs. Both align with v^H\hat v_H (cosine 0.67–0.96) and reproduce its gradient alignment and boundary slope on all five guards, while the base model's refusal direction (cosine 0.11–0.33) barely moves the verdict. The Section 5 finding is therefore not definitional.

GuardGradient alignment (v^H\hat v_H / probe / pairs)Boundary slope (v^H\hat v_H / probe / pairs)
LlamaGuard 3-8B0.38 / 0.38 / 0.383.1 / 2.9 / 3.0
LlamaGuard 3-1B0.26 / 0.28 / 0.271.8 / 1.9 / 1.8
NemoGuard0.22 / 0.22 / 0.231.8 / 1.6 / 1.8
gpt-oss-safeguard0.13 / 0.13 / 0.136.3 / 6.2 / 6.1
Qwen3Guard-8B0.14 / 0.14 / 0.140.4 / 0.4 / 0.4

[q3] gpt-oss-safeguard

[re7] With reasoning, paired accuracy rises from 21.5% to 61–72%. The revision adds this as Appendix D.6 and scopes Section 3.1 to verdicts without reasoning. Routing without reasoning matches low-effort reasoning in paired accuracy (64.9% vs. 62.7%) and TPR (85.5% vs. 85.0%) while generating only the verdict, and adding routing to reasoning brings no further gain.

Dear Reviewer f1aF,

As the discussion period is nearing its end, we would like to kindly follow up on our responses. We hope our clarifications have adequately addressed your concerns, and we would greatly appreciate it if they could be reflected in your final assessment, if you find them satisfactory. Thank you again for your time and consideration.

Thanks to the authors for the additional results. The clearer notation, multi-label analysis, and the reorganized tables and figures address my main concerns on presentation, so I'll raise its score to 3. But routing is barely helping Qwen3Guard, leaves NemoGuard's non-target fpr high, and the layer rule misses the best layer on both. Therefore, unfortunately I cannot increase my soundness score, and I'll have to keep my overall score.

Please add the DynaGuard and independent harmfulness-direction results from your response to the paper.

Thank you for raising the presentation score and for the constructive follow-up.

Requested additions. We will add the DynaGuard comparison and the label-independent harmfulness directions to the appendix of the final version, and refer to them from Section 6.2 and Section 5.2.

Qwen3Guard and NemoGuard. These are the two guards where forced contraction already shows, before routing is applied, that the verdict depends weakly on the harmfulness coordinate at the intervention layer (Appendix D.2). Routing helps least exactly where this diagnostic predicts, so an operator can identify such guards in advance. We will state this link explicitly in Section 6.2.

Layer rule. The rule uses training folds only, so that no evaluation prompt influences the choice. Choosing the best layer on evaluation data would raise NemoGuard to 49.5% and Qwen3Guard to 14.8% (Table 15), but would overstate held-out performance. In the final version we will report both the rule-selected and the best layer for every guard.

Reviewer_3HJxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether safety guard models make decisions based on enabled policies or primarily on the harmfulness of the input. Through counterfactual evaluations across five open-source guards, the authors find that models frequently flag harmful prompts even when the relevant policy is disabled, despite being able to identify the correct policy category. They find that models encode the input–policy relationship but fail to incorporate it into their verdicts. To address this, the authors propose policy-gated harmfulness routing.

Critical strengths
  • The paper distinguishes models’ recognizing relevant categories and making policy-dependent judgments. This is an important distinction to make and provides new insights into the topic.
  • The paper uses clean counterfactual designs to evaluate the effect of swapping/ removing policies.
  • The work has realistic implications for downstream deployers using these LLM guard models.
Critical weaknesses
  • Lack of statistical rigor. None of the results report multiple seeds/ error bars, and thus it’s unclear whether the results are statistically significant.
  • Routing is validated only with a single enabled policy, but the paper is motivated by multi-policy settings, which is also what realistic deployment seeks. Realistic setup should also include studies of combinations of enabled policies and interactions between overlapping rules.
  • The routing intervention improves policy-following accuracy on several guards, but improvements vary across models and safety detection TPR also decreases (Table 3).
Questions for authors
  • The guard models also support response classification to test whether model response violates policy. Do the current findings also apply to response classification?
  • How does safeguard behave with reasoning enabled? Compared with reasoning-stripped setup, do they show better policy-following?
3 comments

[w1] Lack of statistical rigor

[re1] We already report 95% bootstrap intervals over prompts in Figure 2 and a random-direction band in Figures 12–13. The revision adds Table 14 (Appendix D.1) with two measures for routing.

  • A 95% bootstrap interval for the routing gain excludes zero on every guard.
  • Over five runs, each with a new fold split and directions fitted on a random 80% of the training fold, the s.d. is at most 0.9 points.
GuardΔ\DeltaPairAcc [95% CI]Routing PairAcc (mean ±\pm s.d., 5 runs)
LlamaGuard 3-8B+45.2+45.2 [42.2, 48.3]64.2 ±\pm 0.1
LlamaGuard 3-1B+38.7+38.7 [35.6, 41.9]42.9 ±\pm 0.1
NemoGuard+20.3+20.3 [17.8, 22.8]32.1 ±\pm 0.9
gpt-oss-safeguard+46.6+46.6 [44.1, 48.8]67.3 ±\pm 0.2
Qwen3Guard-8B+1.4+1.4 [0.4, 2.5]3.6 ±\pm 0.1

[w2] Routing is validated only with a single enabled policy

[re2] We add two settings with multiple enabled policies to the revision, on the same 936 prompts.

Multiple enabled policies (Table 3 lower block, Appendix D.5). A prompt counts only if it is flagged under the full taxonomy and not once its target is removed. Native paired accuracy drops to 0.5–5.0%, and routing raises it to 44.3% on LlamaGuard 3-8B and 58.5% on gpt-oss-safeguard without raising benign false positives, with k∈{2,4,8}k \in \{2, 4, 8\} random policies in Appendix D.5.

Overlapping policies (Appendix D.5). On 186 pairs that GPT-4o labels as violating both c1c_1 and c2c_2, a correct guard flags under [c1][c_1], [c2][c_2] and [c1,c2][c_1, c_2] and not under an unrelated policy. The first two columns are unsafe rates and the last is the share correct under all four, in %, native / routing.

Guard[c1,c2][c_1, c_2] ↑\uparrowUnrelated ↓\downarrowAll four ↑\uparrow
LlamaGuard 3-8B94.6 / 84.485.5 / 0.06.5 / 45.7
LlamaGuard 3-1B94.1 / 68.891.4 / 20.41.1 / 10.8
NemoGuard98.9 / 96.897.8 / 88.20.5 / 3.8
gpt-oss-safeguard97.8 / 91.987.6 / 0.58.6 / 55.9
Qwen3Guard-8B93.5 / 93.093.0 / 93.00.5 / 0.0

Routing improves all-four accuracy on four of the five guards, most on LlamaGuard 3-8B and gpt-oss-safeguard, where unrelated flags drop to almost zero and the share correct rises six- to sevenfold. A majority vote of three annotators (230 pairs) gives the same conclusions.

[w3] improvements vary across models and TPR also decreases

[re3] TPR decrease. Routing buys policy following at the lowest detection cost among the methods we compare.

  • Matched cost. At the same non-target FPR on LlamaGuard 3-8B, a uniform margin shift keeps 38.7% TPR and routing 71.4% (Figure 6). At the same 2-point TPR loss, routing reaches 59.6% paired accuracy, against 21.0% for prompting the guard to judge only the enabled policy, with the same ordering on NemoGuard and gpt-oss-safeguard.
  • Own reasoning. On gpt-oss-safeguard, routing matches the paired accuracy and TPR of low-effort reasoning without a reasoning trace (Appendix D.6).
  • Adjustable. Restoring native TPR on LlamaGuard 3-8B still leaves 70.3% paired accuracy, against 19.0% natively, at an 11.1% benign FPR.

Variation across guards. Much of it comes from the estimated gate. With an oracle gate set to the true relation, routing reaches up to 96.4%, and a better layer also helps, as NemoGuard reaches 49.5% at layer 24 (Table 15).

GuardRoutingOracle gate
LlamaGuard 3-8B64.379.9
LlamaGuard 3-1B42.969.4
NemoGuard32.143.6
gpt-oss-safeguard67.796.4
Qwen3Guard-8B3.56.2

[q1] Response classification

[re4] The revision adds Appendix D.7, which repeats the evaluation on responses from four datasets, with each guard in its official response-moderation format. Native guards again ignore the enabled policy, and routing raises paired accuracy by 17–59 points on every dataset, up to 74.4% on gpt-oss-safeguard, without raising benign false positives. A gate estimated only on prompts also transfers unchanged to responses. Paired accuracy, native / routing:

DatasetLG3-8BLG3-1BNemoGuardgpt-oss-sg
BeaverTails19.6 / 53.76.2 / 44.313.1 / 31.515.5 / 65.6
PKU-SafeRLHF26.6 / 58.85.8 / 34.48.9 / 31.610.2 / 69.6
Aegis 2.022.9 / 41.77.1 / 25.917.9 / 41.326.9 / 74.4
WildGuardMix15.8 / 54.12.3 / 28.24.3 / 21.28.9 / 52.5

[q2] gpt-oss-safeguard

[re5] With reasoning, paired accuracy rises from 21.5% to 61–72%, mainly by lowering FPRirr\text{FPR}_\text{irr}. The revision adds this as Appendix D.6 and scopes our findings in Section 3.1 to verdicts without reasoning. Routing without reasoning matches low-effort reasoning (64.9% vs. 62.7%) while generating only the verdict, against 91 tokens on average for reasoning.

Dear Reviewer 3HJx,

As the discussion period is nearing its end, we would like to kindly follow up on our responses. We hope our clarifications have adequately addressed your concerns, and we would greatly appreciate it if they could be reflected in your final assessment, if you find them satisfactory. Thank you again for your time and consideration.

Thank you for the detailed response. I have read through the new results and am updating my score from 2 to 3, and will encourage authors to include the new rebuttal results to different reviews in future versions.

Comments on the paper

1 comment

Overview

We thank all reviewers for their constructive feedback. The reviewers (3HJx, f1aF, YpUP) found the counterfactual design clean and the distinction between recognizing a category and following a policy important, and f1aF noted that routing turns the diagnosis into a causal demonstration. Their concerns centered on three aspects: the robustness of the evaluation (3HJx, f1aF, YpUP), the scope and practicality of routing (3HJx, f1aF), and the evaluation of gpt-oss-safeguard without its reasoning mode (3HJx, f1aF, YpUP).

Summary of reply

  • 3HJx. Every bootstrap interval for the routing gain excludes zero, and five resampled runs vary by at most 0.9 points. With the full taxonomy enabled, routing raises paired accuracy from 5.0% to 44.3% on LlamaGuard 3-8B and from 2.2% to 58.5% on gpt-oss-safeguard, handles overlapping policies, and raises paired accuracy by 17–59 points on four response datasets. Routing loses the least detection among the methods we compare, and an oracle gate shows that much of the variation across guards comes from the estimated gate.
  • f1aF. A multi-label annotation of all 936 prompts explains at most 16% of non-target flags and changes the routing gain by at most 3.4 points, and flagged policies are not enriched for each guard's own second category. Harmfulness directions estimated without our prompts or the guard's verdicts reproduce the gradient alignment and boundary slope on all five guards. Routing updates no weights, needs 5 harmful and 5 benign prompts per concept, adds at most 0.25 ms per request, and outperforms DynaGuard, a guard retrained for policy following. We also clarified the notation and the placement of results.
  • YpUP. Under a valid but unrelated policy of the same format, guards still flag 63–88% of harmful prompts but no more benign prompts than under their own policy, so the effect is not a default response to unfamiliar blocks, and routing lowers the unsafe rate under that policy to 0.2% on LlamaGuard 3-8B.

On gpt-oss-safeguard, raised by all three reviewers, its reasoning mode reaches 61–72% paired accuracy, and routing without reasoning matches low-effort reasoning while generating only the verdict.

Summary of revision

We have uploaded a revised PDF, with changes marked in red.

Main

  • Section 3.1. Scope the findings on gpt-oss-safeguard to verdicts produced without reasoning (3HJx, f1aF, YpUP).
  • Section 3.3. Summarize the multi-label annotation of all 936 harmful prompts (f1aF).
  • Section 4.1. Summarize the unrelated-policy control (YpUP).
  • Section 5.1, Eq. 5, Figure 4. Define each quantity once, state how the two relation estimators differ, and explain the directions in the caption (f1aF).
  • Section 6.2, Table 3. Add routing under multiple enabled policies as a lower block of Table 3, with one native baseline (3HJx, f1aF).
  • Figure 2, Tables 2–3. Name the population in each caption (f1aF).

Appendix

  • A.4. Replace the keyword check with the multi-label annotation, and compare non-target flags with each guard's own second category (f1aF).
  • B.4. Add an unrelated-policy control with a valid policy block of the same format (YpUP).
  • D.1. Report bootstrap intervals and five resampled runs for routing in Table 14 (3HJx).
  • D.5. Evaluate routing with k∈{2,4,8}k \in \{2, 4, 8\} random policies and with overlapping policies (3HJx).
  • D.6. Evaluate gpt-oss-safeguard in its reasoning mode (3HJx, f1aF, YpUP).
  • D.7. Repeat the evaluation and routing on response classification with four datasets (3HJx).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.