acceptodds
Under review as a conference paper at ICLR 2027

Boundary-Stable PolicyFlip: Training Policy-Sensitive Safety Guards

Abstract

Safety guards deployed under changing policies must sometimes reverse decisions when trusted state changes even if the underlying event does not. We introduce Boundary-Stable PolicyFlip, a fine-tuning objective for matched policy-state interventions. Writing a pair by its unsafe–safe score gap g and score center c, a strict threshold crossing requires g>2|c|: conventional pairwise ranking controls g but leaves c unconstrained. We therefore optimize the weaker-endpoint slack b=g/2-|c|=min(-d_safe,d_unsafe). At the default settings, its penalty is exactly worst-endpoint cross-entropy, yielding a threshold-aware aggregation of the same endpoint labels. BS-PF combines this boundary term with transition ranking; BG-PF additionally attenuates ranking as a pair moves inside the correct decision regions. Neither changes the guard architecture or inference path. On a controlled policy-intervention benchmark, BS-PF improves Qwen3.5-4B balanced accuracy over pointwise training by 5.25–6.50 percentage points across four raw-input evaluations. Against a ranking-plus-paired-CE control on Policy OOD, BS-PF improves transition direction by 4.08 points (95% interval [2.69, 5.47]). On a fresh machine-rendered holdout, BG-PF improves pair exactness by 2.30 points and transition direction by 5.25 points [3.04, 7.42] over Pair-CE. With unchanged loss constants on Llama-Guard-3-8B, it improves Policy-OOD BAcc over gap-only training by 7.46 points and pair exactness by 12.07 points. Across 12,434 public moderation records, aggregate macro BAcc remains within 0.03 points of Decision-only (85.27% versus 85.24%). Overall, threshold-aware pair optimization strengthens policy-transition tracking under a fixed training-step budget without adding inference-time machinery.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM safety, safety guards, policy-sensitive classification, pairwise learning, threshold-aware fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6BNGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes training safety classifiers that change their decisions when the given policy state changes. The paper introduces Boundary-Stable PolicyFlip to fine-tune models to match policy-state interventions. By constructing a pairwise ranking with a score center c, Boundary-Stable PolicyFlip optimizes the weaker-endpoint slack. Experimental results suggest that BS-PF improves Qwen3.5-4B balanced accuracy over pointwise training on a controlled policy-intervention benchmark.

Critical strengths
  1. The paper performs good ablation controls. Pair-CE, PF+PairCE, and Boundary-only are important additions beyond comparing against training or ranking alone.

  2. Experimental evaluations show that Boundary-Stable PolicyFlip improves the safeguard performance.

Critical weaknesses
  1. The paper is hard to read and follow. As disclosed in the AI Use Statement, this paper is likely to be written by AI. I think the authors should rewrite the paper to improve readability. For example, what are BS-PF and BG-PF in your abstract? At least, the authors should write their full names the first time they are mentioned.

  2. Improvements are not convincing for out-of-domain setups. On Policy OOD, Pair-CE has a higher mean BAcc than both proposed variants. These results fail to prove that Boundary-Stable PolicyFlip can generalize well.

  3. No actual new policy semantics are tested even for the OOD setups. All four Boolean policy mappings remain fixed and are learned during training. For example, Policy OOD only changes cardinality and code morphology, while Attack OOD only changes wording.

Questions for authors
  1. What is a softplus function? I cannot see any explanation of it in this paper.

  2. I tried to follow your formulations in Sec. 2, but I cannot understand your derivations. Could you update your writing to provide more details?

  3. Can you provide more details on Policy-state benchmark construction?

1 comment

Thank you for the questions about the derivation and benchmark construction, and for highlighting the scope of the OOD evaluation. We have rewritten the abstract and opening sections to start with the safety problem rather than the notation: the same operation may become unsafe when its authorization is revoked, even though the governing rule does not change. Table 1 follows this before-and-after intervention, and Appendix A.3 provides the fixed system instruction and both complete model-visible user messages. The revised benchmark description also explains the roles of the compiler, renderer, and EventIR.

The key distinction behind the method is between correctly ranking a safe/unsafe pair and giving both endpoints the correct verdict. For example, the score pairs (1, 3) and (-1, 1) have the same ranking gap, but only the second crosses the zero threshold correctly. Moving both scores together can preserve their order while changing the decisions. Let d_safe and d_unsafe be the unsafe-minus-safe scores, with gap g = d_unsafe - d_safe and center c = (d_unsafe + d_safe)/2. A correct flip requires g > 2|c|, which gives the weaker-endpoint slack b = g/2 - |c| = min(-d_safe, d_unsafe). Using the standard softplus function, softplus(t) = log(1 + exp(t)), the boundary term becomes softplus(-b) = max(CE_safe, CE_unsafe) at the default settings (Eq. 9). The methodological step is to use this crossing-derived limiting endpoint to allocate supervision: BS-PF combines that signal with ranking, while BG-PF adjusts the ranking weight according to boundary placement. Neither changes inference.

The benchmark contains 4,608 training programs and 768 programs per raw evaluation split. Each program fixes a policy profile and event, varies the relevant trusted registry state, and obtains exact safe/unsafe labels from one of four Boolean policy functions. The resulting examples are rendered into model inputs without revealing the gold labels. Policy OOD changes registry size and identifier morphology; Attack OOD uses held-out mechanism contexts; Tool OOD uses the held-out tool-call surface. The governing functions remain unchanged, so these splits isolate a guard's response to state and representation changes rather than introducing a different rule definition.

The revised results connect this setting to the training objectives. On raw Policy OOD, BS-PF improves Direction over PF+PairCE by 4.08 points (95% CI [2.69, 5.47]). On independently rendered External-B inputs, BG-PF improves PairExact over matched Pair-CE by 2.30 points and Direction by 5.25 points (95% CI [3.04, 7.42]). The program-aligned analysis finds 280 corrected versus 227 regressed pairs (Table 5 and Appendix Table 16). PairExact requires both endpoint decisions to be correct, making it a direct measure of the crossing behavior that motivated the objective.

Reviewer_HBJWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper targets an important problem. Namely, safeguard models face scenarios where trusted state changes and guard models need to adjust its decision based on the new state. The paper makes 3 main contributions. First, it provides the analysis of the policy transition rankings, which leads to the weaker-endpoint slack. Second, the paper provides boundary-gated and additive objectives. Third, it provides a comprehensive evaluations with representation shifts, boundary-only controls and etc.

Critical strengths
  1. The paper provides a credible protocol. The paper also provides useful controls: Pair-CE, PF+PairCE, Boundary-only.

  2. Objective introduces no inference cost. The equations cleanly show matched transition, translation-invariance argument and its reduction to CE (Eq 10). Figure 1 cleanly conveys that ranking and crossing are different geometrically.

Critical weaknesses
  1. The proposed method does not significantly outperform strongest baselines on splits or on Llama.

    • Pair-CE already provides most of the gains over Decision-only baseline. The proposed method by the paper (BS-PF) which combines ranging loss with the boundary loss, gains only +1.53 / +1.23 / −0.40 / +1.52 on the four splits over simple Pair-CE. It has worse performance than Pair-CE (91.61 vs 92.01) on the Policy OOD (Table 2) and based on the Table 10 the gains are not statistically significant (all 95% Cis include 0, Table 10).
    • The new boundary loss (Boundary-only on Table 2) does not provide substantial gains over the Decision-only baseline and performs slightly worse than Decision only baseline on 4 out of 4 splits (Test, Attack OOD, Policy OOD, Tool OOD).
    • In the abstract, BG-PF gains over the gap only training baseline (7.46 points) is reported to show the transfer to Lllama-Guard-3-8B. However, the gap-only PF is the weakest baseline in table 4 for Lllama-Guard-3-8B Policy OOD. BG-PF’s gains against a stronger baseline (Pair-CE) is not significant (p value of 0.49, Table 18).
  2. In the benchmark, paper never evaluates deployment under changing policies, although the abstract motivates this. In the benchmark, the policies are fixed and a guard is expected to respond to state changes. Therefore, there is no setting to evaluate whether a guard can accommodate a policy given in context.

Questions for authors

Please see the Critical Weaknesses section

1 comment

Thank you for the careful assessment of the matched-pair baselines and ablations. We have added a program-aligned analysis of the frozen External-B predictions to examine whether the proposed objectives improve both endpoint decisions, rather than only the ordering of their scores.

The most direct result is PairExact, which counts a policy-state intervention as correct only when both endpoint verdicts are correct. On the independently rendered External-B holdout, BG-PF improves PairExact over matched Pair-CE by 2.30 percentage points. The paired analysis finds 280 cases corrected by BG-PF and 227 regressions across the same 768 programs and three training seeds: a net 53 additional jointly correct seed-program outcomes. The exploratory seed/program bootstrap 95% interval is [+0.26, +4.38] points. The corresponding Direction gain is 5.25 points (95% CI [3.04, 7.42]), and both comparisons are positive for each seed. BS-PF has the highest mean PairExact on External-B (57.60%), while BG-PF has the highest Direction (80.60%).

We have also added a structural control that already combines pair ranking with ordinary mean endpoint CE. Relative to PF+PairCE, BG-PF improves External-B PairExact by 4.30 points (exploratory 95% CI [+0.78, +7.47]) and Direction by 6.73 points. BS-PF improves PairExact by 4.77 points (exploratory 95% CI [+1.00, +9.68]). Table 5 reports all seven methods, and Appendix Table 16 provides the paired changes. The PF+PairCE evaluation is a retrospective comparison on the same holdout using the reported fixed coefficients; it is not presented as another independent test.

The ablations help clarify why we combine endpoint placement and pair ordering. Boundary-only supplies supervision at the limiting endpoint, whereas ranking supplies an explicit transition-order signal. On raw Policy OOD, adding ranking in BS-PF increases BAcc from 84.06% to 91.61%, a 7.55-point difference (95% CI [2.64, 10.81]); Direction rises by 8.68 points (95% CI [4.08, 15.76]). This shows the benefit of the coupled objective under the common training budget. Pair-CE remains a strong raw BAcc comparator, and the Llama experiment demonstrates use of the objective with a second guard and a different canonical input view rather than serving as the main strong-baseline improvement claim.

Finally, the deployment intervention in this study is a change in trusted policy state, such as a revoked capability, while the governing rule remains fixed. The revised introduction and evaluation protocol distinguish this from interpreting an entirely new policy definition supplied at runtime.

Reviewer_JNPfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Boundary-Stable PolicyFlip (BS-PF), a fine-tuning objective for LLM-based safety classifiers that must change their decisions when trusted policy state changes. It combines pairwise ranking with a loss emphasizing the harder-to-classify example in a matched safe–unsafe pair. The method is evaluated on a synthetic policy-state benchmark, machine-rendered inputs, a second guard model, and public moderation datasets.

Critical strengths
  • The benchmark uses controlled policy-state changes while holding the underlying event fixed, allowing targeted evaluation of decision reversals.
  • The experiments include relevant matched-pair training baselines and ablations under comparable training budgets.
Critical weaknesses
  • The abstract introduces abbreviations, mathematical details and metrics before explaining the task. In the main text, many terms such as compiler, latent program, renderer, and canonical EVENT IR are introduced without sufficient context. A complete example connecting the model input, policy state, and expected output is missing.
  • The paper trains an LLM to classify requests as safe or unsafe based on policy rules and trusted state. However, the experiments do not establish why this fine-tuning is necessary. For example, if a request involves sending a file to destination D, the LLM could simply identify D, and ordinary code could check whether D is on the approved list. The paper does not compare against this simpler approach or adequately demonstrate what specialized fine-tuning adds.
  • The proposed objective combines pairwise ranking with a boundary penalty equivalent to worst-endpoint cross-entropy at the default settings. Although it improves some metrics, its balanced-accuracy gains over strong matched-pair baselines are generally modest and inconsistent.
Questions for authors

Please check weaknesses.

1 comment

Thank you for the detailed comments on the presentation and the practical motivation. We have rewritten the abstract and opening sections around a concrete authorization change rather than starting with the loss formulation. Table 1 shows the same operation changing from safe to unsafe when a required capability is removed from the trusted registry. Appendix A.3 now provides the complete system instruction and both model-visible user messages, and distinguishes the compiler's hidden rule mapping and gold labels from what the guard receives.

Your example of extracting a destination and checking it with ordinary code raises an important system-design question. Such a pipeline is sensible when the relevant facts can be reliably extracted and the governing rule is available for deterministic execution. Our work examines the learned-guard interface itself: given a text-conditioned safety classifier, how should it be trained to change its verdict when authoritative state changes but the action does not? The controlled benchmark holds the event and governing rule fixed within each matched intervention, providing exact labels with which to measure that behavior. The proposed objectives improve this existing decision interface without requiring a new inference-time component.

We evaluate the contribution of specialized training against conventional objectives under the same backbone and training budget. BS-PF improves balanced accuracy over pointwise fine-tuning by 5.25–6.50 percentage points across the four raw-input splits. The objective also addresses a specific limitation of pair ranking: the safe/unsafe score pairs (1, 3) and (-1, 1) have the same ranking gap, but only the latter yields two correct decisions. The weaker-endpoint crossing condition identifies which prediction is preventing a correct flip; BS-PF couples supervision of that endpoint with ranking, while BG-PF adjusts ranking pressure according to boundary placement.

The independently rendered External-B holdout offers a further comparison beyond the original input templates. Relative to matched Pair-CE, BG-PF improves PairExact, which requires both endpoint verdicts to be correct, by 2.30 points and Direction by 5.25 points (95% CI [3.04, 7.42]). A program-aligned analysis finds 280 corrected versus 227 regressed pairs, a net gain of 53 jointly correct seed-program outcomes (Table 5 and Appendix Table 16). Together, the worked example and these comparisons clarify both the behavior we study and the measurable benefit of training explicitly on trusted-state interventions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.