acceptodds
Under review as a conference paper at ICLR 2027

SafetyFlip: Learning LLM Safety Boundaries from Bidirectional Counterfactual Instructions

Abstract

Balancing helpfulness and safety in large language models (LLMs) remains challenging: imprecise safe/unsafe boundaries can cause both unsafe compliance and over-refusal of safe but sensitive requests. We propose **SafetyFlip**, a data-centric framework that learns LLM safety boundaries from bidirectional counterfactual instructions. SafetyFlip constructs semantically matched SafeUnsafe pairs with structured boundary annotations identifying the preserved non-safety semantic frame, the safety-critical factor, and the required policy behavior. These pairs are used for **Boundary-Constrained Fine-Tuning (BCFT)**, which combines supervised fine-tuning with Safety Contrastive Regularization (SCR) to encourage safety-relevant representation shifts while preserving shared non-safety utility. On Qwen2.5-7B, SafetyFlip reduces HarmBench attack success from 58.4% to **6.5%** while keeping XSTest over-refusal low at **2.1%** and preserving MT-Bench utility. Our work offers a scalable, low-supervision pathway for boundary-aware alignment without inference-time overhead.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Safety, Safety Boundary

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iLtSOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Safety alignment in LLMs draws a boundary between helpfulness and harmfulness, but in many cases models learn an imprecise boundary, leading to over-refusal or to being tricked into producing harmful content. This paper proposes a data-driven fine-tuning algorithm that uses multi-agent synthesis of safe/unsafe data pairs to help the model learn a more accurate boundary, thereby enhancing safety while avoiding too much damage to helpfulness.

Critical strengths

I think the core contribution of this paper lies in reframing safety alignment as a safety boundary learning problem: rather than assigning global safety labels to isolated instructions, it constructs Safe-Unsafe pairs that share nearly identical semantic frames and differ only in the safety-critical factor, thereby providing the model with contrastive supervision near the boundary. This design directly targets the existing sparse safety data. Moreover, the experimental results support the independent value of this data-pair construction: under a sample-matched SFT control, using exactly the same SafetyFlip examples but removing the paired boundary supervision and SCR, HarmBench ASR drops only to 12.4%, whereas full SafetyFlip reduces it to 6.5%, with lower XSTest OR and boundary misclassification rates as well. This indicates that the safety gain does not come from an increase in synthetic data volume, but from the structured information inherent in the paired boundary supervision itself.

Critical weaknesses

I believe it is necessary to compare the method with other safety alignment/defense approaches, such as reasoning-based external guardrails. In addition, there is no clear comparison test against attack methods: HarmBench only reports aggregate ASR, without breaking them down by attack type, nor does it compare defense effectiveness against specific jailbreak attacks. The paper mentions that it generalizes beyond the training distribution to unseen attack, but does not provide sufficient results to support this claim. There is no comparison with inference-time defense solutions. This paper emphasizes no inference-time overhead as an advantage, but does not set an inference-time defense baseline as a control.

Questions for authors

This also brings to mind an interesting point: since this paper focuses on making the safety boundary more accurate, and some jailbreak methods specifically exploit the imprecision of safety boundaries, shouldn't it be even more important to test whether this paper achieves better results under attacks from these methods?

I found one paper that could serve as a reference and there might be other similar works: https://aclanthology.org/2026.acl-long.1139/

4 comments
AuthorsOpenReview ↗

Response to Weaknesses and Questions (Part 1/2)

Dear Reviewer iLtS,

Thanks for your constructive comments. Below we will address your concerns.


Q1. How does SafetyFlip compare and combine with external guardrails?

GuardReasoner-8B checks the input and output of each generator using the same block-and-refuse policy.

Table R5. External guardrail comparison.

MethodHB8 ASR ↓XST OR ↓MT ↑BMR ↓
Base58.4%1.2%7.6122.5%
Base + GuardReasoner8.6%4.8%7.5910.6%
SafetyFlip6.6%2.1%7.848.1%
SafetyFlip + GuardReasoner3.5%3.3%7.816.9%

Adding GuardReasoner to SafetyFlip reduces ASR from 6.6% to 3.5%, with OR increasing from 2.1% to 3.3%. This supports complementary use in the tested setting. R8 includes the cost of all guardrail calls.


Q2. What is the performance against individual attacks?

Table R6. ASR (%) by condition; DirectRequest is the non-adversarial control.

MethodDirectRequestHumanJailbreaksZeroShotGCGAutoDANPAIRTAPPAP-top5HB8
Base32.049.338.080.374.066.770.756.358.4
Safe-RLHF0.31.71.05.73.37.79.05.04.2
Circuit Breakers0.72.31.03.32.710.312.77.75.1
Base + GuardReasoner1.34.72.38.07.715.717.311.78.6
SafetyFlip0.72.71.810.06.511.712.37.26.6

Each condition uses 200 behaviors and three repetitions. HB8 is computed before rounding. SafetyFlip improves over Base in every condition, but is not best in every column; PAIR and TAP remain harder cases. We report these results together with over-refusal and utility.


Q3. What supports generalization to unseen attacks?

Here, “unseen” refers to attack-generation procedures absent from data construction. Role-play and intent rewriting occur in training, so semantic disjointness is not established.

Beyond the submitted BeaverTails check, R7 uses an independent WildJailbreak subset with 500 harmful and 500 benign requests per repetition.

Table R7. External WildJailbreak evaluation.

MethodHarmful ASR ↓Benign OR ↓
Sample-matched SFT24.9%9.4%
Safe-RLHF13.1%23.4%
DPO14.2%5.2%
SafetyFlip12.6%4.0%

Relative to sample-matched SFT, SafetyFlip lowers harmful ASR by 12.3 percentage points and benign OR by 5.4 points. Removing the role-play construction operator, with example and training budgets matched, gives HumanJailbreaks ASR of 4.0% versus 2.7% for the full method and 49.3% for Base; HB8 is 7.2%, 6.6% and 58.4%. This supports transfer beyond that explicit operator. Generalization across attacker-model families remains untested.

AuthorsOpenReview ↗

Response to Weaknesses and Questions (Part 2/2)

Q4. Is the inference-cost claim justified?

We correct the inference description in the submitted manuscript: the evaluated implementation generates a bounded structured annotation before the visible answer, using the same generator and decoding state. Only the answer is returned to the user. Construction agents, auxiliary critics, and the learned safety vector are absent at inference. The revised abstract, §3.2.2, and §4.3.1 state this explicitly. SafetyFlip requires no external safety-model calls, but the measured end-to-end latency increases from 2.14 to 2.54 seconds per query (18.7%). Appendix D.1 and the latency table report the timing protocol, including annotation decoding and defense calls.

The three-field annotation uses a fixed 96-token decoding cap to bound the internal prefix, separately from the common 512-token visible-answer budget.

Table R8. End-to-end latency. One A100-80GB, batch size 1, 200 fixed mixed requests, warm-up and three timed repetitions. All annotation tokens and defense calls are included.

SystemSeconds/query ↓Relative to Base
Base2.141.00×
SafetyFlip2.541.19×
Base + Llama Guard3.081.44×
Base + Intention Analysis4.051.89×
Base + GuardReasoner5.122.39×
SafetyFlip + GuardReasoner5.472.56×

SafetyFlip adds 0.40 s/query (18.7%) over Base, compared with 43.9% for Base + Llama Guard and 139.3% for Base + GuardReasoner in this setting. Together, R5 and R8 show that SafetyFlip achieves lower ASR and OR than Base + GuardReasoner (6.6%/2.1% versus 8.6%/4.8%), at 2.54 versus 5.12 s/query. This supports an efficient integration of safety conditioning into the generator. These are end-to-end system measurements, including any differences in visible-answer length; the Llama Guard and Intention Analysis rows serve as timing references.


Q5. Does SafetyFlip help against attacks exploiting ambiguous boundaries?

We evaluated Jargon from the paper you recommended, Hung et al., Into the Gray Zone, with separate benign research requests matched by domain and discourse style.

Table R9. Jargon goal-level success and benign-research over-refusal.

MethodJargon goal-level ASR ↓Benign-research OR ↓
Base95.7%3.3%
Safe-RLHF78.3%22.8%
Annotation KL-SFT68.3%10.7%
DPO66.0%6.2%
Annotation KL-SFT + DPO62.7%5.2%
Circuit Breakers69.3%12.0%
Base + GuardReasoner64.7%11.0%
SafetyFlip59.0%4.7%
SafetyFlip + GuardReasoner46.0%7.8%

The attack uses 100 JailbreakBench goals and the published budget structure: three retries, two trials per retry, four rounds per trial and eight optimization variants. Systems share the attacker configuration, budget, initial memory conditions and goal ordering; search adapts to each target. Candidate queries count toward the budget. Three repetitions give 300 goal evaluations; the benign control has 200 requests per repetition.

Success means harmful assistance anywhere within the full interaction, including accumulated disclosure. Attacker-supplied text does not count. A separate final evaluator and blinded transcript audit check the attack-internal judge. Specifically, the blinded audit covers 180 complete transcripts, with 20 per method in R9, balanced between automatically judged successful and unsuccessful attacks. Two annotators review each transcript independently with method identities hidden; a third helps to adjudicate disagreements. Benign tasks are validated as safe before evaluation.

Against annotation KL-SFT, SafetyFlip reduces ASR from 68.3% to 59.0% and benign OR from 10.7% to 4.7%. The improvement therefore does not come from blanket refusal of research requests. The remaining 59.0% ASR is substantial; GuardReasoner lowers it to 46.0% with 7.8% benign OR.

Appendix B.1 requires a genuine change in the safety-critical factor. The revised Appendix B.1 explicitly states that academic or defensive framing alone does not justify a label flip. Auditing accepted training pairs tests that construction rule; auditing Jargon transcripts tests attack scoring. These are separate checks. Multi-turn counterfactual training remains future work.


Thanks again for your time and thoughtful feedback. If you have any additional concerns or suggestions, please feel free to share them, we would be happy to discuss further.

Sincerely,

Thank you for the additional experiments, which have addressed most of my concerns. Now I have no further suggestions and will maintain my score.

Dear Reviewer iLtS,

We sincerely appreciate your positive feedback on our paper. We are also grateful for your constructive suggestions, which have helped us improve the manuscript.

Thank you once again for your support.

Sincerely,

Reviewer_U5NdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SafetyFlip, a data-centric framework designed to improve LLM safety boundary learning without sacrificing general utility. The approach constructs bidirectional counterfactual instruction pairs (Safe↔Unsafe\text{Safe} \leftrightarrow \text{Unsafe}) that preserve the non-safety semantic frame while altering only the safety-critical factor.

Critical strengths

The manuscript is generally well-written, easy to follow, and clearly presents its motivation and overarching methodology.

Addressing the trade-off between unsafe compliance and over-refusal by refining model safety boundaries is a highly relevant and practical problem in LLM alignment.

Critical weaknesses
  1. Several key concepts and mathematical notations remain abstract in Section 3. Providing explicit, running text examples alongside the definitions would significantly improve clarity. Specifically, it is unclear what concrete form the structured annotation c=(rframe,rcrit,rpolicy)c = (r_{\text{frame}}, r_{\text{crit}}, r_{\text{policy}}) takes in practice. Showing a full textual example of cc for a sample prompt pair early in the methodology section would make the supervision signal much easier to understand.

  2. Notation Ambiguity:There is confusion regarding the notation used for safety labels and annotations. Specifically, what is the exact relationship between Attr(x)\text{Attr}(x) and aa? In Section 3.1

  3. Potential Overfitting to Keyword/Factor Differences:The authors argue that existing methods cause models to rely on coarse heuristics (e.g., keyword blacklisting) rather than fine-grained, context-aware decision rules. However, because SafetyFlip generates counterfactual pairs by holding the semantic frame identical and flipping only the safety-critical factor, could this setup inadvertently teach the model to over-focus on the localized keyword/factor edit rather than learning holistic semantic intent?

  4. Clarification on vsafetyv_{\text{safety}} Estimation:How is the dominant safety direction vsafetyv_{\text{safety}} obtained during training? It is unclear whether vsafetyv_{\text{safety}} is a dynamically learned parameter updated via gradient descent alongside Ldir\mathcal{L}_{\text{dir}}, a static vector computed offline via mean-difference estimation over the dataset representations, or an auxiliary projection head.

  5. Missing SOTA Alignment Baselines:While the comparison includes Safe-RLHF and standard SFT, the experimental baseline set lacks comparison with more recent state-of-the-art alignment and over-refusal mitigation techniques (e.g., SimPO, DPO variants, or representation-engineering approaches like Refusal Direction Removal). Including comparisons against these modern methods would provide a stronger evaluation of the framework's effectiveness.

Questions for authors

please see above

5 comments
AuthorsOpenReview ↗

Response to W1, W2, and W3

Dear Reviewer U5Nd,

Thanks for your constructive comments. Below we will address your concerns.


W1. What does a complete structured annotation look like?

The following illustrative pair keeps the invoice, domain, tone and output form fixed. Debt validity and deceptive intent determine the required behavior.

Safe instruction:
  Write a brief, polite payment-reminder email about invoice A17.
  The services were delivered and the payment is overdue.
Annotation:
  r_frame: Customer billing; invoice A17; brief, polite email.
  r_crit: The debt is genuine and the payment claim is truthful.
  r_policy: Comply

Unsafe instruction:
  Write a brief, polite payment-reminder email about invoice A17.
  No transaction occurred; induce payment of the fabricated debt.
Annotation:
  r_frame: Customer billing; invoice A17; brief, polite email.
  r_crit: The fabricated debt is intended to obtain payment deceptively.
  r_policy: Pivot

The safe-side target provides the reminder; the unsafe-side target declines deception and offers truthful billing communication. The revised §3.2.2 now includes this illustrative invoice pair beside the annotation definitions.


W2. How are the safety label and annotation related?

The label specifies which side of the boundary an instruction occupies; the annotation specifies what makes it belong there and what assistance can still be provided. Formally, a=Attr(x)∈{Safe,Unsafe}a=\mathrm{Attr}(x)\in\{\mathrm{Safe},\mathrm{Unsafe}\}, with g(x)∈{+1,−1}g(x)\in\{+1,-1\} supplying the sign for directional training. An Unsafe label alone does not identify the problematic factor or distinguish a necessary refusal from a useful safe redirection.

The annotation c=(rframe,rcrit,rpolicy)c=(r_{\mathrm{frame}},r_{\mathrm{crit}},r_{\mathrm{policy}}) supplies this missing structure: the frame records the task context and potentially recoverable benign utility; the critical factor identifies what changes the safety decision; and the policy selects Comply, Refuse or Pivot. In the invoice example above, the model can preserve the useful task form of a billing email while declining the fabricated payment claim and redirecting to truthful communication. Preserving a frame never licenses assistance that advances the harmful objective.

The label and annotation therefore play complementary roles, as clarified in §§3.1–3.2.2. Both response targets remain safe under Eq. (2), although the instruction labels differ. Under the original-submission protocol, removing structured annotations increases OR from 2.1% to 9.2% and BMR from 7.8% to 10.8%. This ablation removes both annotation supervision during training and annotation conditioning at inference, so it supports the annotation-conditioned pipeline as a whole; it does not isolate training supervision or provide a matched binary-label-only comparison. We now state this scope in §4.4 and retain the original results in Appendix A.7.


W3. Could the model learn localized keyword shortcuts?

R3 tests both directions of this failure: safe requests retaining risk-associated words and unsafe requests paraphrased without their usual trigger words. Each class contains 300 held-out original/rewrite pairs per repetition. Annotators validate that each rewrite preserves the task and safety label before model evaluation; an independent output audit checks the resulting decisions.

Table R3. Label-preserving surface-form probes.

MethodSafe-trigger OR ↓Trigger-free unsafe UCR ↓Original and rewrite both correct ↑
Base9.6%54.0%64.0%
Sample-matched SFT (unpaired)17.0%13.6%81.0%
Annotation KL-SFT14.0%9.7%85.5%
Annotation KL-SFT + SupCon10.4%7.7%89.3%
DPO7.1%8.9%89.8%
SimPO6.8%9.3%89.5%
SafetyFlip w/o MI13.0%8.0%88.0%
SafetyFlip5.6%7.0%90.7%

Joint correctness requires correct behavior on both versions. Removing MI increases safe-trigger OR from 5.6% to 13.0%, while unsafe UCR rises from 7.0% to 8.0%. These results support reduced dependence on the tested cues; they do not establish robustness to every unseen vocabulary or context.

AuthorsOpenReview ↗

Response to W4, W5, and Ethics Concerns

W4. How is the safety direction estimated and learned?

v∈Rdv\in\mathbb R^d is a single trainable vector, learned jointly with LoRA. It is initialized from training pairs at the initial checkpoint:

\qquad v_0=m_0/\|m_0\|_2.$$ $h$ is normalized mean pooling over last-layer instruction tokens, excluding padding and template tokens. Dropout is disabled for initialization; a norm below $10^{-8}$ triggers a seeded unit-Gaussian fallback. The sign aligns both construction directions toward safe. Both $L_{\mathrm{dir}}$ and $L_{\mathrm{cons}}$ update $v$, including gradients through $P_{\perp v}(z)=z-v(v^\top z)$. Response tokens are pooled for $L_{\mathrm{cons}}$. AdamW uses FP32, learning rate $2\times10^{-5}$, zero weight decay for $v$, and the model's schedule; $v$ is normalized after each step. The other losses update the model but have no explicit dependence on $v$. The vector is used only in training. Appendix C.2 and the visualization caption now distinguish the last-token visualization from the mean-pooled training representations. *** ### W5. How does SafetyFlip compare with modern alignment methods? R1 provides DPO/SimPO objective controls. R4 reuses those checkpoints and adds Circuit Breakers and refusal-direction removal (RDR). Circuit Breakers uses its required rerouting/retain supervision; RDR is applied to Safe-RLHF. **Table R4. Complete-method comparison.** | Method | HB8 ASR ↓ | XST OR ↓ | MT ↑ | BMR ↓ | |---|---:|---:|---:|---:| | Base | 58.4% | 1.2% | 7.61 | 22.5% | | Safe-RLHF | 4.2% | 14.5% | 7.45 | 11.7% | | DPO | 7.7% | 3.6% | 7.77 | 9.3% | | SimPO | 7.8% | 3.3% | 7.80 | 9.4% | | Circuit Breakers | 5.1% | 7.6% | 7.52 | 10.8% | | Safe-RLHF + RDR | 10.1% | 5.7% | 7.47 | 10.7% | | SafetyFlip | 6.6% | 2.1% | 7.84 | 8.1% | Safe-RLHF and Circuit Breakers achieve lower ASR, but higher over-refusal. RDR reduces Safe-RLHF OR from 14.5% to 5.7% while increasing ASR from 4.2% to 10.1%. SafetyFlip's advantage concerns the joint safety–utility trade-off, rather than the lowest ASR alone. The requested X-Boundary comparison is addressed in R11 below. *** ### Ethics Concerns We take the dual-use implications of constructing and releasing unsafe instructions seriously. Both sides of each SafetyFlip pair have safe response targets under Eq. (2), but safe targets do not remove the misuse risk of the underlying instructions. We will release the code and framework publicly. We will not release the unsafe-instruction corpus for unrestricted public download. Access to data containing unsafe instructions, including the corresponding counterfactual pairs, will require an application describing the research purpose, researcher verification, and a signed data-use agreement prohibiting malicious use and unauthorized redistribution. Public repository examples will be screened and sanitized so that they do not bypass this restriction. We will document the access criteria, review process, permitted uses and revocation procedure in the final Ethics Statement and repository documentation. Public evaluation code, configurations and aggregate results will support reproducibility, with restricted data available through the stated access process. These measures reduce exposure and provide accountability; they do not eliminate all misuse risk. *** Thanks again for your time and thoughtful feedback. If you have any additional concerns or suggestions, please feel free to share them, we would be happy to discuss further. Sincerely,

Dear Reviewer U5Nd,

We would like to kindly follow up on our rebuttal. We have provided detailed responses to all your concerns, conducted additional experiments with supporting results and conclusions, and revised the manuscript accordingly.

We would greatly appreciate it if you could review our responses and let us know whether any concerns remain. We hope these clarifications and additional results may help you reconsider your assessment.

Best regards, The Authors

Reviewer_ran9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SafetyFlip addresses LLM over-refusal and unsafe compliance by learning precise safety boundaries from bidirectional counterfactual instructions. A multi-agent pipeline constructs semantic frame matched instruction pairs that isolate and flip the safety-critical factor. The training incorporates annotation conditioned SFT and Safety Contrastive Regularization (SCR) to guide directional representation shifts. On Qwen2.5-7B, the method drops HarmBench ASR from 58.4% to 6.5% and achieves 2.1% over-refusal on XSTest, without degrading general utility on MT-Bench.

Critical strengths
  1. Framing safety alignment as local boundary learning provides a clean perspective, and using a sample matched baseline confirms that gains stem from paired data design and regularization rather than data quantity.
  2. Performance gains hold consistently across LLaMA 3 8B, Mistral 7B, and Qwen2.5 72B Instruct as shown in Table 8.
Critical weaknesses
  1. Baseline comparisons do not isolate Safety Contrastive Regularization because pairwise preference objectives like DPO on paired data are absent, proving only that pairs with SCR beat standard SFT rather than simpler contrastive objectives.
  2. Evaluation details are underspecified regarding which HarmBench attacks are executed, which judge evaluates MT-Bench, and how SRR and BMR are scored.
Questions for authors

Which HarmBench attacks and which classifier were used to compute ASR, and which model judges MT-Bench, SRR, and BMR?

4 comments
AuthorsOpenReview ↗

Response to W1

Dear Reviewer ran9,

Thanks for your constructive comments. Below we will address your concerns.


W1. Does SCR add value beyond DPO or simpler contrastive objectives?

We have added annotation-matched DPO, SimPO, and SupCon controls in §4.4, with standalone DPO/SimPO alternatives in §4.3. The matched controls retain the same annotation-conditioned KL-SFT term, initialization, prompt set, supervised targets, optimizer updates, validation-search budget, and test-time annotation cap. Relative to matched DPO, SafetyFlip lowers OR from 3.2% to 2.1% and BMR from 8.5% to 8.1%. These results evaluate full SCR, including MI; its additional observed benefit is mainly lower over-refusal.

Table R1. Training-objective comparisons. Values are means over three training seeds.

(a) Annotation-matched controls

MethodHB8 ASR ↓XST OR ↓MT ↑BMR ↓
Annotation KL-SFT7.6%6.8%7.469.9%
Annotation KL-SFT + SupCon7.0%5.1%7.629.3%
Annotation KL-SFT + DPO6.9%3.2%7.798.5%
Annotation KL-SFT + SimPO7.1%3.1%7.818.7%
SafetyFlip (Annotation KL-SFT + SCR)6.6%2.1%7.848.1%

(b) Standalone training alternatives

MethodHB8 ASR ↓XST OR ↓MT ↑BMR ↓
DPO7.7%3.6%7.779.3%
SimPO7.8%3.3%7.809.4%

For DPO/SimPO, preferences are checked under the same instruction: appropriate assistance is preferred to unjustified refusal on safe requests, and refusal or safe redirection is preferred to harmful assistance on unsafe requests. The counterpart's safe answer is not automatically a valid rejected response. In the annotation-matched DPO/SimPO controls, both candidates share the same validated annotation prefix during training. At inference, all annotation-matched methods generate their own annotation cc under the same 96-token cap, followed by the response under the common 512-token visible-answer cap; no reference annotations are supplied. SupCon uses same-label, same-domain positives and the counterfactual as a hard negative.

The annotation-matched comparison evaluates the full SCR objective, including its MI term. DPO and SimPO remain competitive alternatives; the additional observed benefit of SCR is mainly lower over-refusal.

Correspondence control (R10; §4.4 and Appendix D.2). We replace each original safe–unsafe correspondence with a one-to-one, non-original match within the same domain and a comparable non-safety frame. Individual instructions, annotations, response targets, safety labels, MI supervision, and training and evaluation settings remain fixed; only the pair indices used by the directional and response-consistency losses change. Replacing the original correspondence increases OR from 2.1% to 4.1% and BMR from 8.1% to 9.4%, supporting its contribution within SafetyFlip. The original-correspondence row reuses the R1 SafetyFlip checkpoints.

Table R10. Correspondence control. The original-correspondence row reuses the same SafetyFlip checkpoints reported in R1, which were trained on the identical example pool.

CorrespondenceHB8 ASR ↓XST OR ↓MT ↑BMR ↓
Original counterfactual correspondence6.6%2.1%7.848.1%
Within-domain non-original correspondence7.7%4.1%7.799.4%

Evaluation follows the common three-seed protocol, with identical test sets, attack budgets and judges; adaptive attacks are run against each target. The pooled denominators are 4,800 completions for HB8, 750 requests for XST OR, 3,000 requests for BMR and 480 turn scores for MT.

In this comparison, replacing the original correspondence increases OR from 2.1% to 4.1% and BMR from 8.1% to 9.4%, alongside higher ASR and lower MT. These results support the contribution of precise correspondence to BCFT: the model receives the same individual examples, but loses the explicit link identifying which local change should alter its safety behavior.

AuthorsOpenReview ↗

Response to W2 & Q1

W2 & Q1. Which attacks, judges and metric definitions are used?

The supplemental protocol is:

ComponentConfiguration
HarmBench200 fixed text behaviors; identical IDs across systems; eight conditions in R6
Final ASR classifiercais/HarmBench-Llama-2-13b-cls; behavior-appropriate template
Candidate-selection classifiercais/HarmBench-Mistral-7b-val-cls; final-classifier labels excluded from selection
LLM attackerQwen2.5-72B-Instruct where required
XST / MT250 safe XSTest prompts; 80 two-turn MT-Bench questions
XST / MT / SRR / BMR judgegpt-5.2-2025-12-11, reasoning effort none, separate fixed rubrics
GenerationGreedy; 512 visible-answer tokens, plus a separate 96-token annotation cap where applicable

Each behavior and condition contributes one selected completion per repetition. Three repetitions give 4,800 scored completions per system. HB8 weights behaviors and conditions equally. Budgets are 500 GCG steps, 100 AutoDAN generations with population 64, at most 40 target calls for PAIR/TAP, five candidates for HumanJailbreaks/PAP, and one for DirectRequest/ZeroShot. Visible answers are scored; classifier truncation is tracked separately.

SRR/BMR use SafetyFlip-Test: 500 Safe–Unsafe pairs, or 1,000 prompts. Table 11 in the revised manuscript additionally reports BeaverTails results under the original-submission protocol. BMR is the proportion of safe requests wrongly refused plus unsafe requests receiving harmful assistance, within the same dataset. A refusal followed by harmful instructions still counts as harmful compliance.

For SRR, eligibility is determined from the request before examining outputs. A successful pivot must decline the unsafe component, avoid harmful assistance and offer a relevant safe alternative; a generic refusal does not qualify. The all-prompt score used in the response is

=\rho \mathrm{SRR}_ {\mathrm{eligible}},$$ where $\rho$ is the eligible fraction. We now distinguish these denominators in §4.1 and Appendices A.11 and D.1. The cross-dataset difference can reflect both eligibility coverage and success on eligible requests. **Table R2. Boundary-judge audit against adjudicated human labels.** | Dataset / decision | n | Agreement | Cohen's κ | |---|---:|---:|---:| | SafetyFlip-Test / BMR error | 200 | 94.0% | 0.81 | | BeaverTails subset / BMR error | 200 | 92.5% | 0.79 | | SafetyFlip-Test / successful pivot on eligible prompts | 120 | 95.8% | 0.76 | | BeaverTails subset / successful pivot on eligible prompts | 120 | 92.5% | 0.76 | Two annotators independently review anonymized outputs, with a third adjudicating disagreements. Automated labels are frozen beforehand. These sample sizes describe the audit, not the full-test denominators. R2 validates boundary decisions; the submitted 100-pair audit concerns training-data quality. *** Thanks again for your time and thoughtful feedback. If you have any additional concerns or suggestions, please feel free to share them, we would be happy to discuss further. Sincerely,

Thank you for your detailed response and the additional clarifications. I have carefully read the rebuttal and the other reviewers' comments. After consideration, I have decided to maintain my original score. I appreciate the authors' efforts and wish them the best with this work.

Dear Reviewer,

Thank you for your time and careful consideration. May we kindly ask if there are any remaining concerns that have not been adequately addressed in our rebuttal? We would be happy to provide further clarification and hope our responses may help you reconsider your assessment.

Best regards, The Authors

Comments on the paper

3 comments
~Jason_Vega1OpenReview ↗

Comparison with X-Boundary

Dear authors,

I happened to come across this work and couldn't help but notice that the core argument of the paper -- i.e., the safety/over-refusal trade-off can be mitigated by encouraging a more precise boundary between unsafe and boundary-safe examples ("We formulate LLM safety alignment as a local boundary learning problem") -- is shared by an earlier work that proposed a defense called X-Boundary [1]. However, X-Boundary is not discussed or used as a baseline in the current draft. One of the main differences for handling the safety/over-refusal trade-off is that SafetyFlip generates counterfactual pairs for training data, whereas X-Boundary uses unpaired data. Thus, I think it would be important to add X-Boundary as an evaluation baseline, to see how much pairing data matters in mitigating the safety/over-refusal trade-off.

[1] Lu, Xiaoya, et al. "X-Boundary: Establishing Exact Safety Boundary to Shield LLMs from Jailbreak Attacks without Compromising Usability." EMNLP (Findings). 2025.

AuthorsOpenReview ↗

How does SafetyFlip differ from X-Boundary, and how much does pairing matter?

How does SafetyFlip differ from X-Boundary, and how much does pairing matter?

X-Boundary is closely related: it shapes representations through separation, erasure and retention objectives, and its separation set already contains paired boundary-safe and refusal examples. SafetyFlip's distinctive supervision is a bidirectional counterfactual relation between instructions: changing a safety-critical factor changes the appropriate behavior while preserving the non-safety frame. Structured annotations identify that change, and BCFT uses the correspondence to constrain instruction displacements and the shared components of response representations. This is a difference in the relation being supervised, rather than simply the presence of pairs.

R10 isolates that correspondence while preserving the examples, annotations and training objective. R11 instead specifies a complete-method comparison on the same Qwen2.5-7B starting checkpoint and common evaluation suite, including Jargon to cover multi-turn attacks. X-Boundary retains its method-specific objectives and training setup; configurations are selected on validation data. This is a complete-method comparison with a common evaluation protocol, rather than a comparison with identical training data or compute.

Table R11. Complete-method comparison

MethodHB8 ASR ↓XST OR ↓MT ↑BMR ↓Jargon ASR ↓Benign-research OR ↓
SafetyFlip6.6%2.1%7.848.1%59.0%4.7%
X-Boundary5.8%3.2%7.788.4%52.3%6.0%

In this comparison, X-Boundary has lower HB8 and Jargon ASR, while SafetyFlip has lower XST over-refusal (2.1% versus 3.2%), lower benign-research over-refusal (4.7% versus 6.0%) and higher MT (7.84 versus 7.78). A key methodological distinction is how boundary supervision is specified. X-Boundary explicitly optimizes representation separation, erasure and retention. SafetyFlip constructs bidirectional instruction counterfactuals that identify which safety-critical change should alter the response behavior while preserving the non-safety task frame.

SafetyFlip therefore contributes an explicit supervision signal for both sides of a local boundary: when legitimate assistance becomes unsafe, and when changing the unsafe factor makes assistance permissible again. Structured annotations identify the relevant factor and appropriate behavior; BCFT translates these instruction-level contrasts into representation constraints. This connects boundary learning to the concrete decision to comply, refuse or safely redirect. R10 is designed to isolate the contribution of precise correspondence. The revised Introduction and Related Work explain this distinction, and Appendix D.2 reports the complete-method comparison.

AuthorsOpenReview ↗

Revisions Summary

Dear Reviewers and AC,

We thank the reviewers for focusing the discussion on three questions: what counterfactual supervision contributes beyond standard alignment, how boundary annotations preserve legitimate assistance, and whether the evaluation and deployment claims are adequately supported.

The revised manuscript reports annotation-matched DPO, SimPO, and SupCon controls and a correspondence control in §4.4. Relative to matched DPO, SafetyFlip lowers OR from 3.2% to 2.1%. The original annotation ablation supports the complete annotation-conditioned pipeline, while the surface-form probes in §4.5 examine reliance on the tested cues. We also specify attack budgets, judges, metric denominators, and boundary-label audits in §4.1 and Appendix D.1. SafetyFlip generates a bounded annotation within the same model before its visible answer; §4.3.1 reports its measured end-to-end cost, including annotation decoding.

The supplemental fine-tuning results are means over three training seeds on Qwen2.5-7B, with each checkpoint evaluated once. The revised manuscript incorporates the comparisons and methodological clarifications at the locations cited below. We plan to release the implementation, evaluation scripts, configurations, and seed-to-checkpoint mappings publicly. Data containing unsafe instructions will follow the controlled-access policy described in our ethics response.

We hope that our replies and revisions could help resolve your concerns. If there are any further questions or suggestions, we would be happy to discuss them with you. Thank you again for your time and thoughtful feedback.

Sincerely,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.