- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Safety alignment in LLMs draws a boundary between helpfulness and harmfulness, but in many cases models learn an imprecise boundary, leading to over-refusal or to being tricked into producing harmful content. This paper proposes a data-driven fine-tuning algorithm that uses multi-agent synthesis of safe/unsafe data pairs to help the model learn a more accurate boundary, thereby enhancing safety while avoiding too much damage to helpfulness.
Critical strengths
I think the core contribution of this paper lies in reframing safety alignment as a safety boundary learning problem: rather than assigning global safety labels to isolated instructions, it constructs Safe-Unsafe pairs that share nearly identical semantic frames and differ only in the safety-critical factor, thereby providing the model with contrastive supervision near the boundary. This design directly targets the existing sparse safety data. Moreover, the experimental results support the independent value of this data-pair construction: under a sample-matched SFT control, using exactly the same SafetyFlip examples but removing the paired boundary supervision and SCR, HarmBench ASR drops only to 12.4%, whereas full SafetyFlip reduces it to 6.5%, with lower XSTest OR and boundary misclassification rates as well. This indicates that the safety gain does not come from an increase in synthetic data volume, but from the structured information inherent in the paired boundary supervision itself.
Critical weaknesses
I believe it is necessary to compare the method with other safety alignment/defense approaches, such as reasoning-based external guardrails. In addition, there is no clear comparison test against attack methods: HarmBench only reports aggregate ASR, without breaking them down by attack type, nor does it compare defense effectiveness against specific jailbreak attacks. The paper mentions that it generalizes beyond the training distribution to unseen attack, but does not provide sufficient results to support this claim. There is no comparison with inference-time defense solutions. This paper emphasizes no inference-time overhead as an advantage, but does not set an inference-time defense baseline as a control.
Questions for authors
This also brings to mind an interesting point: since this paper focuses on making the safety boundary more accurate, and some jailbreak methods specifically exploit the imprecision of safety boundaries, shouldn't it be even more important to test whether this paper achieves better results under attacks from these methods?
I found one paper that could serve as a reference and there might be other similar works: https://aclanthology.org/2026.acl-long.1139/
4 comments
Response to Weaknesses and Questions (Part 2/2)
Q4. Is the inference-cost claim justified?
We correct the inference description in the submitted manuscript: the evaluated implementation generates a bounded structured annotation before the visible answer, using the same generator and decoding state. Only the answer is returned to the user. Construction agents, auxiliary critics, and the learned safety vector are absent at inference. The revised abstract, §3.2.2, and §4.3.1 state this explicitly. SafetyFlip requires no external safety-model calls, but the measured end-to-end latency increases from 2.14 to 2.54 seconds per query (18.7%). Appendix D.1 and the latency table report the timing protocol, including annotation decoding and defense calls.
The three-field annotation uses a fixed 96-token decoding cap to bound the internal prefix, separately from the common 512-token visible-answer budget.
Table R8. End-to-end latency. One A100-80GB, batch size 1, 200 fixed mixed requests, warm-up and three timed repetitions. All annotation tokens and defense calls are included.
| System | Seconds/query ↓ | Relative to Base |
|---|---|---|
| Base | 2.14 | 1.00× |
| SafetyFlip | 2.54 | 1.19× |
| Base + Llama Guard | 3.08 | 1.44× |
| Base + Intention Analysis | 4.05 | 1.89× |
| Base + GuardReasoner | 5.12 | 2.39× |
| SafetyFlip + GuardReasoner | 5.47 | 2.56× |
SafetyFlip adds 0.40 s/query (18.7%) over Base, compared with 43.9% for Base + Llama Guard and 139.3% for Base + GuardReasoner in this setting. Together, R5 and R8 show that SafetyFlip achieves lower ASR and OR than Base + GuardReasoner (6.6%/2.1% versus 8.6%/4.8%), at 2.54 versus 5.12 s/query. This supports an efficient integration of safety conditioning into the generator. These are end-to-end system measurements, including any differences in visible-answer length; the Llama Guard and Intention Analysis rows serve as timing references.
Q5. Does SafetyFlip help against attacks exploiting ambiguous boundaries?
We evaluated Jargon from the paper you recommended, Hung et al., Into the Gray Zone, with separate benign research requests matched by domain and discourse style.
Table R9. Jargon goal-level success and benign-research over-refusal.
| Method | Jargon goal-level ASR ↓ | Benign-research OR ↓ |
|---|---|---|
| Base | 95.7% | 3.3% |
| Safe-RLHF | 78.3% | 22.8% |
| Annotation KL-SFT | 68.3% | 10.7% |
| DPO | 66.0% | 6.2% |
| Annotation KL-SFT + DPO | 62.7% | 5.2% |
| Circuit Breakers | 69.3% | 12.0% |
| Base + GuardReasoner | 64.7% | 11.0% |
| SafetyFlip | 59.0% | 4.7% |
| SafetyFlip + GuardReasoner | 46.0% | 7.8% |
The attack uses 100 JailbreakBench goals and the published budget structure: three retries, two trials per retry, four rounds per trial and eight optimization variants. Systems share the attacker configuration, budget, initial memory conditions and goal ordering; search adapts to each target. Candidate queries count toward the budget. Three repetitions give 300 goal evaluations; the benign control has 200 requests per repetition.
Success means harmful assistance anywhere within the full interaction, including accumulated disclosure. Attacker-supplied text does not count. A separate final evaluator and blinded transcript audit check the attack-internal judge. Specifically, the blinded audit covers 180 complete transcripts, with 20 per method in R9, balanced between automatically judged successful and unsuccessful attacks. Two annotators review each transcript independently with method identities hidden; a third helps to adjudicate disagreements. Benign tasks are validated as safe before evaluation.
Against annotation KL-SFT, SafetyFlip reduces ASR from 68.3% to 59.0% and benign OR from 10.7% to 4.7%. The improvement therefore does not come from blanket refusal of research requests. The remaining 59.0% ASR is substantial; GuardReasoner lowers it to 46.0% with 7.8% benign OR.
Appendix B.1 requires a genuine change in the safety-critical factor. The revised Appendix B.1 explicitly states that academic or defensive framing alone does not justify a label flip. Auditing accepted training pairs tests that construction rule; auditing Jargon transcripts tests attack scoring. These are separate checks. Multi-turn counterfactual training remains future work.
Thanks again for your time and thoughtful feedback. If you have any additional concerns or suggestions, please feel free to share them, we would be happy to discuss further.
Sincerely,
Thank you for the additional experiments, which have addressed most of my concerns. Now I have no further suggestions and will maintain my score.
Thank you
Dear Reviewer iLtS,
We sincerely appreciate your positive feedback on our paper. We are also grateful for your constructive suggestions, which have helped us improve the manuscript.
Thank you once again for your support.
Sincerely,
Response to Weaknesses and Questions (Part 1/2)
Dear Reviewer iLtS,
Thanks for your constructive comments. Below we will address your concerns.
Q1. How does SafetyFlip compare and combine with external guardrails?
GuardReasoner-8B checks the input and output of each generator using the same block-and-refuse policy.
Table R5. External guardrail comparison.
Adding GuardReasoner to SafetyFlip reduces ASR from 6.6% to 3.5%, with OR increasing from 2.1% to 3.3%. This supports complementary use in the tested setting. R8 includes the cost of all guardrail calls.
Q2. What is the performance against individual attacks?
Table R6. ASR (%) by condition; DirectRequest is the non-adversarial control.
Each condition uses 200 behaviors and three repetitions. HB8 is computed before rounding. SafetyFlip improves over Base in every condition, but is not best in every column; PAIR and TAP remain harder cases. We report these results together with over-refusal and utility.
Q3. What supports generalization to unseen attacks?
Here, “unseen” refers to attack-generation procedures absent from data construction. Role-play and intent rewriting occur in training, so semantic disjointness is not established.
Beyond the submitted BeaverTails check, R7 uses an independent WildJailbreak subset with 500 harmful and 500 benign requests per repetition.
Table R7. External WildJailbreak evaluation.
Relative to sample-matched SFT, SafetyFlip lowers harmful ASR by 12.3 percentage points and benign OR by 5.4 points. Removing the role-play construction operator, with example and training budgets matched, gives HumanJailbreaks ASR of 4.0% versus 2.7% for the full method and 49.3% for Base; HB8 is 7.2%, 6.6% and 58.4%. This supports transfer beyond that explicit operator. Generalization across attacker-model families remains untested.