- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a dual-adversarial framework for LRMs to internalize the intrinsic knowledge of unsafety by getting teacher LLM's understanding of the attack into reasoning traces for the student to learn. The pipeline has three stages, including adversarial synthesis, adversarial extraction, and student model training. In stage 1, the attacker agent with a 17 strategy toolbox iteratively attacks a target model. In the second stage, a teacher receives each successful attacker and a scaffold prompt asking it to unmask the intent, extract the jailbreak technique, and derive a defense strategy. In stage 3, the student model are sfted on the samples from stages 2. Through experiments, the proposed dual attacking pipeline improve the model's safety across multiple harmful benchmarks, while maintaining the utility performance.
Critical strengths
- The idea of collecting high quality reasoning data from a powerful teacher model makes sense in terms of improving the student model's reasoning performance. This paper proposes the scaffold prompt to extract useful information from the teacher model.
- The experiments and ablations are informative. It analyzes trace components, and compare different data sizes and teacher choices. This gives the reader useful signals when selecting the right setup.
- The gain in ASR is consistent and significant. ASR drops airplay on all students across different model families, using only 1k high quality examples and LoRA training. The proposed approach is efficient and effective on training for safety of the model.
Critical weaknesses
- The novelty of the paper is limited. The prior work (e.g. WildTeaming) already use similar synthesizes adversarial jailbreaks at scale and trained on them for safety. For the stage 2, there are existing work who studies strategy-library red-teaming agents and reasoning based safety training (AutoDan-Turbo, PAIR, TAP). The novelty of the paper seems to me a combination of existing approaches, which is effective but limited.
- The authors show the performance of the trained model on utility benchmarks, but no over-refusal is reported with related benchmarks. The most important measurement for a model trained with safety data is the over-refusal rate on the boundary benign prompts. There is no result on this, which makes me hard to understand whether the model is really learning how to tell a harmful prompt, or just be skeptical with any prompt when it is on the boundary.
- The prompt for the teacher has privilege information, the model was told to analyze a jailbreak prompt. And this information was leaked into the traces the student learns to imitate. At inference time, the student see the prompt, but it was trained to presuppose that every input is a jailbreak to be refused. So the over-refusal behavior needs to be analyzed more carefully.
Questions for authors
1 Is there any result on over-refusal benchmarks, e.g. XSTest, OR-Bench, WildJailbreak benign adversarial. 2. How was the raw refusal variant trained? and why it doesn't reduce ASR that much?
6 comments
Q1: Is there any result on over-refusal benchmarks, e.g. XSTest, OR-Bench, WildJailbreak benign adversarial.
Response: Yes. We additionally evaluated AdvSafe on XSTest using WildGuard, obtaining an over-refusal rate of 25.20%. Please see our responses to W2 and W3 for the detailed analysis and discussion.
Q2: How was the raw refusal variant trained? and why it doesn't reduce ASR that much?
Response: The Raw Refusal variant is trained on jailbreak prompts paired with a fixed refusal response (“I'm sorry, but I can't help with that request.”), without the teacher’s structured reasoning. Its limited ASR reduction suggests that learning a refusal output alone provides insufficient supervision for recognizing harmful intent concealed by diverse attack strategies. In contrast, AdvSafe supplies explicit analysis of harmful intent, bypass mechanisms, and defense principles, offering richer signals for generalization to unseen jailbreaks. The performance gap supports the value of this structured reasoning supervision beyond refusal-only training.
Follow-up on rebuttal
Dear Reviewer Vq5c,
Thank you again for your valuable comments! We have carefully considered your comments and have provided our responses.
Please let us know if our replies have satisfactorily addressed your concerns. Please do not hesitate to let us know if you have any further questions or if you require any additional clarification.
Thank you very much!
W1: The idea of collecting high quality reasoning data from a powerful teacher model makes sense in terms of improving the student model's reasoning performance. This paper proposes the scaffold prompt to extract useful information from the teacher model.
Response: Thank you for raising this concern. Agentic red-teaming and adversarial prompt synthesis are established techniques, and we do not claim them as standalone contributions. AutoDAN-Turbo, PAIR, and TAP primarily develop attack-generation methods, while WildTeaming demonstrates the value of large-scale adversarial safety data.
Our core contribution lies in the dual-adversarial coupling of successful-attack discovery and structured teacher-reasoning distillation. After discovering prompts that breach the teacher, we elicit the same teacher’s analysis of their harmful intent, bypass mechanisms, and defense principles. This transforms demonstrated vulnerabilities into explicit safety-reasoning supervision for student LRMs. The distinctive design concerns how successful attacks are converted into informative supervision.
We additionally evaluated UnsafeChain, Self-RedTeam, and MAGIC:
All values are percentages; Avg. ASR averages the four safety benchmarks. UnsafeChain uses our common SFT configuration, while Self-RedTeam and MAGIC use published checkpoints.
AdvSafe achieves the lowest average ASR and highest GSM8K accuracy in this comparison. Some baselines achieve stronger ASR on individual benchmarks, but their evaluated checkpoints show substantially lower GSM8K accuracy. These results support the practical value of distilling structured threat analyses: effective safety alignment with only 1,000 examples, alongside strong general reasoning performance. We will clarify this contribution and its relationship to prior work in the revised manuscript.
W2: The authors show the performance of the trained model on utility benchmarks, but no over-refusal is reported with related benchmarks. The most important measurement for a model trained with safety data is the over-refusal rate on the boundary benign prompts. There is no result on this, which makes me hard to understand whether the model is really learning how to tell a harmful prompt, or just be skeptical with any prompt when it is on the boundary.
Response: AdvSafe obtains an XSTest over-refusal rate of 25.20% using WildGuard, numerically below STAR-1’s published 38.00%, suggesting a potential advantage in mitigating over-refusal. Together with its lower average harmful ASR (6.66% vs. 46.86%) and higher GSM8K accuracy (85.44% vs. 70.66%), these results indicate a favorable balance between safety and utility.
W3: The prompt for the teacher has privilege information, the model was told to analyze a jailbreak prompt. And this information was leaked into the traces the student learns to imitate. At inference time, the student see the prompt, but it was trained to presuppose that every input is a jailbreak to be refused. So the over-refusal behavior needs to be analyzed more carefully.
Response: Thank you for raising this concern. We train the student on successful jailbreaks paired with the teacher’s structured analysis of harmful intent, bypass mechanisms, and defense principles, aiming to distill its ability to analyze deceptive harmful requests.
We acknowledge that this harmful-only supervision may encourage refusal on some benign requests. On XSTest using WildGuard, AdvSafe obtains an over-refusal rate of 25.20%, numerically below STAR-1’s published 38.00%, provided as an external reference. AdvSafe also achieves lower average harmful ASR (6.66% vs. 46.86%) and higher GSM8K accuracy (85.44% vs. 70.66%) in our main evaluation. Although over-refusal remains, AdvSafe’s rate is numerically lower than STAR-1’s published result, suggesting that training on successful jailbreaks can provide effective safety supervision without necessarily producing a higher refusal rate than existing safety-training approaches.