acceptodds
Under review as a conference paper at ICLR 2027

Dual-Adversarial Safety Alignment: Cultivating Intrinsic Threat Comprehension in LRMs

Abstract

Large reasoning models (LRMs) achieve remarkable success on complex tasks but remain vulnerable to harmful prompts that induce unsafe outputs. Recent methods align LRMs using direct refusals or safety rationales, yet often focus on prompt patterns rather than intrinsic attack mechanisms. As a result, these pattern-centric alignments struggle to generalize across diverse jailbreaks, compromising adversarial robustness and reasoning utility. We propose AdvSafe, a dual-adversarial framework that enables LRMs to internalize unsafety knowledge by explicitly deconstructing adversarial mechanisms. This moves beyond pattern-dependent traces, fostering robust cognitive defense without compromising reasoning utility. Our pipeline operates via a two-phase adversarial game. First, in adversarial synthesis, an autonomous agent dynamically crafts deceptive jailbreak prompts, adapting its strategies to breach a strong teacher model. Second, in adversarial extraction, the breached teacher executes a cognitive counter-attack. For every successful jailbreak, the teacher unmasks the camouflage, explaining why the attack succeeds and how such prompts can be identified and mitigated. This dual-adversarial process yields a compact reasoning dataset capturing rich, generalizable unsafety knowledge. Student models trained on this dataset implicitly acquire safety alignment through intrinsic threat comprehension. Experiments show that with only 1K synthesized samples, AdvSafe-aligned LRMs achieve significantly stronger jailbreak robustness than existing baselines, with almost no utility degradation. Furthermore, AdvSafe improves robustness against out-of-distribution prompts, demonstrating that learning unsafety knowledge enables a superior robustness-utility trade-off and generalizes beyond seen attack patterns.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, chain-of-thought

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Vq5cOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a dual-adversarial framework for LRMs to internalize the intrinsic knowledge of unsafety by getting teacher LLM's understanding of the attack into reasoning traces for the student to learn. The pipeline has three stages, including adversarial synthesis, adversarial extraction, and student model training. In stage 1, the attacker agent with a 17 strategy toolbox iteratively attacks a target model. In the second stage, a teacher receives each successful attacker and a scaffold prompt asking it to unmask the intent, extract the jailbreak technique, and derive a defense strategy. In stage 3, the student model are sfted on the samples from stages 2. Through experiments, the proposed dual attacking pipeline improve the model's safety across multiple harmful benchmarks, while maintaining the utility performance.

Critical strengths
  1. The idea of collecting high quality reasoning data from a powerful teacher model makes sense in terms of improving the student model's reasoning performance. This paper proposes the scaffold prompt to extract useful information from the teacher model.
  2. The experiments and ablations are informative. It analyzes trace components, and compare different data sizes and teacher choices. This gives the reader useful signals when selecting the right setup.
  3. The gain in ASR is consistent and significant. ASR drops airplay on all students across different model families, using only 1k high quality examples and LoRA training. The proposed approach is efficient and effective on training for safety of the model.
Critical weaknesses
  1. The novelty of the paper is limited. The prior work (e.g. WildTeaming) already use similar synthesizes adversarial jailbreaks at scale and trained on them for safety. For the stage 2, there are existing work who studies strategy-library red-teaming agents and reasoning based safety training (AutoDan-Turbo, PAIR, TAP). The novelty of the paper seems to me a combination of existing approaches, which is effective but limited.
  2. The authors show the performance of the trained model on utility benchmarks, but no over-refusal is reported with related benchmarks. The most important measurement for a model trained with safety data is the over-refusal rate on the boundary benign prompts. There is no result on this, which makes me hard to understand whether the model is really learning how to tell a harmful prompt, or just be skeptical with any prompt when it is on the boundary.
  3. The prompt for the teacher has privilege information, the model was told to analyze a jailbreak prompt. And this information was leaked into the traces the student learns to imitate. At inference time, the student see the prompt, but it was trained to presuppose that every input is a jailbreak to be refused. So the over-refusal behavior needs to be analyzed more carefully.
Questions for authors

1 Is there any result on over-refusal benchmarks, e.g. XSTest, OR-Bench, WildJailbreak benign adversarial. 2. How was the raw refusal variant trained? and why it doesn't reduce ASR that much?

6 comments

W1: The idea of collecting high quality reasoning data from a powerful teacher model makes sense in terms of improving the student model's reasoning performance. This paper proposes the scaffold prompt to extract useful information from the teacher model.

Response: Thank you for raising this concern. Agentic red-teaming and adversarial prompt synthesis are established techniques, and we do not claim them as standalone contributions. AutoDAN-Turbo, PAIR, and TAP primarily develop attack-generation methods, while WildTeaming demonstrates the value of large-scale adversarial safety data.

Our core contribution lies in the dual-adversarial coupling of successful-attack discovery and structured teacher-reasoning distillation. After discovering prompts that breach the teacher, we elicit the same teacher’s analysis of their harmful intent, bypass mechanisms, and defense principles. This transforms demonstrated vulnerabilities into explicit safety-reasoning supervision for student LRMs. The distinctive design concerns how successful attacks are converted into informative supervision.

We additionally evaluated UnsafeChain, Self-RedTeam, and MAGIC:

MethodHarmBench ↓AdvBench ↓WildJailbreak ↓StrongREJECT ↓Avg. ASR ↓GSM8K ↑
UnsafeChain42.757.8822.9017.2522.7030.48
Self-RedTeam29.500.0027.250.6414.3532.37
MAGIC18.500.1918.251.289.5627.22
AdvSafe14.251.924.406.076.6685.44

All values are percentages; Avg. ASR averages the four safety benchmarks. UnsafeChain uses our common SFT configuration, while Self-RedTeam and MAGIC use published checkpoints.

AdvSafe achieves the lowest average ASR and highest GSM8K accuracy in this comparison. Some baselines achieve stronger ASR on individual benchmarks, but their evaluated checkpoints show substantially lower GSM8K accuracy. These results support the practical value of distilling structured threat analyses: effective safety alignment with only 1,000 examples, alongside strong general reasoning performance. We will clarify this contribution and its relationship to prior work in the revised manuscript.

W2: The authors show the performance of the trained model on utility benchmarks, but no over-refusal is reported with related benchmarks. The most important measurement for a model trained with safety data is the over-refusal rate on the boundary benign prompts. There is no result on this, which makes me hard to understand whether the model is really learning how to tell a harmful prompt, or just be skeptical with any prompt when it is on the boundary.

Response: AdvSafe obtains an XSTest over-refusal rate of 25.20% using WildGuard, numerically below STAR-1’s published 38.00%, suggesting a potential advantage in mitigating over-refusal. Together with its lower average harmful ASR (6.66% vs. 46.86%) and higher GSM8K accuracy (85.44% vs. 70.66%), these results indicate a favorable balance between safety and utility.

W3: The prompt for the teacher has privilege information, the model was told to analyze a jailbreak prompt. And this information was leaked into the traces the student learns to imitate. At inference time, the student see the prompt, but it was trained to presuppose that every input is a jailbreak to be refused. So the over-refusal behavior needs to be analyzed more carefully.

Response: Thank you for raising this concern. We train the student on successful jailbreaks paired with the teacher’s structured analysis of harmful intent, bypass mechanisms, and defense principles, aiming to distill its ability to analyze deceptive harmful requests.

We acknowledge that this harmful-only supervision may encourage refusal on some benign requests. On XSTest using WildGuard, AdvSafe obtains an over-refusal rate of 25.20%, numerically below STAR-1’s published 38.00%, provided as an external reference. AdvSafe also achieves lower average harmful ASR (6.66% vs. 46.86%) and higher GSM8K accuracy (85.44% vs. 70.66%) in our main evaluation. Although over-refusal remains, AdvSafe’s rate is numerically lower than STAR-1’s published result, suggesting that training on successful jailbreaks can provide effective safety supervision without necessarily producing a higher refusal rate than existing safety-training approaches.

Q1: Is there any result on over-refusal benchmarks, e.g. XSTest, OR-Bench, WildJailbreak benign adversarial.

Response: Yes. We additionally evaluated AdvSafe on XSTest using WildGuard, obtaining an over-refusal rate of 25.20%. Please see our responses to W2 and W3 for the detailed analysis and discussion.

Q2: How was the raw refusal variant trained? and why it doesn't reduce ASR that much?

Response: The Raw Refusal variant is trained on jailbreak prompts paired with a fixed refusal response (“I'm sorry, but I can't help with that request.”), without the teacher’s structured reasoning. Its limited ASR reduction suggests that learning a refusal output alone provides insufficient supervision for recognizing harmful intent concealed by diverse attack strategies. In contrast, AdvSafe supplies explicit analysis of harmful intent, bypass mechanisms, and defense principles, offering richer signals for generalization to unseen jailbreaks. The performance gap supports the value of this structured reasoning supervision beyond refusal-only training.

AuthorsOpenReview ↗

Follow-up on rebuttal

Dear Reviewer Vq5c,

Thank you again for your valuable comments! We have carefully considered your comments and have provided our responses.

Please let us know if our replies have satisfactorily addressed your concerns. Please do not hesitate to let us know if you have any further questions or if you require any additional clarification.

Thank you very much!

Reviewer_EkjPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ADVSAFE, a two-phase adversarial framework for safety alignment of Large Reasoning Models (LRMs):

  1. Adversarial Synthesis Phase: An automated agent generates jailbreak prompts and adapts its strategies to bypass the safety guardrails of a teacher model.
  2. Adversarial Extraction Phase: For each prompt that successfully breaches the teacher model, the teacher model is prompted to generate a chain-of-thought (CoT) reasoning trace that identifies the underlying unsafe intent, analyzes the attack mechanism, formulates mitigation principles, and concludes with a refusal.

Using this two-phase pipeline, the authors construct a 1,000-sample safety dataset containing threat-deconstruction reasoning traces, and use it to fine-tune student LRMs. According to the reported results, fine-tuning on this dataset reduces the Attack Success Rate (ASR) by a factor of 4x to 10x compared to five baseline alignment methods while maintaining general reasoning task performance and generalizing to unseen jailbreak formats.

Critical strengths
  • Problem Formulation: The work distinguishes between output-level behavior imitation (mapping harmful prompt patterns directly to refusals) and explicit reasoning over attack mechanisms (analyzing how a jailbreak disguises unsafe intent).
  • Pipeline Design: The data generation framework couples automated red-teaming (generating attacks that breach the teacher model) with analytical trace extraction from the same teacher model on its own failure cases.
Critical weaknesses
  • W1: Novelty of the adversarial alignment should be clarified: Adversarial alignment for LLMs has been broadly researched. Earlier works introduced game-theoretic adversarial frameworks, typically formulated as a zero-sum game between an attacker and a defender trained via SFT or RL, such as safety-selfplay [1], MAGIC [2], and TriPlay-RL [3]. Subsequent studies have further incorporated auto-research mechanisms to expand the search space of both attack and defense surfaces, thereby improving alignment effectiveness [4]. The current manuscript does not discuss these prior works or clarify its specific novelty and contributions relative to this established line of research.
  • W2: Experimental validation needs to be more comprehensive:
    1. W2.1 More baselines should be included: The authors should include standard RL-based alignment baselines (such as naive GRPO alignment) as well as the adversarial alignment methods discussed in W1 [1, 2, 3, 4] for a fair and complete comparison.
    2. W2.2 Overrefusal ratio should be reported: Reporting the overrefusal ratio is necessary to evaluate whether the model overfits to harmful jailbreak patterns and becomes overly sensitive to benign inputs. The authors should conduct overrefusal evaluations on standard benchmarks, specifically the WildJailbreak adversarial benign split [5] and XSTest [6].

References

[1] Liu, Mickel, et al. "Chasing moving targets with online self-play reinforcement learning for safer language models." arXiv preprint arXiv:2506.07468 (2025). [2] Wen, Xiaoyu, et al. "MAGIC: A co-evolving attacker-defender adversarial game for robust LLM safety." arXiv preprint arXiv:2602.01539 (2026). [3] Tan, Zhewen, et al. "TriPlay-RL: Tri-Role Self-Play Reinforcement Learning for LLM Safety Alignment." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026. [4] A3: An Automated Alignment Agent for Safety Finetuning. https://alignment.anthropic.com/2026/automated-alignment-agent/ [5] Jiang, Liwei, et al. "Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models." Advances in Neural Information Processing Systems 37 (2024): 47094-47165. [6] Röttger, Paul, et al. "Xstest: A test suite for identifying exaggerated safety behaviours in large language models." Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Questions for authors

refer to weakness

6 comments

Thank you for your thorough comments.

W1: Novelty of the adversarial alignment should be clarified

Response: Thank you for highlighting these relevant works. Self-RedTeam, MAGIC, and TriPlay-RL emphasize continuous co-evolution through reinforcement learning, while A3 automates safety-data generation and fine-tuning decisions. AdvSafe contributes a dual-adversarial supervision pipeline: adversarial synthesis discovers prompts that successfully breach the teacher, and adversarial extraction then elicits the same teacher’s reasoning about their harmful intent, bypass mechanisms, and defense principles. These structured analyses become supervision for student LRMs through lightweight SFT.

The distinctive contribution is converting teacher failure cases into explicit, mechanism-oriented reasoning supervision. This coupling uses adversarial search to identify informative vulnerabilities and leverages the teacher’s reasoning ability to explain how to recognize and counter them. Our experiments demonstrate that this recipe enables substantial safety improvements with only 1,000 examples, while preserving strong general reasoning performance.

We have also added baseline comparisons and reported over-refusal results, as detailed in our response to W2.

W2.1:

Thank you for suggesting these additional comparisons. Our study focuses on SFT-based safety alignment, so we did not conduct a separate naive GRPO training experiment. To broaden the evaluation, we added Self-RedTeam and MAGIC following your suggestions, together with UnsafeChain, suggested by Reviewer yYy6.

MethodHarmBench ↓AdvBench ↓WildJailbreak ↓StrongREJECT ↓Avg. ASR ↓GSM8K ↑
UnsafeChain42.757.8822.9017.2522.7030.48
Self-RedTeam29.500.0027.250.6414.3532.37
MAGIC18.500.1918.251.289.5627.22
AdvSafe14.251.924.406.076.6685.44

All values are percentages; Avg. ASR is the arithmetic mean across the four safety benchmarks. UnsafeChain uses our common SFT configuration, while Self-RedTeam and MAGIC use their published checkpoints.

AdvSafe achieves the lowest average ASR and highest GSM8K accuracy. Self-RedTeam and MAGIC obtain strong ASR results on individual benchmarks, but their evaluated checkpoints show substantially lower GSM8K accuracy, highlighting AdvSafe’s favorable balance between safety and general reasoning.

Our contribution is the dual-adversarial pipeline, which couples successful-jailbreak discovery with the teacher’s structured analysis of harmful intent, bypass mechanisms, and defense principles. Distilling these analyses enables effective safety alignment with only 1,000 examples while preserving strong reasoning performance.

W2.2:

AdvSafe obtains an XSTest over-refusal rate of 25.20% using WildGuard, numerically below STAR-1’s published 38.00%, suggesting a potential advantage in mitigating over-refusal. Together with its lower average harmful ASR (6.66% vs. 46.86%) and higher GSM8K accuracy (85.44% vs. 70.66%), these results indicate a favorable balance between safety and utility.

AuthorsOpenReview ↗

Follow-up on rebuttal

Dear Reviewer EkjP,

Thank you again for your valuable comments! We have carefully considered your comments and have provided our responses.

Please let us know if our replies have satisfactorily addressed your concerns. Please do not hesitate to let us know if you have any further questions or if you require any additional clarification.

Thank you very much!

AuthorsOpenReview ↗

Further follow-up on rebuttal

Dear Reviewer EkjP,

Thank you very much for your time and helpful comments. We have addressed your concerns in our rebuttal. As the discussion period is drawing to a close, we would appreciate any further comments or suggestions you may have on our responses.

Thank you again for your time and consideration.

The Authors

Reviewer_yYy6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AdvSafe, a safety alignment pipeline for large reasoning models. An agent first synthesizes jailbreak prompts that succeed against a teacher model. The teacher is then prompted to explain the harmful intent, bypass technique, and defense rationale, and to provide a refusal. Student models are supervised on 1,000 resulting examples. The evaluation covers four safety benchmarks, five reasoning benchmarks, eight student models, and additional attack methods. On DeepSeek-R1-Distill-Qwen-7B, the reported average ASR decreases from 66.70% to 6.66%, while average utility changes from 64.73% to 65.17%. Ablations examine adversarial prompt construction, trace components, dataset size, and teacher choice. The results suggest that structured safety traces from a strong teacher are useful, although the adaptive robustness and intrinsic-comprehension interpretations require stronger evidence.

Critical strengths
  1. A practical and empirically promising training recipe. Table 1 reports a reduction in average ASR to 6.66%, compared with 27.41% for DirectRefusal and 31.33% for ThinkSafe, using a 1K-example budget. Average utility is approximately preserved on the default student. Table 2 reports safety improvements across eight student models. These results make the method potentially useful beyond a single checkpoint, even though utility preservation is not uniform.

  2. Relevant controlled ablations support the usefulness of the supervision. Table 5 holds the successful prompts, student initialization, sample count, and optimization setup fixed while changing supervision. Raw refusal gives 62.64% average ASR, whereas the full trace gives 6.66%; removing individual components gives substantially worse results than the full method. Figure 3 additionally compares raw seeds with agentically constructed prompts. These experiments provide meaningful support for the proposed data construction and supervision design.

Critical weaknesses
  1. W1 — The strongest robustness claims are not established by the documented attack protocol. Appendix C.3, lines 1066–1070, states that attacks are constructed against DeepSeek-R1-Distill-Qwen-7B and then evaluated on other models by transfer. It is unclear whether Table 3 and Figure 2 re-optimize attacks against each aligned checkpoint or reuse attacks optimized against the base model. Transfer resistance does not establish resistance to an attacker adapting to the defense. PAIR and TAP also restrict target responses during search to 150 tokens, which may give incomplete feedback for reasoning models. The different mean attack-turn counts in Table 3 need reconciliation with the transfer description. Separately, the label audit in Table 9 contains 62 Llama-Guard-unsafe responses out of 400, implying 15.50% ASR rather than the 14.25% reported in Tables 1 and 8. The transfer results remain useful evidence, but the threat model should be explicit. Supporting the stronger interpretation requires adaptive evaluation; alternatively, the claims should be restricted to the tested transfer setting. The response-level audit also needs to be reconciled.

  2. W2 — Substantial overlap with prior work is not addressed through direct comparison. SCoT [1] already combines adversarial prompt construction, structured intent and safety analysis, and supervised fine-tuning. UnsafeChain [2] selects prompts that trigger unsafe outputs and constructs corrected reasoning supervision, while AdvChain [3] trains on adversarial reasoning traces with explicit danger identification and correction. None is discussed or compared against. Although AdvSafe introduces a specific successful-attack mining procedure and intent/bypass/defense decomposition, the paper does not establish their incremental value over these closely related approaches. Direct comparisons under matched conditions are needed to substantiate the claimed contribution.

References

[1] Xianglin Yang, Gelei Deng, Jieming Shi, Tianwei Zhang, and Jin Song Dong. Enhancing Model Defense Against Jailbreaks with Proactive Safety Reasoning. arXiv:2501.19180, 2025.

[2] Raj Vardhan Tomar, Preslav Nakov, and Yuxia Wang. UnsafeChain: Enhancing Reasoning Model Safety via Hard Cases. arXiv:2507.21652, 2025.

[3] Zihao Zhu, Xinyu Wu, Gehan Hu, Siwei Lyu, Ke Xu, and Baoyuan Wu. AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models. ICLR, 2026.

Questions for authors
  1. Evaluation clarification and validation. For each Table 3 attack and Figure 2 baseline, which exact checkpoint was attacked during optimization? If these are transfer attacks, can you provide adaptive attacks optimized separately against AdvSafe and the strongest baselines under matched budgets, with enough target generation to observe their responses? Please also define how mean attack turns are calculated and how failures enter that statistic.

  2. Audit clarification. Why does Table 9 imply 62/400 = 15.50% ASR while Tables 1 and 8 report 14.25%? Are the checkpoint, prompts, decoding, and judge identical? Please reconcile these results and the 252 unsafe-to-safe pairs in Appendix G using a single aligned set of response identifiers and labels.

  3. Discriminative validation. How does AdvSafe perform on benign requests matched to the attack framing and on harmless requests involving sensitive topics? A comparison of benign refusal rates and harmful ASR against the strongest baselines would help determine whether the improvement reflects selective safety rather than increased refusal of these contexts.

  4. Mechanistic clarification. How exactly are the Table 5 component-removed traces constructed, and do the remaining traces still contain the omitted information implicitly? What attribution method, layers, aggregation, and normalization produce Figures 6–7? Please distinguish the evidence for a useful supervision recipe from the stronger claim of intrinsic threat comprehension.

7 comments

Thank you for your thorough comments.

W1: The strongest robustness claims are not established by the documented attack protocol.

  1. Our in-house agentic attack in Table 3 is adaptive: we conduct separate attack runs against the base and AdvSafe-aligned checkpoints. During each run, the attacker directly queries the checkpoint being evaluated and uses its responses and the evaluator’s feedback to revise subsequent attacks. Under this adaptive evaluation, ASR decreases from 72.75% to 18.00%. The other attack comparisons use transferred prompts rather than attacks separately re-optimized against each aligned checkpoint. Our adaptive evidence therefore applies specifically to the in-house agentic attacker; the other results establish resistance under the evaluated transfer protocol.

  2. We acknowledge that the 150-token response limit may provide incomplete feedback for reasoning models. However, we use the same transferred attack prompts across checkpoints for a controlled comparison, under which AdvSafe still demonstrates improved robustness.

  3. The reported means of 4.86 and 9.08 come from separate adaptive agentic runs. Each run has a budget of 10 agent iterations, and the average includes all attack instances, with unsuccessful runs assigned a value of 10. We chose this cap because adaptive searches against AdvSafe could be lengthy; applying the same cap to both models keeps evaluation time and computational cost manageable. A turn denotes one attacker-agent iteration, which may involve strategy selection, a target query, or a formatting retry. Thus, this statistic is a budget-capped average over all attempts, rather than the number of target queries or the mean turns among successful attacks only.

  4. Table 9 reports an earlier evaluation run used for judge selection and agreement analysis, whereas Tables 1 and 8 report a subsequent main evaluation later on. The reviewer is correct that Table 9 corresponds to 62/400 = 15.50% ASR, while the main evaluation reports 57/400 = 14.25%. We apologize for failing to distinguish these runs clearly. We will identify their provenance explicitly and tie the audit labels and paired-response analyses consistently to the earlier set of response identifiers. The difference is 1.25 percentage points, and both values remain below the best baseline HarmBench ASR reported in Table 1 (44.25%). This numerical comparison does not resolve the reporting inconsistency.

W2: Substantial overlap with prior work is not addressed through direct comparison

Response: Thank you for pointing out these closely related works and suggesting additional comparisons. We added UnsafeChain following your suggestion, together with Self-RedTeam and MAGIC, suggested by Reviewer EkjP.

MethodHarmBench ↓AdvBench ↓WildJailbreak ↓StrongREJECT ↓Avg. ASR ↓GSM8K ↑
UnsafeChain42.757.8822.9017.2522.7030.48
Self-RedTeam29.500.0027.250.6414.3532.37
MAGIC18.500.1918.251.289.5627.22
AdvSafe14.251.924.406.076.6685.44

All values are percentages; Avg. ASR is the arithmetic mean across the four safety benchmarks. UnsafeChain uses our common SFT configuration, while Self-RedTeam and MAGIC use their published checkpoints with different base models. AdvSafe achieves the lowest average ASR and highest GSM8K accuracy in this comparison.

Our key contribution is the dual-adversarial pipeline: we first discover successful jailbreaks, then elicit the same teacher’s reasoning to analyze their harmful intent, bypass mechanisms, and defense principles. Distilling these analyses provides effective safety supervision while preserving strong general reasoning performance.

Specifically, SCoT emphasizes proactive intent and safety analysis, UnsafeChain constructs corrected reasoning for prompts eliciting unsafe outputs, and AdvChain teaches recovery from deliberately introduced reasoning deviations. AdvSafe connects successful-attack discovery with the teacher’s explanation of why those attacks work and how to counter them. The added UnsafeChain comparison supports the effectiveness of this overall recipe; we will expand the related-work discussion to clarify these distinctions.

Q1: Evaluation clarification and validation.

Response: Thank you for requesting this clarification. Our in-house agentic evaluation is adaptive: we separately attack DeepSeek-R1-Distill-Qwen-7B and its DirectRefusal- and AdvSafe-aligned counterparts, using feedback from each evaluated checkpoint to revise subsequent attacks. Under the same 10-iteration agent budget, ASR is 72.75% for Base, 40.75% for DirectRefusal, and 18.00% for AdvSafe, with reported logged turn statistics of 4.86, 8.42, and 9.08, respectively. A turn denotes an attacker-agent control-loop iteration, which may involve strategy selection, a target query, or a formatting retry; it therefore does not equal one target-model query. The remaining attack comparisons use transferred prompts rather than separate optimization against each aligned checkpoint. We acknowledge that the 150-token PAIR/TAP search-response limit may provide incomplete feedback for reasoning models. Although the shared transferred prompts support controlled comparisons within that setting, we will explicitly document this limitation and distinguish those results from the adaptive evidence provided by our in-house agentic evaluation.

Q2: Audit clarification.

Response: As clarified in W1, the first run was conducted for judge-model selection, and the second was repeated for the main results table later on, with the model and attack protocol unchanged. The ASRs differ by only 1.25 percentage points (15.50% versus 14.25%), preserving the overall trend; we apologize for our oversight in not consistently distinguishing and reporting these runs.

Q3: Discriminative validation.

Response: AdvSafe obtains an XSTest over-refusal rate of 25.20% using WildGuard, numerically below STAR-1’s published 38.00%, suggesting a potential advantage in mitigating over-refusal. Together with its lower average harmful ASR (6.66% vs. 46.86%) and higher GSM8K accuracy (85.44% vs. 70.66%), these results indicate a favorable balance between safety and utility.

Q4: Mechanistic clarification.

Response: The Table 5 ablations are performed at the prompt level: we remove the corresponding instruction from the teacher’s extraction prompt and regenerate the reasoning trace and refusal. This avoids disrupting coherence through direct passage deletion.

Figures 6–7 use token-level Integrated Gradients through Inseq, with 100 integration steps, an internal batch size of 10, and a 15-token generation limit. Columns represent selected prompt tokens and rows represent generated tokens; darker colors indicate larger attribution scores within each panel.

Together, the ablations and attribution analysis provide complementary evidence: the former quantify the benefit of structured threat-analysis supervision, while the latter help characterize how input cues contribute to generation. This supports our design rationale and offers evidence relevant to threat comprehension.

Thank you for the rebuttal and the clarification of the adaptive evaluation. I am maintaining my rating. My main concern is methodological novelty. SCoT already combines adversarial prompt construction with structured safety reasoning and SFT, while UnsafeChain already uses observed failures to build corrective supervision. AdvSafe’s main additions appear to be teacher-targeted jailbreak search and explicit bypass analysis. The manuscript needs a direct comparison of these design choices with the closest prior work to establish the extent of the contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.