- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether safety-trained open-weight LLMs remain robust when attackers jointly remove multiple refusal directions rather than a single one. The authors first propose Refusal Subspace Ablation (RSA), an efficient adaptive attack that iteratively selects complementary directions evaluated under joint ablation, exposing vulnerabilities missed by single-direction tests. Building on this, they introduce Refusal Subspace Ablation Training (ReSAT), which combines mixed supervised fine-tuning with iterative safety-training rounds that supervise safe refusals under cumulative subspace removal, refreshing candidate directions between rounds. Evaluations on three model families show that ReSAT achieves lower attack success rates than most baselines under various ablation and jailbreak attacks while largely preserving general capability, with mechanism analysis confirming the value of retaining direction history.
Critical strengths
S1. Clear presentation throughout. The paper is easy to follow from motivation through method design to experiments, with each section logically building on the previous one. A minor suggestion: the visual polish of tables and figures could be improved.
S2. A relatively complete research. The paper starts from a concrete gap in existing work (single-direction robustness does not imply subspace robustness), proposes an attack (RSA) that exposes this gap, then develops a defense (ReSAT) and validates it with thorough experiments and mechanism analysis.
Critical weaknesses
W1. The safety improvement may stem from increased sensitivity rather than genuine robustness. On XSTest (Table 2), all three ReSAT-trained models exhibit over-refusal rates near 50%, indicating substantial exaggerated safety behavior on benign prompts. This suggests the defense may work partly by making the model more broadly cautious rather than by genuinely preserving refusal under adversarial ablation.
W2. Evaluation is limited to non-thinking models. All three evaluated backbones (Llama-3-8B, Mistral-7B, Qwen2.5-14B) are standard instruction-tuned models. Given that reasoning/thinking models have become mainstream, it remains unclear whether ReSAT's defense transfers to settings where refusal behavior is mediated through long internal reasoning traces rather than direct first-token responses.
W3. Defense underperforms baselines in some settings. For example, on Mistral under SOM attack, ReSAT achieves 28.57% ASR compared to MixAT's 4.40%.
Questions for authors
See weakness1 & 2
3 comments
Thanks to the authors for the reply, which addressed some of my concerns, especially weakness 2. Overall I'm inclined to accept this paper; there are still a few minor issues, but I'll keep my current score.
Thank you very much for taking the time to read our rebuttal and for continuing to engage in the discussion. We are pleased that we have addressed all of your major concerns. If there are any remaining questions or concerns, we would be happy to address them further.
W1 Response. The trade-off should also be compared with safety-training baselines.** Thank you for providing this comment. Balancing safety and helpfulness is a long-standing challenge in alignment, as discussed by Bai et al. (2022) and the XSTest study of exaggerated safety by Röttger et al. (2024). Huang et al. (2025) further document a “safety tax” in Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable: in their studied sequential training pipeline, safety alignment restores safety but degrades reasoning capability, which is a longlasting problems and some other baselines are even worse than us for over-refusal and worse than our safety as well.
Thank you for providing this comment. Table 2 in the paper reports XSTest benign compliance (higher is better). On Llama-3-8B, ReSAT achieves the highest benign compliance among the compared safety-training baselines: 53%, compared with 48% for DeepRefusal, 49% for MixAT, and 26% for CAT. At the same time, ReSAT achieves substantially lower RSA endpoint HarmBench ASR: 11.20% versus 23.44%, 82.03%, and 45.31%, respectively (submitted Tables 1–2). Thus, on Llama-3-8B, ReSAT improves refusal-ablation robustness without exhibiting worse over-refusal than the compared baselines. On Mistral-7B, the trade-off also remains competitive: ReSAT retains 57% XSTest compliance, compared with MixAT’s 62%, while reducing RSA endpoint ASR from 65.62% to 20.00%. ReSAT additionally achieves higher MMLU (59.22% versus 52.93%) and GSM8K (42.99% versus 34.27%). These results demonstrate a favorable safety–utility trade-off rather than uniformly worse utility or safety relative to existing baselines.
W2 Response. We thank the reviewer for making this commnet. Our original comparison evaluated ReSAT and the defense baselines on standard instruction-tuned backbones. To test whether the defense transfers to reasoning models, we also trained and evaluated ReSAT on Qwen3-8B with native thinking enabled. Prior defense studies used a setting similar to our original comparison. Bui et al. (AGOPNs, Findings of IJCNLP-AACL 2026) evaluated their defense on three Llama Instruct checkpoints, and Shen et al. (Safety Instincts, ICLR 2026) evaluated SIRL on Llama and Qwen2.5 Instruct models. Xie et al. (Findings of EMNLP 2025) also evaluated DeepRefusal, one of our baselines, on instruction-tuned models.
Using our RSA to attack thinking mode, adaptive RSA raised HarmBench ASR from 9.6% to 88.8% (Table 8). Lexical refusal rate fell from 68.8% to 14.4%, and MMLU accuracy went from 79.75% to 78.25%.
Table 8: RSA on Qwen3-8B with native thinking enabled, before ReSAT training.
Using our ReSAT safe training for thinking mode. With native thinking enabled, HarmBench ASR was 11.20% without ablation and 8.00% under both Single and cumulative RSA (Table 9). The ReSAT trained thinking model resisted both refusal ablations in this evaluation. We also observed XSTest compliance was 100.00%, no overrefusal on the benign prompts evaluated.
Table 9: ReSAT on Qwen3-8B with native thinking reasoning enabled. Clean, Single, RSA (ours), and HumanJB report HarmBench ASR. XSTest reports benign compliance and MMLU reports accuracy.
W3 Response. We agree that MixAT provides stronger protection against SOM on Mistral-7B. Its HarmBench ASR under SOM is 4.40%, compared with 28.57% for ReSAT. Under RSA, the ordering reverses. MixAT reaches 65.62% ASR, compared with 20.00% for ReSAT. Also, for model in deployment, an attacker always choose the method that works best against that model. We therefore also compare the highest ASR observed for each defense across the four refusal-ablation attacks in Table 1 of the submission (Single, Geometry, SOM, and RSA). MixAT's highest ASR is 65.62% under RSA. ReSAT's highest is 28.57% under SOM, a difference of 37.05 percentage points in ReSAT's favor.