- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a defense against abliteration attacks on open-weight LLMs. The main idea is to train models so that removing their refusal directions either preserves refusal or causes the model to collapse, rather than producing a capable, non-refusing model.
The authors introduce a training objective with three terms: standard SFT, refusal preservation under partial ablation, and EOS-token prediction under stronger ablation. During training, they simulate ablation over randomly sampled directions, layers, and strengths.
They evaluate the defense on eight models across the Llama, Qwen, and Gemma families. Standard abliteration achieves 0.00 attack success rate across all defended models, although adaptive attacks remain effective on some models, particularly Qwen-2.5-72B.
Critical strengths
-
New/Interesting approach to abliteration defense: I think the idea of making strong abliteration cause model collapse rather than simply trying to preserve refusal is interesting. The distinction between maintaining refusal under weaker interventions and deliberately inducing collapse under stronger interventions is a potentially useful defense objective.
-
Evaluation across multiple models and attack settings: The authors evaluate eight models from three families, ranging from 7B to 72B parameters. I like that they include adaptive attacks and report cases where the defense fails, particularly Qwen-2.5-72B. The observation that some models collapse under abliteration while others retain their refusal behavior is also interesting.
Critical weaknesses
-
Limited novelty relative to existing abliteration defenses. My main concern is that the proposed approach seems closely related to existing work, particularly ReFAT (Yu et al., ICLR 2025) and Abliteration-Resistant Tuning (ART; Kuo et al., 2026). Both already study training models against simulated refusal-feature ablation.
The main difference appears to be the additional collapse objective, which trains models to emit EOS after sufficiently strong ablation, along with the use of randomized refusal subspaces. I think this is a reasonable extension, but the paper does not establish how much of the robustness comes from these additions versus the previously studied adversarial-training approach.
I also think Decoy Direction Optimization (DDO; Muhamed et al., 2026) is an important related work that is missing. DDO addresses the same refusal-direction ablation threat model, but takes a different approach by modifying the attacker's estimated refusal direction rather than training the model to collapse under ablation.
I would particularly like to see comparisons against ReFAT, DDO and ART, as well as an ablation removing the collapse loss while keeping the refusal-preservation loss. Without these comparisons, it is difficult to determine the contribution of the proposed objective.
-
The attack evaluation does not adequately show robustness against adaptive abliteration. I also have a more substantial concern about the evaluation protocol. Appendix H states that the defended models are evaluated using refusal-only attack selection, whereas the undefended controls use stronger fluency-gated selection. This seems important given that Section 5.3 demonstrates that attack selection alone can change measured ASR from 0.00 to 0.56 on an undefended model.
This confounder makes the headline reduction in ASR difficult to interpret, since the defended and undefended models are not evaluated under the same attack-selection procedure.
More generally, the adaptive attack only searches over a restricted family of ablation configurations and is described as a simplified version of Heretic. Since the attacker has full white-box access, it could also try to find interventions that avoid triggering the learned collapse response. It is unclear whether the defense would remain effective against such attacks.
The reported adaptive results already show limitations: Qwen-2.5-72B reaches 0.77 ASR, while Qwen-2.5-7B and 32B remain partially vulnerable. The defense also has a substantial clean capability cost on Gemma-1-7B, where LAMBADA accuracy drops from 0.62 to 0.33. These findings suggest that neither robustness nor capability preservation is consistent across models.
Questions for authors
-
On novelty and baselines: How does the proposed approach compare with ReFAT and ART, which already train models against simulated refusal-feature ablation? Could the authors include a baseline that only uses SFT and the refusal-preservation objective, without the collapse term? It would be nice to include additional baselines like DDO as well.
-
On attack evaluation: Why are defended models evaluated with refusal-only attack selection while undefended controls use fluency-gated selection? Given the large differences reported in Section 5.3, could the authors rerun the defended evaluations with the stronger, matched attack-selection procedure?
-
On adaptive attacks: Could an adaptive attacker explicitly optimize to avoid the learned EOS-collapse behavior while suppressing refusal? Have the authors evaluated a full Heretic-style attack or other adaptive weight-space interventions outside the training distribution?
-
On capability preservation: LAMBADA accuracy drops substantially on Gemma-1-7B, and the best training configuration differs between models. How sensitive is the method to these hyperparameters? Could the authors also evaluate general capabilities using benchmarks such as MMLU or instruction-following tasks?
We thank the reviewer for taking the time to read our paper and for providing us with feedback that will allow us to improve our work.
As for the adaptive attackers, we will include results with other types of adaptive attackers. We will also include experiments that study hyperparameter sensitivity. As for other benchmarks, we have run additional experiments: