- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper evaluates the robustness of safety-aligned language models against abliteration and prefilling attacks. The authors propose ART, a training method that simulates abliteration during training to defend against these attacks.
Critical strengths
The authors provide a broad evaluation of robustness against these attacks across a variety of models and three datasets.
Critical weaknesses
- The paper would benefit from a more thorough discussion of closely related prior work, such as Circuit Breakers (NeurIPS 2024), ReFAT (ICLR 2025), and DeepRefusal (EMNLP Findings 2025). Including these as baselines, or at least discussing how ART differs from them, would help clarify the paper’s contribution.
- The evaluation does not appear to include adaptive attacks. For instance, SOM (AAAI 2026) performs attacks using multiple refusal directions, and it would be valuable to know whether ART remains robust in this setting.
- As noted in line 802, ART is trained on a relatively small dataset. However, the paper does not report variance across random seeds. Reporting this would help establish the stability of the training procedure.
- The paper offers limited analysis of the ART-trained models. A deeper explanation of why ART is effective, for example in relation to the training parameters or the resulting changes in model representations, would strengthen the paper.
Questions for authors
- Could the authors expand the related work section and clarify how ART differs from the methods mentioned above?
- Could the authors provide additional experiments against a broader range of adaptive attacks?
- Could the authors report the mean and variance of the results across multiple random seeds?
- Could the authors provide further insight into why ART-trained models remain better aligned under these attacks?
2 comments
Thank you for the detailed response and additional experiments. However, I remain unconvinced by the novelty of the main contribution, as similar robustness evaluations have already been explored in prior work. The differences between ART and existing methods also remain unclear, and the experimental comparisons are insufficient. Therefore, I maintain my score.
We thank the reviewer for appreciating the breadth of our evaluation and for the pointers to closely related refusal-direction defenses.
We would first like to clarify the scope of our work: our primary contribution is a systematic evaluation of fine-tuning safeguards (SEAM and TAR) against abliteration and prefilling across several models and benchmarks. Our observation that ASR rises from below 10% to 16-96% leads us to conclude that FT safeguards are still susceptible to these simple attacks. We believe this is consequential and relevant to practitioners trying to build safe open models, irrespective of the exact defenses they’re using.
ART is a secondary contribution, and thus we can only dedicate some of the paper to ART and leave future investigations on ART to future work. However, it seems to us that the reviewer’s ratings ignore the rest of our primary valuable contributions and are highly concentrated on ART. Our primary contributions have impressed the other reviewers who find our work to be “important for the community” (chZv) and “relevant” and “clearly presented” (j8Rj). As such, we request the reviewer to please reconsider their rating in the light of this clarification about primary contributions.
Nevertheless, we really value the reviewer’s comments on ART and they will indeed help us strengthen this part of the paper. We address these concerns below.
W1/Q1. Related prior work and novelty. Thanks for these references. We’re sorry that we missed these and will cite them in the paper. We agree that ReFAT and DeepRefusal are particularly related, while Circuit Breakers is more distinct. ReFAT recomputes refusal directions over a fixed set of layers during training, while DeepRefusal fixes a global refusal direction and randomly samples subsets of layers to ablate during training. ART differs from the two prior works by both resampling the ablated layer and recomputing refusal directions during training.
Due to limited response time, we only evaluated DeepRefusal as it is more recent and is a more comprehensive defense than ReFAT. We evaluate DeepRefusal on Llama3.1-8B-Instruct and find that abliteration is similarly effective against DeepRefusal as it is against TAR (27 vs. 28).
W2/Q2. Adaptive attacks. Thanks for the suggestion to evaluate whether ART can defend models against SOM. We test this on three different model architectures (Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3). We optimize SOM with 7 refusal directions. To select the layer to attack with SOM, we use the layer with the highest ASR based on single-direction abliteration. Overall, ART significantly reduces the ASR of SOM from over 60 to under 20.
W3/Q3. Random seeds. Thanks for this suggestion to verify the stability of ART training. We ran 3 seeds of ART training on top of Base, SEAM, and TAR models. We then evaluate the ASR of abliteration against each model on BeaverTails and judge the outputs using HarmBench Llama2-13B. Across all models and seeds, we find that ART significantly reduces ASR to about half of the original value.
W4/Q4. Analysis of ART. Thanks for suggesting further analysis of ART. We present an analysis that measures how well the refusal direction (from test data) is preserved after abliteration (from attack training data). We first measure the refusal direction on the test data (BeaverTails), and then measure the cosine similarity of the model’s representations on the same data after applying abliteration to a base or ART-tuned model. We report results across 3 layers (11-13) and report the cosine similarity to the pre-ART refusal direction. We find that as abliteration weight is increased, Refusal similarity decreases, as expected (abliteration makes the refusal direction weaker). However, if ART is applied before abliteration, the drop in refusal similarity is smaller.
We hope our responses answered your questions. Please let us know if there are any additional concerns. We look forward to hearing from you!
Thanks, Authors