acceptodds
Under review as a conference paper at ICLR 2027

Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attacks

Abstract

Recent defenses for safeguarding open-weight large language models (LLMs) are intended to prevent adversarial usage tamirisa2025tamper,wang2025self. Underlying these defenses is an assumption that new harmful behavior is learned through fine-tuning rather than elicited by jailbreaking the model. Yet, pretrained LLMs already encode substantial harmful knowledge across many domains, which raises an important question: can an adversary jailbreak safeguarded models, to achieve harmful usage without fine-tuning at all? In this paper, we show that open-weight safeguards are susceptible to simpler strategies that, despite being well known, have not been systematically evaluated against these safeguards. Specifically, we evaluate two low-cost attacks–abliteration and prefilling–that do not rely on gradient-based optimization. Across three harmfulness evaluation benchmarks (BeaverTails, HarmBench, and AdvBench), these attacks increase attack success rates against safeguarded open-weight models from below 10% to a range of 16%-96%. To mitigate this vulnerability, we introduce abliteration-resistant tuning (ART), which incorporates an abliteration-based objective into training. ART can be layered onto existing defenses and reduces the success rates of abliteration, prefilling, and their combination by 10%-20%. These findings indicate that the attack surface for open-weight models is broader than previously characterized, and that evaluations of safeguarding defenses should incorporate a more diverse set of attack strategies beyond adversarial fine-tuning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Open-Weight LLM, Fine-Tuning Defenses, SEAM, TAR, Attack

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Q8qpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper evaluates the robustness of safety-aligned language models against abliteration and prefilling attacks. The authors propose ART, a training method that simulates abliteration during training to defend against these attacks.

Critical strengths

The authors provide a broad evaluation of robustness against these attacks across a variety of models and three datasets.

Critical weaknesses
  1. The paper would benefit from a more thorough discussion of closely related prior work, such as Circuit Breakers (NeurIPS 2024), ReFAT (ICLR 2025), and DeepRefusal (EMNLP Findings 2025). Including these as baselines, or at least discussing how ART differs from them, would help clarify the paper’s contribution.
  2. The evaluation does not appear to include adaptive attacks. For instance, SOM (AAAI 2026) performs attacks using multiple refusal directions, and it would be valuable to know whether ART remains robust in this setting.
  3. As noted in line 802, ART is trained on a relatively small dataset. However, the paper does not report variance across random seeds. Reporting this would help establish the stability of the training procedure.
  4. The paper offers limited analysis of the ART-trained models. A deeper explanation of why ART is effective, for example in relation to the training parameters or the resulting changes in model representations, would strengthen the paper.
Questions for authors
  1. Could the authors expand the related work section and clarify how ART differs from the methods mentioned above?
  2. Could the authors provide additional experiments against a broader range of adaptive attacks?
  3. Could the authors report the mean and variance of the results across multiple random seeds?
  4. Could the authors provide further insight into why ART-trained models remain better aligned under these attacks?
2 comments

We thank the reviewer for appreciating the breadth of our evaluation and for the pointers to closely related refusal-direction defenses.

We would first like to clarify the scope of our work: our primary contribution is a systematic evaluation of fine-tuning safeguards (SEAM and TAR) against abliteration and prefilling across several models and benchmarks. Our observation that ASR rises from below 10% to 16-96% leads us to conclude that FT safeguards are still susceptible to these simple attacks. We believe this is consequential and relevant to practitioners trying to build safe open models, irrespective of the exact defenses they’re using.

ART is a secondary contribution, and thus we can only dedicate some of the paper to ART and leave future investigations on ART to future work. However, it seems to us that the reviewer’s ratings ignore the rest of our primary valuable contributions and are highly concentrated on ART. Our primary contributions have impressed the other reviewers who find our work to be “important for the community” (chZv) and “relevant” and “clearly presented” (j8Rj). As such, we request the reviewer to please reconsider their rating in the light of this clarification about primary contributions.

Nevertheless, we really value the reviewer’s comments on ART and they will indeed help us strengthen this part of the paper. We address these concerns below.

W1/Q1. Related prior work and novelty. Thanks for these references. We’re sorry that we missed these and will cite them in the paper. We agree that ReFAT and DeepRefusal are particularly related, while Circuit Breakers is more distinct. ReFAT recomputes refusal directions over a fixed set of layers during training, while DeepRefusal fixes a global refusal direction and randomly samples subsets of layers to ablate during training. ART differs from the two prior works by both resampling the ablated layer and recomputing refusal directions during training.

Due to limited response time, we only evaluated DeepRefusal as it is more recent and is a more comprehensive defense than ReFAT. We evaluate DeepRefusal on Llama3.1-8B-Instruct and find that abliteration is similarly effective against DeepRefusal as it is against TAR (27 vs. 28).

ModelPre-attackAttack
Base2086
Base + ART244
SEAM681
SEAM + ART220
TAR928
TAR + ART012
DeepRefusal127

W2/Q2. Adaptive attacks. Thanks for the suggestion to evaluate whether ART can defend models against SOM. We test this on three different model architectures (Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3). We optimize SOM with 7 refusal directions. To select the layer to attack with SOM, we use the layer with the highest ASR based on single-direction abliteration. Overall, ART significantly reduces the ASR of SOM from over 60 to under 20.

ModelNo attackSOM
Llama13%63%
Llama + ART2%14%
Qwen12%61%
Qwen + ART4%6%
Mistral22%35%
Mistral + ART2%4%

W3/Q3. Random seeds. Thanks for this suggestion to verify the stability of ART training. We ran 3 seeds of ART training on top of Base, SEAM, and TAR models. We then evaluate the ASR of abliteration against each model on BeaverTails and judge the outputs using HarmBench Llama2-13B. Across all models and seeds, we find that ART significantly reduces ASR to about half of the original value.

modelno ARTseed 1seed 2seed 3
Base85444740
SEAM73203226
TAR26121316

W4/Q4. Analysis of ART. Thanks for suggesting further analysis of ART. We present an analysis that measures how well the refusal direction (from test data) is preserved after abliteration (from attack training data). We first measure the refusal direction on the test data (BeaverTails), and then measure the cosine similarity of the model’s representations on the same data after applying abliteration to a base or ART-tuned model. We report results across 3 layers (11-13) and report the cosine similarity to the pre-ART refusal direction. We find that as abliteration weight is increased, Refusal similarity decreases, as expected (abliteration makes the refusal direction weaker). However, if ART is applied before abliteration, the drop in refusal similarity is smaller.

LayerWeightBaseART
11none1.000.98
111.00.810.92
112.00.710.87
12none1.000.92
121.00.740.81
122.00.580.70
13none1.000.95
131.00.730.88
132.00.280.80

We hope our responses answered your questions. Please let us know if there are any additional concerns. We look forward to hearing from you!

Thanks, Authors

Thank you for the detailed response and additional experiments. However, I remain unconvinced by the novelty of the main contribution, as similar robustness evaluations have already been explored in prior work. The differences between ART and existing methods also remain unclear, and the experimental comparisons are insufficient. Therefore, I maintain my score.

Reviewer_j8RjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper evaluates safeguarded OSS models under adversarial jailbreaking, showing that attacks requiring less effort than the (supposedly defended against) finetuning can already elicit harmful outputs. In particular, the authors study abliteration, which removes the model's refusal direction, and prefilling. Across three model families and three benchmarks, they show that these attacks (especially abliteration), can substantially increase attack success rates on models defended with SEAM and TAR. Based on this, the authors then incorporate abliteration into the inner training objective by identifying the most affected layers and constructing a typical adversarial loss term. This reduces attack success rates (~30%) at some cost to utility.

Critical strengths
  • The defense of open-source models are becoming an increasingly relevant topic and the paper provides a clearly presented work in this direction.
Critical weaknesses
  1. I am not convinced by the novelty of the main finding. The motivation emphasizes that prior work has primarily considered finetuning attacks, but existing work has already shown that abliteration and prefilling can be sufficient to bypass safeguards (e.g., [1]). Also, having been in the space for some time, I don't think many people would have expected these attacks to have no effect (especially abliteration). Further, the fact that defenses like TAR do not erase harmful knowledge is generally accepted, and prior work has already shown that it can be reconstructed (in one case even by just switching the chat template [2], which is not far from what is done here). As the paper acknowledges, neither the attacks themselves nor their specific execution here is novel, leaving it unclear what additional insight the main results provide. In general, one might ask why we should even inspect SEAM or TAR to transfer well into these settings.

  2. Several choices in the evaluation make the results difficult to contextualize. It is unclear whether the evaluated SEAM and TAR configurations effectively defended against the finetuning attacks they were intended to counter. In particular, in Table 8, SEAM seems to provide essentially no protection against finetuning, while TAR also performs "not so great". The experiments further switch models frequently, making consistent comparisons difficult, and I could not find the exact model identifiers anywhere in the main paper. The benign capability evaluation is limited to Alpaca and MMLU, while the utility costs in Table 4 and the refusal behavior of the proposed SEAM+ART configuration seem substantial. It also remains unclear whether harmful responses suffer from a comparable loss in capability, as in a jailbreak tax [3].

For some other points I refer to my questions below.

[1] Zhao et al. Improving LLM Safety Alignment with Dual Objective Optimization, ICML 2025

[2] Qi et al. On Evaluating the Durability of Safeguards for Open Weight LLMs, ICLR 2025

[3] Nikolić et al. The Jailbreak Tax: How Useful are Your Jailbreak Outputs?, ICML 2025

Questions for authors
  1. Why did you choose TAR and SEAM? In particular, were the configurations used here effective defenses against finetuning attacks before applying abliteration or prefilling?

  2. Are these safeguard objectives (SEAM/TAR) used in practical model deployments? Given the utility impact in Table 4, I am uncertain how attractive they would be to a model deployer.

  3. Could you consider using a stronger judge than Llama 2? I know that it is sometimes still the default judge of some benchmarks, but such weaker judges can sometimes produce unexpected scores when evaluating stronger models on these benchmarks.

  4. Why does combining abliteration with prefilling perform worse than abliteration alone in several cases (i.e., could you do some qualitative analysis)? Have you tried prefills that are more aligned with the model's natural response pattern?

  5. In Figure 2, what exactly are the representations before and after generation? It is unclear to me what the PCA visualization exactly shows here.

2 comments

We thank the reviewer for the clear and detailed review. FT stands for Fine-Tuning.

W1. Limited novelty of the main finding. We thank the reviewer for bringing up this point. While we agree that prefilling and abliteration have been shown to be effective attacks against safeguards and mention this in the paper, there is no systematic study showing how these attacks can break FT safeguards. Our evaluation fills this gap and leads to a broader consequential conclusion: FT safeguards are still susceptible to simple attacks. We believe this to be very relevant to practitioners trying to build safe open models, irrespective of the exact defenses they’re using. Hence, if practitioners want to use these defenses, they should test open models for their susceptibility to such attacks before deployment. Furthermore, open models should be deployed with additional safeguards that prevent such attacks while keeping the FT safeguard intact.

Q1. Effectiveness of SEAM/TAR. We chose SEAM and TAR because they are recent state-of-the-art FT safeguards. We trained SEAM and TAR against FT attacks using the Anthropic-HH dataset, and chose the most robust configurations where FT only increases ASR from 4% to 5% against TAR and from 1% to 19% for SEAM. However, the safeguards have limited effectiveness when the FT attack uses unseen BeaverTails and SST-2 data (Table 8).

ModelBefore attackAfter attack
Base1726
Vaccine024
Booster119
SEAM119
TAR45

Q2. Practical deployments. We are not sure about the exact industry practices for safe open model deployment, but believe FT safeguards are relevant to industry practitioners. For instance, safety grants from Thinking Machines (https://thinkingmachines.ai/news/safety-research-grants/) discuss and cite FT safeguards, including TAR. These types of defenses are an important direction for developing safe open models and may be widely adopted once they are made more practical. Hence, the academic community will still value research that pushes these methods forward irrespective of current industry state.

Q3. Stronger judge. Thanks for this suggestion. We use Claude Sonnet 5.5 as a judge and evaluate abliterated models on BeaverTails. The Llama2 judge reports around 2-4% higher ASR than Sonnet 5.5, which leaves the overall conclusion unchanged: SEAM and TAR are vulnerable to abliteration, and ART helps reduce this vulnerability.

ModelLlama2-13BSonnet 5.5
Base8682
Base + ART4440
SEAM8179
SEAM + ART2016
TAR2825
TAR + ART1210

Q4. Abliteration with prefilling. We show that combining abliteration with prefilling retains more of the refusal direction when abliteration is applied with a larger scaling factor of 2 as opposed to 1. We measure the cosine similarity of the model’s representations along the refusal direction of out-of-distribution data before and after prefilling (i.e. different from the actual refusal direction subtracted during abliteration). We find that abliteration with a scaling factor of 1 overall benefits more (lower refusal similarity) from prefilling than a scaling factor of 2.

LayerWeightAbliterateAbliterate+Prefill
121.00.740.47
122.00.580.65
131.00.730.50
132.00.280.81

Q5. PCA visualization. Figure 2 shows the representations at the last token and last layer of an abliterated model. In the figure, we show that harmful/harmless representations are initially relatively separable; after generating 128 tokens, the representations of jailbroken inputs significantly overlap with those of harmless inputs. This suggests that it is important for safeguards to keep harmful/harmless representations far from each other, but the observation that several correctly refused prompts lie in the same cluster as jailbroken and harmless prompts keeps this from being a general conclusion.

Other points.

  • Model identifiers. For most experiments, we use meta-llama/Llama-3.1-8B-Instruct. For the purpose of evaluating over a diverse set of models, we evaluate several smaller models. These are meta-llama/Llama-3.2-1B, meta-llama/Llama-3.2-3B-Instruct, Qwen/Qwen2.5-1.5B-Instruct and google/gemma-3-1b-it. All experiments in our response use Llama-3.1-8B-Instruct and we will include them in the revision.

  • Benign capability evaluation. Thanks for the suggestion. We present results on GSM8K, which shows that TAR performs much worse than SEAM, while ART sacrifices little utility on BASE and SEAM models.

ModelAcc
Base77.5
Base+ART76.9
SEAM72.6
SEAM+ART73.4
TAR31.3
TAR+ART20.5

We hope our responses answered your questions. Please let us know if there are any additional concerns. We look forward to hearing from you!

I thank the authors for their rebuttal. My concerns with respect to the novelty of the work / relation to prior work, as well as the practicality of the overall method, remain however. As such I will retain my score.

Reviewer_chZvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is a timely, well-executed and useful evaluation paper. Its core message—that open-weight safeguards suppress refusal without removing knowledge, and hence fall to cheap inference-time attacks—is important for the community and changes how such defenses should be benchmarked. The experiments are broad: two safeguards, five defenses, three benchmarks, three model families, plus attack-budget, model-size, utility, and representation analyses. The writing is clear. My reservations are scope and depth: abliteration and prefilling are well-known, so the conceptual novelty is limited to the systematic evaluation; ART’s improvement is modest and residual ASR stays high; most models are small; and best-of-N reporting flatters the attack. As an empirical/positioning contribution it is above the bar; as a methods paper it is not.

Critical strengths

(1) Definitive threat-model correction: The paper shows safeguards built exclusively to defend against fine-tuning can be circumvented via gradient-free inference-time attacks, directly undermining the central premise of TAR/SEAM-style methods. This addresses the key question of whether the work is valuable to the community, and the answer is affirmative.

(2) Attack cost characterization: Most of the ASR can be recovered with a small random budget, and the required budget does not grow with model layers. This suggests the attack is not a narrow exhaustive lab trick but a practical low-cost threat, supporting the external validity of the results. Collectively, these two arguments provide solid justification for the recommendation that evaluations ought to incorporate gradient-agnostic attacks.

Critical weaknesses

(1) ART does not deliver a substantive fix. When layered on existing safeguards, ASR stays elevated with a large gap to the clean unattacked baseline, paired with a clear rise in benign refusals. The observed defensive benefit mismatches its advertised capability as a composable mitigation, which confines the core contribution to an empirical evaluation report.

(2) The novelty claim is weakly grounded. Abliteration and prefilling are established, publicly available attacks. Concurrent lines of work such as Fool’s Gold and Decoy Direction Optimization (from the same group) already investigate abliteration-focused. The manuscript does not sufficiently situate itself relative to these prior results, undermining its assertion of being the first systematic evaluation.

Questions for authors

(1) What is the ASR of TAR and SEAM under a realistic single-shot (non best-of-N) threat model without adversary access to validation-set filtering? (2) How does ART relate to and differ from concurrent abliteration defenses (DDO, Fool’s Gold)? Why is the comparison omitted? (3) How to balance the utility penalty from increased benign refusals?

Future work: (4) Scale ART to larger models and stronger adaptive attacks. (5) Investigate defenses that erase harmful knowledge rather than suppress refusal outputs.

1 comment

We thank the reviewer for recognizing our work as a timely and useful evaluation, and for highlighting the threat-model correction and the low cost of the attack as key strengths.

Below we address your concerns.

W1. ART does not deliver a substantive fix. While we agree that it is not ideal that all the baselines have elevated ASR relative to the unattacked baseline, we believe this highlights a broader challenge in safety where defenses are vulnerable to out-of-distribution adversaries in terms of the data, attack method, or hyperparameters. Therefore, ART is offered as an initial mitigation that shows abliteration-resistance can be significantly improved at a modest cost. Furthermore, we show that ART composed with TAR achieves near-lowest ASR against both fine-tuning and abliteration compared to other safeguards. We leave the rest to future work.

W2/Q2. Comparison to concurrent works. Thanks for pointing out these references, which we will add to the related work. As noted by the reviewer, these are concurrent works to us and we didn’t know about them at the time of writing this manuscript.

These two methods draw on different classes of defenses than ART. ART follows adversarial training: like TAR, which simulates fine-tuning attacks during training, ART simulates the attack (abliteration) during training and optimizes the model to remain safe after it is applied. DDO is a backdoor- or trapdoor-style defense: it deliberately plants a harmless decoy direction in the model so that the attacker's direction-extraction step finds the decoy rather than the true refusal direction. Fool's Gold is a self-sabotaging defense: rather than preventing the attack, it ensures the attacked model produces misleading outputs, which is similar in spirit to self-destructing models (Henderson et al., 2023; SEAM). In conclusion, while all methods defend against abliteration, they take fundamentally different approaches to doing so.

Q1. Single-shot selection without validation filtering. We report this in Table 3 (Random, N=1). With no selection at all, a single random configuration is close to the unattacked baseline: SEAM gets 4/4/5% and TAR 5/13/30% on BeaverTails/AdvBench/HarmBench, against baselines of 2/0/1% and 5/4/29%.

We would like to note that when evaluating or building defenses it is important to evaluate attackers across a range of resource budgets so we can build the strongest defense. As such, we use multi-shot selection because it is a simple yet effective way to scale up the budget of the attack.

Q3. How to balance the utility penalty from increased benign refusals? We can control the utility-robustness tradeoff by tuning defense hyperparameters such as the number of steps or learning rate. Below, we show the tradeoff of these two quantities across multiple ART configurations on Llama3.1-8B-Instruct. We note that utility can degrade due to overall poor model quality, not just benign refusals (LR=1e-3).

Learning rateASRBenign refusalROUGE-L
Base86%0.1%0.316
1e-571%1.5%0.329
1e-444%4.0%0.343
1e-338%6.0%0.276

Future work. Thank you for these suggestions, which we will incorporate into the next revision. We agree that realistic deployment of safe open-weight models requires further investigation into larger models, stronger adaptive attacks, and defenses that erase harmful knowledge rather than suppress harmful behavior. However, these are out of scope for this paper and we leave them to future work.

We hope our responses answered your questions. Please let us know if there are any additional concerns. We look forward to hearing from you and hope that you continue supporting our work!

Thanks, Authors

Comments on the paper

2 comments

We thank all reviewers for their feedback. Reviewers found the evaluation broad (chZv, Q8qp) and clearly presented (j8Rj), and found the threat-model question timely and relevant to open-weight safety (chZv, j8Rj). Two concerns were shared across reviews, which we summarize here, with details in the individual responses.

  1. Novelty and scope. While prior work has shown individual safeguards can be bypassed without fine-tuning, our work makes this limitation explicitly clear: safeguards share an implicit threat model in which harmful capability must be introduced through fine-tuning. We show that this assumption does not hold by systematically evaluating fine-tuning safeguards against cheaper inference-time attacks across benchmarks, model sizes, and model families. We will revise the abstract and introduction to state this scope precisely, and expand the related work with the references raised by reviewers.

  2. Role of ART and related defenses. ART is a secondary contribution of the paper and was offered as an initial mitigation showing that fine-tuning safeguards can be layered with an abliteration-specific defense. This is as much the scope of the paper allows and we leave further work on ART for future. However, we will discuss closely related refusal-direction defenses (ReFAT, DeepRefusal, Circuit Breakers, DDO, Fool’s Gold) and clarify how ART differs.

We hope our responses answered the reviewers’ questions. Please let us know if there are any additional concerns. We look forward to hearing from you!

Thanks, Authors

Area_Chair_5GkEOpenReview ↗

Reminder: Please review the author rebuttal before the deadline

Dear reviewers,

The rebuttal period is about to close. If you haven’t responded yet, please read the authors’ rebuttal and leave a brief comment before the deadline.

If you’ve already responded, please disregard this reminder. Thanks for your time, and apologies for the extra email.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.