acceptodds
Under review as a conference paper at ICLR 2027

Capability Changes Under Safety-Alignment Removal: A Stage-Resolved Study of LLMs

Abstract

Harmful fine-tuning can remove the safety alignment of open-weight language models. Studies of such attacks usually report two final numbers: attack success on harmful prompts, and one average utility score compared with the aligned model. We show that this pair can mislead. We follow four models from three families through each stage of a removal pipeline, including branches that start from adapted defenses. At every stage, we measure harmful compliance and task utility separately, and we add matched evaluation controls. Three lessons emerge. First, similar attack success comes with different utility outcomes. Final HarmBench attack success reaches 88–97% for every model. Yet average utility ranges from a small gain to a loss of 9.3 points over three training seeds. Different skills decline at different stages. Second, the reference point decides the verdict. A model can recover strongly from a defended start and still remain below the original aligned model. Third, closing a gap is not the same as explaining it. When we change only how answers are requested, one large math gap disappears, but only because the aligned model's score falls to the trained model's level. The trained model now answers without working, and the new request makes the aligned model do the same. Instruction-following losses persist under every control we test. Claims that a removal preserves capability therefore need comparisons that match task, reference, and evaluation protocol.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, AI safety and alignment, s afety unlearning, post-training reinforcement learning, capability evaluation and benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GX2DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that judging safety-alignment removal by attack success and an average utility score is misleading. Across experiments on four open-weight models, all achieve 88–97% HarmBench attack success, yet utility changes range from +0.6 to −9.3 points, with losses emerging at different training stages and across different capabilities. Additionally even if one runs recovery after applying a defense, the utility value often does not reach one of the aligned model.

Critical strengths
  1. The paper applies each scoring or regeneration control to both checkpoints and reports the interaction, rather than only checking whether the attacked model’s score improves. This exposes a failure mode that endpoint comparisons would miss. The authors also explain why by showing that the instruction pushes the aligned model into short answers, which is the same style warmup induces.
Critical weaknesses
  1. The paper says that even when an attack “bounces back” after a defense, the model is still worse than the original. To show this, the authors built their own versions of four published defenses. Several of those versions didn’t work as intended as SEAM is supposed to significantly reduce the model’s abilities when someone trains it on harmful data, but that self-destruct behavior didn’t happen, RepNoise mostly stopped the model from writing normal answers, so its low starting score is confounded, and TAR had a key part switched off. They also only tested Qwen3-8B on this stage.
Questions for authors

To pick which checkpoint to report, the authors scored candidates on a small validation set and kept the one with the best utility. That validation set was drawn from the same question pools as the final test benchmarks, so some questions may appear in both? How many questions actually overlap? Do the main utility results change if those questions are removed?

The RepNoise defense makes the model mostly stop writing normal answers. Free-form tasks score low because the model produces broken text, even if it still knows the answer, and since multiple-choice tasks are scored by checking the option that is selected, the model doesn’t need to write anything. This raises the question of how effective this defense actually is, especially regarding the 8.8 gap?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer GX2D

Defense re-implementations.

You are right on all three points: SEAM did not collapse in our check, our RepNoise model mostly stopped writing normal answers, and a key stage of TAR was inactive on our base model. We therefore ran the pipeline on the authors' released TAR-Refusal and RepNoise checkpoints (general response, point 1). The attack removes both (96.6% and 97.3% HarmBench, three seeds), and both also fail a simple harmful fine-tuning check (77.4% and 86.2%).

Comparing each final model with both the defended model and the aligned base separates two costs. The released TAR-Refusal checkpoint already starts 22.6 points below Llama-3-8B-Instruct (for example, GSM8K drops from 79.7 to 35.0, IFEval from 67.8 to 18.1, and HumanEval+ from 56.7 to 42.7). This matches the degradation Qi et al. (2025) report for TAR. The attack itself then changes utility by only -0.9 points. A study that compares only with the aligned model would see a 23.6-point loss and blame the attack for it.

SEAM, run on its own base model, again did not collapse. The paper keeps our original re-implementations as observed starting points, with the same caveat. The main conclusion now also holds on two released defenses and two more base models, so it no longer depends on Qwen3-8B alone.

Q1: overlap between validation and test sets.

We compared every validation item with every test item after text normalization. Exactly 65 of 25,110 items overlap (15 ARC, 20 GSM8K, 30 MMLU), all as exact matches. No IFEval, MATH-500, or JMMLU item overlaps. Removing the 65 items changes no task score of the 28 main checkpoints by more than 0.26 points, and no six-task average by more than 0.04 points. The 15 code prompts in the validation set are HumanEval+ problems. We did not keep per-item code results, so we report this overlap but cannot give a corrected code score.

Q2: is the RepNoise gap only a writing failure, and how effective is the defense?

Partly, and we now separate the two parts. In Tab. 3, our RepNoise model drops from 67.9 (aligned) to 36.4 (defended), and the attack brings it back to 59.1: a 22.6-point recovery that still ends 8.8 points below the aligned model.

The defended model scores 0 on the three tasks that require a written answer (GSM8K, MATH-500, IFEval), which confirms your reading. But it also loses on tasks scored by answer likelihood, which require no writing: ARC -12.9 and JMMLU -6.3 points against the aligned model. At the final checkpoint, these likelihood-scored tasks remain 4.0 points below the aligned model (three-seed mean; ARC -5.9, JMMLU -4.9, MMLU -1.0). So most of the 22.6-point recovery is restored writing, but part of the remaining gap is a lasting loss that does not depend on writing. Sec. 4.4 now states this.

On effectiveness: the authors' released RepNoise checkpoint did not resist our attacks. Harmful fine-tuning reaches 86.2% HarmBench with utility 28.1 (aligned base: 28.7), and the full pipeline reaches 97.3%.

Reviewer_GX2DOpenReview ↗

Official Comment from Reviewer GX2D

Thank you for the thorough response and for testing the released TAR and RepNoise checkpoints. The overlap question is resolved for the six-task suite. My main remaining concern is that the released checkpoints do not support the recovery is not restoration claim. The released TAR checkpoint already sits 22.6 points below the aligned model before the attack, and the attack adds only 0.9 points of further loss, so nearly the entire gap is defense cost, not attack cost. Calling this incomplete recovery from an attack is misleading. The released RepNoise checkpoint goes in the opposite direction: after fine-tuning, utility nearly matches the aligned base (28.1 vs. 28.7), which is close to full restoration. The paper's main example of a large recovery that still falls short therefore comes from the your broken RepNoise, not from a working defense. Two smaller issues also remain: the 8.8-point gap breakdown mixes numbers from two different evaluation suites, and the 15 HumanEval+ problems shared between checkpoint selection and testing have not been corrected. I will keep my score unless the authors can address these further points.

AuthorsOpenReview ↗

Follow-up reply to Reviewer GX2D

Thank you. We agree with each point and have revised the paper (new PDF uploaded).

1. Released checkpoints. We removed the "recovery is not restoration" claim. On TAR, the defense causes nearly the whole gap. On RepNoise, harmful fine-tuning nearly restores the aligned model. To confirm this, we added two seeds and a control on the undefended base model. Our earlier reply quoted one seed (28.1 at 86.2%).

Checkpoint, attackDefense costAttack changeFinal minus alignedHarmBench
TAR, our pipeline (3 seeds)-22.6-0.9-23.696.6%
TAR, harmful fine-tuning (1 seed)-22.6-5.1-27.777.4%
RepNoise, our pipeline (3 seeds)-3.4+0.2-3.297.3%
RepNoise, harmful fine-tuning (3 seeds)-3.4+2.9-0.682.0%
No defense, harmful fine-tuning (1 seed)-0.482.4%

The paper now makes only the claim that these rows support: on a defended model, the gap to the aligned model does not measure the cost of the attack, so studies need both references. We rewrote Sec. 4.4, RQ2, Tab. 5, and the conclusion accordingly. The RepNoise port is now only a confounded example.

2. The 8.8-point gap. A new table (Tab. 39) splits the gap within the nine-task suite: 1.3 points from likelihood-scored tasks, 3.6 from written tasks, 1.5 from code, and 2.4 from Japanese. Only 1.3 points are independent of the writing failure, and Sec. 4.4 now states this.

3. The 15 HumanEval+ problems. We reran HumanEval+ on all 28 main checkpoints, kept per-problem results, and scored the 149 problems outside the validation set. The nine-task average changes by at most 0.23 points per checkpoint, and the stage differences change by at most 0.11 points (Apx. B).

Reviewer_KHE5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the degradation and shifts in LLM utility during the safety-alignment removal (harmful fine-tuning) process. By decomposing the removal process into two stages (SFT warmup and RL) and evaluating across four open-source models (8B to 14B) from three families (Qwen, Llama and Gemma), the authors analyze fine-grained item transitions and evaluate how different evaluation controls affect observed performance gaps.

The work highlights several key observations, e.g.,

  1. One cannot merely look at the attack success rate (ASR) and a single average utility score. Capability changes must be reported stage by stage and task by task.
  2. The "recovery" after defense does not equal to "restoration". It is necessary to simultaneously compare the defense model and the aligned model.
  3. Net Scores may not reflect the truth. Item transitions such as loss rate and gain rate should be noticed.
Critical strengths

The strengths of the paper include:

  • Interesting research topic: The paper investigates the evaluation of harmful finetuning from traditional ASR and average utility scores to two-stage, detailed aspects.
  • Fine-grained analysis framework: The inclusion of item-level transition analysis (tracking lost vs. gained items) and control-variable evaluation protocols offers valuable methodological insights into LLM safety and utility evaluation.
  • Extensive experiments: The study conducts extensive empirical experiments as well as discussions to analyze the experimental results, giving valuation observations.
Critical weaknesses

The weaknesses of the paper include:

  • Obscure presentation: The overall presentation of the manuscript is overly dense and difficult to navigate. Although I have tried to follow the authors’ thinking, but it is still difficult to understand what the authors want to express, how they performed an experiment and get such results, and why they did in such way. Key notation, mathematical formulations, and symbols in both the text and figures are insufficiently explained, making it challenging for readers to understand.
  • Lack of clear principal line: The paper presents an overwhelming amount of empirical observations and figures without a clear, focused narrative structure. The main takeaways get buried under excessive details.
  • Limited model scale: The empirical study is constrained to small-to-medium models ranging from 8B to 14B parameters from three families. It remains questionable whether these observations can generalize to other model families or larger models.
Questions for authors
  1. In page 3, section 3.2, why is harmful-only RL only available for Qwen3-8B and Llama, but not all four models?
1 comment
AuthorsOpenReview ↗

Reply to Reviewer KHE5

Presentation and main line.

Thank you for the specific diagnosis. The revised paper is built around one claim: attack success plus one average utility score can misjudge an alignment-removal attack. The abstract and introduction now name the method, a paired evaluation by stage, reference, and protocol, and the two lines of work it serves (harmful fine-tuning attacks and tamper-resistance defenses).

The introduction states three research questions: (1) does similar attack success mean similar utility, (2) does recovery from a defended model restore the original model, and (3) does a smaller gap under an evaluation control mean recovery. Each question is answered in one section with one main table, and Tab. 5 links every finding to a reporting recommendation. Each results section now opens with its main finding. Within the 10-page limit and without renumbering any table, we also made these changes:

  • A roadmap paragraph links each contribution to its section and table.
  • A new paragraph in Sec. 3.1, "The warmup stage", explains what warmup is (a short supervised pass, not a learning-rate schedule), why RL needs it, and how to read the warmup checkpoint (a model that still mostly refuses). Fig. 1 labels it "supervised stage".
  • A notation block at the end of Sec. 3 defines all symbols (A, W, F, D_d, T1, T2, T1*, L, G, Delta, the interaction I, the utility score, the item set, ASR, and the seed interval).
  • Every symbol is defined where it first appears and has one meaning only. For example, Delta means only the net item change, and table columns read F-A and T2-T1. The control table states its sign convention.

Model scale.

We add Qwen3.8-27B, which uses a different (hybrid-attention) architecture, as a one-seed check (Apx. O):

Qwen3.8-27BAlignedAfter warmupFinal
Utility (eight-task average)76.972.077.9
HarmBench0.6%0.0%88.1%

We leave out the Japanese llm-jp score because a parsing error in the evaluation tool corrupts the aligned model's value. IFEval drives the warmup dip (74.7, then 27.9, then 79.1 at the final stage), partly offset by a MATH gain (43.8 to 52.0).

The released checkpoints also add two more base models (Llama-3-8B and Llama-2-7B). Larger models remain a limitation.

Q1: harmful-only RL (T1) on all four models.

The original runs were limited by compute. We have now run T1 on Qwen3-14B and Gemma-2-9B with three seeds, so Tab. 3 covers all four models. Utility score; each cell is a three-seed mean with a 95% interval.

ModelT1 (harmful-only)T2 (mixed)HarmBench T1 / T2
Qwen3-8B68.8 ± 2.968.5 ± 0.797.9% / 95.8%
Qwen3-14B72.5 ± 0.671.7 ± 0.795.2% / 97.1%
Llama-3.1-8B57.4 ± 0.655.4 ± 0.284.7% / 93.1%
Gemma-2-9B58.0 ± 6.552.6 ± 8.186.4% / 88.3%

T2 does not have a higher average than T1 on any model. The seed-paired differences (T2 minus T1) are -0.3 ± 2.9 (Qwen3-8B), -0.8 ± 0.8 (Qwen3-14B), -2.1 ± 0.5 (Llama), and -5.5 ± 14.4 (Gemma; one unusual T1 seed widens this interval).

For HarmBench, the paired bootstrap differences over behaviors (T2 minus T1, 95% intervals) are +8.4 [+2.7, +14.1] for Llama and -2.1 [-4.4, +0.2] for Qwen3-8B. This supports the paper's view that the benefit of adding utility tasks to RL is an open question.

We also asked whether Llama's IFEval loss is specific to RL, using a decision rule fixed before seeing the result. Plain harmful fine-tuning of the Llama warmup lowers IFEval by 18.7 points (86.8% HarmBench), close to the 20.3-point loss under mixed RL from the same warmup, so Llama's loss is not specific to RL. For Gemma, the comparison is only partial (IFEval -24.4, but HarmBench reaches only 69.2%). These two fine-tuning checks use one seed each.

Reviewer_xFd1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how alignment removal would affect the utility of LLMs. The authors argue that the current safety removal processes only emphasize improving ASR and maintaining average model performance, which they believe is not a comprehensive understanding of model utility. Experiments test multiple models from different families across many utility tasks and harmful benchmarks. Results show that models with similar ASR may perform differently across models and even for different tasks on the same model. Addational results also show that utility recovery does not mean the performance is restored.

Critical strengths
  • This topic is meaningful, I do agree that we should not simply look at the average utility of models after the alignment removal. This work brings up an important but often neglected perspective.
  • Experiments are comprehensive, the authors test multiple models from different families across many utility tasks and harmful benchmarks. There are also many ablation studies and additional experiments to support the key idea.
Critical weaknesses
  • The novelty of this paper is to some degree limited. Although this problem itself is meaningful, the main conclusion of this paper is straightforward and has been studied in prior works. I think this work is trying to combine those existing observations into the alignment removal case, so the insights are less impressive.
  • Some experimental results are less convincing. There are part of results seems unexpected and likely abnormal, such as Table 1 and 3. I think this problem would harm the soundness of this paper. Please find more information about this in the Questions section.
Questions for authors
  • Can the authors explain the significant performance degradation in Table 3 for the RepNoise defense method? RepNoise reduces utility from 67.9 to 36.4, and it is unexpectedly large. This raises my concern about the validity of this defended baseline. Therefore, I would suspect that the following large recovery seems to be simply introduced by this unusually low starting point. Considering that the conclusion "recovery is not restoration" largely comes from this evidence, I would suggest the authors make further clarifications on this.
  • After reading Table 1, I am a little curious about the model ASR at the warm-up stage (W). So I checked Table 19 in the appendix and found something interesting. After the warm-up, only Qwen3-8B decreases in ASR for all three benchmarks, while the other models generally improve. Could the authors explain why Qwen3-8B exhibits this unexpected behavior?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer xFd1

Novelty.

We agree that each ingredient has precedent, and our nearest-work table (Apx. L) shows this. Item-level changes hidden behind a small average change are documented between model versions (Cacioli, 2026) and after post-training (Harmon et al., 2026). Gan et al. (2026) show that answer format or generation length can reverse a measured fine-tuning effect. Qi et al. (2025) show that a prompt template can change the conclusion about a defended model, and they report the utility cost of a TAR checkpoint itself.

However, among the work we checked, no study of safety-alignment removal does all of the following: follow the same models through every stage, compare each attack on a defended model with both the defended model and the original aligned model, and re-score the aligned model under each evaluation control. This combination, and what it reveals, is our contribution. The revised abstract and introduction now state it as a paired evaluation: by stage, by reference, and by protocol.

The new runs make it concrete. On the released TAR and RepNoise checkpoints, the attack itself changes utility by only -0.9 and +0.2 points. Comparing with the aligned model alone would wrongly assign the defense's own cost (-22.6 and -3.4 points) to the attack. We do not claim a new metric.

Q1: the RepNoise drop in Table 3.

The drop from 67.9 to 36.4 is mostly because our RepNoise model barely writes normal answers: GSM8K, MATH-500, and IFEval all score 0 at the start. We agree that the large recovery partly reflects this low start. The conclusion does not depend on it, for two reasons. First, on tasks scored by answer likelihood, which need no writing, the final model stays 4.0 points below the aligned model (ARC -5.9, JMMLU -4.9, MMLU -1.0; three seeds).

Second, the authors' released RepNoise checkpoint writes normally (GSM8K 25.2 for the defended model versus 25.5 for the aligned model) and shows the same pattern. The defense costs 3.4 points, mostly on code (HumanEval+ 12.2 to 2.4), and after the attack the final model is still 3.2 points below the aligned model.

Q2: why Qwen3-8B's attack success falls after warmup.

In short, the drop is real and specific to Qwen3-8B: during warmup, the model becomes more likely to refuse some borderline requests. Attack success at the final stage is unaffected (95-96% for every seed). We report this as an observation and do not claim a mechanism.

  • First, a clarification that the original text lacked. Warmup is a short supervised pass on 1,130 harmful prompts paired with compliant responses. Its only purpose is to make the model comply occasionally, because the RL stage learns nothing from a model that refuses every time. It is not meant to raise attack success: the warmup checkpoint still mostly refuses (HarmBench below 17% for every model), and almost all of the increase comes from the RL stage. The revised Sec. 3.1 now explains this in its own paragraph.

  • Second, we checked every HarmBench behavior that the classifier flagged for the aligned Qwen3-8B. All 21 flags are real compliance, not classifier errors: none of the 21 responses refuses, and most are persuasive-writing or security how-to requests.

After warmup, 12 to 14 of these 21 behaviors (depending on the seed) become refusals. The number of responses containing a refusal phrase rises from 119 to 138-139 of 159. A paired bootstrap over behaviors gives -7.5 points from aligned to warmup (95% interval [-12.2, -3.4]). Qwen3-14B does not show this drop (+1.1 [-1.7, +4.0]), and Llama rises (+16.1).

Thank you for your further clarification and additional experiments. I have raised my score to 3.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (to all reviewers)

We thank the reviewers for their careful reading. The reviews raise four main points. For each, we ran new experiments or analyses, and all results are in the revised PDF. Table and appendix numbers cited in the reviews are unchanged; new material is in the last appendix (Apx. O).

1. Our defended models were our own re-implementations, and some did not behave as intended (GX2D, xFd1).

We therefore ran the same pipeline, without changes, on two checkpoints released by the defense authors: TAR-Refusal (base model Llama-3-8B-Instruct) and RepNoise (base model Llama-2-7b-chat). Utility is the paper's nine-task average. Warmup and final values are means over three seeds with 95% intervals.

Aligned baseDefended modelAfter warmupFinalFinal minus defendedFinal minus aligned
TAR-Refusal, utility56.333.732.0 ± 1.232.7 ± 1.6-0.9-23.6
TAR-Refusal, HarmBench3.8%25.2%44.2%96.6%
RepNoise, utility28.725.326.3 ± 0.625.5 ± 1.5+0.2-3.2
RepNoise, HarmBench1.9%1.3%0.6%97.3%

The attack removed both defenses (about 97% HarmBench attack success). Almost all of the utility loss was already there before the attack: the attack itself changed utility by only -0.9 and +0.2 points. The released TAR-Refusal checkpoint is also less safe than its base model before any attack (25.2% versus 3.8%).

We also ran a simple harmful fine-tuning check (one epoch on 8,000 BeaverTails pairs, learning rate 2e-5, one seed). It reached 77.4% (TAR) and 86.2% (RepNoise), so neither checkpoint resisted harmful fine-tuning in our setting. Our setting differs from the attacks tested in each defense paper.

For SEAM, no checkpoint is released. We ran our re-implementation with the original hyperparameters on SEAM's own base model, Llama-2-7b-chat. It cost 5.0 points of utility, and harmful fine-tuning then reached 84.3% without the intended collapse in utility (one seed). We see this as a failure of our re-implementation to reproduce SEAM, not as evidence against SEAM.

2. The RepNoise start is weak because the model barely writes normal answers (GX2D, xFd1).

We agree for our re-implementation. However, the main conclusion also holds on tasks that need no written answer, and on the released RepNoise checkpoint, which writes normally (see the replies to GX2D and xFd1).

3. Checkpoint selection and test overlap (GX2D).

65 of 25,110 test items also appear in the validation set. Removing them changes no reported six-task average by more than 0.04 points.

4. Presentation and scope (KHE5).

The paper is now organized around one main claim, with a roadmap, three research questions, and a notation block. Harmful-only RL now covers all four models with three seeds each, and we add Qwen3.8-27B as a one-seed check at larger scale.

AuthorsOpenReview ↗

Update after the discussion

Following Reviewer GX2D's follow-up, we made three further changes to the revised PDF. Details are in our reply to GX2D.

  1. Narrower claim on defended models. On the released checkpoints, the defense causes nearly all of the gap to the aligned model, and harmful fine-tuning nearly restores the released RepNoise model. We therefore removed the claim that recovery from a defense is not restoration. The paper now claims only that, on a defended model, the gap to the aligned model does not measure the cost of the attack, so studies need both references.
  2. More seeds for the harmful fine-tuning check. On the released RepNoise checkpoint, three seeds give 82.0% HarmBench and a utility score of 28.2 ± 0.4 (aligned base: 28.7). Point 1 of the general response quoted one seed (86.2%). The same fine-tuning on the undefended base model gives 28.3.
  3. Validation overlap in code. Point 3 of the general response covered the six-task suite. We also reran HumanEval+ without the 15 problems shared with the validation set. The nine-task average of the 28 main checkpoints changes by at most 0.23 points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.