acceptodds
Under review as a conference paper at ICLR 2027

When Safety Fails Collectively: Robust Alignment Against Harmful Fine-Tuning via Joint Weight Perturbations

Abstract

Harmful fine-tuning can rapidly erode the safety alignment of large language models, creating substantial risks when aligned models are adapted to downstream tasks. Existing alignment-stage defenses use harmful examples and simulated harmful perturbations during safety alignment to prepare models for later harmful fine-tuning, yet the collective effects of harmful weight perturbations remain underexplored. We identify a collective failure pattern: disjoint parts of a harmful perturbation have limited effects in isolation, but their joint application increases the model's harmfulness beyond the sum of their individual effects. Guided by this observation, we propose Joint Weight Perturbations (JWP), an alignment-stage defense that masks different parts of a harmful perturbation estimated from the current model and trains the model to respond safely in the resulting states. This exposes the model to different combinations of harmful perturbations and reduces its sensitivity to their accumulation during subsequent harmful fine-tuning. Experiments across downstream tasks and model families show that JWP offers stronger resistance to harmful fine-tuning than prior defenses while largely preserving task performance. Our code is available at https://anonymous.4open.science/r/JWP-D977.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Model Safety, Harmful Fine-Tuning, Safety Alignment, Weight Perturbation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UaCFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an alignment-stage defense against harmful fine-tuning. It first investigates the cumulative effects of harmful weight perturbations and finds that individual components have limited impact when applied separately, whereas their joint application increases model harmfulness beyond the sum of their individual effects. Motivated by this observation, the paper introduces Joint Weight Perturbations (JWP), which estimates harmful perturbations for the current model, masks different subsets of their components, and trains the model to produce safe responses under the resulting perturbed states.

Critical strengths

The paper offers a novel perspective on safety degradation caused by harmful fine-tuning by examining the joint effects of weight perturbations. Building on this insight, it introduces JWP, a simple and straightforward method to implement. The paper is also clearly organized, with a coherent progression from motivation and method design to experimental evaluation, making it easy to follow.

Critical weaknesses
  1. Although the paper acknowledges in its limitations that joint amplification measured by the Harmful Score (HS) captures only behavioral consequences rather than an intrinsic property of the model, the current experiments do not sufficiently establish the existence of specific interactions or genuine joint amplification among different weight-perturbation components. As the perturbation retention ratio increases, the overall magnitude of the perturbation applied to the model also grows. The resulting increase in HS may therefore simply reflect the model moving progressively farther from its safety-aligned state and eventually crossing a safety boundary, rather than mutual reinforcement among perturbation components. Consequently, the logical connection between the empirical observation and the design of JWP is not yet fully substantiated.

 

  1. JWP constructs partially perturbed model states by randomly masking a harmful weight perturbation estimated through several simulated fine-tuning steps. Although the experimental results show that training on these states improves robustness to subsequent harmful fine-tuning, these states may not correspond to checkpoints along an actual fine-tuning trajectory. The paper does not establish whether training on randomly masked perturbation states is more effective than training directly on intermediate checkpoints from the simulated harmful fine-tuning trajectory.

 

  1. The paper primarily compares JWP with SFT, Vaccine, RepNoise, and Booster, but does not include more recent defense methods developed under the same threat model. Consequently, the current comparisons are insufficient to fully establish JWP’s advantages over state-of-the-art approaches. Moreover, in Table 1, when the harmful ratio reaches 0.15 or 0.20, the HS of Booster and the other baselines indicate near-complete defense failure. The authors are therefore encouraged to verify the implementations and hyperparameter settings of these methods to rule out potential reproduction issues.

 

  1. The breadth of the experimental evaluation remains limited.
    (1) In terms of task coverage, the downstream evaluations primarily focus on classification and reasoning tasks, with limited assessment of open-ended generation and instruction-following capabilities. The authors are encouraged to include generative benchmarks such as AlpacaEval.
    (2) Regarding safety evaluation, the paper relies primarily on BeaverTails for harmful data and evaluation prompts, while using Beaver-Dam-7B to compute the Harmful Score. Additional evaluations on safety benchmarks such as AdvBench and HarmBench, together with multiple safety judges or human evaluation, would help establish the robustness of the results.
    (3) Moreover, because safety alignment, harmful fine-tuning, and safety evaluation all rely heavily on BeaverTails, the experiments may be affected by same-source distribution bias.
    (4) The paper also fixes the safety-alignment dataset at 5,000 examples and the downstream fine-tuning dataset at 1,000 examples without examining how dataset size affects defense performance. The method should be evaluated across different data scales, and complete results should be reported for all downstream tasks under varying harmful-data ratios.
    (5) Finally, the sensitivity analysis should be extended to key hyperparameters, including the learning rates and so on.
Questions for authors
  1. Does the increase in Harmful Score genuinely demonstrate joint amplification among harmful weight perturbation components, rather than simply reflecting the larger perturbation magnitude caused by a higher retention probability?
  2. The experimental evaluation is insufficiently comprehensive, with missing recent baselines, limited tasks and safety benchmarks, potential same-source distribution bias, and inadequate analysis of data scale and hyperparameter sensitivity.
Reviewer_yNzgOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper offers a method to prevent harmful alignment by providing alignment-stage defense. The method, first, estimates a harmful weight perturbation by simulating multiple steps of harmful fine-tuning from the current model. Second, it masks perturbation to create partially perturbed model state. Then the model is trained to produce safe responses in this state. This way exposes the model to different combinations of harmful weight perturbations and aims to reduce its sensitivity to their joint effects. The paper offers great results in terms of Harmful Score reduction, 4.3 - 5.4x reduction in average harmful score across four downstream tasks preserving utility after safety alignment.

Critical strengths
  • The paper is easy to read and easy to understand.
  • JWP offers a great reduction in HS while preserving downstream task performance compared to competitive methods such as Vaccine and Booster. Especially, Figure 1 is impressive
  • The method is not computationally expensive even though it performs many iterations in inner loop.
  • The experiments offer an ablation and sensitivity study.
Critical weaknesses
  • My main criticism is that the observations made in the paper are very obvious and predictable. For example, why is the joint effect of disjoint subsets exceeding the sum of their separate effects surprising? Also, increasing the fraction of perturbation naturally increases the harmful effect.
  • The paper lacks mathematical motivation and theoretical analysis that support their claims. If the coordinate sets are disjoint, how is the sum different than their joint? If A and B are disjoint, then A U B = A + B - A n B = A + B. Because the authors claim that "applying the two subsets jointly increases HS more than the sum of their separate increases." Do we have only two subsets, A and B, in each iteration?
  • There are some claims that need more support: "increasing pp allows more harmful weight perturbations to reinforce one another." The cause underlying the overall increase in HS may not be weight perturbations reinforcing one another. Do the authors have other observations for their claims?
  • I suspect overfitting and data leakage. The harmful data is sampled from BeaverTails, and for calculating the Harmful Score, the Beaver-Dam-7B model, which was fine-tuned using the same dataset, is used. Does all the gain come from overfitting to the harmfulness data that is leaked during training? To make sure, can the authors carry out a different experiment using different harmful data from a domain that is different than BeaverTails
Questions for authors

I asked my questions in the weaknesses section. If the authors can answer them, I am happy to raise my score.

1 comment

I thank the authors for answering my questions; therefore, I raise my score to 3. However, I would like to make an observation. When harmfulness is judged by HarmBench ASR instead of Beaver-Dam HS, the baselines change by at most about 15%. JWP’s harmfulness roughly doubles (11.90 → 23.89 with BeaverTails attacks, 10.93 → 18.06 with HarmfulQA). Its reduction relative to SFT drops from 4.4–5.2× to 2.7–3.1×. Also, W1 is answered partly. Randomly masking coordinates also shrinks the perturbation’s norm, so this may be the same safety-basin edge reached in a different way. A direct experiment would be more convincing.

Reviewer_cmxnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper reports the steadiest, largest empirical improvements of the four, with an interesting weight-level observation of joint harmful amplification and threshold behavior connected to safety basins. Key ablations show uniform retention sampling and K are critical. Limitations include missing comparisons against TAR/Antibody/AsFT/Safe LoRA, an outdated default model, single-judge HS-only safety evaluation, a behavioral collective-failure metric, and K-dependent fragility.

Critical strengths

(1) The collective-failure finding is pivotal. It recasts defense collapse under persistent harmful fine-tuning from an amorphous robustness issue into a measurable weight-level interaction effect (joint amplification and sharp thresholding with applied fraction). This provides a grounded derivation for the method and distinguishes it from ad-hoc regularizers. (2) Large, consistent gains across tasks and model families: HS is markedly reduced versus Booster/Vaccine/RepNoise/SFT across four downstream tasks, three model families and 0–20% poisoning, with no meaningful utility loss. The efficiency results show manageable one-time alignment cost, supporting real-world deployment potential.

Critical weaknesses

(1) The baseline set is limited and weak. It only compares against SFT, Vaccine, RepNoise and Booster, excluding stronger alignment and fine-tuning defenses including TAR, Antibody and AsFT. Whether the advantage persists against these baselines is unknown, limiting the external validity of its stronger-resistance claim. (2) Concerns apply to the amplification metric and hyperparameter robustness. Joint amplification is formulated using the behavioral HS score and lacks mechanistic grounding at the model level. HS drops sharply from 68.6 to 10.3 when K grows from 10 to 250 before bouncing back at K=500, showing strong sensitivity to the tuned block length K.

Questions for authors

Clarification requests: (1) Please include TAR and Antibody as alignment baselines and confirm if JWP retains its advantage. (2) Report ASR and a second judge (human or model) besides HS to validate consistency. (3) With strong sensitivity to K and p, is there a cross-model, cross-task default setting that requires no tuning?

Future work: (4) Design a model-level metric to quantify joint amplification. (5) Test combinations with jailbreak and fine-tuning defenses (jailbreaks are not evaluated in the current submission).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.