- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes an alignment-stage defense against harmful fine-tuning. It first investigates the cumulative effects of harmful weight perturbations and finds that individual components have limited impact when applied separately, whereas their joint application increases model harmfulness beyond the sum of their individual effects. Motivated by this observation, the paper introduces Joint Weight Perturbations (JWP), which estimates harmful perturbations for the current model, masks different subsets of their components, and trains the model to produce safe responses under the resulting perturbed states.
Critical strengths
The paper offers a novel perspective on safety degradation caused by harmful fine-tuning by examining the joint effects of weight perturbations. Building on this insight, it introduces JWP, a simple and straightforward method to implement. The paper is also clearly organized, with a coherent progression from motivation and method design to experimental evaluation, making it easy to follow.
Critical weaknesses
- Although the paper acknowledges in its limitations that joint amplification measured by the Harmful Score (HS) captures only behavioral consequences rather than an intrinsic property of the model, the current experiments do not sufficiently establish the existence of specific interactions or genuine joint amplification among different weight-perturbation components. As the perturbation retention ratio increases, the overall magnitude of the perturbation applied to the model also grows. The resulting increase in HS may therefore simply reflect the model moving progressively farther from its safety-aligned state and eventually crossing a safety boundary, rather than mutual reinforcement among perturbation components. Consequently, the logical connection between the empirical observation and the design of JWP is not yet fully substantiated.
- JWP constructs partially perturbed model states by randomly masking a harmful weight perturbation estimated through several simulated fine-tuning steps. Although the experimental results show that training on these states improves robustness to subsequent harmful fine-tuning, these states may not correspond to checkpoints along an actual fine-tuning trajectory. The paper does not establish whether training on randomly masked perturbation states is more effective than training directly on intermediate checkpoints from the simulated harmful fine-tuning trajectory.
- The paper primarily compares JWP with SFT, Vaccine, RepNoise, and Booster, but does not include more recent defense methods developed under the same threat model. Consequently, the current comparisons are insufficient to fully establish JWP’s advantages over state-of-the-art approaches. Moreover, in Table 1, when the harmful ratio reaches 0.15 or 0.20, the HS of Booster and the other baselines indicate near-complete defense failure. The authors are therefore encouraged to verify the implementations and hyperparameter settings of these methods to rule out potential reproduction issues.
- The breadth of the experimental evaluation remains limited.
(1) In terms of task coverage, the downstream evaluations primarily focus on classification and reasoning tasks, with limited assessment of open-ended generation and instruction-following capabilities. The authors are encouraged to include generative benchmarks such as AlpacaEval.
(2) Regarding safety evaluation, the paper relies primarily on BeaverTails for harmful data and evaluation prompts, while using Beaver-Dam-7B to compute the Harmful Score. Additional evaluations on safety benchmarks such as AdvBench and HarmBench, together with multiple safety judges or human evaluation, would help establish the robustness of the results.
(3) Moreover, because safety alignment, harmful fine-tuning, and safety evaluation all rely heavily on BeaverTails, the experiments may be affected by same-source distribution bias.
(4) The paper also fixes the safety-alignment dataset at 5,000 examples and the downstream fine-tuning dataset at 1,000 examples without examining how dataset size affects defense performance. The method should be evaluated across different data scales, and complete results should be reported for all downstream tasks under varying harmful-data ratios.
(5) Finally, the sensitivity analysis should be extended to key hyperparameters, including the learning rates and so on.
Questions for authors
- Does the increase in Harmful Score genuinely demonstrate joint amplification among harmful weight perturbation components, rather than simply reflecting the larger perturbation magnitude caused by a higher retention probability?
- The experimental evaluation is insufficiently comprehensive, with missing recent baselines, limited tasks and safety benchmarks, potential same-source distribution bias, and inadequate analysis of data scale and hyperparameter sensitivity.
I thank the authors for answering my questions; therefore, I raise my score to 3. However, I would like to make an observation. When harmfulness is judged by HarmBench ASR instead of Beaver-Dam HS, the baselines change by at most about 15%. JWP’s harmfulness roughly doubles (11.90 → 23.89 with BeaverTails attacks, 10.93 → 18.06 with HarmfulQA). Its reduction relative to SFT drops from 4.4–5.2× to 2.7–3.1×. Also, W1 is answered partly. Randomly masking coordinates also shrinks the perturbation’s norm, so this may be the same safety-basin edge reached in a different way. A direct experiment would be more convincing.