- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how large language models, despite safety alignment, can still suffer safety degradation after fine-tuning with a small number of samples. It focuses on how such safety degradation manifests internally, analyzing whether refusal behavior concentrates in specific representation directions or network locations and how safety-related changes are distributed after fine-tuning. From the perspective of internal model mechanisms, it explores the fragility of safety alignment and provides experimental observations and analytical insights for understanding why few-shot fine-tuning may weaken LLM safety capabilities. However, the explanation of the safety damage relocation mechanism remains insufficient, and some conclusions rely mainly on empirical observations, lacking more direct causal verification.
Critical strengths
1.The research question is important and reveals that safety behavior localization and defense robustness are not necessarily correlated. 2.The layer-freezing experiments cover multiple models with relatively sufficient controls, and the finding regarding changes in the recovery transition position is of research value. 3.The authors distinguish between explicit refusal rates and actual unsafe output rates, and analyze different scenarios of defense effectiveness and failure.
Critical weaknesses
1.The logical connection among research motivation, experimental design, and core conclusions is not sufficiently natural, and writing clarity needs improvement. 2.The paper's novelty lies mainly in analyzing the failure of existing defense methods, and some core conclusions are limited by evaluation metrics and experimental coverage, making the novelty moderate. 3.Does a refusal rate near zero truly prove complete failure of safety defense? An explicit refusal rate near zero does not equate to a complete loss of safety capability. Some models still produce a large number of responses judged safe by safety evaluators after freezing, so the interpretation of defense failure requires more caution. 4.Insufficient evidence for the relocation mechanism. The change in the depth of the recovery transition layer is a clear experimental observation, but it is not sufficient to fully prove that safety-related damage undergoes mechanism-level relocation within the network. 5.Limited generality of adaptive evaluations. All attacks use the same type of supervised fine-tuning objective, and the most in-depth spectral repair adaptive evaluation is mainly limited to one model, weakening the generality of the conclusions.
Questions for authors
First, further explain the difference between refusal rate and safety. It would be better to provide more direct judgments of output harmfulness, especially explaining why some models have explicit refusal rates near zero yet still produce a considerable proportion of safe responses. Second, strengthen the mechanistic evidence for "Damage Relocates." At minimum, the shift in recovery transition position should be more clearly distinguished from a genuine change in damage location. If stronger evidence cannot be provided, it is recommended to reduce the strength of mechanistic claims. Third, add representative independent attack settings. In particular, extend the most critical weight-space adaptive evaluation to other models to test whether the findings are stable across models.
14 comments
Response to Reviewer ugUR (2/12)
W2. The missing adaptive comparison
2.The paper's novelty lies mainly in analyzing the failure of existing defense methods, and some core conclusions are limited by evaluation metrics and experimental coverage, making the novelty moderate.
Answer. The new matched comparison tests whether SPPFT's published localization adds protection under malicious training and finds no separable refusal benefit.
What the two defenses assume, and how SPPFT was evaluated. Freezing the safety layers (SPPFT, Li et al., 2025) and projecting the update (Safe LoRA, Hsu et al., 2024) were proposed on the premise that safety lives in a findable place. In a mixed setting of a thousand ordinary examples with malicious examples added, SPPFT's own Table 3 reports harmful rates of 68.3 and 78.8 percent at a malicious fraction of 0.2 against initial rates of 1.35 and 0.77 in its Table 2, so the paper reads the defense as mitigation.
That reading rests on one comparison the paper does not run, the published layers against a freeze of as many layers under malicious data. Its stated threat model gives the attacker "Limited Knowledge" of "the model's internal structure, parameters, and the location of safety layers" (Li et al., 2025, Appendix A.4.1), so an attacker who knows which layers are frozen falls outside that knowledge assumption. Our attack does not need knowledge of which layers are frozen. The three conditions share the same PKU-SafeRLHF pairs at a hundred examples, the same seeds, the same response-masked loss and the same three epochs at batch four, and differ only in which layers the provider freezes. The paper notes at Lines 297-298 that SPPFT was evaluated against backdoor and ordinary instruction data rather than against such an attacker. That is the attacker the lower block of Table 1 supplies, beside a freeze of as many layers.
At the layers Li et al. (2025) published as the ones to freeze, refusal after the attack is not separable from a freeze of as many layers on all four checkpoints they report bands for (Table 1, lower block, see Lines 292-296).
What this paper adds. Wei et al. (2024) show that restricting modifications to the sparse safety-critical regions they identify does not stop fine-tuning attacks, and Qi et al. (2025b) that small changes to the fine-tuning configuration defeat safeguards built into alignment. Li et al. (2025) ran that neuron freeze themselves at a matched proportion and report it "proved ineffective in preventing security degradation during fine-tuning, consistent with their original findings". They conclude that SPPFT "proves to be the most effective parameter-freezing defense" because it holds whole layers rather than discrete neurons. After Wei et al. (2024), then, the open question was precisely whether a whole-layer defense survives an attacker who knows it, and that is the question answered here.
What this response adds on metrics and coverage. On metrics and coverage, this response adds two further harm scorers (Table R1), a second harmful corpus (Table R2), two more harmful benchmarks (Table R3), a second training objective in DPO with the Llama 3 likelihood term (Table R9), and the weight-space adaptive evaluation on two more checkpoints (Table R6).
Response to Reviewer ugUR (3/12)
W2. Two further harmful benchmarks, continued.
some core conclusions are limited by evaluation metrics and experimental coverage
To test whether the freeze failure depends on which harmful benchmark scores it, we evaluated every frozen adapter of Table 1 on two further benchmarks, a hundred HarmBench prompts and sixty StrongREJECT prompts.
Table R3. The frozen adapters of Table 1 on two further harmful benchmarks, a hundred HarmBench prompts and sixty StrongREJECT prompts, at a dose of a hundred harmful examples, ranges over three training seeds. Refusal is coherent refusal, higher is better, and LG unsafe is the Llama-Guard-3-8B unsafe rate, lower is better. Frozen is the band the freeze protects, taken from Table 1.
| Benchmark | Model | Frozen | Refusal clean | Attack | Freeze | LG unsafe clean | Attack | Freeze |
|---|---|---|---|---|---|---|---|---|
| HarmBench | Llama-3.1-8B | [0, 17] | .84 | .00-.01 | .00 | .14 | .88-.90 | .69-.84 |
| HarmBench | Tulu-3-8B-DPO | [0, 15] | .93 | .00-.01 | .00-.05 | .05 | .83-.89 | .68-.79 |
| HarmBench | OLMo-2-7B | [0, 16] | .92 | .01-.02 | .00-.01 | .03 | .85-.92 | .65-.68 |
| HarmBench | OLMo-2-13B | [0, 20] | .86 | .00 | .00-.02 | .06 | .88-.92 | .50-.70 |
| HarmBench | Qwen2.5-14B | [0, 31] | .70 | .00-.03 | .00-.01 | .10 | .90-.92 | .56-.66 |
| HarmBench | Yi-1.5-9B | [0, 39] | .66 | .00-.01 | .02 | .30 | .90-.94 | .62-.81 |
| StrongREJECT | Llama-3.1-8B | [0, 17] | .97 | .00 | .00 | .00 | .88-.93 | .83-.85 |
| StrongREJECT | Tulu-3-8B-DPO | [0, 15] | .92 | .00 | .00-.03 | .00 | .88-.93 | .80-.82 |
| StrongREJECT | OLMo-2-7B | [0, 16] | .93 | .00 | .00 | .00 | .90-.93 | .77-.90 |
| StrongREJECT | OLMo-2-13B | [0, 20] | .92 | .00-.02 | .00 | .00 | .92-.95 | .67-.78 |
| StrongREJECT | Qwen2.5-14B | [0, 31] | .70 | .00-.03 | .00 | .00 | .92-.93 | .67-.75 |
| StrongREJECT | Yi-1.5-9B | [0, 39] | .77 | .00-.02 | .00-.02 | .23 | .85-.92 | .63-.68 |
On both benchmarks the freeze leaves coherent refusal at or below .05 while Llama-Guard rates .50 to .90 of the frozen attack's completions unsafe, so the failure does not depend on the benchmark that scores it.
W2. Contribution and coverage, continued.
some core conclusions are limited by evaluation metrics and experimental coverage
To test whether the added lineages support the failure claim, we repeated the full procedure on the additional checkpoints.
Table R8. Three further checkpoints through the full procedure. AdvBench, a hundred harmful examples, ranges over three training seeds. Depth is the transition layer by seed, Frozen is the band given by the three-seed mean, and Matched freezes as many layers at the top. The last two rows repeat the two checkpoints whose deepest seed lies above the mean band with the freeze one layer deeper, so the frozen band covers the measured depth of every seed.
| Model | Depth | Frozen | Refusal clean | Attack | Freeze | Matched | LG unsafe clean | Attack | Freeze | Matched |
|---|---|---|---|---|---|---|---|---|---|---|
| gemma-2-9b-it (42 layers) | 24, 24, 24 | [0, 24] | .99-1.00 | .00 | .00 | .00-.06 | .00 | .95-.97 | .94-.97 | .87-.96 |
| Ministral-8B-Instruct-2410 (36) | 20, 20, 17 | [0, 19] | .81-.85 | .00-.01 | .00-.05 | .01-.02 | .16-.18 | .93-.97 | .57-.86 | .89-.95 |
| Mistral-Nemo-Instruct-2407 (40) | 19, 19, 15 | [0, 18] | .68-.76 | .00-.01 | .00-.01 | .00-.01 | .22-.32 | .95-.98 | .88-.92 | .96-.98 |
| Ministral-8B-Instruct-2410 (36), one deeper | 20, 20, 17 | [0, 20] | .81-.85 | .00-.01 | .00-.01 | .00-.02 | .16-.18 | .93-.97 | .76-.88 | .94-.95 |
| Mistral-Nemo-Instruct-2407 (40), one deeper | 19, 19, 15 | [0, 19] | .68-.76 | .00-.01 | .00-.01 | .01-.03 | .22-.32 | .94-.98 | .83-.93 | .90-.98 |
All added checkpoints lose refusal under the freeze, with no separation from the equal-sized control.
It also adds three checkpoints from the Gemma and Mistral lineages, gemma-2-9b-it, Ministral-8B-Instruct-2410 and Mistral-Nemo-Instruct-2407, through the full procedure, with the transition depth measured on each seed and the freeze applied from layer 0 to the three-seed mean depth. All nine attacked seeds pass the validity checks of the depth measurement, and refusal after the frozen attack is 0.00 to 0.05 against clean rates of 0.68 to 1.00. That range includes Mistral-Nemo, which refuses only 0.68 to 0.76 when clean. On the two checkpoints whose deepest seed lies above the mean band, freezing one layer deeper so that the band covers every seed leaves refusal at 0.00 to 0.01. Appendix C of the revision reports that table.
Response to Reviewer ugUR (1/12)
Response overview and the paper's argument
Under both harm scorers we added, and under a second harmful corpus, the frozen attack stays above the clean model on every checkpoint and seed, and with the second corpus Llama-Guard rates it within 0.04 of the unrestricted attack trained on the same seed (Table R1 and Table R2). A direct intervention on the weight update locates the damage above the frozen boundary, where switching the update off restores refusal to 0.70 to 0.97 while keeping it only there leaves 0.00 to 0.35 (Table R5). We thank the reviewer for a careful review and for recognizing the controls and the separation of refusal from unsafe output.
Before writing this response, we ran the new experiments the reviews ask for and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new measurements.
Answer. The revision follows one question through localization, defense tests and the resulting checks, making their connection explicit in the main text.
The question is whether a place where refusal can be found is also a place where it can be defended (Section 1, see Lines 065-071). Section 4 finds the place, with a frozen probe and with clean-state patching that locates a transition depth. Section 5 defends it by freezing every layer up to that depth and attacking again with an attacker who knows the frozen layers. Section 6 asks the same question in weight space, where top-two removal repairs one training regime and an adaptive attacker defeats it. Section 7 turns each failure into a check that a localization must pass before it supports a defense. That chain is the connection the review found missing. The revision also rewords the abstract and Sections 1 to 7 for clarity, replaces the submission's nonstandard terms with standard ones defined at first use, and states the conclusion in the main text as Section 8. We would be grateful if the reviewer would read the chain in that form.
Localization, defense evaluation and the resulting checks form the argument stated in the revised main text.
The revised main text follows one question through each test, so the argument can be followed without the appendix.
Revised-PDF location. The revision states the chain in a new closing paragraph of Section 1, opens Sections 6 and 7 and the subsections of Sections 4 to 6 with what each tests.