acceptodds
Withdrawn from ICLR 2027

When Heads Reveal Safety: Causal Layer Freezing for LLM Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-tuning aligned large language models (LLMs) can weaken their safety behavior, even when the adaptation data are benign. Freezing safety-related components during fine-tuning offers a direct way to preserve alignment, but the component that reveals where safety behavior is localized may not be sufficient as a unit of protection. In particular, freezing individual attention heads can leave other computation pathways in the same layer trainable, whereas freezing a broad contiguous block may unnecessarily restrict task adaptation. We propose Head-Causal Layer Freezing (HeaCaLF), which separates safety localization from protection. HeaCaLF uses activation interventions to estimate the safety contribution of individual attention heads, aggregates these signals by layer, and freezes the selected layers during fine-tuning. Across four aligned LLMs and multiple downstream tasks, HeaCaLF reduces harmful responses relative to standard fine-tuning while retaining competitive task performance. Our granularity analysis further shows that freezing identified heads alone provides limited protection, whereas layer-level freezing offers a more effective safety–utility trade-off.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safety Alignment, Fine-Tuning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FEuhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work investigates the ability to preserve safety alignment while performing benign performance fine-tuning. Considering that freezing individual safety-related attention heads might leave adaptation paths for other heads and the MLP in the same layer, while directly freezing consecutive layers might limit too many task capabilities, HeaCaLF uses activation patching to measure the contribution of each head to rejection behavior, aggregates head-level scores to layers, and selects several non-consecutive layers to freeze as a whole.

Critical strengths

This design problem is valid. Identifying the heads involved in the safety behavior does not mean that simply freezing these heads guarantees safety after fine-tuning. Table 1 shows the safety gains across multiple models. For example, Llama-2's average HS decreased from 48.75 with SFT to 25.91, and its average task accuracy improved from 63.71 to 63.40. Table 3 compares random selection, SPPFT, and HeaCaLF with the same number of frozen layers; when k=3, the MMLU HS is 31.20, superior to SPPFT's 40.35. This demonstrates useful selection strategy control.

The authors also honestly acknowledge that utility is not always maintained: for example, Mistral's GSM8K accuracy decreased from 41.40 to 29.27. Therefore, the results should be understood as a safety-utility tradeoff, not lossless protection.

Critical weaknesses

W1: The core measurement protocol is inconsistent across different sections.

Section 3.4 states that QwQ receives the original instruction and generated text, outputting two non-negative scores; Appendix D, however, states that the target model's logits are input into QwQ. These are different processes, directly affecting how SCS is reproduced.

W2: The metrics in Figure 3(a) do not match the formula.

The figure caption indicates SCS, but the colored bar reads "Logit change," with a range of approximately [−4,4][-4,4].

W3: The data scale is inconsistent between the main text and the appendix.

The main text states that 1,400 BeaverTails samples are used, while Appendix B lists 1,177; the appendix also lists 983 AdvBench harmful instructions, which do not correspond to the standard's 520 behaviors.

W4: Appendix D references Equations 10 and 12, but the formula numbering in this paper only goes up to 8; Equation 8 actually corresponds to the layer selection, not the activation intervention mentioned there.

Questions for authors

See weaknesses and below

Q1: Does the actual implementation of SCS score the generated text or process logits? Please provide the complete judge prompt, numerical analysis, and code path.

Q2: Please unify the counting across all datasets, provide the specific division between the localization set and the evaluation set, and correct the formula numbers in the appendix.

Q3: After correcting the protocol, can you provide multiple sub-outcomes, localization costs, and stable comparisons under equal frozen budgets?

Reviewer_2fFGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Fine-tuning after safety alignment can end up degrading safety, even if the data used is harmless. State-of-the-art works attempt to mitigate this safety by detecting where in the model the safety resides (layers, heads, etc.) and attempt to freeze them during fine-tuning to preserve the safety while also fitting on downstream tasks. This paper proposes HeaCaLF, which uses head-level mechanistic determination of safety-critical paths by comparing harmless and harmful prompts, i.e. it detects which heads are most helpful for safety by turning them on/off on safe vs. harmful prompts. It then aggregates the individual heads into finding which layers are most impactful (based on the heads' impact). This results in denoting scattered layers instead of contiguous layers of blocks or heads themselves. These layers are then frozen while fine-tuning, the claim here being that this leaves more of the model free to adapt to fine-tunable tasks. In doing so, they get better results such as harmfulness dropping from 59.82 to 23.66 on Llama-3-8B versus plain fine-tuning, and higher average task accuracy than state-of-the-art SPPFT on all models tested. They also provide ablation studies testing freezing granularity, layer-selection strategy, the number of frozen layers, and robustness to learning rate and to harmful data mixed into training, demonstrating that freezing the identified heads alone barely helps while freezing their containing layers does.

Critical strengths

Strengths -

  1. The paper is well written and fairly easy to follow.
  2. The topic is of great importance in the current research landscape.
Critical weaknesses

W1. Limited novelty. The two pieces of the method both exist already. Zhou et al. (ICLR 2025) score individual attention heads by intervening on them. SPPFT and SSAH/Freeze+ freeze localized components during fine-tuning. SafeSeek (ICML 2026), uncited, already lets the user pick the granularity to protect — heads or layers. What is left is the step from Equations 5–6 to Equation 7: turning per-head scores into a count that picks layers. That seems to be a design choice only.

W2. Compute cost is impractical. Scoring every head means one full generation per head per prompt pair, plus a judge call on each. For Llama-2-7B that is 32 layers × 32 heads × 100 pairs ≈ 102,400 generations and the same number of QwQ-32B calls — for one model, and a small one. Cost grows with both depth and width, so it scales worse than model size. The paper reports no runtime or GPU hours, and the gains over SPPFT do not obviously justify this.

W3. Results are inconsistent across tables. Tables 1 and 4 report the same model, task, and method with different numbers. The text never says these are separate runs or different setups. Until this is explained, the comparisons built on these tables are confusing.

W4. Key settings are unjustified and untested. The threshold τ is fixed at 0 with no explanation and never varied, even though it decides which heads count. N = 100 prompt pairs is never varied either. The per-model values of K contain no selection procedure, so if they were chosen using the harmful evaluation set, the safety comparison is contaminated. The ε in Equation 5 is never given at all. Sensitivity analysis for all of them are required.

W5. RQ3’s comparison is unclear. The paper notes for Table 3 that it is “a layer-count-controlled comparison, not a parameter-matched comparison” but the parameter-matched version is what RQ3 actually needs. Head-only and layer-level freezing differ in granularity and in how much is frozen, with layer freezing locking far more, most of it MLP that head-only never touches. Without a budget-matched control, the "surrounding components route around a frozen head" cannot be distinguished from the claim that “freezing more is simply safer".

Questions for authors
  1. How have all the parameters like K, epsilon, etc. been chosen?
  2. Please clarify the discrepancies between the tables 1 and 4, where Mistral-7B with SFT on GSM8K gives a score of 79.82 in Table 1 and 60.12 in Table 4. Gemma-2-9B and Llama-3-8B also do not match.
  3. What is the logic behind deriving Eq 5?
Reviewer_ubgNOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HeaCaLF, which uses heads as probes and layers as protection units. Across four aligned LLMs and six tasks, the paper reports lower Harmfulness Score than SFT and SPPFT, higher average accuracy than SPPFT.

Critical strengths

The paper provides broader empirical base: the method is validated on four model families, six datasets, and four harmful benchmarks.

Critical weaknesses
  1. Several baseline results appear substantially degraded. The authors should clarify whether the baseline implementations, hyperparameters, and tuning procedures were carefully validated. In addition, Table 1 should also report the number of frozen layers used by each baseline.

  2. The evaluation is limited to full-parameter supervised fine-tunning, without evaluating LoRA or other parameter-efficient fine-tuning methods.

  3. Section 3.4 refers to a QwQ/classifier agreement analysis in the supplementary material, but this analysis does not appear in appendices.

  4. Appendix D cites equation (10) and equation (12), which however do not appear to exist.

  5. The references also contain apparent duplicate entries, e.g., Jianhui Chen, Xiaozhi Wang, Zijun Yao, Yushi Bai, Lei Hou, and Juanzi Li. Finding safety neurons in large language models. arXiv preprint arXiv:2406.14144,2024a. Jianhui Chen, Xiaozhi Wang, Zijun Yao, Yushi Bai, Lei Hou, and Juanzi Li. Finding safety neurons in large language models. arXiv preprint arXiv:2406.14144,2024b.

Questions for authors
  1. Several baseline results appear substantially degraded. The authors should clarify whether the baseline implementations, hyperparameters, and tuning procedures were carefully validated. In addition, Table 1 should also report the number of frozen layers used by each baseline.
  2. The evaluation is limited to full-parameter supervised fine-tunning, without evaluating LoRA or other parameter-efficient fine-tuning methods.
  3. Section 3.4 refers to a QwQ/classifier agreement analysis in the supplementary material, but this analysis does not appear in appendices.
  4. Appendix D cites equation (10) and equation (12), which however do not appear to exist.
  5. The references also contain apparent duplicate entries.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.