- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper audits the modular account of safety in instruction-tuned LLMs, specifically the claim that “safety neurons” can be localized and pruned to remove refusal behavior while preserving general competence. The authors compare Wanda, SNIP, and Representation Perturbation under Harmful Consistency and Contrastive Differential scoring on Llama-3-8B, Gemma-7B, and Qwen2.5-7B. The paper evaluates pruning using random pruning, benign-salience pruning, unpruned self-variance, KL divergence, NLI, embedding distance, hidden-state trajectories, and small-scale human annotation. It also restores either the higher- or lower-scoring half of the removed neurons to test whether refusal and semantic stability can be separated. The main conclusions are that neuron localization is architecture- and criterion-dependent, ASR can mistake generative collapse for safety, safety-targeted pruning induces semantic and hidden-state changes beyond random pruning, and restoring high-scoring neurons jointly restores refusal and semantic stability. The authors therefore argue that existing pruning- and ASR-based evidence is insufficient to establish a modular safety system with preserved competence.
Strengths
- Technical novelty and innovation: The paper addresses an often-neglected premise of prior work: whether general capabilities are actually preserved after safety-neuron removal. The locate-remove-restore experimental structure is well motivated. The observation that Harmful Consistency pruning can produce generative collapse rather than genuine refusal removal is also valuable.
- Experimental rigor and validation: The study includes random-pruning, benign-salience, and unpruned self-variance controls. The scoring and evaluation prompts are intended to be disjoint. Multiple models, scoring methods, pruning ratios, semantic metrics, hidden-state analyses, and human checks provide a broader evidence base than papers relying only on ASR.
Critical strengths
Strengths
- Technical novelty and innovation: The paper addresses an often-neglected premise of prior work: whether general capabilities are actually preserved after safety-neuron removal. The locate-remove-restore experimental structure is well motivated. The observation that Harmful Consistency pruning can produce generative collapse rather than genuine refusal removal is also valuable.
- Experimental rigor and validation: The study includes random-pruning, benign-salience, and unpruned self-variance controls. The scoring and evaluation prompts are intended to be disjoint. Multiple models, scoring methods, pruning ratios, semantic metrics, hidden-state analyses, and human checks provide a broader evidence base than papers relying only on ASR.
Critical weaknesses
Weaknesses
-
The paper does not fully separate “the current pruning intervention cannot achieve disentanglement” from “refusal and general semantics are intrinsically non-separable.” Structured pruning is a destructive, nonlinear intervention. The restoration experiment tests only one scoring method, one pruning scope, and one median split, so it cannot rule out finer grained or alternative.
-
The study covers only a few dense models in the 7B to 8B range. Its claims do not generalize strongly to larger models, mixture-of-experts systems, alternative alignment pipelines, or other languages.
-
The restoration experiment contains only two points, restoring the higher or lower scoring half. It lacks restoration curves, random-half restoration, layer-wise restoration, matrix wise restoration, and independent ranking controls. It therefore cannot determine whether a finer-grained disentanglement exists.
Questions for authors
- Could the Llama-3-70B judge introduce family-specific bias against Llama-3-8B? Have judges from different model families or blinded human evaluations been tested?
- Have the authors tested restoration fractions such as 10%, 25%, 50%, 75%, and 90%, as well as random, layer-wise, and matrix-wise restoration? Was the median split selected before observing the results?
- How are semantic contradiction, factual error, conceptual inversion, repetition degeneration, and refusal distinguished?
4 comments
Q2: Generalization Across Model Scales and Alignment Pipelines
We appreciate the reviewer for highlighting the importance of evaluating larger models and diverse alignment pipelines. To address the scaling concern, we extended our evaluation to Qwen2.5-32B-Instruct, enabling a within-family comparison with Qwen2.5-7B-Instruct under a broadly shared post-training framework. We replicated the original experiments on safety-neuron identification (Table 1) and semantic stability (Table 2), using the same evaluation protocols.
- (1) ASR (%) across scoring configurations
| Model | Unpruned | Random | Benign | CD-Wanda | CD-RP | CD-SNIP | HC-Wanda | HC-RP | HC-SNIP |
|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-32B-Instruct | 0.67±0.3 | 0.78±0.4 | 0.67±0.3 | 80.89±0.5 | 91.78±1.0 | 98.89±0.8 | 79.89±0.5 | 64.78±0.7 | 73.11±1.3 |
- (2) Semantic contradiction (%) under CD-RP
| Condition | Probe-Contr. | H-Contr. | B-Contr. |
|---|---|---|---|
| Safety-pruned | 22.35±5.2 | 39.96±3.9 | 8.05±0.9 |
| Randomly-pruned | 6.11±1.1 | 13.78±4.1 | 7.33±1.0 |
| Unpruned | 5.03±0.9 | 11.43±2.8 | 7.50±1.4 |
These results reproduce our original findings at a larger model scale. Under CD-RP, Qwen2.5-32B reaches 91.78% ASR, comparable to 92.33% for Qwen2.5-7B. More importantly, harmful-domain contradiction increases from 11.43% to 39.96%, a 28.53-percentage-point increase, and substantially exceeds the randomly-pruned baseline (13.78%). Meanwhile, benign-domain contradiction remains relatively stable. These results extend our findings to 32B models, showing that the disproportionate semantic changes in safety-sensitive contexts persist at a larger scale.
Regarding alignment pipelines, our original evaluation already covers models with distinct post-training recipes. Llama 3 uses SFT, rejection sampling, PPO, and DPO; Gemma employs SFT and RLHF; and Qwen2.5 adopts SFT with multi-stage preference optimization, including DPO and GRPO. Together with Mistral in Appendix B.1.4, our evaluation spans four model families with diverse post-training recipes. Our layer-wise structural pruning analysis (Figure 2) also reveals distinct localization patterns across model families: Llama exhibits distributed Attention–MLP sensitivity, while Gemma and Qwen show more concentrated mid-layer MLP effects.
We agree with the reviewer that MoE architectures and multilingual settings are important directions for assessing broader generalization. Given the time and computational constraints of the rebuttal period, we will discuss these extensions explicitly in the final manuscript and include further experiments where feasible. We will also incorporate the larger-model results and clarify the scope and limitations of our findings.
Q3: Cross-Family Judge Validation
We agree with the reviewer's concern about potential family-specific bias in LLM-based evaluation. We therefore conducted additional evaluations using two judges from different model families, Qwen2.5-72B-Instruct and Gemma-4-31B-IT, alongside the original Llama-3-70B judge. We applied the same evaluation protocol across all three judges and compared ASR across the original baseline and CD-based pruning configurations.
| Model | Condition | Method | Llama-3-70B | Qwen2.5-72B-Instruct | Gemma-4-31B-IT |
|---|---|---|---|---|---|
| Llama-3-8B | Baseline | Unpruned | 2.11 ± 0.2 | 2.11 ± 0.2 | 2.22 ± 0.2 |
| Random | 1.67 ± 0.3 | 1.67 ± 0.3 | 1.67 ± 0.3 | ||
| Benign | 1.67 ± 0.2 | 1.67 ± 0.3 | 1.67 ± 0.3 | ||
| CD | Wanda | 22.34 ± 0.3 | 22.22 ± 0.2 | 22.34 ± 0.3 | |
| RP | 42.67 ± 0.3 | 42.67 ± 0.3 | 42.67 ± 0.0 | ||
| SNIP | 93.33 ± 0.3 | 93.44 ± 0.4 | 93.33 ± 0.3 | ||
| Gemma-7B | Baseline | Unpruned | 6.11 ± 0.2 | 6.11 ± 0.2 | 6.11 ± 0.2 |
| Random | 6.67 ± 0.3 | 6.67 ± 0.0 | 6.67 ± 0.3 | ||
| Benign | 2.67 ± 0.2 | 2.67 ± 0.0 | 2.67 ± 0.2 | ||
| CD | Wanda | 61.44 ± 0.5 | 61.44 ± 0.5 | 61.22 ± 0.4 | |
| RP | 85.00 ± 0.3 | 85.00 ± 0.3 | 85.00 ± 0.3 | ||
| SNIP | 99.67 ± 0.0 | 99.67 ± 0.0 | 99.67 ± 0.0 | ||
| Qwen2.5-7B | Baseline | Unpruned | 1.67 ± 0.3 | 1.67 ± 0.3 | 1.78 ± 0.2 |
| Random | 2.00 ± 0.3 | 2.00 ± 0.3 | 2.00 ± 0.3 | ||
| Benign | 1.67 ± 0.3 | 1.67 ± 0.3 | 1.67 ± 0.3 | ||
| CD | Wanda | 81.44 ± 0.2 | 81.44 ± 0.2 | 81.44 ± 0.4 | |
| RP | 92.33 ± 0.3 | 92.33 ± 0.3 | 92.33 ± 0.3 | ||
| SNIP | 100.00 ± 0.0 | 100.00 ± 0.0 | 100.00 ± 0.0 |
The results show highly consistent ASR measurements across all three judge families, with a maximum difference of only 0.22 percentage points in mean ASR across all evaluated configurations. In particular, all three judges report identical ASR (42.67%) for Llama-3-8B under CD-RP pruning. The consistency across Llama, Gemma, and Qwen supports the robustness of our primary findings to judge selection, with no systematic family-specific discrepancy evident in these results.
Together with our existing human-annotation validation in Appendix A.2.3, these additional cross-family evaluations strengthen the reliability of our safety assessment. We thank the reviewer for highlighting this concern and will add the comparison to the appendix to provide a more comprehensive validation of our evaluation protocol.
Q4: Clarifying Semantic and Behavioral Changes
We thank the reviewer for raising this question. In our study, these terms describe related but distinct aspects of generation behavior, which may overlap. Refusal-related behavior is assessed through ASR, which measures successful harmful responses under our dual-stage safety evaluation protocol (Section 3 and Appendix A.2). Semantic contradiction measures NLI-based incompatibility between pruned and unpruned responses (Section 5 and Appendix B.2.5). Conceptual inversion and repetitive degeneration are qualitatively distinct phenomena illustrated in Figure 3: the former involves reversing or distorting a concept's meaning, while the latter involves repetitive or tautological generation. Factual errors may overlap with conceptual inversion and are discussed qualitatively without separate quantitative measurement.
We agree that clearer distinctions would improve the presentation of these concepts. We appreciate the reviewer's suggestion and will clarify their definitions and usage in the relevant sections of the revised manuscript.
keep my score unchanged
Thank you for the thorough experiments. However, I have reservations about the paper’s novelty. I will keep my score unchanged but lower my confidence, leaving the final assessment to the Area Chair.
Q1: Finer-grained Modularity Test
We agree with the reviewer that finer-grained restoration can strengthen the assessment of functional disentanglement. We therefore conducted four complementary analyses across Llama, Gemma, and Qwen: (1) restoration curves (1%–99%) with random controls, (2) restoration across six scoring configurations, (3) independent layer-wise and module-wise restoration, and (4) cumulative layer-wise restoration. All experiments evaluate both ASR and harmful-domain NLI contradiction, with the remaining settings following the main paper. The original median split was simply a 50/50 division for comparing equally sized neuron groups, without optimizing the threshold based on restoration outcomes.
These experiments extend our original median-split findings across a broader range of restoration strategies. In Llama, both CD-ranked and random restoration progressively recover refusal behavior and semantic stability. In Gemma, all six scoring configurations yield similar post-restoration ASR (12.78–14.00%) and contradiction rates (14.73–16.12%). In Qwen, recovery is concentrated in middle-layer MLPs, consistent with our localization findings in Section 4. Cumulative restoration brings both metrics close to their unpruned baselines by layer 16 (ASR: 2.00%; contradiction: 17.45%). Together, these results demonstrate broadly joint recovery across the tested interventions.
The finer-grained analyses also reveal meaningful recovery asymmetries. Restoring Qwen's MLP layers 13–16 reduces harmful-domain contradiction from 41.37% to 19.91% and ASR from 92.33% to 42.78%. Contradiction approaches its unpruned baseline more closely, suggesting partial dissociation under structural partitioning. This finding supports the reviewer's insight that finer partitions can reveal functional relationships that a median split may obscure. The additional experiments thus provide a more detailed understanding of how refusal and semantic stability interact under pruning-based interventions. We will add these results to the appendix and refine the Discussion to address both the joint recovery patterns and the observed asymmetries, providing a more precise characterization of the evidence for functional modularity.