acceptodds
Under review as a conference paper at ICLR 2027

Locating, Removing, and Restoring: Auditing the Modular Account of Safety in Instruction-Tuned LLMs

Abstract

Safety alignment is typically realised through post-training interventions, and a prominent line of work has proposed a modular account in which sparse sets of “safety neurons” mediate refusal. This account carries an operational claim: that such neurons can be located and pruned to remove refusal while leaving broader generative competence largely intact — a claim evaluated almost entirely through attack success rate (ASR). We examine it at three stages: locating the neurons, removing them, and putting them back. Locating the neurons is not a settled procedure: across four open-weight instruction-tuned families (three in the main text), localisation is architecture-dependent, ranging from concentrated mid-layer MLP blocks to structure distributed across attention and MLP subspaces, and the two scoring strategies in common use are not interchangeable — under one of them, one architecture does not de-align at all but collapses into incoherent generation, and a second does so at a higher ratio, which ASR records as perfect safety. Removing the neurons changes more than refusal: measured against random pruning and intrinsic self-variance, next-token distributions, semantic content, and hidden-state trajectories all shift, with harmful-domain contradiction rising 25–29 points over the unpruned baseline; benign-domain effects are weaker and model-dependent, and most deviations are not accompanied by conspicuous surface breakage, so many would not be caught by inspection for obvious degeneration. Putting the neurons back shows that the two effects are not separated under a median split of the criterion's own ranking: reinstating the higher-scoring half of the removed neurons recovers 91–95% of the ASR shift and 84–86% of the contradiction shift together, while the lower-scoring half recovers 8–18% of either, in all three architectures. That the semantic effect reverses at all rules out irreversible damage inflicted by the act of pruning; under this split, neither half recovers refusal without also recovering semantic stability, leaving the modular reading unsupported by the criterion's own ranking. We conclude that pruning-based evidence does not support the modular reading, and that ASR is an insufficient basis for claims about preserved competence.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Safety, Safety Alignment, Safety Neurons, Mechanistic Interpretability, Refusal Mechanisms, Neuron Pruning, Representation Analysis, Model Editing, Semantic Stability, Alignment Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ru1NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits the modular account of safety in instruction-tuned LLMs, specifically the claim that “safety neurons” can be localized and pruned to remove refusal behavior while preserving general competence. The authors compare Wanda, SNIP, and Representation Perturbation under Harmful Consistency and Contrastive Differential scoring on Llama-3-8B, Gemma-7B, and Qwen2.5-7B. The paper evaluates pruning using random pruning, benign-salience pruning, unpruned self-variance, KL divergence, NLI, embedding distance, hidden-state trajectories, and small-scale human annotation. It also restores either the higher- or lower-scoring half of the removed neurons to test whether refusal and semantic stability can be separated. The main conclusions are that neuron localization is architecture- and criterion-dependent, ASR can mistake generative collapse for safety, safety-targeted pruning induces semantic and hidden-state changes beyond random pruning, and restoring high-scoring neurons jointly restores refusal and semantic stability. The authors therefore argue that existing pruning- and ASR-based evidence is insufficient to establish a modular safety system with preserved competence.

Strengths

  • Technical novelty and innovation: The paper addresses an often-neglected premise of prior work: whether general capabilities are actually preserved after safety-neuron removal. The locate-remove-restore experimental structure is well motivated. The observation that Harmful Consistency pruning can produce generative collapse rather than genuine refusal removal is also valuable.
  • Experimental rigor and validation: The study includes random-pruning, benign-salience, and unpruned self-variance controls. The scoring and evaluation prompts are intended to be disjoint. Multiple models, scoring methods, pruning ratios, semantic metrics, hidden-state analyses, and human checks provide a broader evidence base than papers relying only on ASR.
Critical strengths

Strengths

  • Technical novelty and innovation: The paper addresses an often-neglected premise of prior work: whether general capabilities are actually preserved after safety-neuron removal. The locate-remove-restore experimental structure is well motivated. The observation that Harmful Consistency pruning can produce generative collapse rather than genuine refusal removal is also valuable.
  • Experimental rigor and validation: The study includes random-pruning, benign-salience, and unpruned self-variance controls. The scoring and evaluation prompts are intended to be disjoint. Multiple models, scoring methods, pruning ratios, semantic metrics, hidden-state analyses, and human checks provide a broader evidence base than papers relying only on ASR.
Critical weaknesses

Weaknesses

  • The paper does not fully separate “the current pruning intervention cannot achieve disentanglement” from “refusal and general semantics are intrinsically non-separable.” Structured pruning is a destructive, nonlinear intervention. The restoration experiment tests only one scoring method, one pruning scope, and one median split, so it cannot rule out finer grained or alternative.

  • The study covers only a few dense models in the 7B to 8B range. Its claims do not generalize strongly to larger models, mixture-of-experts systems, alternative alignment pipelines, or other languages.

  • The restoration experiment contains only two points, restoring the higher or lower scoring half. It lacks restoration curves, random-half restoration, layer-wise restoration, matrix wise restoration, and independent ranking controls. It therefore cannot determine whether a finer-grained disentanglement exists.

Questions for authors
  • Could the Llama-3-70B judge introduce family-specific bias against Llama-3-8B? Have judges from different model families or blinded human evaluations been tested?
  • Have the authors tested restoration fractions such as 10%, 25%, 50%, 75%, and 90%, as well as random, layer-wise, and matrix-wise restoration? Was the median split selected before observing the results?
  • How are semantic contradiction, factual error, conceptual inversion, repetition degeneration, and refusal distinguished?
4 comments

Q1: Finer-grained Modularity Test

We agree with the reviewer that finer-grained restoration can strengthen the assessment of functional disentanglement. We therefore conducted four complementary analyses across Llama, Gemma, and Qwen: (1) restoration curves (1%–99%) with random controls, (2) restoration across six scoring configurations, (3) independent layer-wise and module-wise restoration, and (4) cumulative layer-wise restoration. All experiments evaluate both ASR and harmful-domain NLI contradiction, with the remaining settings following the main paper. The original median split was simply a 50/50 division for comparing equally sized neuron groups, without optimizing the threshold based on restoration outcomes.

  • Finer-grained restoration curves (Llama-3-8B, restoring lower-scoring neurons first)
MethodMetric0%1%5%10%25%50%75%90%95%99%
CD-RPASR42.67±0.342.56±0.541.33±0.339.44±0.538.78±0.737.67±0.319.44±0.84.89±0.54.78±0.82.11±0.2
Contradiction-H37.19±7.037.04±6.536.57±5.335.59±6.635.03±5.334.87±5.422.88±4.612.32±1.410.11±1.08.94±0.5
CD-RP (random)ASR42.67±0.342.33±0.340.22±0.235.67±0.930.00±1.521.67±2.811.56±2.55.00±2.64.11±0.82.11±0.2
Contradiction-H37.19±7.036.78±6.835.48±7.130.21±5.925.76±6.116.91±5.312.97±5.110.48±3.29.32±2.49.08±1.0
  • Restoration across scoring configurations (Gemma-7B, restoring the higher-scoring half)
ModelMetricCD-WandaCD-RPCD-SNIPHC-WandaHC-RPHC-SNIP
Gemma-7BASR12.89±0.213.34±0.212.89±0.513.00±0.612.78±0.514.00±0.3
Contradiction-H14.73±0.515.28±0.915.14±1.115.78±0.514.99±0.416.12±0.7
  • Layer-wise and module-wise restoration (Qwen2.5-7B)
MethodMetric1–45–89–1213–1617–2021–2425–28
AttnASR92.33±0.392.22±0.292.33±0.392.33±0.392.22±0.492.33±0.392.33±0.3
Contradiction-H41.32±1.141.18±1.041.20±0.841.31±0.841.58±1.340.99±1.041.72±1.0
MLPASR91.89±0.482.56±0.364.67±0.242.78±0.292.00±0.392.11±0.292.11±0.2
Contradiction-H41.05±0.837.66±2.425.19±3.819.91±3.541.47±0.641.53±1.041.31±0.5
Attn and MLPASR91.89±0.482.44±0.264.44±0.242.78±0.292.00±0.392.11±0.292.11±0.2
Contradiction-H41.09±0.737.83±2.725.20±4.319.65±4.441.81±0.841.22±1.041.34±0.5
  • Cumulative layer-wise restoration (Qwen2.5-7B)
MethodMetric1–41–81–121–161–201–241–28
Attn and MLPASR91.89±0.480.33±0.352.89±0.52.00±0.31.78±0.41.67±0.31.67±0.3
Contradiction-H41.09±0.735.88±3.024.30±5.117.45±2.215.99±3.716.31±2.515.82±4.3

These experiments extend our original median-split findings across a broader range of restoration strategies. In Llama, both CD-ranked and random restoration progressively recover refusal behavior and semantic stability. In Gemma, all six scoring configurations yield similar post-restoration ASR (12.78–14.00%) and contradiction rates (14.73–16.12%). In Qwen, recovery is concentrated in middle-layer MLPs, consistent with our localization findings in Section 4. Cumulative restoration brings both metrics close to their unpruned baselines by layer 16 (ASR: 2.00%; contradiction: 17.45%). Together, these results demonstrate broadly joint recovery across the tested interventions.

The finer-grained analyses also reveal meaningful recovery asymmetries. Restoring Qwen's MLP layers 13–16 reduces harmful-domain contradiction from 41.37% to 19.91% and ASR from 92.33% to 42.78%. Contradiction approaches its unpruned baseline more closely, suggesting partial dissociation under structural partitioning. This finding supports the reviewer's insight that finer partitions can reveal functional relationships that a median split may obscure. The additional experiments thus provide a more detailed understanding of how refusal and semantic stability interact under pruning-based interventions. We will add these results to the appendix and refine the Discussion to address both the joint recovery patterns and the observed asymmetries, providing a more precise characterization of the evidence for functional modularity.

Q2: Generalization Across Model Scales and Alignment Pipelines

We appreciate the reviewer for highlighting the importance of evaluating larger models and diverse alignment pipelines. To address the scaling concern, we extended our evaluation to Qwen2.5-32B-Instruct, enabling a within-family comparison with Qwen2.5-7B-Instruct under a broadly shared post-training framework. We replicated the original experiments on safety-neuron identification (Table 1) and semantic stability (Table 2), using the same evaluation protocols.

  • (1) ASR (%) across scoring configurations
ModelUnprunedRandomBenignCD-WandaCD-RPCD-SNIPHC-WandaHC-RPHC-SNIP
Qwen2.5-32B-Instruct0.67±0.30.78±0.40.67±0.380.89±0.591.78±1.098.89±0.879.89±0.564.78±0.773.11±1.3
  • (2) Semantic contradiction (%) under CD-RP
ConditionProbe-Contr.H-Contr.B-Contr.
Safety-pruned22.35±5.239.96±3.98.05±0.9
Randomly-pruned6.11±1.113.78±4.17.33±1.0
Unpruned5.03±0.911.43±2.87.50±1.4

These results reproduce our original findings at a larger model scale. Under CD-RP, Qwen2.5-32B reaches 91.78% ASR, comparable to 92.33% for Qwen2.5-7B. More importantly, harmful-domain contradiction increases from 11.43% to 39.96%, a 28.53-percentage-point increase, and substantially exceeds the randomly-pruned baseline (13.78%). Meanwhile, benign-domain contradiction remains relatively stable. These results extend our findings to 32B models, showing that the disproportionate semantic changes in safety-sensitive contexts persist at a larger scale.

Regarding alignment pipelines, our original evaluation already covers models with distinct post-training recipes. Llama 3 uses SFT, rejection sampling, PPO, and DPO; Gemma employs SFT and RLHF; and Qwen2.5 adopts SFT with multi-stage preference optimization, including DPO and GRPO. Together with Mistral in Appendix B.1.4, our evaluation spans four model families with diverse post-training recipes. Our layer-wise structural pruning analysis (Figure 2) also reveals distinct localization patterns across model families: Llama exhibits distributed Attention–MLP sensitivity, while Gemma and Qwen show more concentrated mid-layer MLP effects.

We agree with the reviewer that MoE architectures and multilingual settings are important directions for assessing broader generalization. Given the time and computational constraints of the rebuttal period, we will discuss these extensions explicitly in the final manuscript and include further experiments where feasible. We will also incorporate the larger-model results and clarify the scope and limitations of our findings.

Q3: Cross-Family Judge Validation

We agree with the reviewer's concern about potential family-specific bias in LLM-based evaluation. We therefore conducted additional evaluations using two judges from different model families, Qwen2.5-72B-Instruct and Gemma-4-31B-IT, alongside the original Llama-3-70B judge. We applied the same evaluation protocol across all three judges and compared ASR across the original baseline and CD-based pruning configurations.

ModelConditionMethodLlama-3-70BQwen2.5-72B-InstructGemma-4-31B-IT
Llama-3-8BBaselineUnpruned2.11 ± 0.22.11 ± 0.22.22 ± 0.2
Random1.67 ± 0.31.67 ± 0.31.67 ± 0.3
Benign1.67 ± 0.21.67 ± 0.31.67 ± 0.3
CDWanda22.34 ± 0.322.22 ± 0.222.34 ± 0.3
RP42.67 ± 0.342.67 ± 0.342.67 ± 0.0
SNIP93.33 ± 0.393.44 ± 0.493.33 ± 0.3
Gemma-7BBaselineUnpruned6.11 ± 0.26.11 ± 0.26.11 ± 0.2
Random6.67 ± 0.36.67 ± 0.06.67 ± 0.3
Benign2.67 ± 0.22.67 ± 0.02.67 ± 0.2
CDWanda61.44 ± 0.561.44 ± 0.561.22 ± 0.4
RP85.00 ± 0.385.00 ± 0.385.00 ± 0.3
SNIP99.67 ± 0.099.67 ± 0.099.67 ± 0.0
Qwen2.5-7BBaselineUnpruned1.67 ± 0.31.67 ± 0.31.78 ± 0.2
Random2.00 ± 0.32.00 ± 0.32.00 ± 0.3
Benign1.67 ± 0.31.67 ± 0.31.67 ± 0.3
CDWanda81.44 ± 0.281.44 ± 0.281.44 ± 0.4
RP92.33 ± 0.392.33 ± 0.392.33 ± 0.3
SNIP100.00 ± 0.0100.00 ± 0.0100.00 ± 0.0

The results show highly consistent ASR measurements across all three judge families, with a maximum difference of only 0.22 percentage points in mean ASR across all evaluated configurations. In particular, all three judges report identical ASR (42.67%) for Llama-3-8B under CD-RP pruning. The consistency across Llama, Gemma, and Qwen supports the robustness of our primary findings to judge selection, with no systematic family-specific discrepancy evident in these results.

Together with our existing human-annotation validation in Appendix A.2.3, these additional cross-family evaluations strengthen the reliability of our safety assessment. We thank the reviewer for highlighting this concern and will add the comparison to the appendix to provide a more comprehensive validation of our evaluation protocol.


Q4: Clarifying Semantic and Behavioral Changes

We thank the reviewer for raising this question. In our study, these terms describe related but distinct aspects of generation behavior, which may overlap. Refusal-related behavior is assessed through ASR, which measures successful harmful responses under our dual-stage safety evaluation protocol (Section 3 and Appendix A.2). Semantic contradiction measures NLI-based incompatibility between pruned and unpruned responses (Section 5 and Appendix B.2.5). Conceptual inversion and repetitive degeneration are qualitatively distinct phenomena illustrated in Figure 3: the former involves reversing or distorting a concept's meaning, while the latter involves repetitive or tautological generation. Factual errors may overlap with conceptual inversion and are discussed qualitatively without separate quantitative measurement.

We agree that clearer distinctions would improve the presentation of these concepts. We appreciate the reviewer's suggestion and will clarify their definitions and usage in the relevant sections of the revised manuscript.

Reviewer_ru1NOpenReview ↗

keep my score unchanged

Thank you for the thorough experiments. However, I have reservations about the paper’s novelty. I will keep my score unchanged but lower my confidence, leaving the final assessment to the Area Chair.

Reviewer_rN49OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper finds that locating and removing certain neurons can remove refusal by language models, but preserve their general competence. Consistent findings emerge across Llama, Gemma, Qwen and Mistral families. Locating the safety neurons is model family and scoring rule dependent. Pruning them switches off refusal and also meaning. Putting back just first half of the neurons brings back both abilities. These results challenge the modularity view of safety research. It is timely.

Critical strengths

Well-posed problem and very careful prose about what is claimed vs. what is not.

The finding that HC pruning produces nonsense text that ASR scores as perfect safety is an important observation.

The benign-salience control is a better baseline than the random pruning used in prior work.

The restoration experiment is good to test separability.

Critical weaknesses

A finer split on the modularity test could make the conclusions more robust.

The competence effect appears mainly on sensitive-topic prompts. A proponent of the modular view could argue that changing how the model talks about bombs is part of what safety neurons do. The benign-domain evidence is weak outside Llama.

The localization results (MLP-centric, layers 10 to 15) conflict with prior work (Zhao et al.), who report attention-dominant, early-layer safety neurons. This deserves explicit discussion, as does recent work localizing refusal to single neurons.

Questions for authors

Do standard benchmarks (MMLU, ARC) move for CD-RP? I think this could further strengthen your paper.

Can you do a finer partition, or perhaps use context-aware partitions, like searching for a subset of neurons that preserves meaning while still ensuring safety?

4 comments

Q1: Finer-grained Modularity Test

We agree with the reviewer that finer-grained partitioning can better assess the disentanglement of refusal and semantic stability. We therefore conducted four complementary restoration analyses across Llama, Gemma, and Qwen: (1) fine-grained restoration curves with random controls, (2) restoration across six scoring configurations, (3) layer-wise and module-wise restoration, and (4) cumulative layer-wise restoration. All experiments evaluate both ASR and harmful-domain NLI contradiction, with other experimental settings following the main paper.

  • Finer-grained restoration curves (Llama-3-8B, restoring lower-scoring neurons first)
MethodMetric0%1%5%10%25%50%75%90%95%99%
CD-RPASR42.67±0.342.56±0.541.33±0.339.44±0.538.78±0.737.67±0.319.44±0.84.89±0.54.78±0.82.11±0.2
Contradiction-H37.19±7.037.04±6.536.57±5.335.59±6.635.03±5.334.87±5.422.88±4.612.32±1.410.11±1.08.94±0.5
CD-RP (random)ASR42.67±0.342.33±0.340.22±0.235.67±0.930.00±1.521.67±2.811.56±2.55.00±2.64.11±0.82.11±0.2
Contradiction-H37.19±7.036.78±6.835.48±7.130.21±5.925.76±6.116.91±5.312.97±5.110.48±3.29.32±2.49.08±1.0
  • Restoration across scoring configurations (Gemma-7B, restoring the higher-scoring half)
ModelMetricCD-WandaCD-RPCD-SNIPHC-WandaHC-RPHC-SNIP
Gemma-7BASR12.89±0.213.34±0.212.89±0.513.00±0.612.78±0.514.00±0.3
Contradiction-H14.73±0.515.28±0.915.14±1.115.78±0.514.99±0.416.12±0.7
  • Layer-wise and module-wise restoration (Qwen2.5-7B)
MethodMetric1–45–89–1213–1617–2021–2425–28
AttnASR92.33±0.392.22±0.292.33±0.392.33±0.392.22±0.492.33±0.392.33±0.3
Contradiction-H41.32±1.141.18±1.041.20±0.841.31±0.841.58±1.340.99±1.041.72±1.0
MLPASR91.89±0.482.56±0.364.67±0.242.78±0.292.00±0.392.11±0.292.11±0.2
Contradiction-H41.05±0.837.66±2.425.19±3.819.91±3.541.47±0.641.53±1.041.31±0.5
Attn and MLPASR91.89±0.482.44±0.264.44±0.242.78±0.292.00±0.392.11±0.292.11±0.2
Contradiction-H41.09±0.737.83±2.725.20±4.319.65±4.441.81±0.841.22±1.041.34±0.5
  • Cumulative layer-wise restoration (Qwen2.5-7B)
MethodMetric1–41–81–121–161–201–241–28
Attn and MLPASR91.89±0.480.33±0.352.89±0.52.00±0.31.78±0.41.67±0.31.67±0.3
Contradiction-H41.09±0.735.88±3.024.30±5.117.45±2.215.99±3.716.31±2.515.82±4.3

These experiments extend our original median-split findings across a broader range of restoration strategies. In Llama, both CD-ranked and random restoration progressively recover refusal behavior and semantic stability. Across six scoring configurations in Gemma, the restored models exhibit similar ASR (12.78–14.00%) and contradiction rates (14.73–16.12%). In Qwen, restoration effects are concentrated in middle-layer MLPs, consistent with our localization findings in Section 4. Cumulative restoration brings both metrics close to their unpruned baselines by layer 16 (ASR: 2.00%; contradiction: 17.45%). Together, these results show that refusal and semantic stability generally recover jointly across the tested interventions, although their recovery rates can differ.

The finer-grained analyses also reveal meaningful recovery asymmetries, highlighting the value of the reviewer's suggested partitions. For example, restoring Qwen's MLP layers 13–16 reduces harmful-domain contradiction from 41.37% to 19.91% and ASR from 92.33% to 42.78%. The stronger recovery in semantic stability suggests partial dissociation under structural partitioning and reveals a more nuanced relationship than the original median split captured. We agree that context-aware partitioning offers a promising direction for extending this analysis. We will add these experiments to the appendix and discuss the observed recovery asymmetries and opportunities for more targeted partitioning in the Discussion section.

Q2: General Competence and Sensitive-Domain Semantic Stability

We appreciate the reviewer's suggestion to assess general competence using standard benchmarks. We additionally evaluated Llama-3-8B under CD-RP pruning (0.5%) on MMLU, ARC Challenge, and GSM8K. Performance declines modestly across all three benchmarks, by 0.51, 1.38, and 0.41 percentage points, respectively. These results complement the benign-domain findings for Llama reported in Section 5.

OriginalCD-RP
MMLU65.02±0.864.51±1.0
ARC Challenge80.71±0.279.33±0.3
GSM8K76.33±0.375.92±0.4

As the reviewer notes, benign-domain semantic effects are less pronounced in Gemma and Qwen. Across the three architectures, the strongest and most consistent semantic changes occur in benign-intent prompts concerning safety-sensitive topics, while effects on general benign-domain prompts are weaker and model-dependent. This distinction helps characterize the scope of the observed semantic changes across architectures.

Importantly, safety-related behavioral regulation and semantic stability serve distinct purposes. Safety mechanisms appropriately regulate responses to harmful requests, while benign informational questions about sensitive topics still require accurate conceptual understanding. As illustrated in Section 5, pruning can cause models to describe malware as protective software or confuse identity theft with identity verification. These conceptual distortions alter the meaning of otherwise legitimate answers and extend beyond changes in refusal behavior. Our findings therefore highlight the importance of evaluating semantic stability alongside ASR when assessing safety-neuron interventions.

We appreciate the reviewer for highlighting this distinction and helping us better characterize the implications of our findings. We will add the benchmark results and clarify the architecture-dependent effects and their implications for evaluating safety-neuron interventions in the revised manuscript.

Q3: Localization Differences and Recent Single-Neuron Findings

We agree with reviewer's recongition on our different localization results from Zhao et al. (2025). This is because the two studies adopt different neuron identification and intervention protocols.

First, the identification criteria differ. Zhao et al. identify neurons whose removal produces consistently large representation perturbations across harmful queries, whereas our main CD-RP framework ranks neurons by their normalized importance difference between harmful and benign inputs. These criteria need not select the same neurons. Second, the localization measurements differ. Zhao et al. report the distribution of identified neurons across modules, while our analysis evaluates the functional sensitivity of Attention and MLP components through pruning-induced ASR changes. A higher concentration of identified neurons in one module does not necessarily imply a larger functional effect when that module is pruned. Third, the intervention configurations differ. Zhao et al. use threshold-based neuron identification, whereas we apply a fixed per-matrix pruning budget (1% for the layer-wise analysis), with differences also in the targeted projection groups and layer coverage. The studies additionally use different identification datasets and partially different model families.

Those discrepancies illustrate the methodological sensitivity that motivates our Section 4 audit. By systematically comparing identification criteria, pruning ratios, and model architectures, we show that localization results can vary substantially with these choices. We believe this is an important observation for the community: claims about where safety-related computation resides should be interpreted in the context of the identification and intervention procedures used to obtain them.

We also appreciate the reviewer's reference to recent single-neuron research. Kazemi et al. [1] demonstrate that intervening on a single MLP refusal neuron can substantially disrupt refusal and distinguish refusal-related neurons from concept-related neurons. Their findings complement our investigation in several important respects.

First, the research objectives differ. Kazemi et al. examine whether a single neuron is sufficient to bypass refusal, while we investigate whether safety-targeted pruning can alter refusal while preserving semantic stability. Second, the units of intervention differ: they identify individual scalar coordinates of MLP hidden activations, whereas our framework operates on structured channels of projection matrices. Third, they manipulate activations during inference by fixing them at selected values, while we remove weights through structured pruning. These approaches provide complementary evidence on the mechanisms of refusal and the consequences of intervening on safety-associated parameters. Fourth, the studies evaluate collateral effects from different perspectives. Kazemi et al. assess general capabilities using MMLU and GSM8K, while we examine semantic stability on benign-intent prompts involving safety-sensitive topics. Our additional results illustrate the importance of this distinction: under CD-RP pruning, Llama-3-8B shows modest declines on MMLU (0.51 points), ARC Challenge (1.38 points), and GSM8K (0.41 points), while harmful-domain contradiction increases by 28.46 percentage points.

Taken together, the findings are consistent and complementary. Single-neuron interventions demonstrate fine-grained control over refusal, while our structured pruning experiments reveal accompanying changes in semantic stability. Both studies also highlight the role of MLPs, consistent with our localization findings for Gemma and Qwen. We thank the reviewer for highlighting this connection and will clarify these methodological distinctions and their implications in the revised manuscript.

[1] Kazemi, H., Chegini, A., & Safi, M. (2026). "A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models." arXiv.

Thanks for the response, I maintain my rating. But as I do not work directly in this area (indicated confidence of 2), I leave it to the AC and other reviewers.

Reviewer_KWFXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper audits the claim that safety-related neurons can be removed from instruction-tuned LLMs while preserving broader capabilities. Across several model families, it examines neuron identification, pruning, and selective restoration. The results show that localization depends on the scoring strategy and architecture, pruning can alter semantic content or cause generation collapse, and restoring selected neurons recovers refusal and semantic stability together. The authors conclude that pruning-based evidence alone does not establish safety modularity and that ASR is insufficient to assess capability preservation.

Critical strengths

I find it difficult to identify many strengths in this paper. Its main merit may be highlighting overclaims and evaluation issues in safety-related work from several years ago. However, these reflected the limitations of the field at the time and are now widely recognized and acknowledged within the research community.

Critical weaknesses
  1. The paper reads like an LLM-generated paper-mill submission, with very poor writing and presentation.
  2. It offers no meaningful novelty and falls well below the standard expected of an ICLR paper.
Questions for authors

N/A

2 comments

As explicitly stated in our AI Use Statement, generative AI tools were used solely for editorial and linguistic refinement. Research ideas, experimental designs, analyses, and conclusions are our original work. We remain open to addressing any specific technical concerns or questions.

As I mentioned in the critical Strengths part, the conclusion or your claimed contribution is not some new things. It's the limitation of the entire research domain of AI Safety or Security at 2024. After that, the domain already recognized the these limitation, the previous evaluation of AI safety is not perfect. From my perspective, this shouldn't be a ICLR level paper. The reviewer ignored this largest concern in their rebuttal.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the Area Chair and reviewers for their valuable feedback and constructive suggestions. During the rebuttal period, we conducted additional experiments and clarified several methodological questions, strengthening our findings in four areas:

  • Expanded Restoration Analysis (rN49, ru1N). Following both reviewers' suggestions, we conducted finer-grained restoration experiments across Llama, Gemma, and Qwen, revealing both joint recovery and partial dissociation between refusal and semantic stability.
  • Extended Generalization and Competence Evaluation (rN49, ru1N). We validated our findings on Qwen2.5-32B-Instruct (ru1N) and evaluated MMLU, ARC Challenge, and GSM8K (rN49), confirming substantial sensitive-domain semantic changes alongside modest benchmark declines.
  • Strengthened Evaluation Reliability (ru1N). We introduced two independent judges from Qwen and Gemma, achieving consistent ASR measurements across three judge families (maximum difference: 0.22 percentage points).
  • Clarified Methodology and Related Work (rN49, ru1N). We addressed localization discrepancies and recent single-neuron findings (rN49), and clarified the distinctions among semantic and behavioral changes (ru1N).

We would also like to reiterate three core technical contributions of our work:

  1. Identification Reliability. We reveal that common safety-neuron identification criteria are not interchangeable. Harmful-consistency scoring can induce generative collapse that ASR incorrectly records as 0% attack success.
  2. Semantic Effects Beyond Refusal. We demonstrate that safety-targeted pruning increases harmful-domain contradiction by 25–29 percentage points on benign-intent prompts, often without disrupting fluency, exposing limitations of general-domain utility evaluations.
  3. Restoration and Functional Coupling. We show that refusal and semantic stability recover together under the criterion's own neuron ranking, establishing the reversibility of semantic disruption and revealing that ASR alone can overstate restoration.

Together, these contributions establish our locate–remove–restore audit of pruning-based safety interventions. The additional rebuttal experiments strengthen the original findings and sharpen their interpretation. We thank the reviewers again for helping us improve the rigor and clarity of our work, and will incorporate these improvements into the revised manuscript. As the rebuttal period is still ongoing, we welcome any further comments or questions and would be happy to provide additional clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.