acceptodds
Under review as a conference paper at ICLR 2027

Towards Understanding LLM Backdoors: A Mechanistic Perspective from Backdoor Neurons

Abstract

Large language models (LLMs) have achieved advanced performance on multiple language tasks. However, they remain vulnerable to backdoor attacks and few works provide a comprehensive understanding of LLM backdoors, leaving the internal mechanisms of these attacks largely a black box. To address this gap, we explore the interpretable mechanisms of LLM backdoors through the lens of mechanistic interpretability, focusing on identifying and analyzing backdoor neurons within LLMs that are responsible for backdoor behaviors. Our findings reveal that the neurons crucial to backdoors are extremely sparse and are predominantly located in MLP modules of the early few layers. Ablating only a very small number of backdoor neurons can reduce the attack success rate by over 95% (only modifying  0.00068% of the parameters of Qwen3-0.6B, 0.036% of Llama3.2-1B, and 0.018% of Llama3.2-3B, in contrast to the  3% modification required in prior work). Furthermore, we hypothesise a two-stage backdoor encoding workflow to explain why early-MLP is critical for backdoors, and conduct empirical experiments to validate this hypothesis. Together, our findings provide a novel perspective for unpacking the black box of LLM backdoors, and revealing actionable insights for the community.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Backdoor Attacks and Defenses, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3LBsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The method in this paper has practical value for the engineering goal of “identifying vulnerable nodes in backdoor behavior,” and its observations regarding sparsity and early-layer concentration merit serious attention. The claim that “backdoor-related computation is more concentrated in early layers and involves very few parameters” is also consistent with similar statements in the SoK paper (Reference [1]); therefore, this phenomenon appears reliable. In addition, I consider the methodology reasonable in two respects: (1) using representational change instead of ASR as a per-neuron screening signal avoids the prohibitive cost of computing ASR for every neuron individually; and (2) setting layer-wise thresholds accounts for the fact that representations gradually converge with depth, thereby avoiding direct cross-layer comparison of absolute importance. Nevertheless, after reading the paper, I have a corresponding concern: the paper defines a row of a parameter matrix as a “neuron” and assumes that it can be ablated and attributed independently. This may conflict with the theory of neuronal polysemanticity.

[1] Zhao W, Li Z, Sun J. SoK: a Comprehensive Causality Analysis Framework for Large Language Model Security[J]. arXiv preprint arXiv:2512.04841, 2025.

Critical strengths
  • Using representational change instead of ASR as a per-neuron screening signal avoids the prohibitive cost of computing ASR for every neuron individually. Within the “intervention–comparison” framework of causal inference, ablation at least directly manipulates candidate variables and is therefore closer to a causal test than purely correlational attribution methods such as activation-mean differences or attention-weight visualization.

  • Setting layer-wise thresholds accounts for the fact that representations gradually converge with depth, thereby avoiding direct cross-layer comparison of absolute importance. This technical detail indicates that the authors are aware of the cross-layer incomparability problem, and the method is internally coherent at the operational level.

  • Related work has confirmed that security-related mechanisms are highly localized in the early-to-middle layers, with only 1–2% of neurons exhibiting causal influence. This is consistent with the paper’s observations that backdoor neurons are extremely sparse and concentrated in early layers. Theoretically, I consider this direction promising.

Critical weaknesses

My primary concern is the following. Individual neurons in LLMs generally encode multiple unrelated features, and even “highly significant” concept-related neurons consistently exhibit polysemantic behavior. This means that a “backdoor neuron” selected by ablation simultaneously carries other functions; labeling it a “backdoor neuron” conceptually presupposes an unproven premise: that the neuron’s primary functional role is the backdoor. However, the paper’s detection procedure does not rule out the possibility that a neuron is selected merely because it exhibits a large representational perturbation in response to backdoor queries, while the source of this large perturbation may be another, backdoor-unrelated polysemantic feature that happens to be activated. (Reference [2])

More challenging is superposition. Neurons in early layers are especially in a state of superposition, where features are distributed across polysemantic neurons as sparse linear combinations rather than corresponding one-to-one with individual neurons. The paper’s key claim is precisely that “backdoor neurons are concentrated in early-layer MLPs,” and early layers are among the regions where superposition is most severe. Under superposition, using “a row of parameters” as the discrete carrying unit of a backdoor is theoretically most sharply mismatched. (Reference [3])

[2] Haider M U, Rizwan H, Sajjad H, et al. Neurons speak in ranges: Breaking free from discrete neuronal attribution[J]. arXiv preprint arXiv:2502.06809, 2025.

[3] Gurnee W, Nanda N, Pauly M, et al. Finding neurons in a haystack: Case studies with sparse probing[J]. arXiv preprint arXiv:2305.01610, 2023.

Questions for authors
  1. Among the detected backdoor neurons, does there exist a “matched control feature” unrelated to the backdoor semantics that can likewise induce activation magnitudes in these neurons no lower than those elicited by backdoor queries?

  2. Do the detected sets of backdoor neurons highly overlap across different random seeds, different detection thresholds, and different poisoned-sample subsets?

  3. In the settings where “ablating a small number of neurons causes a substantial drop in ASR,” how much does task performance on clean queries change?

Reviewer_Z7QSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates backdoor behavior in large language models (LLMs) from a neuron-level perspective. The authors define neurons as rows of parameter matrices in attention and multilayer perceptron (MLP) modules. They measure neuron importance by examining changes in hidden states after ablating individual neurons. Based on this measure, they identify neurons associated with backdoor behavior.

The authors conduct experiments on three models: Llama3.2-1B, Llama3.2-3B, and Qwen3-0.6B. They evaluate several backdoor attacks, including AddSent, Sleeper, and VPI. By analyzing neuron importance across different layers and modules, they find that only a small number of neurons are strongly associated with backdoor activation. These neurons mainly appear in the MLP modules of early layers. Further ablation experiments show that removing a small number of highly important neurons can significantly reduce the attack success rate (ASR). In contrast, ablating the same proportion of neurons in middle or deeper layers usually has a smaller effect.

Based on these findings, the authors propose a two-stage hypothesis for backdoor encoding. They suggest that early layers mainly embed trigger information, while deeper layers further process the semantic information related to backdoor behavior. To investigate this hypothesis, they compare the cosine similarities of hidden states between poisoned inputs and between clean and backdoored models on poisoned inputs. They analyze how these representations change across layers and discuss the potential implications for backdoor detection and defense.

Critical strengths

1. The paper investigates LLM backdoor mechanisms at the neuron level and provides valuable insights.

Existing studies on LLM backdoors mainly focus on attack success rates, overall model behavior, or coarse-grained changes in internal representations. This paper further examines how individual neurons affect backdoor behavior. The authors also compare neuron importance across different layers and between attention and MLP modules. This perspective helps us better understand how backdoor behavior relates to model parameters. It may also guide the development of more fine-grained backdoor detection and repair methods.

2. The neuron ablation experiments provide direct evidence and reveal interesting sparsity and layer-wise differences.

The authors identify candidate backdoor neurons based on changes in hidden states. They then ablate these neurons and measure the changes in attack success rates. Table 1 shows that removing a small number of highly important neurons from early layers can significantly reduce ASR across multiple models and attack settings. In contrast, ablating the same proportion of neurons in middle or deeper layers usually has a limited effect. Figures 2 and 3 further illustrate how candidate backdoor neurons are distributed across different modules and how they affect backdoor behavior. These results provide useful empirical evidence that backdoor-related parameters are sparse and tend to concentrate in early layers.

Critical weaknesses

1. The neuron importance metric does not clearly distinguish backdoor-specific importance from general representation importance. The ablation experiments also lack important controls.

Section 3.2 measures neuron importance based on changes in hidden states after ablating individual neurons. The authors then use this metric to identify backdoor neurons. However, a large change in hidden states only indicates that a neuron strongly affects the representation of a given input. It does not necessarily show that the neuron plays a specific role in backdoor behavior. Some neurons may contribute to both normal semantic processing and backdoor activation. Therefore, ranking neurons based only on representation changes for triggered inputs may also select neurons that are generally important.

Although Table 1 shows that ablating some candidate neurons can significantly reduce ASR, the paper does not provide sufficient comparisons with random neurons from the same layers, neurons that are important for clean inputs, or other ablation strategies with similar parameter budgets. More importantly, the paper mainly reports changes in ASR but provides limited analysis of clean task performance after ablation. Therefore, the current results cannot rule out the possibility that some ASR reductions come from degraded generation or task performance rather than selective backdoor removal. This weakens the claim that the identified neurons serve backdoor-specific functions.

2. The two-stage backdoor encoding hypothesis mainly relies on representation similarity and lacks sufficient evidence for strong mechanistic claims.

Section 5 proposes that early layers mainly embed trigger information, while deeper layers process backdoor-related semantic information. The authors support this hypothesis by comparing the cosine similarities of Poisoned–Poisoned (P-P) and Poisoned–Clean (P-C) representations. However, these similarity changes may also reflect normal representation changes across layers or general differences between clean and backdoored models after fine-tuning. They do not necessarily indicate two distinct stages of backdoor encoding.

Furthermore, the strong effect of early-layer neuron ablation on ASR does not directly prove that early layers only handle trigger embedding. It also does not establish that deeper layers perform specific backdoor-related semantic processing. The paper lacks causal experiments that separately test the roles of these two proposed stages. Therefore, the current evidence better supports an empirical observation about how backdoor-related representations change across layers than a complete two-stage mechanistic explanation.

Finally, the experiments mainly cover three relatively small models, a limited set of backdoor attacks, and fixed poisoning and training settings. Since the importance of early-layer neurons is one of the paper's main findings, the authors should clarify how broadly this conclusion applies across different model sizes, attack strategies, and training conditions.

Questions for authors

Q1. Specificity of the backdoor neuron identification metric.

Section 3.2 identifies backdoor neurons based on changes in hidden states for triggered inputs. However, this metric may also select neurons that strongly affect general input representations. How do the authors distinguish backdoor-specific neurons from neurons that are generally important? If the authors apply the same importance metric to clean inputs, how much overlap would they find between the two sets of highly important neurons?

Q2. Ablation controls and model utility.

Table 1 shows that ablating a small number of candidate neurons in early layers can significantly reduce ASR. Does this approach outperform the ablation of random neurons from the same layers or neurons that are important for clean inputs? In addition, how does neuron ablation affect clean task accuracy and general generation capabilities? How do the authors demonstrate that the ASR reduction comes mainly from selective backdoor suppression rather than a loss of normal model capabilities?

Q3. Causal evidence for the two-stage backdoor encoding hypothesis.

Figure 4 shows how P-P and P-C similarities change across different layers. How do the authors rule out the possibility that these differences come from normal representation changes across layers or general representation shifts caused by fine-tuning? Can the authors provide further intervention-based evidence to separately verify the role of early layers in trigger embedding and deeper layers in backdoor-related semantic processing?

Q4. Generalizability of the findings.

The experiments mainly cover three relatively small models and a limited set of backdoor settings. Do the authors expect the stronger influence of early-layer MLP neurons on backdoor activation to hold for larger models, different poisoning ratios, and more complex contextual trigger attacks? What experimental evidence supports this expectation?

Reviewer_jx5mOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the internal mechanisms of LLM backdoors through neuron-level attribution. By examining three language models under different backdoor attacks, the authors find that a small number of neurons, primarily in early MLP layers, have a substantial influence on attack success. The paper further proposes a two-stage explanation of backdoor encoding based on representation similarity across layers.

Critical strengths

The paper addresses an interesting problem that remains insufficiently understood. Moving from layer-level or attention-head attribution to individual neurons provides a potentially useful perspective on backdoor mechanisms.

Critical weaknesses
  • Section 5 interprets the representation similarity patterns in Figure 4 as evidence that early layers perform trigger embedding while deeper layers perform semantic understanding. However, the observed differences could simply reflect representation changes introduced by poisoned fine-tuning. The analysis does not establish that the identified neurons implement either stage. Activation patching or causal interventions that restore and reproduce backdoor behavior would provide stronger mechanistic evidence.

  • Early-layer interventions naturally affect all subsequent computations, which may partly explain why they have a larger impact than interventions in deeper layers. The authors should determine whether this effect is specific to backdoor neurons or reflects general sensitivity to early-layer perturbations. The results are also not always monotonic. For example, in Table 1, ablating more neurons in the third layer of Llama3.2-1B increases the AddSent ASR from 53.5% to 75.5%. This deserves investigation before concluding that the selected neurons consistently promote backdoor activation.

  • The paper claims to be the first to attribute LLM backdoors at neuron level, although the cited SafeSeek work already examines backdoor circuits through fine-grained attribution. The comparison based on the percentage of modified parameters is also incomplete because it does not account for clean utility or differences in experimental settings. Furthermore, the experiments are limited to models of at most 3B parameters, with ASR measured on 200 sampled inputs. It remains unclear whether the findings generalize to larger models or independently trained backdoors.

Questions for authors
  1. Can the authors demonstrate that these neurons causally encode the backdoor through activation patching, rather than merely disrupting its execution?
  2. After ablating the identified neurons, how much benign instruction-following capability is preserved?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.