- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The method in this paper has practical value for the engineering goal of “identifying vulnerable nodes in backdoor behavior,” and its observations regarding sparsity and early-layer concentration merit serious attention. The claim that “backdoor-related computation is more concentrated in early layers and involves very few parameters” is also consistent with similar statements in the SoK paper (Reference [1]); therefore, this phenomenon appears reliable. In addition, I consider the methodology reasonable in two respects: (1) using representational change instead of ASR as a per-neuron screening signal avoids the prohibitive cost of computing ASR for every neuron individually; and (2) setting layer-wise thresholds accounts for the fact that representations gradually converge with depth, thereby avoiding direct cross-layer comparison of absolute importance. Nevertheless, after reading the paper, I have a corresponding concern: the paper defines a row of a parameter matrix as a “neuron” and assumes that it can be ablated and attributed independently. This may conflict with the theory of neuronal polysemanticity.
[1] Zhao W, Li Z, Sun J. SoK: a Comprehensive Causality Analysis Framework for Large Language Model Security[J]. arXiv preprint arXiv:2512.04841, 2025.
Critical strengths
-
Using representational change instead of ASR as a per-neuron screening signal avoids the prohibitive cost of computing ASR for every neuron individually. Within the “intervention–comparison” framework of causal inference, ablation at least directly manipulates candidate variables and is therefore closer to a causal test than purely correlational attribution methods such as activation-mean differences or attention-weight visualization.
-
Setting layer-wise thresholds accounts for the fact that representations gradually converge with depth, thereby avoiding direct cross-layer comparison of absolute importance. This technical detail indicates that the authors are aware of the cross-layer incomparability problem, and the method is internally coherent at the operational level.
-
Related work has confirmed that security-related mechanisms are highly localized in the early-to-middle layers, with only 1–2% of neurons exhibiting causal influence. This is consistent with the paper’s observations that backdoor neurons are extremely sparse and concentrated in early layers. Theoretically, I consider this direction promising.
Critical weaknesses
My primary concern is the following. Individual neurons in LLMs generally encode multiple unrelated features, and even “highly significant” concept-related neurons consistently exhibit polysemantic behavior. This means that a “backdoor neuron” selected by ablation simultaneously carries other functions; labeling it a “backdoor neuron” conceptually presupposes an unproven premise: that the neuron’s primary functional role is the backdoor. However, the paper’s detection procedure does not rule out the possibility that a neuron is selected merely because it exhibits a large representational perturbation in response to backdoor queries, while the source of this large perturbation may be another, backdoor-unrelated polysemantic feature that happens to be activated. (Reference [2])
More challenging is superposition. Neurons in early layers are especially in a state of superposition, where features are distributed across polysemantic neurons as sparse linear combinations rather than corresponding one-to-one with individual neurons. The paper’s key claim is precisely that “backdoor neurons are concentrated in early-layer MLPs,” and early layers are among the regions where superposition is most severe. Under superposition, using “a row of parameters” as the discrete carrying unit of a backdoor is theoretically most sharply mismatched. (Reference [3])
[2] Haider M U, Rizwan H, Sajjad H, et al. Neurons speak in ranges: Breaking free from discrete neuronal attribution[J]. arXiv preprint arXiv:2502.06809, 2025.
[3] Gurnee W, Nanda N, Pauly M, et al. Finding neurons in a haystack: Case studies with sparse probing[J]. arXiv preprint arXiv:2305.01610, 2023.
Questions for authors
-
Among the detected backdoor neurons, does there exist a “matched control feature” unrelated to the backdoor semantics that can likewise induce activation magnitudes in these neurons no lower than those elicited by backdoor queries?
-
Do the detected sets of backdoor neurons highly overlap across different random seeds, different detection thresholds, and different poisoned-sample subsets?
-
In the settings where “ablating a small number of neurons causes a substantial drop in ASR,” how much does task performance on clean queries change?