acceptodds
Under review as a conference paper at ICLR 2027

Backdoors Live in Shallow MLPs: State-Level Discriminative Attribution for Localized Pruning in LLM Backdoor Defense

10 views

Abstract

Reducing the attack success rate while preserving benign utility is the central goal of backdoor defense for large language models (LLMs). Existing post-training defenses implicitly treat the backdoor as a shortcut but never analyze in fine granularity how it works across layers or which layers matter, so they intervene across all layers and easily damage the clean-task parameters that dominate these layers. In this paper, we rethink this assumption and find that the backdoor comprises two functionally decoupled stages: attack-specific trigger recognition anchored in shallow-layer MLPs, and target output that carries no attack-specific computation in deep layers. Guided by this finding, we propose BadScalp, a backdoor defender requiring only 500 clean samples without knowing the trigger, the target label, or a clean reference model. BadScalp further refines localization within shallow-layer MLPs to the channel level, as most channels there serve benign computation. It quantifies each channel's contribution to driving the representation toward the poisoned state, thereby repairing backdoor channels. To preserve benign utility, BadScalp further adopts Fisher-guided protection and soft pruning. Extensive experiments across representative models, datasets, and attacks show that BadScalp achieves a 53.5% lower average attack success rate than the second-best defense while maintaining the utility. The code is available at https://anonymous.4open.science/r/BadScalp/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
backdoor defense, large language model, backdoor mechanism analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UZdYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors proposed a backdoor defender that requires only limited clean data. The design is based on insights into how backdoors work across layers and which layers are important. Experimental results demonstrate the effectiveness of their solution.

Critical strengths
  • The paper offers new insights.
  • The experiment is sufficient, and the results are good.
Critical weaknesses
  • The discussion on adaptive attacks is insufficient.
Questions for authors

Why were only 7B and 8B models used? What would happen with larger or smaller models?

Reviewer_dRCSOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies post-training defense against backdoored LLMs under a realistic setting where the defender has access only to a potentially compromised model and a small amount of clean data, without the original clean model, trigger, or target output. The paper first conducts a layer-wise analysis of backdoored LLMs and argues that the backdoor consists of two functionally decoupled stages: trigger recognition occurs primarily in shallow-layer MLPs, particularly the up_proj and gate_proj components, whereas the target output is produced in deeper layers. Based on this observation, the paper proposes BadScalp, which performs localized channel-level repair in shallow MLPs. BadScalp consists of four stages. First, it learns a trigger embedding from clean data by forcing the model away from its original top-1 prediction, producing a simulated poison-like representation. Second, it trains linear probes to perform state-level discriminative attribution and ranks individual MLP channels according to their contribution to the poison-like state. Third, it uses a diagonal empirical Fisher score on clean data to protect utility-critical channels. Finally, it softly scales selected channel weights and performs recovery fine-tuning. The evaluation covers two LLMs, two tasks, and five backdoor attacks, resulting in 20 scenarios. The paper reports that BadScalp achieves substantially lower ASR than the compared defenses while maintaining relatively high clean utility. Ablation and sensitivity experiments are provided to support the individual components.

Critical strengths
  1. A useful mechanistic perspective for localized defense The strongest aspect of the paper is the attempt to understand where the backdoor operates before designing the defense. The layer-swapping experiments suggest a potentially useful distinction between shallow trigger recognition and deeper target-output computation. If this phenomenon generalizes, it provides a principled explanation for why full-depth pruning or fine-tuning can unnecessarily damage benign capabilities.
  2. Practical defense setting without a clean reference model or trigger BadScalp operates using only a small clean dataset and does not require the original clean model, known trigger, or target output. This is an important practical setting for post-training backdoor defense. The reported ASR reductions are also substantial across the evaluated scenarios, and the ablations indicate that localized intervention is materially better than random or full-depth intervention.
Critical weaknesses
  1. The key simulated-trigger assumption is insufficiently validated The most important weakness is the gap between “the simulated trigger produces a poison-like representation” and “the simulated trigger reveals the true backdoor channels.” BadScalp does not have access to real poisoned examples, so it constructs a learnable embedding by maximizing the loss of the original top-1 prediction. The paper then uses the resulting representations to localize backdoor channels. However, the optimization objective does not explicitly encode the actual backdoor mechanism. Therefore, it is unclear why the resulting direction should consistently coincide with the attacker's trigger-sensitive direction. Figure 5b showing similar clean/poison separation at layer 4 is suggestive, but does not directly validate channel-level correspondence. In particular, similar separability does not imply that the same channels are responsible. This is consequential because the simulated trigger is not merely an auxiliary visualization: it is the input used to compute the attribution scores that determine which parameters are modified.
  2. “Attribution” is not demonstrated to be causal, and the shallow-localization claim may be overgeneralized The state-level attribution score is based on a linear probe. This provides a discriminative association between channels and the poison-like state, but it does not establish that those channels causally implement the backdoor. A high attribution score could reflect a correlated representation feature rather than a mechanism necessary for attack execution. Similarly, layer swapping demonstrates that replacing shallow layers can suppress the attack, but this does not necessarily imply that the entire backdoor mechanism is confined to those layers. The deeper layers may still participate in propagating, transforming, or decoding the trigger-induced state. The paper's wording should therefore distinguish between “layers whose replacement is sufficient to suppress the attack” and “layers where the backdoor resides.” These distinctions are important for the claimed mechanistic contribution.
Questions for authors
  1. Most important question: does simulated-trigger localization agree with real-trigger localization? Can the authors directly measure whether the channels selected using the simulated trigger overlap with channels that are actually necessary for the real backdoor? For example, if the real trigger is available only for analysis, could the authors:
  • localize channels using the real trigger;
  • localize channels using the simulated trigger;
  • report channel-level overlap (e.g., Precision/Recall, Jaccard, rank correlation);
  • and compare attack suppression when pruning each set? This experiment would directly validate the central assumption of BadScalp.
  1. How causal is the state-level attribution? Can the authors perform an intervention-based validation of the attribution scores? For example, progressively suppress channels according to the proposed score and compare the resulting ASR curve against channels ranked by activation gap, random ranking, Fisher ranking, and other attribution criteria. This would clarify whether the proposed attribution actually identifies causally important backdoor channels rather than merely discriminative channels.
  2. How general is the shallow-layer finding? The paper evaluates two models and five attacks. Are there attack/model combinations for which the backdoor is not localized in shallow up_proj/gate_proj components? In particular, how does BadScalp behave for backdoors introduced through different mechanisms, such as weight editing, RLHF manipulation, or attacks specifically designed to distribute the trigger representation across layers? The answer would materially affect whether the central claim should be interpreted as a general property of LLM backdoors or an empirical property of the evaluated attack family.
  3. Does recovery fine-tuning risk restoring the backdoor? After soft pruning, the model is fine-tuned on clean data. Since the defense has no poisoned examples or explicit backdoor-removal objective, what prevents recovery fine-tuning from reconstructing the previously suppressed backdoor functionality? An experiment measuring ASR immediately after soft pruning and after recovery fine-tuning would help establish that utility recovery does not inadvertently restore the backdoor.
1 comment
AuthorsOpenReview ↗

Follow-up on rebuttal: any remaining concerns?

Dear reviewer, we've addressed all the concerns raised in your review during the rebuttal. Could you kindly let us know if any issue remains unresolved on your end? We'd be grateful for any updated assessment before the discussion period ends. Thank you!

Reviewer_4VFkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies post-training backdoor removal for LLMs, where the defender has only 500 clean samples and does not know the trigger or the target. Using t-SNE, layer swapping and the logit lens, mainly on Llama-3-8B / SST-2 / BadNets, the authors argue that a backdoor has two decoupled stages: trigger recognition in shallow-layer MLPs, and target output in deep layers.

Based on this, they propose BadScalp. It learns a soft trigger embedding to create poison-like inputs, uses linear probes to find up/gate channels in the first five layers that push activations toward the poison state, scales those channels down (skipping Fisher-important ones), and then fine-tunes on clean data. Across two 7–8B models, two tasks and five attacks, BadScalp lowers average ASR to 22.82%, versus 49.07% for the next-best defense, and the authors report that utility is maintained.

Critical strengths

Major

  • Practical threat model. The defender needs no trigger, target or clean reference model, only 500 clean samples.
  • Persuasive localization evidence in the showcased setting.
    • The layer-4 t-SNE separation (Fig. 2a) is clear.
    • Layer swapping is a causal intervention, not a correlational probe (Fig. 2b, Table 3).
    • The component swap (Fig. 3) narrows the effect to up_proj/gate_proj.
    • Both analyses are repeated across all 20 settings (Fig. 7, Fig. 8), and in many of them the shallow layers do carry the trigger.

Minor

  • Consistent ASR reduction across 20 cells.
    • Mean ASR is 22.82%, vs 49.07% for FinePrune and 75.92% for BadSign.
    • On WebQuestions, BadScalp keeps CACC at ≈71–84, while FinePrune and BadSign fall to ≈46–60 (Table 1).
  • Cheap. It takes 370 s, vs 5,141 s for BadSign (Table 4).
  • Informative ablations on BadNets (Fig. 6a).
    • Random channels in the same layers leave ASR at 0.71.
    • Editing mid or deep layers instead leaves it at 0.85 / 0.84.
    • Without recovery SFT, ASR is 0.27 (vs 0.22 with it).
    • So in these cells the channel selection, not SFT, does most of the work.
  • Adaptive attack is considered (App. D.6), and code is released.
Critical weaknesses

Major

W1. The headline numbers cannot be reproduced from the paper's own runs at the stated defaults.

  • Fig. 6b sweeps four hyperparameters around the defaults (5 layers, ρ = r0 = 0.2, γ = 0.1).
    • All four panels agree at the default point: ASR ≈ 37 / 90 / 27 / 73 for Llama/SST-2, Llama/WebQ, DeepSeek/SST-2 and DeepSeek/WebQ.
    • Table 1 gives 16.56% / 21.36% / 8.44% / 42.18% (BadNets) for the same cells. No other attack row matches either.
  • Fig. 6a (also BadNets, same four cells) gives 0.22, consistent with Table 1. Fig. 6b is the outlier.
  • At Fig. 6b's default values, the mean BadNets ASR would be ≈ 0.57. That is worse than FinePrune's BadNets mean of 41.4% (Table 1).
  • Fig. 6b's utility values also differ from Table 1: ≈ 0.40 vs 46.00 / 43.17 for the two DeepSeek cells.
  • No seeds are reported. The "±" in Table 10 is the spread across four cells: its 500-sample column equals the Table 1 BadNets means.
  • Please state the exact configuration behind Table 1 and behind Fig. 6b, and confirm that one fixed configuration is used for all 20 cells and that it was chosen without triggered data. If possible, please report mean ± std over at least 3 seeds.

W2. There is no security–utility trade-off analysis.

  • Each baseline is reported at a single operating point, its original settings, while BadScalp's settings were chosen by the authors.
  • There are no ASR–CACC or ASR–utility curves for BadScalp or the baselines (e.g., FinePrune at several pruning rates, FineTune at several budgets). It is therefore unclear whether BadScalp dominates the baselines or just sits at a different point on the trade-off.
  • The only sweep (Fig. 6b) shows utility nearly flat (≈ 0.33–0.51) while ASR ranges from 0.01 to 1.0. This suggests the utility metric cannot register the cost.
  • Please add trade-off curves that use CACC and the informative benchmarks separately.

W3. The claimed mechanism does not explain why the method works.

  • Swapping in the clean model's weights is the ideal undo of whatever the attack wrote into those layers. If the backdoor were anchored in layers 0–4, scaling a subset of up/gate rows there should not beat swapping those layers.

  • Yet in several cells BadScalp, which edits only layers 0–4, removes backdoors that no swap window of up to seven layers removes. Fig. 8 vs Table 1 (swap values are read from the figure):

  • From experiments in Figure 6a and Appendix D.5, it seems SFT is not the factor. It should have further investigation

    CellBest swap ASR, any window (Fig. 8)BadScalp ASR (Table 1)
    DeepSeek/WebQ/CTBA≈0.701.62%
    DeepSeek/WebQ/MTBA≈0.654.28%
    Llama/WebQ/VPI≈0.9012.01%
    Llama/WebQ/CTBA≈0.7018.90%
    Llama/SST-2/Sleepershallow ≈0.9–1.0; ≈0.08 only at layers 9–1229.82%
    DeepSeek/SST-2/VPIshallow ≈0.95; ≈0.57 only at layers 23–2626.86%

W4. The simulated trigger is not shown to match the real one.

  • The paper states that the learned embedding is not meant to reconstruct the trigger. Instead it finds "abnormal" directions to which backdoored models are "particularly sensitive" (Sec. 5.2). This is plausible but untested.
  • Fig. 5b only shows that simulated inputs separate from clean ones at layer 4.
    • The simulated shift is much larger than the real one: two distant clusters, vs two adjacent, slightly overlapping ones for real triggers.
    • A t-SNE separation cannot not show that the same channels are involved.
  • The RandTrig ablation (0.38 ± 0.39 over four BadNets cells) suggests that a random trigger sometimes works about as well.
  • Please consider investigating the overlap between the channels selected using real poisoned inputs and those selected using the simulated trigger (and random triggers);

W5. Model coverage is narrow and dated.

  • The paper uses only two models, Llama-3-8B-Instruct and DeepSeek-LLM-7B-Chat, released in 2023–2024.
    • Both are dense, Llama-style 7–8B models with 30–32 layers, so this is effectively one architecture at one scale.
    • I understand prior works use those model settings, but I highly recommend authors choose state-of-the-art LLM models instead.
  • Size. Does the shallow localization, and the fixed "first 5 layers" rule, carry over to smaller (< 5B) and larger (≈ 14B, ≈ 30B) models with different depths? The layer budget should be defined relative to depth, or chosen without triggered data.
  • (optional) MoE. In MoE models, the MLP becomes routed experts plus a router.
    • Trigger recognition may live in routing decisions or in particular experts, rather than in rows of a dense up/gate projection.
    • With 500 samples, Fisher and probe statistics for rarely used experts will be noisy.
    • Both the localization result and the channel-level method need re-validation (e.g., Qwen3-30B-A3B, gpt-oss-20b).
  • Suggestion. A size ladder within one family (e.g., Qwen3 dense 1.7B/4B/8B/14B/32B plus 30B-A3B) would separate the effects of size and MoE from architecture.
  • Compute. This does not appear to be the obstacle. App. C reports four A100 GPUs, which is enough for LoRA attacks and BadScalp up to the 14B class, and plausibly for 30B-class models.

Minor

  • Over-strong wording.
    • App. D.2 says the swap pattern holds "in every scenario". In many Fig. 8 panels, shallow windows do not lower ASR (e.g., Llama/SST-2/Sleeper, DeepSeek/SST-2/VPI, WebQ VPI/CTBA). The overall claim can stand with softer wording.
    • The logit lens is generally uninformative in the early layers of Llama-style models. A late rise of the target probability therefore does not show that deep layers do no attack-specific computation.
  • Missing details:
    • LoRA rank and target modules, which determine where a backdoor can live;
    • which checkpoint serves as the "clean model" in the swaps;
    • how ASR and WebQuestions CACC are scored (string match, keyword list or a judge);
    • test-set sizes and the target sentiment label;
    • the recovery-SFT recipe;
    • the value of α and the Fisher dataset D_u;
    • the soft trigger's length and insertion position;
  • The CROW baseline looks like a no-op. Its CACC and ASR equal No-Defense to two decimals in 7/20 cells. Please validate the implementation against published results.
  • Positioning. Lamparth & Reuel (2024), cited in Sec. 2, already identify early-layer MLPs (and the embedding projection) as central to backdoor mechanisms. Sec. 2's statement that prior work does not examine depth-wise variation should be revisited.
  • Typos and small errors:
    • Table 4: "8951", "73,16".
    • Table 3 layer counts: 21–31 is 11 layers and 11–31 is 21, so the "5.5×" in D.3 is off.
    • Sec. 4.2 gives window sizes as "0–3 consecutive layers", while the Fig. 2b legend says 1–4.
    • Sec. 6 lists eight utility tasks while saying nine; TruthfulQA appears only in App. C.
    • DeepSeek-LLM-7B-Chat is cited as the DeepSeek-V3 technical report.
Questions for authors
  • please explain concerns raised in W1, W3 as much as possible in your time window
  • W2 tradeoff experiments and W4 channel overlapping study are highly recommended to add
  • W5 model choice suggestions are somehow optional but worth consideration, especially for your future work.

Comments on the paper

1 comment

Dear Reviewers,

With the discussion period closing on November 18, please take a moment to read the authors’ responses. Please also indicate whether your concerns have been addressed and whether any further clarification or follow-up is needed.

Thank you for your time and contributions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.