acceptodds
Under review as a conference paper at ICLR 2027

SPARSE: An Explainability-Inspired Defense Against Neural Trojans

Abstract

Deep Neural Networks are increasingly available as pretrained checkpoints from public hubs or fine-tuned by third-party providers. This exposes them to Trojan attacks: an adversary poisons the training data and/or implants a backdoor during training so that the CNN emits an attacker-chosen output on inputs containing a specific trigger. Existing defenses either target a single architecture or, when representation-based and architecture-agnostic, operate offline on a poisoned training set rather than flagging individual test inputs online. None has been validated across CNN, encoder-only, and decoder-only LLM architectures within a single evaluation framework. We propose SPARSE (SAE-based Poisoned Anomaly Recognition and Selective Exclusion), an online defense method built on a hypothesis coming from mechanistic interpretability: a trigger is an out-of-distribution signal that the backdoor keys on, so it should activate a distinct subset of the model's internal "concepts" that the clean inputs almost never activate. SPARSE attaches a small SAE to a chosen layer of the target model as a read-only side-car (leaving the forward pass unmodified), fits it together with a lightweight anomaly detector over its fit statistics computed using only clean data, and flags any input whose SAE output departs from the output of clean-data concepts. The same mechanism works for CNNs, encoder-only LMs and generative LLMs. Across 4 models, 12 datasets, and 14 attacks, SPARSE achieves an F1-score of 87%, outperforming the best baselines by 15 percentage points with a common SAE-based detection mechanism, instantiated per modality, for CNNs, encoder-only LMs and generative LLMs. The implementation of SPARSE is publicly available at https://anonymous.4open.science/r/SPARSE-54FD

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
ML Security, DNN Trojan, ML Testing

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_AfYQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents SPARSE, an SAE-based anomaly detection framework that is generally applicable to many deep learning models. The authors train a sparse autoencoder on the models' latent states and then use 3 informative statistics, compared to their typical distribution over a clean dataset, to detect anomalies. The method shows promise across various modalities, beating baselines.

Critical strengths
  • Clear presentation, method as explained is easy to follow. Additionally, the method proposed is quite interesting and would be a valuable contribution to the literature if shown to be strong empirically.
  • Well written background/related work section.
  • Multimodality of the method is a nice property, and well explored by the paper.
  • A reasonable range of attacks considered.
Critical weaknesses
  • The paper tests against only a few white box baselines, none for BERT and only one for generative LLMs. Additionally, the blackbox LLM baselines are quite dated. The authors should compare SPARSE against more recent and stronger baselines, especially for LLMs.
  • In LLMs, it appears the detector acts on statistics from generated tokens (rather than prompt tokens), where the harmful output may be trivially recognizable/seperable.
  • Baselines seem poorly tuned: ONION scores similarly for all datasets in the BERT case, while all other methods vary significantly - suggest looking at implementation/# of flagged cases here. Additionally Mahalanobis distance numbers seem very low compared to other pieces of work, suggesting there has not been much consideration into making the baselines strong for comparison.
  • The paper only reports F1 scores (in main text). It would be good to see results in true positive rate at 1% false positive rate or AUROC, alongside F1.
  • The background section could be updated with more recent literature on detecting anomalies in LLMs, as the field is progressing quickly.
  • No ablation studies over the combination of statistics used for detection or the pooling used over response tokens, e.g. mean pooling instead of max pooling.
Questions for authors
  • Can the authors provide an ablation of combinations of the three SAE statistics used for detection? And for the type of pooling?
  • Can the authors show how detection performance is impacted by using other decision rules than 3-sigma/other thresholds?
  • Are the Mahalanobis detectors using prompt or response tokens?
  • How much training data was used, and was it matched for each method?
  • How do you set the threshold for the Mahalanobis detector?
  • Typos:
    • 014: CNN should be DNN?
    • fig 1 caption: 'reprot'
    • fig 5 caption: 'exhamined'
  • What happens if the fit dataset is slightly noisy (e.g. 1% backdoored data)
  • Did you try multivariate Mahalanobis distance for the 3 statistics rather than OR between 3 univariate thresholds?
  • Do you also test multi-layer Mahalanobis distance (given you use multi-layer SAEs)?
  • Do you do any analysis of which SAE features seem most helpful for detection? Is it just a few critical ones? This would be interesting.
4 comments
AuthorsOpenReview ↗

Response to Official Review of Submission31908 by Reviewer AfYQ - part 1

(Part 1/3)

Dear Reviewer,

We thank you for the careful and constructive feedback, and we revised the manuscript accordingly. The supplementary material contains a version of the revised manuscript with all changes highlighted in blue. Below, each question or weakness is followed by our answer; points that share the same answer are grouped together.

Weakness 3: "The paper only reports F1 scores (in main text). It would be good to see results in true positive rate at 1% false positive rate or AUROC, alongside F1."

Q2: "Can the authors show how detection performance is impacted by using other decision rules than 3-sigma/other thresholds?"

Answer: Section 6.1 now reports AUROC, TPR at 1% FPR, and the measured clean FPR alongside F1, and Appendix D.1.1 reports them, with precision, recall, and bootstrap confidence intervals, for every configuration. AUROC and TPR at 1% FPR do not depend on the three-sigma rule. Appendix D.3.4 compares the three-sigma rule with a threshold calibrated on held-out clean data and with an exact small-sample threshold with a Bonferroni correction. For ResNet-18, the calibrated rule meets its target and slightly improves F1 (0.89 to 0.92). For the language models, both rules lower the clean FPR (e.g., Gemma from 16.2% to 3.8% with the Bonferroni rule) at a cost in recall, which raises F1 when poisoned inputs are rare (Gemma at 1% prevalence: 0.20 to 0.57).


Q1: "Can the authors provide an ablation of combinations of the three SAE statistics used for detection? And for the type of pooling?"

Q8: "Did you try multivariate Mahalanobis distance for the 3 statistics rather than OR between 3 univariate thresholds?"

Q9: "Do you also test multi-layer Mahalanobis distance (given you use multi-layer SAEs)?"

Weakness 6: "No ablation studies over the combination of statistics used for detection or the pooling used over response tokens, e.g. mean pooling instead of max pooling."

Answer: Appendix D.3 adds an ablation study on the full evaluation. Appendix D.3.1 evaluates all combinations of the three fit statistics and, following your suggestion, a multivariate Mahalanobis test over them; Appendix D.3.2 evaluates each layer strategy and a multivariate test across layers; Appendix D.3.3 compares max, mean, and first-token pooling. Combining the statistics improves F1 over any single statistic on ResNet-18 and BERT; the best single statistic changes across modalities, and the combination of the paper is within 0.04 F1 of the best variant for every model. The full layer ensemble has the best F1 of all OR-based configurations for every model (e.g., Llama: 0.83 against at most 0.77 for a single layer strategy). The multivariate tests are slightly better on ResNet-18 and BERT but worse on the generative LLMs, with the highest clean FPR, since their covariance is estimated from few clean samples. Pooling changes the operating point more than the ranking (average AUROC 0.92–0.96 for all variants). Each subsection discusses the trade-offs; the main threshold configuration of the paper is unchanged.


Weakness 5: "In LLMs, it appears the detector acts on statistics from generated tokens (rather than prompt tokens), where the harmful output may be trivially recognizable/separable."

Q3: "Are the Mahalanobis detectors using prompt or response tokens?"

Answer: SPARSE reads the activations of the generated tokens, whereas the Mahalanobis baseline reads the prompt tokens; both choices are now stated explicitly (Appendices A.2 and B.3). Section 6.1 and Appendix D.3.3 now test whether SPARSE recognizes the output rather than the trigger. First, restricting SPARSE to a single state, the one that produces the first generated token, removes the response entirely, since this state is computed from the prompt alone; the average F1 stays close to that of the full response (0.90 on Llama and 0.86 on Gemma), and far above the prompt-based Mahalanobis baseline. Second, on SST-2, the backdoored answer is a valid label that clean inputs also produce; SPARSE flags clean inputs with this same output no more often than the others and still separates them from poisoned inputs. Finally, SPARSE never observes the target behavior: it only flags departures from the clean behavior of the model, whereas recognizing a harmful output would require knowing the target.

AuthorsOpenReview ↗

Response to Official Review of Submission31908 by Reviewer AfYQ - part 2

(Part 2/3)

Weakness 1: "The paper tests against only a few white box baselines, none for BERT and only one for generative LLMs. Additionally, the blackbox LLM baselines are quite dated. The authors should compare SPARSE against more recent and stronger baselines, especially for LLMs."

Weakness 2: "Baselines seem poorly tuned: ONION scores similarly for all datasets in the BERT case, while all other methods vary significantly - suggest looking at implementation/# of flagged cases here. Additionally Mahalanobis distance numbers seem very low compared to other pieces of work, suggesting there has not been much consideration into making the baselines strong for comparison."

Answer: We compare against the inference-time detectors provided by the benchmark frameworks used in our study (BackdoorBench, OpenBackdoor); the defenses of BackdoorLLM repair the model or modify its decoding and do not flag individual inputs (Section 5, Appendix B.3). Following your suggestion, we added DAN, a white-box multi-layer Mahalanobis detector, as a baseline for BERT (Tables 2 and 18). Your remark on ONION led us to find two implementation issues (padded positions in the perplexity computation and input truncation); after fixing them, our implementation reproduces the official perplexities, and we now evaluate ONION as a detector under the same protocol as the other baselines. The threshold and the token source of the Mahalanobis baseline are now documented in Appendix B.3. These baselines remain standard in recent work (e.g., ParaFuzz, NeurIPS 2023; Liu et al., NAACL 2024; BEAT, ICLR 2025).


Q5: "How do you set the threshold for the Mahalanobis detector?"

Answer: An input is flagged when its squared Mahalanobis distance exceeds the 99.73% quantile of a χ² distribution with 50 degrees of freedom (the number of PCA components), i.e., the same nominal false-positive rate as a single three-sigma test of SPARSE. This is now documented in Appendix B.3.


Q4: "How much training data was used, and was it matched for each method?"

Answer: Appendix B.1 now details the data splits: the training split is used only to train the target model, and all clean data used by SPARSE comes from disjoint parts of the test split, with the same proportions for every dataset. Baselines that need clean reference data use the same samples as SPARSE.


Q7: "What happens if the fit dataset is slightly noisy (e.g. 1% backdoored data)"

Answer: SPARSE addresses a different setting from defenses that operate on poisoned data: it is not fitted on the training data of the model, which poisoning attacks contaminate, but on a small clean set collected by the defender. Contaminating this set would require access to the defender's own data, which is outside our threat model; Appendix B.1 and the Limitations now state this assumption. Following your suggestion, Appendix D.3.6 evaluates SPARSE with one or two poisoned inputs added to its clean set. The larger clean sets of ResNet-18 and BERT are barely affected, whereas the small clean sets of the generative LLMs can break detection on some configurations. A robust estimator removes this sensitivity at the cost of a higher clean FPR, so with small clean sets the defender should collect more clean data or screen it; the Limitations now discuss this.


Q10: "Do you do any analysis of which SAE features seem most helpful for detection? Is it just a few critical ones? This would be interesting."

Answer: Our analysis suggests that triggered inputs cause a diffuse activation of the SAE latents rather than activating a few dedicated ones (Figure 2, Appendix D.2), which is why SPARSE relies on aggregate statistics. A latent-level analysis is an interesting direction, which we added to the Future Work.


Weakness 4: "The background section could be updated with more recent literature on detecting anomalies in LLMs, as the field is progressing quickly."

Answer: Section 2 now covers recent inference-time defenses, including DAN, BEAT, and recent text-classification detectors, as well as LLM defenses that modify the decoding or the model.


Q6: "Typos: 014: CNN should be DNN?; fig 1 caption: 'reprot'; fig 5 caption: 'exhamined'."

Answer: Thank you, all three typos have been corrected.

AuthorsOpenReview ↗

Response to Official Review of Submission31908 by Reviewer AfYQ - part 3

(Part 3/3)

Final note - corrected results. While implementing the revised evaluation protocol, we found and fixed a bug in SPARSE which led it to have a different data split with respect to the one used for the baselines. The seed of SPARSE was also not always fixed in the submitted version. All results of SPARSE were recomputed with the corrected implementation, ensuring a perfect match between the datasets, and with a fixed seed. The average F1 of the generative LLMs is unchanged or higher, while that of the image and encoder-only models is lower. Appendix D.3.5 shows that the results vary little across SAE seeds. On average, SPARSE still outperforms the best baseline of every modality, and the conclusions of the paper are unchanged.

Hi, thanks for responding.

Firstly, I'll note that this response was posted with less than a day left in the discussion period, and contains an extra 20 pages of material, meaning I have not read all of the new additions. I consider this to be too onerous an ask from the authors. Further, the summaries the authors have written are not especially helpful in summarising these many new experiments (where are the numbers and the plain language results?) which means that I do not have much confidence about how clean these experiments are or if they show what they purport to.

It is good that the authors run ablations on the choice of SAE statistics. I find the response to my q3 on prompt/response tokens to be somewhat difficult to understand. For example, the sentence: "First, restricting SPARSE to a single state, the one that produces the first generated token, removes the response entirely, since this state is computed from the prompt alone" is very difficult to parse. Does state mean activation? In what way does it 'remove the response entirely'? I take it to maybe mean that it is not acting on the response any more? If so, this addresses my concern, and it should be stated plainly in the paper. Given that First-1 performs better than the main configuration on Llama (F1 0.90 vs 0.83, with lower FPR; Table 31), why not use it as the default?

My main remaining concern is that SPARSE and the baselines are shown at different operating points. At t = 3, SPARSE’s measured clean FPR is 11% for BERT, 22.5% for Llama and 16.2% for Gemma (Tables 24–25). ONION is thresholded at a 1% false-rejection rate (Appendix B.3).

I am also concerned by the bug disclosed in the final note. SPARSE was evaluated on a different data split from the baselines and without a fixed seed, and fixing this changed every SPARSE result (e.g., the BERT average drops by roughly 10 F1 points?). I appreciate the authors reporting this, but a bug this central to the results being found only during the rebuttal, leaves me unconfident that the remaining results are free of similar issues, particularly as I have not been able to check thoroughly.

Overall, the revisions improve the paper’s transparency. However, the central comparison is not made at matched operating points, the only white-box LLM baseline still differs from SPARSE in important ways, and the late bug fix lowers my confidence in the results. I think the paper needs substantial further work/revision before it is ready for publication, and I will keep my score at a 2.

Reviewer_oNGWOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SPARSE, an inference-time detector for Trojan-triggered inputs based on a side-car sparse autoencoder (SAE). The authors train an SAE on clean data at a selected layer of a frozen target model and use reconstruction error, the number of active latent units, and explained variance as fit statistics. Clean means and standard deviations are estimated from a calibration set, and deviations are detected with a three-sigma rule. SAEs can be attached independently to multiple layers and combined by an OR rule. The evaluation covers ResNet image classification, BERT text classification, and Llama/Gemma generation, with five image datasets, four BERT datasets, three generation tasks, and 14 attacks. The paper reports an average F1 of about 0.87 and analyzes SAE variants, layer selection, and failure cases.

Critical strengths
  1. A unified clean-only, read-only detection framework. The target model is not modified, and the detector needs only white-box access and a small clean reference set. The same fit-statistics idea is applied across several model modalities.
  2. Broad evaluation with failure cases. The paper covers many attacks, datasets, and SAE variants, and reports failure settings such as StyleBkd rather than presenting only successful cases.
Critical weaknesses
  1. The three-sigma rule and OR aggregation do not support an overall FPR claim. Even if each statistic were approximately Gaussian, OR aggregation over multiple statistics and layers can make the overall clean FPR much larger than the single-statistic value of 0.27%, and the statistics can be correlated. The paper does not report overall clean-only FPR, AUROC, precision/recall, or confidence intervals. For generative models, the worst value over tokens is used while clean and poisoned inputs are not explicitly length-matched, which can mix length/extreme-value effects into trigger detection.
  2. The configuration-selection and comparison protocols weaken the generalization claim. The preliminary study uses clean and poisoned samples to choose layers, SAE variants, and ensembling, so it is necessary to establish that these attacks are fully independent of the main evaluation. Baselines use framework defaults and usually one seed, while SPARSE uses a fixed three-sigma threshold; the primary metric is F1, which makes the claimed 15-point improvement difficult to interpret fairly. The evaluation also covers only ResNet-18, BERT, Llama-2, and Gemma, without adaptive, multi-trigger, or clean-label attacks.
Questions for authors
  1. Please report clean-only FPR, precision, recall, AUROC, and confidence intervals for every modality and ensemble. How should the three-sigma interpretation be understood after OR aggregation?
  2. Did the layer and SAE choices in the preliminary study use any attack type, model, or test sample from the main evaluation? Please provide a strict nested/held-out selection protocol and results on attacks not used for selection.
  3. For generative models, please report length-matched, fixed-generation-length, and token-count-corrected results, separating trigger signal from input/output length. Please also report per-token computational cost.
  4. Please state the clean/poison prevalence for each F1 result and evaluate very low trigger prevalence, multiple triggers, and adaptive attacks.
2 comments
AuthorsOpenReview ↗

Response to Official Review of Submission31908 by Reviewer oNGW - part 1

(Part 1/2)

Dear Reviewer,

We thank you for the careful and constructive feedback, and we revised the manuscript accordingly. The supplementary material contains a version of the revised manuscript with all changes highlighted in blue. Below, each question or weakness is followed by our answer; points that share the same answer are grouped together.

Q1: "Please report clean-only FPR, precision, recall, AUROC, and confidence intervals for every modality and ensemble. How should the three-sigma interpretation be understood after OR aggregation?"

Weakness 1: "The three-sigma rule and OR aggregation do not support an overall FPR claim. Even if each statistic were approximately Gaussian, OR aggregation over multiple statistics and layers can make the overall clean FPR much larger than the single-statistic value of 0.27%, and the statistics can be correlated. The paper does not report overall clean-only FPR, AUROC, precision/recall, or confidence intervals."

Answer: We agree: the three-sigma value refers to a single test, not to the full detector. Section 4.3 now explains that each test added by the OR is one more chance of a false alarm, so we no longer rely on the nominal value and measure the clean FPR directly. Appendix D.1.1 reports precision, recall, clean FPR, TPR at 1% FPR, and AUROC for every configuration, with bootstrap confidence intervals, and discusses why the measured FPR exceeds the nominal one; Section 6.1 summarizes the averages. Averaged over the four models, SPARSE reaches an AUROC of 0.94 and a TPR at 1% FPR of 0.65, with a measured clean FPR of 12.9%; the excess over the nominal rate comes mainly from the small clean sets of the language models and from non-Gaussian statistics. Appendix D.3.4 adds two rules that target the FPR of the full detector; both lower the clean FPR of the language models (e.g., Llama from 22.5% to 10.9% with the Bonferroni rule), at a cost in recall.


Q2: "Did the layer and SAE choices in the preliminary study use any attack type, model, or test sample from the main evaluation? Please provide a strict nested/held-out selection protocol and results on attacks not used for selection."

Weakness 2 (first part): "The preliminary study uses clean and poisoned samples to choose layers, SAE variants, and ensembling, so it is necessary to establish that these attacks are fully independent of the main evaluation."

Answer: Section 5.1 and Appendix A.3 now describe how the preliminary study is separated from the main evaluation. It uses samples disjoint from all test sets, one model (Gemma) and most datasets and attacks are never used in it, and its design choices are fixed once per modality. Appendix A.3 also groups the main results by their overlap with the preliminary study: the configurations used for selection do not perform better than unseen attacks, datasets, or models (average F1 0.79 against 0.86).


Q3: "For generative models, please report length-matched, fixed-generation-length, and token-count-corrected results, separating trigger signal from input/output length. Please also report per-token computational cost."

Weakness 1 (last part): "For generative models, the worst value over tokens is used while clean and poisoned inputs are not explicitly length-matched, which can mix length/extreme-value effects into trigger detection."

Answer: Appendix D.3.3 now reports the response lengths per task and compares the max pooling of the paper with mean pooling and with pooling over only the first one or four activations (First-1, First-4). First-1 uses a single state, computed from the prompt before any output is generated, and still reaches an average F1 of 0.90 on Llama and 0.86 on Gemma; so a long response is not needed for detection. On NegSentiment, First-4 pools about as many activations for clean as for poisoned inputs and keeps a similar F1. On SST-2, where clean and poisoned responses have the same length and the same possible outputs, SPARSE separates poisoned inputs from clean inputs that produce exactly the same output. Section 6.1 summarizes these controls. Appendix D.4 reports the computational cost: the SAE ensemble adds a few milliseconds per token, or a fraction of a millisecond with batching.

AuthorsOpenReview ↗

Response to Official Review of Submission31908 by Reviewer oNGW - part 2

(Part 2/2)

Q4: "Please state the clean/poison prevalence for each F1 result and evaluate very low trigger prevalence, multiple triggers, and adaptive attacks."

Weakness 2 (last part): "The evaluation also covers only ResNet-18, BERT, Llama-2, and Gemma, without adaptive, multi-trigger, or clean-label attacks."

Answer: Section 5 and Appendix B.1 now describe the composition of the test sets (most are balanced, some have up to twice as many clean inputs), and the precision and F1 of SPARSE are reported at a common 50% prevalence. Appendix D.1.1 also reports them at 1%, 5%, and 10% prevalence for every configuration, and Appendix D.3.4 shows that a stricter decision rule improves F1 at low prevalence. Multi-trigger, clean-label, and adaptive attacks, as well as further architectures, are not evaluated; the Limitations and the Future Work now discuss them.


Weakness 2 (middle part): "Baselines use framework defaults and usually one seed, while SPARSE uses a fixed three-sigma threshold; the primary metric is F1, which makes the claimed 15-point improvement difficult to interpret fairly."

Answer: With the corrected results, the average improvement of SPARSE over the best baseline of each modality is 13 percentage points (Section 6.1). All baselines come from public implementations in the benchmark frameworks used in our study and run in a common pipeline with identical models, attacks, and data splits (Section 5, Appendix B.3). We re-checked them, fixed two implementation issues in ONION, and added DAN as a white-box baseline for BERT (Tables 2 and 18). Besides F1, we now report threshold-free metrics (Section 6.1, Appendix D.1.1). Appendix D.3.5 retrains the SAEs of SPARSE with three additional seeds on 12 configurations: the average F1 is 0.82 ± 0.03 across four seeds, against 0.84 for the main run, and the variability concentrates on three configurations where clean and poisoned inputs are less separated. The target models are trained once, with the seeds reported in Appendix C.


Final note - corrected results. While implementing the revised evaluation protocol, we found and fixed a bug in SPARSE which led it to have a different data split with respect to the one used for the baselines. The seed of SPARSE was also not always fixed in the submitted version. All results of SPARSE were recomputed with the corrected implementation, ensuring a perfect match between the datasets, and with a fixed seed. The average F1 of the generative LLMs is unchanged or higher, while that of the image and encoder-only models is lower. Appendix D.3.5 shows that the results vary little across SAE seeds. On average, SPARSE still outperforms the best baseline of every modality, and the conclusions of the paper are unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.