- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work presents SPARSE, an SAE-based anomaly detection framework that is generally applicable to many deep learning models. The authors train a sparse autoencoder on the models' latent states and then use 3 informative statistics, compared to their typical distribution over a clean dataset, to detect anomalies. The method shows promise across various modalities, beating baselines.
Critical strengths
- Clear presentation, method as explained is easy to follow. Additionally, the method proposed is quite interesting and would be a valuable contribution to the literature if shown to be strong empirically.
- Well written background/related work section.
- Multimodality of the method is a nice property, and well explored by the paper.
- A reasonable range of attacks considered.
Critical weaknesses
- The paper tests against only a few white box baselines, none for BERT and only one for generative LLMs. Additionally, the blackbox LLM baselines are quite dated. The authors should compare SPARSE against more recent and stronger baselines, especially for LLMs.
- In LLMs, it appears the detector acts on statistics from generated tokens (rather than prompt tokens), where the harmful output may be trivially recognizable/seperable.
- Baselines seem poorly tuned: ONION scores similarly for all datasets in the BERT case, while all other methods vary significantly - suggest looking at implementation/# of flagged cases here. Additionally Mahalanobis distance numbers seem very low compared to other pieces of work, suggesting there has not been much consideration into making the baselines strong for comparison.
- The paper only reports F1 scores (in main text). It would be good to see results in true positive rate at 1% false positive rate or AUROC, alongside F1.
- The background section could be updated with more recent literature on detecting anomalies in LLMs, as the field is progressing quickly.
- No ablation studies over the combination of statistics used for detection or the pooling used over response tokens, e.g. mean pooling instead of max pooling.
Questions for authors
- Can the authors provide an ablation of combinations of the three SAE statistics used for detection? And for the type of pooling?
- Can the authors show how detection performance is impacted by using other decision rules than 3-sigma/other thresholds?
- Are the Mahalanobis detectors using prompt or response tokens?
- How much training data was used, and was it matched for each method?
- How do you set the threshold for the Mahalanobis detector?
- Typos:
- 014: CNN should be DNN?
- fig 1 caption: 'reprot'
- fig 5 caption: 'exhamined'
- What happens if the fit dataset is slightly noisy (e.g. 1% backdoored data)
- Did you try multivariate Mahalanobis distance for the 3 statistics rather than OR between 3 univariate thresholds?
- Do you also test multi-layer Mahalanobis distance (given you use multi-layer SAEs)?
- Do you do any analysis of which SAE features seem most helpful for detection? Is it just a few critical ones? This would be interesting.
4 comments
Response to Official Review of Submission31908 by Reviewer AfYQ - part 2
(Part 2/3)
Weakness 1: "The paper tests against only a few white box baselines, none for BERT and only one for generative LLMs. Additionally, the blackbox LLM baselines are quite dated. The authors should compare SPARSE against more recent and stronger baselines, especially for LLMs."
Weakness 2: "Baselines seem poorly tuned: ONION scores similarly for all datasets in the BERT case, while all other methods vary significantly - suggest looking at implementation/# of flagged cases here. Additionally Mahalanobis distance numbers seem very low compared to other pieces of work, suggesting there has not been much consideration into making the baselines strong for comparison."
Answer: We compare against the inference-time detectors provided by the benchmark frameworks used in our study (BackdoorBench, OpenBackdoor); the defenses of BackdoorLLM repair the model or modify its decoding and do not flag individual inputs (Section 5, Appendix B.3). Following your suggestion, we added DAN, a white-box multi-layer Mahalanobis detector, as a baseline for BERT (Tables 2 and 18). Your remark on ONION led us to find two implementation issues (padded positions in the perplexity computation and input truncation); after fixing them, our implementation reproduces the official perplexities, and we now evaluate ONION as a detector under the same protocol as the other baselines. The threshold and the token source of the Mahalanobis baseline are now documented in Appendix B.3. These baselines remain standard in recent work (e.g., ParaFuzz, NeurIPS 2023; Liu et al., NAACL 2024; BEAT, ICLR 2025).
Q5: "How do you set the threshold for the Mahalanobis detector?"
Answer: An input is flagged when its squared Mahalanobis distance exceeds the 99.73% quantile of a χ² distribution with 50 degrees of freedom (the number of PCA components), i.e., the same nominal false-positive rate as a single three-sigma test of SPARSE. This is now documented in Appendix B.3.
Q4: "How much training data was used, and was it matched for each method?"
Answer: Appendix B.1 now details the data splits: the training split is used only to train the target model, and all clean data used by SPARSE comes from disjoint parts of the test split, with the same proportions for every dataset. Baselines that need clean reference data use the same samples as SPARSE.
Q7: "What happens if the fit dataset is slightly noisy (e.g. 1% backdoored data)"
Answer: SPARSE addresses a different setting from defenses that operate on poisoned data: it is not fitted on the training data of the model, which poisoning attacks contaminate, but on a small clean set collected by the defender. Contaminating this set would require access to the defender's own data, which is outside our threat model; Appendix B.1 and the Limitations now state this assumption. Following your suggestion, Appendix D.3.6 evaluates SPARSE with one or two poisoned inputs added to its clean set. The larger clean sets of ResNet-18 and BERT are barely affected, whereas the small clean sets of the generative LLMs can break detection on some configurations. A robust estimator removes this sensitivity at the cost of a higher clean FPR, so with small clean sets the defender should collect more clean data or screen it; the Limitations now discuss this.
Q10: "Do you do any analysis of which SAE features seem most helpful for detection? Is it just a few critical ones? This would be interesting."
Answer: Our analysis suggests that triggered inputs cause a diffuse activation of the SAE latents rather than activating a few dedicated ones (Figure 2, Appendix D.2), which is why SPARSE relies on aggregate statistics. A latent-level analysis is an interesting direction, which we added to the Future Work.
Weakness 4: "The background section could be updated with more recent literature on detecting anomalies in LLMs, as the field is progressing quickly."
Answer: Section 2 now covers recent inference-time defenses, including DAN, BEAT, and recent text-classification detectors, as well as LLM defenses that modify the decoding or the model.
Q6: "Typos: 014: CNN should be DNN?; fig 1 caption: 'reprot'; fig 5 caption: 'exhamined'."
Answer: Thank you, all three typos have been corrected.
Response to Official Review of Submission31908 by Reviewer AfYQ - part 3
(Part 3/3)
Final note - corrected results. While implementing the revised evaluation protocol, we found and fixed a bug in SPARSE which led it to have a different data split with respect to the one used for the baselines. The seed of SPARSE was also not always fixed in the submitted version. All results of SPARSE were recomputed with the corrected implementation, ensuring a perfect match between the datasets, and with a fixed seed. The average F1 of the generative LLMs is unchanged or higher, while that of the image and encoder-only models is lower. Appendix D.3.5 shows that the results vary little across SAE seeds. On average, SPARSE still outperforms the best baseline of every modality, and the conclusions of the paper are unchanged.
Hi, thanks for responding.
Firstly, I'll note that this response was posted with less than a day left in the discussion period, and contains an extra 20 pages of material, meaning I have not read all of the new additions. I consider this to be too onerous an ask from the authors. Further, the summaries the authors have written are not especially helpful in summarising these many new experiments (where are the numbers and the plain language results?) which means that I do not have much confidence about how clean these experiments are or if they show what they purport to.
It is good that the authors run ablations on the choice of SAE statistics. I find the response to my q3 on prompt/response tokens to be somewhat difficult to understand. For example, the sentence: "First, restricting SPARSE to a single state, the one that produces the first generated token, removes the response entirely, since this state is computed from the prompt alone" is very difficult to parse. Does state mean activation? In what way does it 'remove the response entirely'? I take it to maybe mean that it is not acting on the response any more? If so, this addresses my concern, and it should be stated plainly in the paper. Given that First-1 performs better than the main configuration on Llama (F1 0.90 vs 0.83, with lower FPR; Table 31), why not use it as the default?
My main remaining concern is that SPARSE and the baselines are shown at different operating points. At t = 3, SPARSE’s measured clean FPR is 11% for BERT, 22.5% for Llama and 16.2% for Gemma (Tables 24–25). ONION is thresholded at a 1% false-rejection rate (Appendix B.3).
I am also concerned by the bug disclosed in the final note. SPARSE was evaluated on a different data split from the baselines and without a fixed seed, and fixing this changed every SPARSE result (e.g., the BERT average drops by roughly 10 F1 points?). I appreciate the authors reporting this, but a bug this central to the results being found only during the rebuttal, leaves me unconfident that the remaining results are free of similar issues, particularly as I have not been able to check thoroughly.
Overall, the revisions improve the paper’s transparency. However, the central comparison is not made at matched operating points, the only white-box LLM baseline still differs from SPARSE in important ways, and the late bug fix lowers my confidence in the results. I think the paper needs substantial further work/revision before it is ready for publication, and I will keep my score at a 2.
Response to Official Review of Submission31908 by Reviewer AfYQ - part 1
(Part 1/3)
Dear Reviewer,
We thank you for the careful and constructive feedback, and we revised the manuscript accordingly. The supplementary material contains a version of the revised manuscript with all changes highlighted in blue. Below, each question or weakness is followed by our answer; points that share the same answer are grouped together.
Weakness 3: "The paper only reports F1 scores (in main text). It would be good to see results in true positive rate at 1% false positive rate or AUROC, alongside F1."
Q2: "Can the authors show how detection performance is impacted by using other decision rules than 3-sigma/other thresholds?"
Answer: Section 6.1 now reports AUROC, TPR at 1% FPR, and the measured clean FPR alongside F1, and Appendix D.1.1 reports them, with precision, recall, and bootstrap confidence intervals, for every configuration. AUROC and TPR at 1% FPR do not depend on the three-sigma rule. Appendix D.3.4 compares the three-sigma rule with a threshold calibrated on held-out clean data and with an exact small-sample threshold with a Bonferroni correction. For ResNet-18, the calibrated rule meets its target and slightly improves F1 (0.89 to 0.92). For the language models, both rules lower the clean FPR (e.g., Gemma from 16.2% to 3.8% with the Bonferroni rule) at a cost in recall, which raises F1 when poisoned inputs are rare (Gemma at 1% prevalence: 0.20 to 0.57).
Q1: "Can the authors provide an ablation of combinations of the three SAE statistics used for detection? And for the type of pooling?"
Q8: "Did you try multivariate Mahalanobis distance for the 3 statistics rather than OR between 3 univariate thresholds?"
Q9: "Do you also test multi-layer Mahalanobis distance (given you use multi-layer SAEs)?"
Weakness 6: "No ablation studies over the combination of statistics used for detection or the pooling used over response tokens, e.g. mean pooling instead of max pooling."
Answer: Appendix D.3 adds an ablation study on the full evaluation. Appendix D.3.1 evaluates all combinations of the three fit statistics and, following your suggestion, a multivariate Mahalanobis test over them; Appendix D.3.2 evaluates each layer strategy and a multivariate test across layers; Appendix D.3.3 compares max, mean, and first-token pooling. Combining the statistics improves F1 over any single statistic on ResNet-18 and BERT; the best single statistic changes across modalities, and the combination of the paper is within 0.04 F1 of the best variant for every model. The full layer ensemble has the best F1 of all OR-based configurations for every model (e.g., Llama: 0.83 against at most 0.77 for a single layer strategy). The multivariate tests are slightly better on ResNet-18 and BERT but worse on the generative LLMs, with the highest clean FPR, since their covariance is estimated from few clean samples. Pooling changes the operating point more than the ranking (average AUROC 0.92–0.96 for all variants). Each subsection discusses the trade-offs; the main threshold configuration of the paper is unchanged.
Weakness 5: "In LLMs, it appears the detector acts on statistics from generated tokens (rather than prompt tokens), where the harmful output may be trivially recognizable/separable."
Q3: "Are the Mahalanobis detectors using prompt or response tokens?"
Answer: SPARSE reads the activations of the generated tokens, whereas the Mahalanobis baseline reads the prompt tokens; both choices are now stated explicitly (Appendices A.2 and B.3). Section 6.1 and Appendix D.3.3 now test whether SPARSE recognizes the output rather than the trigger. First, restricting SPARSE to a single state, the one that produces the first generated token, removes the response entirely, since this state is computed from the prompt alone; the average F1 stays close to that of the full response (0.90 on Llama and 0.86 on Gemma), and far above the prompt-based Mahalanobis baseline. Second, on SST-2, the backdoored answer is a valid label that clean inputs also produce; SPARSE flags clean inputs with this same output no more often than the others and still separates them from poisoned inputs. Finally, SPARSE never observes the target behavior: it only flags departures from the clean behavior of the model, whereas recognizing a harmful output would require knowing the target.