- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work applies sparse autoencoders (SAE) and multimodal large‑language‑model(MLLM) automated interpretability to improve frozen visual anomaly detectors (primarily PatchCore). The pipeline decomposes anomaly residuals from patch embeddings into sparse SAE features; an MLLM labels each feature as anomaly / distractor / uncertain using high‑activation and contrastive examples. It suppresses distractor features and amplifies anomaly features, then reconstructs patch embeddings for rescoring. Evaluations cover 43 categories across five benchmarks, including synthetic corruptions and real‑world acquisition shifts from RobustAD. Joint intervention lifts macro AUROC from 0.8724 to 0.8857 on source data, and brings larger gains under real distribution shifts. Ablations study SAE hyper‑parameters, MLLM alternatives, other anomaly detectors and vision backbones.
Critical strengths
This work propose a new paradigm, which uses automated mechanistic interpretability to boost downstream anomaly‑detector performance, whereas most prior SAE‑MLLM interpretability work focuses only on understanding model behaviour. This paper turns interpretability into actionable intervention: semantic labels from MLLM directly modify internal representations to improve detection. It establishes a practical use‑case for mechanistic interpretability beyond pure explanation research.
Critical weaknesses
- The proposed method requires calibration set containing anomalous samples, breaking the “normal‑only” premise of unsupervised anomaly detection. As we known, standard industrial anomaly detection only uses normal images for training, but this method needs a balanced calibration set with both normal and anomalous examples to train SAE and select amplification α. Even though the base detector remains normal‑only, the intervention pipeline depends on anomaly examples, limiting real‑world zero‑shot / purely‑normal‑only deployment.
- Ablations show performance gains are not universally transferable. For the FRE detector and WideResNet‑50‑2 backbone, interventions deliver negligible or zero improvements. MLLM feature label agreement between different open‑source LLMs is moderate (~67‑81%). While gains persist on average, there is no guarantee of improvement for arbitrary backbones / detectors, which limits general applicability.
Questions for authors
- How sensitive is the SAE feature quality and subsequent MLLM labeling to the diversity and count of anomalous samples in the calibration set? Specifically, how does performance behave if test anomalies differ fundamentally in type from those present in calibration?
- Could you quantify the end-to-end inference latency added per image by the nearest-neighbor residual computation, SAE encoding/decoding, and secondary memory-bank rematching compared to standard PatchCore?
Thank you for taking the time to review our paper.
W1. We already explicitly acknowledge this limitation in the paper. However, calibration sets containing some anomalous examples are not extraneous to normal-only anomaly detection, as even detectors trained exclusively on normal images may use anomalous examples to select a decision threshold or calibrate anomaly heatmaps. We agree that our approach would not apply in scenarios where no anomalous examples are available. Such scenarios certainly exist, for example when introducing a new product, but we are less convinced that they are representative of industrial anomaly detection as a whole.
W2. We never claimed that our interventions guarantee improvements, and we already explicitly acknowledge this in the limitations paragraph. SAE training and MLLM labels introduce variability, and this is largely an empirical paper. What we show is that interventions improve performance on average across many categories, particularly under synthetic corruptions and even more under real acquisition shifts. Regarding detectors, with FRE we see strong improvements on source images, while results are mixed under synthetic corruptions. Regarding backbones, we see improvements with SwinV2, but not with WideResNet-50-2. We ran several ablations, and using all 43 categories for all of them was not feasible, especially those involving MLLM labeling. Both the detector and backbone ablations therefore cover 5 categories, and further investigation could help us better understand the cases where improvements do not seem to transfer. Regarding the MLLMs, we would not consider 81.1% agreement between GPT and Gemma across 640 feature labels particularly low. Qwen agrees less with the other two models (67.8–69.5%), and its results are slightly lower. Still, all three improve over original PatchCore with joint intervention on both source and corrupted data.
Q1. We added experiments in Appendix H investigating calibration size and sensitivity to the selection of calibration examples, retraining the SAE and regenerating the MLLM labels for each configuration. Increasing calibration size generally improves SAE reconstruction quality, with diminishing returns in downstream performance. Across five calibration splits, AutoInterp consistently outperforms calibration-guided interventions, with relatively limited performance variability. Regarding unseen anomaly types, we added Appendix I, where we evaluate 20 categories with defect types entirely absent from calibration. We found that AutoInterp joint improves over original PatchCore on both seen and unseen defects, under source and corrupted conditions. Interestingly, calibration-guided joint performs better on seen defects, while AutoInterp performs better on unseen defects, where calibration-guided joint even falls below original PatchCore.
Q2. We added Appendix J with an inference latency benchmark across all 43 categories. On an NVIDIA L40S with batch size 1, original PatchCore takes 51.06 ± 6.13 ms per image, while AutoInterp joint takes 75.94 ± 11.34 ms, corresponding to an additional 24.88 ± 5.33 ms per image (48.7% overhead). Most of the overhead comes from SAE processing and secondary nearest-neighbor rematching.
We believe we have addressed all of your points, but if anything remains unclear or unconvincing, we would be happy to clarify. Thank you again for the constructive review.