Which Normal Samples Matter? Query-Relevant Reference Selection for Industrial Anomaly Detection

Recent few-shot methods have achieved strong performance in industrial anomaly detection with only a few normal samples as references. However, in practical industrial scenarios, normal data is often abundant and can be continuously collected over time. This raises a practical problem: when a large pool of normal samples is available, which samples are most suitable to serve as references for a given query image? We find that normal references do not contribute equally, and their relevance to the query can substantially affect anomaly detection performance. Based on this observation, we propose a training-free anomaly detection framework built on a foundation visual encoder. For each query image, our method first retrieves a small set of normal samples with the highest relevance to the query. We then use the retrieved references to model anomalies at the patch level by jointly considering how well each query patch matches normal features, and how well it can be explained by the normal variations across references. The resulting anomaly evidence is aggregated across multi-level features for anomaly detection. Experimental results show that our method achieves competitive performance across multi-class and single-class settings, with particularly strong results on pixel-level anomaly localization. Moreover, applying our reference retrieval strategy to existing few-shot methods consistently improves their performance, further demonstrating the importance of query-relevant normal references.