ERP: Empirical Representation Prototyping for Language-Free Zero-Shot Anomaly Detection

Recent language-free zero-shot anomaly detection (ZSAD) methods have shown that textual prompts can be replaced by learnable normal and abnormal visual representations. However, these representations are typically optimized as free parameters through classification and segmentation objectives, serving primarily as discriminative anchors without being explicitly constrained to represent the visual patterns observed in training data. In this work, we revisit how normality and abnormality should be represented in language-free ZSAD and propose Empirical Representation Prototyping (ERP), which replaces optimization-based decision vectors with data-derived visual prototypes. Specifically, we introduce an EMA-based representation prototyping mechanism that continuously aggregates normal and abnormal features from training samples to construct representative visual references. For anomaly segmentation, we further develop an importance-aware prototype construction strategy that emphasizes ambiguous normal patches while preserving spatially diverse normal patterns, preventing prototypes from being dominated by trivial regions. In addition, a cosine-based multi-scale feature aggregation strategy enhances local representations for fine-grained anomaly localization. Without relying on textual prompts, ERP achieves strong performance across 12 industrial and medical anomaly detection benchmarks, demonstrating that empirical visual prototypes provide an effective alternative to freely optimized decision representations for language-free ZSAD.