Fast-ZSAD: Patch-wise Adaptive Inference for Efficient Zero-Shot Anomaly Detection

Zero-shot anomaly detection (ZSAD) enables the detection and localization of anomalies in unseen categories; however, the substantial inference cost of existing Transformer-based ZSAD models limits their processing throughput and practical deployment in large-scale scenarios. Compared with conventional visual tasks, anomaly detection exhibits a highly imbalanced spatial importance distribution, with anomalous regions of interest typically occupying only a small fraction of an image, making it inefficient to uniformly process all patches through the full Transformer depth. In this paper, we propose Fast-ZSAD, a patch-wise adaptive inference framework that dynamically allocates computation to normal and anomalous patches, thereby substantially accelerating ZSAD models. Specifically, Fast-ZSAD employs an Anomaly-Aware Patch Router to estimate patch-wise anomaly evidence, enabling confidently normal patches to exit early while forwarding suspicious patches for deeper inference. During router training, we find that the highly imbalanced ratio between normal and anomalous patches causes the optimization to be overwhelmingly dominated by abundant normal patches. To this end, we introduce Contrast-Aware Regularization (CAR), which encourages compact normal representations while separating anomaly-related patches from the normal distribution, thereby improving routing robustness. Furthermore, during deep inference after patch routing, we observe that the number and spatial relationships of forwarded tokens differ from those under full-token inference, causing deep-layer attention to become less focused on anomaly-relevant regions. We therefore develop Depth-Spatial Feature Correction (DSFC), which refines forwarded representations by jointly exploiting local spatial support and cross-depth feature evolution. Extensive experiments across AACLIP, AD-DINOv3, and RareCLIP on industrial and medical anomaly detection benchmarks demonstrate a favorable performance–efficiency trade-off: Fast-ZSAD achieves a 1.69× average inference speedup on AACLIP while improving the average mScore from 70.58 to 70.61, and further delivers 1.58× and 1.84× average speedups on AD-DINOv3 and RareCLIP, respectively.