ADoRe: Normality-Guided Token Pruning for Efficient Anomaly Detection

Recent anomaly detectors reach strong accuracy, but their deployment is bound by throughput. A faster decision shortens inspection cycle time and raises the number of objects a surveillance system handles in real time. Defects are small, however, so inputs must stay at high resolution, and the Vision Transformer (ViT) encoder accounts for most of the inference time. Token pruning is a natural remedy, yet existing methods inherit from image classification the intuition that a token which cannot change a single global prediction is safe to discard. This intuition fails in anomaly detection, which labels every location rather than the whole image: faint defects are discarded while salient normal regions are kept. What can be released is a token whose normality is settled. Under ground-truth labels, releasing such tokens carries none of the accuracy cost that pruning usually incurs and can reverse it, for two reasons: anomalous patches would keep their scores while released ones can only lose theirs, and the survivors, encoded with fewer tokens, become easier to separate. Based on this, we propose ADoRe, a training-free module that scores each token against a normality reference trimmed to resist the anomalies it must separate, and accumulates its rank over a band of intermediate layers. The most normal patches are released and enter the anomaly map as zero without interpolation. Across CLIP, DINOv2, and video detectors, ADoRe accelerates the encoder by up to 2.47$\times$ while matching or exceeding the unpruned baseline in most settings, and at a fixed time budget, spending the savings on a larger encoder or more crops surpasses the unpruned pipeline.