Similarity Is Not Normality: Reasoning over Retrieved Evidence for Anomaly Detection

Retrieval-based anomaly detection retains normal exemplars at inference, yet most existing methods fuse retrieval discrepancies without preserving which descriptor produced each of them. Similarity alone does not explain normality. A cracked but visually plausible component may retrieve close appearance neighbors while violating part layout, whereas a normal object observed from an unseen viewpoint may exhibit the opposite pattern. The key question is not only how large a discrepancy is, but also which property causes it and whether the remaining properties agree. Once an aggregation removes that distinction, a predictor observing only the aggregation cannot recover it. We refer to this limitation as the retrieval-to-reasoning gap and show that it is measurable, since permutation-invariant summaries of retrieved evidence are markedly less separable than the evidence itself on real data. R2AD retrieves from four factorized partitions of a frozen prototype bank corresponding to appearance, geometry, structure, and semantics, selecting the partition of the category predicted from the class token without using ground-truth category labels. It calibrates each discrepancy against an empirical distribution of normal costs, placing all factors on a common rank scale while preserving their identities. A conflict-aware mixture of experts then reasons over the resulting evidence spectrum, retaining isolated violations while attenuating the coherent elevations that benign variation tends to produce. R2AD reaches 75.1 pixel AP and 96.6 AUPRO on MVTec-AD and 46.5 pixel AP and 94.1 AUPRO on VisA, and ablation studies demonstrate the contribution of the evidence operators, the category conditioning, and the rank calibration.