AnomalyRAG: Relation-Centric Dual-Bank Evidence Retrieval for Surveillance Anomaly Detection and Reasoning

Surveillance anomaly detection and reasoning must identify context-dependent violations and explain the responsible entities and rules. Similar scenes can require opposite labels when object relations differ, yet global retrieval often overlooks small decisive objects. We present AnomalyRAG, a weakly supervised framework that equips a frozen VLM with Relation-Centric Dual-Bank Evidence Retrieval. It constructs normal and abnormal evidence banks from training data, reranks multi-route candidates through learned object–relation alignment, and derives a calibrated signed margin. Verified bilateral exemplars guide Structured Evidence-Grounded Reasoning Chain-of-Thought (SEGR-CoT), while reliability-gated symmetric fusion admits only grounded corrections. We evaluate detection on UCF-Crime and XD-Violence, reasoning on ECVA and UCF-Crime, and public-safety recognition on the 24-task SMAD dataset. AnomalyRAG improves detection and reasoning while training only lightweight retrieval and calibration modules.