acceptodds
Under review as a conference paper at ICLR 2027

AnomalyRAG: Relation-Centric Dual-Bank Evidence Retrieval for Surveillance Anomaly Detection and Reasoning

Abstract

Surveillance anomaly detection and reasoning must identify context-dependent violations and explain the responsible entities and rules. Similar scenes can require opposite labels when object relations differ, yet global retrieval often overlooks small decisive objects. We present AnomalyRAG, a weakly supervised framework that equips a frozen VLM with Relation-Centric Dual-Bank Evidence Retrieval. It constructs normal and abnormal evidence banks from training data, reranks multi-route candidates through learned object–relation alignment, and derives a calibrated signed margin. Verified bilateral exemplars guide Structured Evidence-Grounded Reasoning Chain-of-Thought (SEGR-CoT), while reliability-gated symmetric fusion admits only grounded corrections. We evaluate detection on UCF-Crime and XD-Violence, reasoning on ECVA and UCF-Crime, and public-safety recognition on the 24-task SMAD dataset. AnomalyRAG improves detection and reasoning while training only lightweight retrieval and calibration modules.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Anomaly detection, Vision language model, Retrieval-Augmented Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Q5eYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes that Anomaly is a relative concept instead of the objective concept where the label is sometimes depends on the scene rules/relations. Thus, it reveals the disadvantages of current VAD method where the zero-shot surveillance judgments remain coarse or inconsistent. They may recognize entities yet miss the interaction or rule that makes an event abnormal.

This paper aims to deal with AnomalyRAG method , a weakly supervised framework that equips a frozen VLM with Relation-Centric Dual-Bank Evidence Retrieval.

Learned object–relation alignment retrieves matched precedents for a frozen VLM, whose Structured Evidence-Grounded Reasoning Chain-of-Thought (SEGR-CoT) contrasts rule satisfaction and violation.

Critical strengths
  1. The method designed is reasonable and easy to follow. It aims in designing the fine-graiend relation with GNN method which is neat and effective and it reaches the SOTA in ucf-crime and XD-violence dataset.
Critical weaknesses
  1. The backbone is a bit old-fashioned. In the paper, it uses qwen3-vl and InternVL2 while I believe the latest version of qwen3.5 and InternVL3 has released for a while which should be tested.

  2. In the section of relation-centric dual-bank evidence retrieval, it has a lot of hyper-parameters such as numbers in evidence reliability. However, in the ablation study, there is no much experiments and discussion on it.

  3. Moreover, the optimization process and details are missing.

Questions for authors
  1. Some of the papers related to context-based VAD is missing.

  2. Although the ucf-crime and XD-violence are great fundamental benchmarks, I believe there are various advanced and comprehensive benchmarks should be tested such as Vad-R1[1].

Huang C, Wang B, Wang W, et al. Vad-r1: Towards video anomaly reasoning via perception-to-cognition chain-of-thought[J]. Advances in neural information processing systems, 2026, 38: 118486-118518.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Q5eY (1/2): Newer backbones and parameter sensitivity

Thank you for recognizing the relation modeling and detection results. We added the newer backbones and sensitivity tests you requested, clarified training, and extended the evaluation to Vad-Reasoning.

C1: Evaluation with newer VLM backbones

Reply:

  1. InternVL2-8B was chosen to support a backbone-matched comparison with VERA. Qwen3-VL-8B-Instruct tests transfer to another model family. Following your suggestion, we added InternVL3-8B and Qwen3.5-9B, replacing only the frozen VLM used for anomaly judgment. Data splits, evidence banks, graph construction, retriever, and input/output budgets stay fixed.

Table F. Detection across judgment backbones. All metrics are percentages. VERA is from main-paper Table 1; the last two rows are newly added experiments.

MethodVLMUCF AUCXD AUCXD AP
VERAInternVL2-8B86.5588.2670.54
AnomalyRAGInternVL2-8B87.9689.0372.95
AnomalyRAGQwen3-VL-8B-Instruct89.9692.1574.16
AnomalyRAGInternVL3-8B90.2391.6975.68
AnomalyRAGQwen3.5-9B91.6692.2277.25
  1. Both newer models improve all three metrics over the earlier backbone in the same family. InternVL3-8B improves UCF AUC, XD AUC, and XD AP by 2.27, 2.66, and 2.73 points over InternVL2-8B. Qwen3.5-9B improves them by 1.70, 0.07, and 3.09 points over Qwen3-VL-8B-Instruct. The same retrieval framework can therefore benefit from the newer judgment models without changing its evidence banks or retriever.

Revision: We extended Supplementary Table 14 and its accompanying description with both new backbones and the controlled replacement protocol.

C2: Reliability parameters and sensitivity

Reply:

  1. Supplementary Tables 15-16 already examine bank size, candidate count, retained evidence count, relation-score scaling, admission thresholds, and graph size. We added Table G to test the reliability weights, confidence threshold, and fusion coefficient more directly.

  2. These parameters have distinct roles. Seven fixed cue weights form the evidence-confidence score. The confidence threshold determines whether evidence can modify a prediction, and the fusion coefficient determines the size of that correction. The threshold and coefficient are selected on validation data; retrieval branch weights are learned during training. This separates evidence reliability from learned candidate ranking.

  3. Table G changes one setting at a time. The default threshold is 0.50 and the default fusion coefficient is 0.25. Evidence use is the fraction of queries that pass both the confidence threshold and verification, enabling an evidence-based correction. Queries without an admitted correction retain their evidence-free predictions.

Table G. Evidence use and parameter sensitivity.

SettingEvidence use (%)UCF AUC (%)
Default27.8389.96
Equal weights for seven cues27.1189.23
Confidence threshold 0.4534.9189.01
Confidence threshold 0.5522.6788.92
Fusion coefficient 0.1027.8389.78
Fusion coefficient 0.5027.8389.02
  1. All tested alternatives stay within 1.04 AUC points of the default; equal cue weights reduce AUC by 0.73 points. Changing the threshold changes how often evidence is used. Changing the fusion coefficient leaves evidence use unchanged and changes only the correction strength. The results show the expected effects of these controls while retaining similar detection performance, with the default achieving the highest AUC among the tested settings.

Revision: Section 3.2 now distinguishes fixed reliability weights from learned retrieval weights. Supplementary Section 7.6 and Table 29 add these sensitivity tests; Supplementary Tables 4-5 document the defaults and gate diagnostics.

AuthorsOpenReview ↗

Response to Reviewer Q5eY (2/2): Training, related work, and Vad-Reasoning

C3: Optimization procedure and reproducibility

Reply:

  1. Training has three stages: construct the evidence banks from training data, train the relation retriever, and calibrate bank scores and fusion parameters on validation data. The detector, OpenCLIP visual encoder, and VLM remain frozen. Training and evidence examples come from the training split; test videos are used only as queries.

  2. Each training query uses a same-task, same-label positive from another source and three opposite-label hard negatives. The objective combines dual-bank discrimination, contrastive retrieval, pairwise ranking, rule compatibility, and visual anchoring, with respective weights 1, 1, 0.5, 0.2, and 0.1. These terms train evidence selection and alignment around the anomaly criterion.

  3. We use AdamW with learning rate and weight decay 1e-4, batch size 64, gradient clipping at 1, and at most 50 epochs. We retain the checkpoint with the lowest validation loss and use early-stopping patience 5. Calibration and fusion settings are selected on validation data and then fixed for testing.

Revision: Main-paper Section 4.1 now states the training sequence and optimization setup. Section 3.3 defines the objective, and Supplementary Sections 3.2-3.3 provide the detailed sampling and optimization procedure.

Q1: Related work on contextual anomaly detection

Reply:

  1. Section 2 and Supplementary Tables 10-11 cover contextual VAD from several perspectives. AnomalyRuler derives normality rules; PANDA retrieves scene-conditioned rules; SlowFastVAD retrieves normal and derived abnormal pattern text. VADER encodes object interactions within a video, while SARD identifies rare scene-graph relations. The revised discussion explains how AnomalyRAG uses this contextual information: it aligns objects and relations across the query and retrieved normal/abnormal images, then compares rule satisfaction and violation at the object level.

  2. We added Vad-R1, which learns perception-to-cognition anomaly reasoning through supervised fine-tuning and reinforcement learning. AnomalyRAG keeps the judgment VLM frozen and learns retrieval and calibration to provide instance-level visual evidence. The comparison clarifies their different ways of supporting anomaly reasoning.

Revision: Section 2 now includes Vad-R1 and the more explicit comparison of contextual reasoning mechanisms.

Q2: Evaluation on Vad-Reasoning

Reply:

  1. We added the requested benchmark to complement the detection and explanation evaluations in main-paper Tables 1-2. The released Vad-Reasoning test file contains 439 videos and evaluates classification, temporal localization, and generated reasoning and answers. Source videos in this test set are excluded from training and both evidence banks.

Table H. Vad-Reasoning results. Metrics range from 0 to 1; higher is better. Vad-R1 values are from its published Table 2.

MetricVad-R1AnomalyRAG
Classification accuracy0.8750.891
Classification F10.8620.882
Temporal mIoU0.7130.698
Temporal [email protected]0.7700.761
Temporal [email protected]0.7060.699
BLEU-20.2330.208
METEOR0.4060.411
ROUGE-20.1940.223
  1. We use the official evaluation scripts. Classification treats abnormality as the positive class. Temporal metrics compare predicted and reference intervals; classification errors receive zero localization credit, and normal videos use the full interval. Text metrics compare the reasoning and answer after format tags are removed.

  2. Vad-R1 uses fine-tuned Qwen2.5-VL-7B; AnomalyRAG uses frozen Qwen3-VL-8B-Instruct with retrieved evidence. In this comparison, AnomalyRAG improves classification accuracy by 1.6 percentage points and F1 by 2.0 points, and obtains higher METEOR and ROUGE-2. The table also reports all three localization metrics and BLEU-2, extending the evaluation to the temporal and textual dimensions you requested.

Revision: We added Section 4.5 and Table 5 to the main paper, with the complete protocol in Supplementary Section 7.7.

We hope the newer-backbone results, sensitivity experiments, training details, and broader evaluation address your concerns. We would appreciate your reconsideration of the rating in light of these additions.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Manuscript Revisions

Dear Reviewer Q5eY,

Thank you again for your constructive suggestions. We have updated the experiments and manuscript to address the points you raised:

  1. Newer VLM backbones. Supplementary Table 14 now includes InternVL3-8B and Qwen3.5-9B, with the evidence banks and retriever held fixed when replacing the judgment backbone.
  2. Parameter sensitivity. Supplementary Table 29 tests the evidence-reliability weights, confidence threshold, and fusion strength. Section 3 also clarifies which parameters are fixed, learned, or selected on validation data.
  3. Training and related work. Section 4.1 and the supplement now detail the training stages, objectives, sampling, and optimization settings. Section 2 expands the contextual anomaly-detection discussion and includes Vad-R1.
  4. Broader reasoning evaluation. New Section 4.5 and main-paper Table 5 report Vad-Reasoning results alongside Vad-R1, covering classification, temporal localization, and generated reasoning and answers.

These updates are included in the revised manuscript and supplement, with revisions marked in blue. Could you let us know whether they address your concerns, or if there are remaining questions we can help clarify?

Thank you for your time and consideration.

Reviewer_Q5eYOpenReview ↗

Answering my Answer

The authors partially answer my questions and I will consider my final score.

Reviewer_LgFVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AnomalyRAG, which uses object relations to retrieve normal and abnormal examples to guide a frozen VLM. Experiments on three public benchmarks and the proposed SMAD dataset show improvements in anomaly detection and reasoning.

Critical strengths
  1. The paper addresses a clear problem: anomaly judgments depend on object relations and scene rules.
  2. Experiments cover several benchmarks, with detailed ablations and human checks.
Critical weaknesses
  1. The novelty is limited. SlowFastVAD already retrieves normal and abnormal patterns, while VADER uses object relations. The main addition is relation matching for retrieving examples from both banks.
  2. The ablation study does not include global retrieval from both banks with the same prompt and gate. This makes it hard to tell how much relation matching itself helps.
  3. It is unclear how the model knows which rule to apply without the task ID or rule as input. For example, fishing with one rod can be normal in one task but abnormal in another.
Questions for authors
  1. How much does relation matching improve over dual-bank global retrieval when the reasoning prompt and gate remain unchanged?
  2. How does the model choose the correct rule when visually similar scenes require different labels?
4 comments
AuthorsOpenReview ↗

Response to Reviewer LgFV (1/3): Core contribution

Thank you for recognizing the problem motivation and experimental coverage. We clarify the contribution first, then provide the requested retrieval and rule-selection experiments in the next two comments.

C1: Core contribution and differences from SlowFastVAD and VADER

Reply:

  1. Anomaly judgments depend on object states, interactions, the number of objects associated with each subject, and scene rules. Recognizing that a person and fishing rods are present does not establish whether that person violates a one-rod-per-person rule. The model needs to identify which rods belong to which person and interpret that configuration under the applicable rule. This motivates using concrete normal and abnormal visual references to support the judgment.

  2. Ordinary RAG can supply the wrong reference because global similarity often favors a similar background over the detail that determines the label. Two fishing scenes can look similar while involving different rod counts or different rules. We therefore retrieve at the object-relation level, preserving each subject's local configuration and seeking precedents with compatible relations and rules. The normal and abnormal references then show what rule satisfaction and violation look like.

  3. Our central contribution is relation-aligned bilateral precedent comparison for anomaly detection and reasoning. AnomalyRAG connects structured evidence construction, relation-centric retrieval, explicit rule comparison, object attribution, and reliability-controlled score correction. Relation matching selects the evidence within this framework; the subsequent comparison explains why the query satisfies or violates a rule and identifies the responsible objects.

  4. SlowFastVAD summarizes normal behavior, derives abnormal pattern text, and retrieves patterns using clip descriptions. Its retrieved units are pattern descriptions. VADER uses CORE to encode dynamic object interactions within the input video. AnomalyRAG retrieves actual normal and abnormal visual precedents and aligns their objects and relations with the query, so the VLM can compare the relevant configurations under their rules. Both works are discussed in Section 2 and Supplementary Tables 10-11.

  5. The retrieval mechanism preserves the correspondence needed for this comparison. Each relation motif contains a subject-relation-object triple and the endpoints' one-hop neighborhoods, retaining the objects and counts associated with each subject. Unary tokens retain state cues such as smoke or damage. Soft object correspondences constrain relation endpoints, and partial Sinkhorn alignment allows extra or missing objects to remain unmatched. Global, object, relation, and rule scores jointly rank the candidates. Training uses compatible positives and opposite-label hard negatives to distinguish visually similar configurations with different anomaly criteria.

  6. The selected correspondences remain available during reasoning. Structured Evidence-Grounded Reasoning Chain-of-Thought (SEGR-CoT) compares the query with a normal and an abnormal precedent, identifies decisive matches and mismatches, and attributes a rule violation to specific query objects. A calibrated signed margin summarizes support from both banks. Verification and reliability-gated fusion determine whether the evidence can adjust the evidence-free anomaly score, either upward or downward. The VLM remains frozen.

  7. The experiments examine these stages separately. Main-paper Table 3 tests the progression from global retrieval to relation retrieval, dual-bank comparison, bilateral reasoning, and gating. Supplementary Table 12 isolates component effects, and Table 13 reports paired significance tests. The new dual-bank control in Supplementary Table 23 keeps the prompt and gate fixed and compares the full retriever with global retrieval and three simpler structured alternatives. Together, these experiments support both the framework and its specific evidence-matching design.

Revision: We revised the motivation and contribution in Section 1, the comparisons with SlowFastVAD and VADER in Section 2, and the explanation of evidence selection and rule comparison in Section 3. Figure 2 presents the complete framework. Section 4.3 summarizes the new controlled retrieval experiment, with its full results in Supplementary Table 23.

AuthorsOpenReview ↗

Response to Reviewer LgFV (2/3): Retrieval with both banks fixed

C2 / Q1: Gain over dual-bank global retrieval with the same prompt and gate

Reply:

  1. Main-paper Table 3 introduces components progressively. It first tests relation-centric retrieval with the abnormal bank, then adds the normal bank, bilateral reasoning, and reliability gating. This establishes the contribution of relation retrieval at that stage. Your requested comparison asks a more specific question: how much does retrieval matter when the complete dual-bank reasoning system is already present? We added Table A to test exactly this.

  2. All methods in Table A use the same normal and abnormal banks, frozen judgment VLM, perception inputs, reasoning prompt, gate, and calibration procedure. The global visual baseline retrieves its own candidates. The structured alternatives share a candidate set, and their trainable components use the same data, objectives, and budget. Object-relation records supplied to the reasoning stage do not feed back into candidate ranking. This keeps the downstream use of evidence fixed while changing how the evidence is selected.

Table A. Retrieval under fixed dual-bank settings. All values are percentages. SMAD uses automatic boxes. UCF match@1 measures human-verified matching of the decisive relation, following Supplementary Table 6.

RetrievalUCF AUCSMAD accuracyUCF match@1
Dual-bank global visual84.1287.3676.12
Relation-triple text86.3588.3878.76
Graph embedding, same encoder85.1989.1279.13
Bidirectional object nearest neighbor86.9889.5480.39
Full AnomalyRAG89.9690.6284.2
  1. With the prompt and gate unchanged, AnomalyRAG improves UCF AUC by 5.84 percentage points and automatic-box SMAD accuracy by 3.26 points over dual-bank global retrieval. The decisive-relation match rate improves by 8.08 points. Better selection therefore improves both the relevance of the retrieved evidence and the final anomaly judgment.

  2. Table A also tests whether simpler structured retrieval is sufficient. Relation-triple text retrieval compares relation descriptions; graph embeddings compare pooled structure; object nearest-neighbor matching uses object correspondences without partial transport. The full method leads on all three metrics. Its gains over object nearest-neighbor matching are 2.98 points in UCF AUC, 1.08 points in SMAD accuracy, and 3.81 points in relation matching. These results support preserving local relation motifs and coordinating relation matches through object alignment.

Revision: We added the complete comparison as Supplementary Table 23 in Section 7.1 and summarized its main finding in main-paper Section 4.3. This table directly answers C2 and Q1 under the fixed-prompt, fixed-gate setting you requested.

AuthorsOpenReview ↗

Response to Reviewer LgFV (3/3): Selecting the applicable rule

C3 / Q2: Rule selection without a task ID and similar scenes with different labels

Reply:

  1. Rules are stored with training precedents. During evidence construction, each instance receives a description and a rule explaining its normal or abnormal configuration. At inference, the query contains no task ID, task name, label, or ground-truth rule. Its visible objects, interactions, and scene cues guide retrieval, and compatibility between the query's local representation and a candidate's stored rule contributes to ranking. The selected precedents supply the rules used for comparison.

  2. In your example, one rod is permitted under a one-rod-per-person rule but prohibited under a no-fishing rule. The judgment therefore requires both the observed configuration and the applicable rule. If visible context distinguishes the settings, retrieval can select a precedent with the relevant rule. If the same image is governed by an externally specified policy, that policy enters the decision stage as the current rule. We evaluate these two cases separately below.

  3. Table B1 tests rule selection from visible cues. Both methods process the same 200 images, comprising 100 normal and 100 abnormal cases, using the same banks and VLM. Neither receives the query task ID or ground-truth rule. Rule selection is correct when all precedent rules supplied to the decision stage apply to the query; conflicting rules or the absence of a reliable rule count as errors.

Table B1. Rule selection from visible cues. Values are percentages. False positives are measured on the normal images.

RetrievalCorrect rule selectionFalse positive rateAccuracy
Dual-bank global retrieval811487
AnomalyRAG92891
  1. Relation-centric retrieval selects the applicable rule more often, reduces false alarms, and improves classification. This tests the standard inference setting directly: the model recovers the relevant rule through visual evidence and retrieved precedents, without being given the task identity.

  2. Table B2 tests your policy ambiguity directly. Each of 100 images is paired with two rules that imply opposite labels, giving 200 image-rule pairs, balanced between normal and abnormal. The images and retrieved evidence are identical across the two conditions; only the current rule is added to the decision input.

Table B2. Identical images under different current rules.

Decision inputAccuracy (%)
Image and retrieved evidence50
Same image and evidence, plus current rule94
  1. Without the current rule, the same prediction is reused for the two opposite labels, so 50% follows from the paired design. Supplying the rule yields 94% measured accuracy. The result shows that the decision stage can combine the same visual evidence with different policies and adjust the judgment accordingly. Table B1 tests selecting a rule from visible context; Table B2 tests applying a specified rule to an unchanged scene.

Revision: Sections 3.1-3.2 now clarify the rule source, query inputs, and query-rule compatibility score. Supplementary Section 7.2 and Tables 24-25 include both experiments and their protocols.

We hope the clarified contribution, controlled retrieval comparison, and rule-selection experiments address your concerns. We would appreciate your reconsideration of the rating in light of these revisions and results.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Manuscript Revisions

Dear Reviewer LgFV,

Thank you again for your careful review. Following your comments, we have revised the manuscript and added experiments addressing your three main concerns:

  1. Contribution and prior work. Sections 1-3 clarify how AnomalyRAG combines relation-aligned normal and abnormal visual precedents, rule comparison, and evidence-grounded reasoning, and explain its differences from SlowFastVAD and VADER.
  2. Controlled dual-bank retrieval. Supplementary Table 23 compares global retrieval and simpler structured alternatives with the same evidence banks, frozen VLM, reasoning prompt, and gate, directly testing the additional benefit of relation matching.
  3. Selecting the applicable rule. Supplementary Tables 24-25 evaluate rule selection without task IDs and judgments on identical images under different supplied rules, addressing the ambiguity raised in your fishing example.

These additions are included in the revised manuscript and supplement, with revisions marked in blue. We would appreciate your feedback on whether they address your concerns and whether any points need further clarification.

Thank you for your time and consideration.

Reviewer_p99gOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AnomalyRAG, a weakly supervised framework for surveillance anomaly detection and reasoning using a frozen vision-language model (VLM). The central idea is to retrieve and contrast normal and abnormal precedents at the object-relation level rather than relying exclusively on global visual similarity. The framework constructs two structured evidence banks containing images, detected objects, scene graphs, descriptions, and rule statements. At inference time, multi-route retrieval and learned object-relation alignment identify relevant precedents from both banks. A calibrated retrieval margin and Structured Evidence-Grounded Reasoning Chain-of-Thought (SEGR-CoT) support rule-based anomaly judgments, while reliability-gated fusion controls whether retrieved evidence can modify the baseline VLM prediction. The authors evaluate the framework on UCF-Crime, XD-Violence, and ECVA, and introduce SMAD, a 24-task benchmark containing 12,500 public-safety images. The reported results show improvements in detection, reasoning, and contextual anomaly recognition. Component ablations, retrieval diagnostics, human audits, significance testing, and runtime analyses provide additional empirical evidence.

Critical strengths
  1. The paper addresses an important limitation of global similarity-based retrieval: visually similar scenes may have different anomaly labels because of differences in object relationships, cardinalities, or context-dependent rules. Retrieving both normal and abnormal relation-level precedents is a meaningful way to make these distinctions explicit.
  2. The combination of local relation motifs, partial object-relation alignment, and bilateral evidence comparison provides a coherent mechanism for supporting more interpretable anomaly judgments. The qualitative multi-rod fishing example illustrates why relation compatibility can be more informative than global scene similarity.
  3. The evaluation covers multiple public benchmarks and a newly introduced dataset with context-sensitive anomaly rules. Importantly, the supplementary material goes beyond standard aggregate performance comparisons: it includes component-isolation experiments, paired bootstrap confidence intervals, independent human audits, cross-backbone evaluations, object-source sensitivity tests, and system-level runtime measurements.
  4. The component analysis indicates that relation-centric retrieval contributes substantial improvements over vanilla global retrieval. The reported bootstrap results also support the smaller incremental gains from reliability-gated fusion. This strengthens the empirical case that the proposed components provide complementary benefits rather than merely increasing prompt complexity.
Critical weaknesses
  1. Although the overall framework is well motivated, several components build on established techniques, including graph attention, partial optimal transport, contrastive retrieval, bilateral memory comparison, structured prompting, and confidence-based fusion. Consequently, the main contribution appears to lie primarily in their integration for context-dependent anomaly reasoning.
  2. The existing ablations establish the importance of relation-aware retrieval compared with global retrieval. However, they do not fully establish whether the particular combination of neighborhood-aware relation motifs and partial Sinkhorn alignment is essential. Simpler structured alternatives, such as relation-triple embedding retrieval, direct graph-level similarity, or object-aware matching without optimal transport, are not sufficiently isolated.
  3. The framework relies on a frozen VLM to generate scene graphs, assign rule statements, and determine whether training frames contain sufficiently visible violations for evidence-bank admission. This introduces a dependency on the quality and consistency of VLM-generated semantic annotations. Although the paper includes bank-purity measurements, human audits, and detector sensitivity experiments, these results do not fully characterize how errors or biases in evidence construction propagate to downstream decisions.
  4. A related concern is that SMAD uses source-group and perceptual-hash separation, but camera-disjoint and location-disjoint generalization cannot be established from the available metadata. The main SMAD results also use annotation-assisted object localization, although the supplementary detector-only evaluation demonstrates that the framework retains much of its performance without manual boxes.
Questions for authors
  1. Could the authors compare the proposed partial Sinkhorn-based object-relation alignment with simpler structured-retrieval alternatives under the same evidence banks and VLM backbone, such as cosine similarity over relation-triple embeddings or graph-level embeddings without explicit partial transport?
  2. How sensitive is AnomalyRAG to incorrect VLM-generated relations, rule statements, or evidence-bank assignments? In particular, can the authors quantify the downstream effect of relation corruption or incorrect rule attribution on retrieval quality, verification acceptance, and final anomaly performance?
  3. Can the authors provide a more targeted evaluation of performance on unfamiliar scene contexts or rule configurations, preferably with source- or scene-disjoint partitions where feasible?
  4. Could the authors provide a more detailed breakdown of the inference cost of each major component and investigate the performance-efficiency trade-offs under different computational budgets?
5 comments
AuthorsOpenReview ↗

Response to Reviewer p99g (1/4): Contribution and structured retrieval controls

Thank you for recognizing the motivation, interpretable mechanism, and breadth of the evaluation. We address the contribution and simpler retrieval alternatives together, then examine evidence errors, unfamiliar scenes and rules, and inference cost.

C1: Contribution of the framework and alignment design

Reply:

  1. Anomaly labels depend on object states, interactions, subject-specific object counts, and scene rules. A useful precedent must preserve the configuration that determines the label. For example, two scenes may each contain two people and two rods. In one, each person uses one rod; in the other, one person uses both. Only the second violates a one-rod-per-person rule. This motivates retrieval that preserves which objects participate in each local relation.

  2. Our contribution is relation-aligned bilateral precedent comparison: retrieving comparable normal and abnormal instances, contrasting rule satisfaction and violation, and attributing the judgment to the responsible objects. Neighborhood-aware motifs preserve each subject's object composition and count. Object correspondences constrain the endpoints of relation matches, while partial alignment lets irrelevant or missing objects remain unmatched. These choices specify what the graph and transport operations must preserve for contextual anomaly reasoning.

  3. The learning objective connects this alignment to anomaly criteria through compatible positives, opposite-label hard negatives, dual-bank discrimination, retrieval ranking, and rule compatibility. SEGR-CoT then uses the aligned evidence to identify decisive matches and mismatches. A calibrated signed margin summarizes bilateral support, and verification plus reliability-gated fusion controls the resulting score correction. The same object-relation correspondences support evidence selection, the decision, and its explanation; the VLM stays frozen.

  4. Main-paper Table 3 and Supplementary Tables 12-13 evaluate the components and their complementary gains. The controlled comparison below tests the additional value of the proposed matching design over simpler ways of using structured evidence.

Revision: Sections 1 and 3 now explain this contribution and the connection between local representation, constrained alignment, and bilateral reasoning.

C2 / Q1: Partial Sinkhorn alignment versus simpler structured alternatives

Reply:

  1. We added the alternatives you requested: cosine retrieval over relation-triple text embeddings, cosine retrieval over pooled graph embeddings from the same encoder, and bidirectional object nearest-neighbor matching without transport. The first captures relation semantics, the second pools scene structure, and the third provides object matching. AnomalyRAG preserves local neighborhoods and coordinates relation matches through soft object correspondences.

  2. The structured variants share the banks, frozen VLM, candidate set, perception inputs, reasoning prompt, gate, and calibration procedure. Their trainable components use the same training data, objectives, and budget. The global visual baseline selects its own candidates; records added for downstream reasoning do not change retrieval rankings.

Table A. Controlled retrieval comparison. Values are percentages. SMAD uses automatic boxes. UCF match@1 measures human-verified matching of the decisive relation, following Supplementary Table 6.

RetrievalUCF AUCSMAD accuracyUCF match@1
Dual-bank global visual84.1287.3676.12
Relation-triple text86.3588.3878.76
Graph embedding, same encoder85.1989.1279.13
Bidirectional object nearest neighbor86.9889.5480.39
Full AnomalyRAG89.9690.6284.2
  1. The full method improves all three metrics over every tested alternative. Against the strongest simpler method, object nearest-neighbor matching, the gains are 2.98 points in UCF AUC, 1.08 points in SMAD accuracy, and 3.81 points in relation matching. Thus, the gain persists when the alternatives already use structured evidence. The improvement in relation matching also connects the detection gain to better selection of the decisive configuration.

Revision: Supplementary Section 7.1 and Table 23 contain the controls. Main-paper Section 4.3 summarizes their results.

AuthorsOpenReview ↗

Response to Reviewer p99g (2/4): Errors in evidence construction

C3 / Q2: Sensitivity to incorrect relations, rules, and bank assignments

Reply:

  1. These errors enter the pipeline in different ways. Incorrect relations can change which precedents are retrieved. Incorrect rule statements can change the rule used for comparison and explanation. Incorrect bank assignments can reverse whether a precedent supports normality or abnormality. We therefore test the three error types separately, so their effects on the final decision can be distinguished.

  2. Supplementary Tables 6 and 20 already audit evidence quality and explanation faithfulness. We added the controlled corruption experiment in Table C to measure the effect of semantic errors after evidence construction. Each condition corrupts 20% of all bank entries: changing object relations or subject/object roles, inserting rules inconsistent with the source image, or swapping equal numbers of normal and abnormal entries between banks.

  3. Identifiers and relation endpoints remain valid, and affected retrieval indices are rebuilt. Model parameters, the verifier, and the default gate remain fixed. The experiment therefore changes the semantic content of the evidence without introducing broken records or retraining the model to accommodate the corruption.

Table C. Effects of evidence-construction errors. All values are percentages. Verification pass rate is measured on queries receiving corrupted evidence; UCF AUC is measured on the complete test set.

Evidence-bank conditionVerification pass rateUCF AUC
Original bankN/A89.96
Incorrect relations or roles95.388.12
Incorrect rule statements95.988.91
Incorrect normal/abnormal assignments94.887.65
  1. Under 20% corruption, the AUC decreases are 1.84, 1.05, and 2.31 percentage points, respectively. Detection retains most of its original performance across all three conditions. Bank-assignment errors produce the largest reduction, consistent with their reversal of which side of the bilateral comparison supports a precedent.

  2. The verification column measures whether the response has the required format, valid object and relation references, and consistency with its cited evidence. Final AUC measures the effect on anomaly detection. Reporting both separates the behavior of the consistency checks from the downstream consequences of errors in the supplied evidence.

Revision: Supplementary Section 7.3 and Table 26 now describe the corruption procedure, verification measurements, and detection results. The existing human audits remain in Supplementary Tables 6 and 20.

AuthorsOpenReview ↗

Response to Reviewer p99g (3/4): Unfamiliar scenes and applicable rules

C4 / Q3: Unfamiliar scenes, automatic boxes, and rule configurations

Reply:

  1. SMAD contains images from multiple cameras and locations, with varied backgrounds and viewpoints. Its standard evaluation pools these images, testing performance across a mixture of scenes. To examine transfer to unfamiliar contexts directly, we added an experiment in which selected scenes are absent from training and both evidence banks.

  2. Automatic object localization is evaluated in Supplementary Table 18 using a shared, fixed detector vocabulary. It retains performance close to the annotation-assisted setting. The new scene experiment also uses automatic detections for every method, so its results cover both unfamiliar backgrounds and detector-provided object inputs.

  3. We reserve scenes with verifiable provenance for testing and remove all their images from training, validation, and both banks. We then retrain the retriever and rebuild the banks using the remaining scenes. The base VLM, dual-bank global retrieval, and AnomalyRAG use the same test images and automatic detections.

Table D. Scenes excluded from training and the evidence banks.

MethodAccuracy (%)F1 (%)
Base VLM83.2383.12
Dual-bank global retrieval84.9684.33
AnomalyRAG87.1187.98
  1. AnomalyRAG exceeds dual-bank global retrieval by 2.15 accuracy points and 3.65 F1 points on these unfamiliar scenes. Its advantage over the base VLM is 3.88 and 4.86 points, respectively. The gains support transferring relation-based evidence comparison to the tested scenes beyond those represented in training and the banks.

  2. We also tested how the model identifies and applies the governing rule. Rules are stored with training precedents; visible objects, interactions, and scene cues guide retrieval, and query-rule compatibility contributes to ranking. Table B1 uses 200 images with visible rule cues, balanced between normal and abnormal. Both methods use the same banks and VLM, without query task IDs or ground-truth rules. A selection is correct only when all supplied precedent rules apply; conflicts or missing reliable rules count as errors.

Table B1. Selecting rules from visible cues. Values are percentages; false positives are measured on normal images.

RetrievalCorrect rule selectionFalse positive rateAccuracy
Dual-bank global retrieval811487
AnomalyRAG92891
  1. Table B2 changes the rule while holding the visual input fixed. Each of 100 images is paired with two rules requiring opposite labels, giving 200 image-rule pairs. Both conditions use the same images and retrieved evidence; the second also supplies the current rule to the decision stage. For example, one-rod fishing is permitted by a one-rod-per-person policy but prohibited by a no-fishing policy.

Table B2. Identical images under different current rules.

Decision inputAccuracy (%)
Image and retrieved evidence50
Same image and evidence, plus current rule94
  1. Without the supplied rule, one prediction is reused for the two opposite labels, yielding 50% by construction. The measured 94% after supplying the rule shows that the decision can change appropriately with the governing policy. These tests examine complementary aspects of contextual judgment: transfer to unfamiliar scenes, rule selection from visible cues, and application of an externally specified rule.

Revision: Supplementary Section 7.4 and Table 27 report the unfamiliar-scene experiment. Sections 3.1-3.2 clarify rule selection, and Supplementary Section 7.2 and Tables 24-25 contain the rule experiments. The automatic-box comparison remains in Supplementary Table 18.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Appreciation for Reviewers' Positive Feedback and Manuscript Revisions

We thank the reviewers for their careful evaluations and constructive suggestions. We appreciate their recognition of AnomalyRAG's motivation, relation-level evidence design, and experimental evaluation. We summarize their positive feedback below, followed by the revisions made in response to their concerns.

1. Motivation and Context-Dependent Anomaly Judgments

  • LgFV (S1): "The paper addresses a clear problem: anomaly judgments depend on object relations and scene rules."
  • p99g (S1): Recognizes that visually similar scenes can have different anomaly labels because of object relationships, cardinalities, and scene rules, and considers normal and abnormal relation-level precedents a meaningful way to distinguish them.

2. Relation-Level Evidence and Interpretable Reasoning

  • p99g (S2): Describes local relation motifs, partial object-relation alignment, and bilateral evidence comparison as "a coherent mechanism for supporting more interpretable anomaly judgments." The reviewer identifies the multi-rod fishing example as an illustration of why relation compatibility matters beyond global visual similarity.
  • Q5eY (S1): "The method designed is reasonable and easy to follow." The reviewer recognizes the effectiveness of the fine-grained relation modeling.

3. Experimental Coverage and Validation

  • LgFV (S2): Recognizes the evaluation across several benchmarks, supported by detailed ablations and human checks.
  • p99g (S3): Appreciates the public-benchmark and SMAD evaluations, component-isolation experiments, bootstrap confidence intervals, independent human audits, cross-backbone comparisons, object-source sensitivity tests, and runtime measurements.
  • Q5eY (S1): Recognizes the strong detection results on UCF-Crime and XD-Violence.

4. Complementary Component Gains

  • p99g (S4): Recognizes the substantial gains from relation-centric retrieval and the statistical support for the additional gains from reliability-gated fusion, concluding that the "proposed components provide complementary benefits."

Responses to Reviewer Comments and Manuscript Revisions

We have revised the manuscript and supplementary material to address the reviewers' concerns. The revisions are marked in blue.

  1. Motivation and contribution. Sections 1-3 now explain relation-aligned bilateral precedent comparison: retrieving comparable normal and abnormal instances, contrasting rule satisfaction and violation, and connecting the decision to specific objects and rules. We clarify the differences from SlowFastVAD and VADER and add the discussion of Vad-R1.

  2. Controlled retrieval comparisons. New Supplementary Table 23 compares dual-bank global visual retrieval, relation-triple retrieval, graph-level retrieval, object nearest-neighbor matching, and AnomalyRAG with the VLM, prompt, and gate fixed. Section 4.3 summarizes the evidence for the proposed alignment design.

  3. Rules, evidence errors, and unfamiliar scenes. Supplementary Tables 24-25 test rule selection without task IDs and identical images under different supplied rules. Table 26 tests incorrect relations, rule statements, and bank assignments. Table 27 evaluates scenes excluded from training and both evidence banks, using automatic detections.

  4. Backbones, efficiency, and sensitivity. We extend Supplementary Table 14 with InternVL3-8B and Qwen3.5-9B. New Tables 28-29 examine reasoning output budgets, total latency, reliability weights, confidence thresholds, and fusion strength. Section 4.4 summarizes the performance-efficiency tradeoff.

  5. Broader reasoning evaluation. New Section 4.5 and main-paper Table 5 report Vad-Reasoning results alongside Vad-R1, covering classification, temporal localization, and text metrics for reasoning and answers. Supplementary Section 7.7 specifies the evaluation protocol.

  6. Training details. Section 4.1 now states the bank-construction, retriever-training, and calibration sequence, together with the principal optimization settings. Section 3 and the supplement explain rule sources, sampling, objectives, and fixed versus learned parameters.

The responses under each review provide the relevant explanations and experimental tables. We thank the reviewers again for helping us improve the paper and hope the revisions and additional evidence support a favorable reassessment.

Area_Chair_ZXm2OpenReview ↗

A Gentle Reminder: About One Day Left in the Discussion Period

Dear Reviewers,

This is a gentle reminder that approximately one day remains in the discussion period. Please review the authors' responses and other reviewers' comments, and engage in the discussion to clarify which concerns have been addressed and which remain unresolved.

Thank you for your time and effort.

Best,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.