acceptodds
Withdrawn from ICLR 2027

ReMem-VAD: Connecting Event Representations and Memory for Anomaly Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video anomaly detection (VAD) is challenging because visually similar actions can correspond to either normal or abnormal events. Recent vision-language models (VLMs) provide broad semantic knowledge, but recognizing an action alone is insufficient when its meaning depends on the participant’s response, the event’s progression, and the evidence supporting interpretations. To address this issue, we propose to transform resuable experience to event representations and structural memory, enhancing VLMs' capabilities with memory-ground reasoning. Specifically, Anomaly Agent is proposed, a memory-augmented VAD framework is built to combine large-model understanding with lightweight anomaly scoring. For each video window, a frozen VLM represents the actions, responses, event phase, and supporting evidence. This event representation specifies what must be explained and guides the verification of competing explanations against the same frames. Under this guidance, candidate patterns are retrieved from memory and matched to the current visual evidence. A lightweight trainable scorer then learns how to combine evidence, explanation judgments, and memory pattern into an anomaly score. Counterexample-guided refinement updates memory and representation during development, and freezes them at test time. Experiments on XDViolence, UCF-Crime and UBnormal demonstrate state-of-the-art performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Video anomaly detection, Anomaly Agent, Memory-Augmented, Representation, Refinement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xLtmOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a memory-augmented video anomaly detection framework that leverages a frozen VLM to extract event evidence and validates explanations against normal and anomalous memory patterns, followed by a lightweight scorer to predict anomaly scores. The system conducts counterexample-guided updates to memories and representations during the development phase and remains frozen during testing.

Critical strengths
  1. The framework explicitly considers participant responses, event phases, and visual evidence, helping distinguish normal and abnormal events with similar-looking actions.

  2. Experiments across three benchmarks, supported by component ablations, provide evidence for the effectiveness of combining structured memory with lightweight anomaly scoring.

Critical weaknesses
  1. The writing and figures are poor and substantially hinder understanding.
  2. The supervision protocol and reported results are unclear.
  3. Some memory patterns may encourage shortcuts rather than evidence-based anomaly detection.
Questions for authors

1.How are window-level training labels and refinement acceptance metrics constructed from video-level supervision? Please clarify the data splits and safeguards against unreliable pseudo-labels.

2.Please resolve the UCF-Crime/XD-Violence score inconsistency in Tables 2–6 and identify which implementation produced the main results. What exactly changes in the Cases-off ablation if the archived implementation does not separately retrieve and display cases?

3.How does performance change after removing news branding, political captions, and protest patterns that lack direct evidence of harm? Can controlled tests show that changing these contextual cues while preserving the underlying actions does not cause inappropriate changes in anomaly scores?

Reviewer_9s3AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ReMem-VAD, a memory-augmented framework for context-dependent VAD. It combines structured event representations, hierarchical Resource/Case/Pattern Memory, and a lightweight anomaly scorer with frozen VLMs. The framework uses C1 for event observation, C2 for explanation verification, and memory-based pattern matching to distinguish normal and abnormal interpretations. Counterexample-guided refinement further updates the memory during development. Experiments on UCF-Crime, XD-Violence, and UBnormal demonstrate competitive detection performance.

Critical strengths
  • This paper explores the ambiguity of visually similar actions in different event contexts. By explicitly modeling the participants' responses, event stages, visual evidence, and time ranges, this paper proposes a reasonable solution for context-dependent anomaly understanding.
  • This paper provides a practical approach for reusing event knowledge with frozen VLM, and the proposed hierarchical memory mechanism is reasonable. The experimental results on three benchmark tests also prove its effectiveness.
Critical weaknesses
  • This experiment failed to clearly determine whether the observed performance improvement was due to structured memory or merely resulted from a large number of multi-round reasoning processes. Moreover, when the framework processes each video window, it approximately needs to call 36.9 times the VLM, taking 10.7 seconds. This cost burden severely limits its scalability in long video tasks.
  • UCF-Crime and XD-Violence rely on video-level supervision to initialize memory, while UBnormal uses frame-level labels to select anomaly anchors. The inconsistent supervision across different datasets undermines the fairness of the comparison.
  • The contribution of Case Memory is unclear, and the isolation degree between the various memory components is insufficient.
  • The final anomaly score depends on the confidence generated by the VLM, the quality assessment of the evidence, the C2 summary fragment, and the degree of support for pattern matching. The implementation details of these components, as well as the method of numerical calibration, require more clarification.
  • Figures 1 and 2 contain too many visual elements and small text, making it difficult for readers to understand the process of event reasoning and memory update.
Questions for authors
  • Under the condition of comparable budget calculation, does the performance of ReMem-VAD outperform simpler multi-round reasoning or self-reflection baseline methods?
  • If anomaly anchors are selected without frame-level labels, how will the performance on the UBnormal dataset change?
  • If case memory is removed while keeping the learned Pattern Memory unchanged, what impact will it have on performance?
  • How are the C1 confidence, evidence quality, and C2 summary ratio calibrated? How stable are these features in different prompts?
  • How much is the performance improvement brought by counterexample-guided refinement? How does this framework prevent the erroneous evidence or explanations generated by VLM from being stored in persistent memory?
  • In the Pattern Matching process, time or causal relationships are not explicitly modeled. How does this framework distinguish events with the same visual features but different temporal sequences or interaction patterns of participants?
Reviewer_YGGcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors address the problem of weakly supervised and open-set video anomaly detection, motivated by the observation that the same visible action can be normal or abnormal depending on the participants' responses and the available evidence. They propose ReMem-VAD, in which a frozen VLM encodes each 96-frame window into a four-part event representation, which in turn drives retrieval from a three-tier structured memory of resources, cases, and patterns. In their approach, a 12-dimensional trainable scorer fuses the strongest direct evidence, four summary fractions, abnormal–normal pattern-matching margin, and six availability flags into a window-level anomaly score.

Critical strengths
  1. The problem framing is convincing, targeting a real challenge: the ambiguity in VAD is not what the action is, but how it should be interpreted. Looking for the evidence needed for interpretation is a well-motivated approach.
  2. Internally, the three memory roles and the L0/L1/L2 hierarchy are coherent and well justified.
  3. Releasing the complete pattern catalog is a clear plus for reproducibility.
  4. Ablations D1–D4 indicate that each element contributes a few points, which speaks for the effectiveness of the design.
Critical weaknesses
  1. The second of the three claimed contributions, counterexample-guided refinement, is never ablated. It is therefore impossible to tell whether the gains come from refinement or from the handcrafted pattern catalog, which is essentially an injected domain knowledge.
  2. All results are single-run, with no seeds, variance, or significance testing, and several claimed margins are within noise. At minimum, Stage-2 training should be repeated over three source-disjoint splits with mean±std reported.
  3. The comparison on UBnormal is not completely fair. Section 3.1 states that on UBnormal, the abnormal windows are taken directly from frame-level labels, whereas most baselines in the table are unsupervised methods trained on normal videos only and required to generalise to unseen anomaly types. Claiming state-of-the-art by 0.1 point under this setting is not convincing. From my point of view, the supervision incongruence should be stated explicitly, or baselines should be ran under matched supervision.
  4. Memory construction is self-confirming and anchor quality is never evaluated. Anchors are selected by the same VLM's confidence g_loc, yet a video-level label does not guarantee that these particular windows are anomalous or normal — especially in long UCF-Crime videos. Due to the fact that no anchor precision is reported and there is no ground-truth-anchor vs. pseudo-label-anchor control, the alternative explanation that the memory amplifies the VLM's existing biases cannot be ruled out.
  5. All prompt templates and a per-video latency/cost comparison should be added to the appendix.
  6. Figure 2 of the paper is overly complicated, which has greatly increased the difficulty of reading.
Questions for authors
  1. In Table 2, which dataset and which metric do the figures 90.89 and 91.01 correspond to? Is AP > AUC a typo? Please make Table 2 consistent with Tables 3–5.
  2. What are the three benchmark numbers without counterexample-guided refinement? How many proposals did the teacher LLM make and what was the acceptance rate?
  3. Given that frame-level labels are used for anchor selection on UBnormal, is the comparison with the unsupervised baselines in the table fair?
  4. A3 and A4 use the same inputs, and differ by roughly 2 points. Does this gap stem from numeric regression being better than verbal judgement, or from explicit feature weighting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.