acceptodds
Under review as a conference paper at ICLR 2027

MemVAU: Agentic Visual Evidence Memory for Video Anomaly Understanding

Abstract

Video Anomaly Understanding (VAU) explains abnormal events along four dimensions: What, Who, Where, and How. Existing single-pass vision-language models (VLMs) do not explicitly check visual support for each dimension, leaving difficult fields such as Who and How insufficiently supported by localized visual evidence. We introduce MemVAU, an agentic visual evidence memory framework that recasts VAU as a closed loop. An Observer generates candidate evidence, a frozen Verifier scores it along the four dimensions, and a rule-based Mem-Optimizer retains complementary evidence in a bounded pool. If any dimension remains below its operational threshold and the round budget is not exhausted, the structured pool snapshot guides the next evidence-acquisition round. The Reporter synthesizes the final report when all dimensions meet their thresholds or the budget is exhausted. The Observer is trained with GRPO under a listwise reward for anomaly-type accuracy and balanced coverage. On HIVAU-70k, MemVAU achieves the best score on 10 of 12 metrics, with its largest gains at the Event and Video granularities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Video Anomaly Understanding, Agentic Visual Evidence Memory, Agentic Closed-Loop Framework, GRPO with Listwise Reward

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Y65yOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an agentic framework for VAU that aims to address insufficient visual evidence in existing methods. The framework consists of an Observer that generates temporally localized evidence, a frozen Verifier that evaluates evidence support across four dimensions, and a Mem-Optimizer that maintains a bounded evidence pool. Experiments on HIVAU-70k demonstrate improvements in VAU performance.

Critical strengths
  1. This paper introduces a structured evidence memory mechanism that explicitly evaluates and maintains visual evidence across four dimensions

  2. The proposed framework achieves competitive performance on HIVAU-70k, with improvements across multiple evaluation metrics.

Critical weaknesses
  1. The experiments do not directly compare evidence sufficiency between baseline methods and MemVAU. Therefore, it remains unclear whether the performance improvements are primarily attributed to acquiring sufficient visual evidence. Besides, the proposed framework employs Qwen3-VL as its visual backbone, while the comparisons mainly include older models such as Qwen2-VL.
  2. The framework relies on a frozen Verifier to evaluate evidence sufficiency. The Observer is trainable while the Verifier keeps frozen. And the Verifier does not engage in the optimization of Observer. It remains unclear whether the two components are sufficiently aligned.
  3. The proposed Mem-Optimizer maintains a bounded evidence pool. However, there are no comparisons with simpler strategy like unbounded evidence accumulation or Top-K selection, which would better demonstrate the necessity of the proposed Mem-Optimizer.
  4. The Observer is optimized with GRPO, but Table 2(b) shows relatively limited improvements over the SFT Observer
Questions for authors

Could the authors clarify the implementation details of the baselines in Table 2? Table 2 is one of the most important ablation studies supporting the effectiveness of the proposed framework. However, the implementation details of the baseline methods are insufficiently described.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Y65y

We thank the reviewer for the comments. The existing experiments examine major performance factors through visual-agent scale controls, fixed-Observer comparisons, and training ablations.

1. Backbone differences and evidence-processing gains

O2/V2/R8 reduces both the Observer and Verifier to 2B and still exceeds Holmes-VAU on five metrics. For example, BLEU-V is 0.661 versus 0.566, and METEOR-E is 0.375 versus 0.165. Thus, some gains persist with smaller visual agents.

Table 2 provides a more direct internal comparison. Raw evidence and MemVAU-SFT use the same SFT Observer, and adding verification and structured memory improves all eight Event/Video metrics. Holding the candidate-generation model fixed supports the contribution of the joint evidence-processing pipeline.

The additional diagnostics measure Verifier cross-prompt consistency and Reporter evidence utilization, each interpreted according to its stated definition.

2. Coordination between the Observer and frozen Verifier

During training, the 4B Judge ranks candidate sets by anomaly-type accuracy, four-dimensional coverage, and diversity. During inference, the 8B Verifier scores the corresponding video segments. Its outputs inform pool updates and the next-round snapshot, allowing the Observer to condition subsequent candidates on retained evidence and unmet dimensions.

Freezing the Verifier keeps this feedback reference fixed. Table 2 compares SFT and GRPO Observers while keeping the Verifier, Mem-Optimizer, and Reporter fixed. GRPO improves all four Video metrics, providing task-level evidence of effective coordination through this interface.

3. Rationale for the Mem-Optimizer

An entry’s value depends on the current pool’s coverage. Equation (13) measures its contribution through the gap increase, surplus loss, and secondary-support loss caused by removing it. An entry that fills a How gap can therefore receive explicit pool-level value. The state is recomputed after each replacement, and S3 preserves all threshold passes.

The capacity analysis also shows a concrete trade-off. In Table 6, increasing K from five to seven lowers the full-four-dimensional pass rate from 91.60% to 88.62%, without consistent improvements in text metrics. K=3 achieves the highest text metrics in this sweep, while K=5 achieves the highest operational coverage. These results support controlling and tuning memory capacity.

The capacity sweep does not directly compare Top-K and unrestricted accumulation. The present rationale for the Mem-Optimizer rests on its explicit pool-level objective, update properties, and full-pipeline results.

4. Contribution of GRPO

Table 8 shows improvements on nine of twelve metrics, slight decreases on two, and one unchanged result. All four Video metrics improve. GRPO therefore provides an incremental improvement over the existing pipeline.

The combined pipeline gain is already present with SFT. MemVAU-SFT achieves CIDEr-E/V of 1.628/1.760, compared with 0.384/0.419 for Raw evidence. The two comparisons separately characterize joint evidence processing and Observer-training optimization.

5. Implementation and control design in Table 2

Table 2 contains three types of controls. For direct inference, Single-pass and CoT use one call, Self-correction uses two, and Multi-round uses three, with shared preprocessing. The evidence-processing comparison holds the SFT Observer fixed. The training comparison fixes the Mem-Optimizer and Reporter and changes only Observer training. Tables 7–8 provide complete results at all three granularities.

Thanks for your response. Howerver, some concerns remain. First, the main baseline results are largely drawn from the Holmes-VAU paper, and comparisons with more recent models would strengthen the evaluation. Second, the effectiveness and necessity of the Mem-Optimizer unclear, particularly without comparisons against simpler evidence accumulation strategies. Overall, my main concerns have not been fully addressed. I will maintain my rating.

Reviewer_2CQwOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses the limitation of single-pass vision-language models in Video Anomaly Understanding (VAU), which map a video directly to a four-dimensional report without explicitly measuring the visual support for each dimension. The authors formalize this problem as evidence insufficiency and propose an agentic framework, MemVAU. MemVAU iterates in a closed loop in which an Observer proposes evidence, a Verifier assigns scores, and a Mem-Optimizer maintains a bounded evidence pool until all dimensions meet their thresholds, after which a Reporter generates the final report. Experiments on the benchmark dataset show strong results across multiple metrics.

Critical strengths
  1. The problem addressed in this paper is clearly described. The authors analyze the deficiency of existing single-pass VAU methods and summarize it as evidence insufficiency, and further design a measurable four-dimensional evidence state. The design is clear and well-reasoned, and constitutes the main conceptual contribution of this work.
  2. The experiments are well conducted. The paper goes well beyond typical practice with a score-blinded audit of cross-prompt Verifier alignment, a report-level Reporter evidence-utilization analysis, end-to-end N/K/T sensitivity screening, and fixed-input offline reward and threshold sensitivity analyses.
  3. The method is highly maintainable and interpretable. The evidence pool scheduler requires no training and runs on fixed rules. The reason why each piece of evidence is retained or dismissed is recorded, so every decision can be traced and explained afterwards.
Critical weaknesses
  1. The central claim, better-supported reports, is never evaluated directly. The paper’s thesis is that MemVAU reduces evidence insufficiency. Yet every main-paper result uses n-gram overlap (BLEU, CIDEr, METEOR, ROUGE) against reference text, which measures stylistic and lexical agreement, not factual or visual support. The only grounding-related evidence is deficient by the audit being circular, as itt covers 150 statements from 30 videos, judged by the same Qwen3-VL-8B weights as the Verifier with a different prompt. That measures self-consistency, as the authors themselves acknowledge.
  2. The contribution analysis of the core mechanism is not sufficiently clear. The key comparison in Table 2 (raw evidence 0.471/0.301 BLEU-E/V vs. MemVAU-SFT 0.999/0.715) jointly introduces Verifier scoring and pool management, so the improvement cannot be attributed to the core evidence-memory mechanism itself. Meanwhile, GRPO adds only a marginal gain over the SFT Observer and slightly hurts Event-level BLEU/CIDEr. I cannot tell which ingredient drives the improvement, which weakens the central claim.
  3. The Verifier reliability evidence is weak. The audit uses the same model with a different prompt, so it only shows the scores are stable across prompts, not that they are correct, and indeed 15.7% of accepted statements are not fully supported with MAE 0.235 on the What dimension. Since the Observer and Verifier are from the same model family, their errors are likely correlated, yet the whole stopping logic depends on these uncalibrated scores.
  4. The pool update uses greedy step-by-step swaps that guarantee only local improvement without any global optimality, and can easily make suboptimal retention decisions when evidence conflicts or the scores are noisy.
Questions for authors
  1. Can you use human evaluation or an independent-model judge or a claim-level factuality measure, such as FActScore-style atomic precision against human annotation to provide direct claim evaluation.
  2. Can you isolate the contribution of the Mem-Optimizer itself, for example with pool updates using random or shuffled Verifier scores, or verification without the competitive pool update? This would address the main attribution gap in Table 2.
  3. How sensitive are the final results to the hand-picked thresholds? The paper only tests jointly shifting all thresholds by ±0.1 with evidence and scores held fixed, so it remains unclear whether re-tuning the thresholds or setting them per-video would materially change end-to-end performance.
  4. MemVAU runs in multiple rounds and therefore costs more inference calls than single-pass models. How does its total inference cost, in calls and processed frames, compare with the strongest baselines at matched report quality, and is the extra cost justified by the gains?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 2CQw

We thank the reviewer for the comments. Below, we clarify the interpretation of each evaluation and address attribution, pool updates, and computational cost using the existing results.

1. Visual-support evaluation and Verifier reliability

The paper evaluates task performance, scoring behavior, and evidence utilization. The main table measures agreement with reference reports, the Verifier audit measures cross-prompt consistency, and the Reporter analysis checks whether reports use pool-supported information.

The audit covers 150 statements from high-, medium-, and low-scoring video strata, with production scores and decisions hidden. Among 83 threshold-accepted statements, 70 receive full four-dimensional audit support, corresponding to 84.3%. Three contain at least one unsupported dimension, while ten involve partial support or uncertainty. This breakdown specifies the nature of the residual disagreements. Because the audit shares model weights with the Verifier, its interpretation is limited to cross-prompt consistency.

Reporter utilization is 96%/92% at Event/Video level, using the criterion that a report contains at least one pool-supported statement. These analyses address scoring consistency and evidence use, respectively. The task-performance conclusions rest on the main results and controlled comparisons.

2. Contributions of joint evidence processing and GRPO

In Table 2, adding verification and structured memory while keeping the SFT Observer fixed improves all eight Event/Video metrics. CIDEr increases from 0.384/0.419 to 1.628/1.760, and ROUGE from 0.305/0.299 to 0.470/0.434. Thus, substantial gains are already present with the SFT Observer.

With the Mem-Optimizer and Reporter fixed, GRPO improves nine of twelve metrics, slightly decreases two, and leaves one unchanged. All four Video metrics improve, including BLEU from 0.715 to 0.748 and CIDEr from 1.760 to 1.795. These comparisons support the gain from joint evidence processing and the additional improvement from GRPO. The joint comparison does not isolate the individual contributions of verification and pool management.

3. Guarantees of the greedy update

The Mem-Optimizer evaluates updates through a pool-level objective that prioritizes reducing the gap, increasing surplus, and improving secondary support. S2 searches swaps between current candidates and retained entries, accepts only strict improvements, and re-evaluates after each update. S3 excludes swaps that violate any threshold.

The algorithm therefore provides monotonic improvement under the supplied scores and threshold preservation in S3. Its guarantees are limited to these checkable local properties. Semantic conflict resolution is outside the current objective, and secondary support represents additional score support.

4. Threshold sensitivity

Table 10 jointly shifts all four thresholds by ±0.1, yielding acceptance counts of 84, 83, and 82. The accepted-set Jaccard similarity with the baseline is approximately 0.988. This quantifies local decision stability under fixed evidence.

The paper also reports end-to-end sweeps over N, K, and T. These analyses examine threshold decisions and overall execution settings, respectively. The threshold conclusions apply specifically to fixed evidence and scores.

5. Inference cost

In Table 4, increasing the maximum number of rounds from one to three raises average calls from 21.32 to 26.58 and processed frames from 326.30 to 421.14. T=3 achieves the highest CIDEr, METEOR, and ROUGE in this sweep, while T=2 achieves the highest BLEU.

These results quantify the quality–cost trade-off within the pipeline. The controls in Table 2 do not match its total inference budget, so the current results make no claim of cross-method efficiency superiority at matched report quality.

Thank you for your response. The clarification in points 2-5 is appreciated, but the response in point 1 does not fully address my concerns about the evaluation. In response to the response being partially satisfactory, I am prepared to improve my rating by 1.

Reviewer_x7VWOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MemVAU, a framework for video anomaly understanding that iteratively collects and manages evidence for four semantic fields: What, Who, Where, and How. A GRPO-trained Observer generates candidate descriptions with temporal intervals, and a frozen visual Verifier assigns each candidate four field-specific support scores. A rule-based memory module maintains a bounded evidence pool, whose scores determine whether to request another observation round or stop. A text-only Reporter then generates the final report from the retained evidence. Experiments on a 2,335-sample subset of HIVAU-70k compare MemVAU with general-purpose and anomaly-specific models, alongside component and sensitivity analyses.

Critical strengths

The paper highlights a meaningful gap between generating a complete-looking anomaly report and having sufficient visual evidence to support its claims. Recognizing that gap and motivating further evidence acquisition is a valuable aspect of the work.

Critical weaknesses
  1. The core problem formulation and its connection to the proposed method are insufficiently justified. Lines 35–40 argue that existing methods generate reports without an explicit state of visual support for each field, making weaknesses in actor descriptions and temporal relations difficult to repair. However, the paper does not adequately explain why single-pass generation leads to these difficulties or why an explicit four-dimensional state would address them. What, Who, Where, and How are introduced through intuitive glosses without clear task boundaries or scoring criteria, particularly for the overlap between What and How in event descriptions and action sequences. In comparison, CUVA [1] specifies the tasks and annotations corresponding to its What, Why, and How questions. More fundamentally, the motivating issue concerns a gap between the acquisition of visual evidence and the evidence needed for anomaly understanding. Vad-R1 [2] already explores the connection between perception and cognition through structured reasoning; the paper does not sufficiently establish the specific remaining limitation or why the proposed four-dimensional state is needed to address it. Although the authors do not equate threshold passing with factual completeness, they still need to justify why these four support scores and their threshold gaps are an effective proxy for the stated evidence insufficiency.
  2. The experimental evidence is insufficient to establish the applicability, necessity, and intended effects of the proposed design. (a) Limited evaluation scope. All main results and ablations use a fixed 2,335-sample subset of HIVAU-70k. Although this benchmark evaluates relevant anomaly understanding abilities, results on one benchmark do not sufficiently support broad conclusions about effectiveness and applicability. Evaluation on other anomaly understanding datasets is needed to determine whether the gains depend on this particular annotation scheme, task format, or data distribution. (b) Incomplete ablation of key operations. The paper includes comparisons of the overall pipeline and Observer training, but does not sufficiently isolate the contributions of its key operations. Moving from Raw evidence to the full memory pipeline jointly introduces verification and pool management. Separate controls removing the Verifier, field-specific feedback, competitive replacement, and secondary support are missing, leaving the independent contribution and necessity of these choices unclear. (c) Insufficient validation that the intended problems are actually resolved. Improvements in final report metrics establish an overall performance benefit, but do not show whether individual modules alleviate the specific problems used to motivate them. For example, if feedback for How is intended to repair missing action sequences or temporal evidence, a qualitative comparison with and without that mechanism should show what was originally missed, what evidence was subsequently acquired, and how the gap was reduced. Depending on the problem, such validation may be qualitative or quantitative. The existing analyses do not sufficiently connect the motivating failure, the module’s behavior, and the resulting improvement. References [1] Hang Du et al. Uncovering What, Why and How: A Comprehensive Benchmark for Causation Understanding of Video Anomaly. CVPR, 2024. https://arxiv.org/abs/2405.00181 [2] Chao Huang et al. Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thought. NeurIPS, 2025. https://arxiv.org/abs/2505.19877
Questions for authors
  1. Why were What, Who, Where, and How selected to characterize evidence support? In particular, what are the boundaries and scoring criteria for What versus How? Can the authors provide analysis or evidence showing that the four support scores and their threshold gaps are an effective proxy for evidence insufficiency, beyond coverage of four categories of descriptive content?
  2. Why is it necessary to use a fine-tuned Observer and a separate frozen Verifier? Since both use models from the same visual-language model family, what evidence supports the reliability of this division of roles and addresses correlated visual errors? Are there controls using a shared model for both roles or removing the separate Verifier that establish the necessity of this design?
1 comment
AuthorsOpenReview ↗

Response to Reviewer x7VW

We thank the reviewer for the comments. Below, we clarify the role of the four-dimensional state and the conclusions supported by the existing comparisons.

1. Rationale for the four-dimensional state and the What/How distinction

The four dimensions correspond to the report structure of HIVAU-70k. What concerns the event category, while How concerns the action sequence and its temporal relations. For example, “theft” identifies the event, whereas “taking, concealing, and leaving with an item” describes its progression. Separate assessment exposes differences in support for event recognition and process explanation.

These scores directly inform decisions. Equation (2) requires the Verifier to score each candidate against its corresponding video segment. Equation (4) accumulates dimension-specific threshold gaps, preventing strong What support from compensating for weak How support. Equation (14) passes these gaps and retained evidence to the next Observer round. The state therefore provides an explicit interface for identifying gaps, acquiring additional evidence, and stopping inference.

2. Relationship to single-pass generation and related work

MemVAU addresses evidence retention, acquisition, and use during report generation. Vad-R1 introduces perception-to-cognition reasoning and self-verification. MemVAU’s specific contribution is to use a four-dimensional support state to control bounded memory updates, subsequent acquisition, and stopping.

Table 2 provides supporting comparisons. With the same SFT Observer, adding verification and structured memory improves all eight Event/Video metrics, including CIDEr from 0.384/0.419 to 1.628/1.760. The tested CoT, self-correction, and multi-round controls do not exceed Single-pass on these metrics. These results support the effectiveness of the proposed evidence-processing pipeline under the evaluated settings.

3. Roles of the Observer and frozen Verifier

The Observer generates descriptions and temporal intervals. The Verifier checks visual support against the corresponding segments. Freezing the Verifier keeps the inference-time scoring function fixed during Observer training.

The training and inference scoring roles are also distinct. GRPO uses a frozen Qwen3-VL-4B Judge to rank candidate sets, while inference uses a frozen Qwen3-VL-8B Verifier for localized four-dimensional scoring. The evidence pool and snapshot connect generation and verification. Section 4.4 quantifies agreement and residual disagreements through a score-blinded audit, interpreted specifically as same-model cross-prompt consistency.

4. Evaluation scope and component contributions

All methods are evaluated on the same fixed 2,335-sample subset. Full MemVAU achieves the highest values on nine of the twelve metrics in Table 1, including all eight Event/Video metrics. Reducing both visual agents to 2B still yields improvements over Holmes-VAU on five metrics.

Table 2 separately examines joint evidence processing and Observer training. The former holds the SFT Observer fixed, while the latter fixes the Mem-Optimizer and Reporter. These comparisons distinguish the combined pipeline gain from the incremental contribution of GRPO.

Equations (7)–(8) provide the rationale for the internal operations. Updates first reduce dimensional gaps, then improve surplus and secondary support. Since four-dimensional maxima can be preserved by at most four entries, secondary support gives additional entries explicit objective value. The existing experiments support the combined contribution of verification and memory management, while Appendix A.2 illustrates changes in descriptions before and after GRPO.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.