acceptodds
Withdrawn from ICLR 2027

From Detection to Translation: Rethinking Evidence Sampling for Video Anomaly Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video Anomaly Understanding (VAU) aims to identify abnormal events in videos and explain their visual evidence, event process, and semantic causes in natural language. Existing VAU pipelines commonly rely on uniform sampling or anomaly-score-based filtering before Multimodal Large Language Model (MLLM) reasoning, where subtle but critical abnormal frames may be removed before the reasoning stage. To address this challenge, we propose a translation-based sampling framework: instead of making coarse anomaly decisions during sampling, our method translates all video clip units into compact visual-semantic representations and performs text-driven redundancy merging to retain representative clips with their corresponding text priors. We further integrate scene context into motion-aware translation and accelerate evidence construction via dynamic token pruning to improve both accuracy and efficiency.Experiments on HIVAU-70k show that, compared withHolmes -VAU baseline, our method improves average BLEU and ROUGE performance by 13.4% and 20.4%. The codes will be public after acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Anomaly Understanding, Video Sampling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JsbwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the risk of discarding important evidence through anomaly-score-based sampling in Video Anomaly Understanding (VAU). The proposed translation-based framework aligns motion and scene representations from fixed-length clips with predefined textual prototypes, groups semantically similar clips, and selects representatives as visual evidence and textual priors for an MLLM. Dynamic Token Pruning reduces feature-extraction costs. Experiments on HIVAU-70k with Holmes-VAU and Qwen2.5-VL report improvements in description generation across multiple temporal granularities.

Critical strengths

S1. Well-motivated problem formulation.

The paper identifies a practical limitation of anomaly-score-based sampling: false negatives during evidence selection can compromise subsequent MLLM reasoning. This is particularly relevant to VAU, which requires contextual and semantic evidence beyond frames with high anomaly scores. The motivation clearly distinguishes anomaly detection from event understanding.

S2. Promising use of semantic diversity.

Selecting evidence based on semantic diversity offers a promising alternative to relying solely on anomaly scores. Different actions, scene contexts, and event transitions may provide complementary information for understanding anomalous events. However, the extent to which this strategy preserves relevant evidence requires more direct validation.

Critical weaknesses

W1. Critical evidence preservation lacks direct quantitative validation.

Although Figure 3 provides qualitative examples, the text-generation improvements in Table 1 do not establish whether the proposed sampler retains more critical evidence. Comparing evidence recall or temporal coverage against existing strategies under identical frame budgets would more directly validate the central claim.

W2. The incremental benefit of the Text Prior remains unclear.

Table 3 supports the Scene Branch’s contribution, particularly to Event- and Video-level BLEU. However, adding the Text Prior yields relatively small BLEU gains of 0.003, 0.009, and 0.012, while Video-level METEOR slightly decreases from 0.199 to 0.198. Further analysis would help establish the robustness and practical significance of the Text Prior’s contribution.

Questions for authors

Q1. Direct validation of evidence preservation.

Could the authors compare evidence recall or temporal coverage using ground-truth anomaly intervals under the same frame budget? This would help establish whether the method retains critical evidence more effectively than existing sampling strategies, beyond improving generated descriptions.

Q2. Evidence selection under category supervision.

Although semantic grouping differs from ranking clips by a single anomaly score, the translator still relies on anomaly category supervision and binary evidence labels. How does the method mitigate evidence loss when relevant clips receive low semantic correspondence scores? Any evaluation on unseen anomaly categories would further clarify the generalizability of this selection strategy.

Reviewer_jpduOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a translation-based sampling framework for VAU with a dedicated translator. The method uses visual-text semantic matching and category-aware redundancy merging to select representative video clips. It further introduces gated motion-scene feature fusion and dynamic token pruning to improve semantic representation and sampling efficiency. Experiments on HIVAU-70k demonstrate improvements in anomaly understanding.

Critical strengths
  1. The proposed dynamic token pruning reduces computational overhead. This design improves sampling efficiency while maintaining competitive anomaly understanding performance

  2. The proposed method achieves consistent improvements across clip-level, event-level, and video-level anomaly understanding on HIVAU-70k

Critical weaknesses
  1. The claimed “translation” primarily relies on visual-text similarity to perform semantic matching and selection. However, most designs focus on the visual side, while the text representations are obtained from predefined prototypes. The technical novelty of the proposed translation-based sampling framework appears limited.

  2. The quality and coverage of the text prototype bank may directly affect the performance of anomaly understanding. Although Appendix E introduces a hierarchical prototype structure, the paper provides limited details about the construction of these prototypes and does not investigate how different prototype designs influence performance.

  3. The proposed redundancy merging retains only the most representative clip from each group. However, clips belonging to the same category are not necessarily redundant, as they may represent different event instances or complementary temporal stages. For example, different stages of a fighting event may be assigned to the same category but provide distinct information about the event progression, which are important for VAU.

  4. In Table 4, performance consistently improves as the number of retained semantic groups increases from 1 to 4. However, the authors do not evaluate larger values of.

Questions for authors
  1. Is it necessary to train a translator? The evidence selection is mainly based on the feature similarity, why not use a frozen CLIP?
Reviewer_EFaEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a translation-based video sampling framework that combines motion and scene features, matches clips to text prototypes, and selects representative clips within semantic groups to provide compact visual and textual evidence to an MLLM. On HIVAU-70k, Holmes-VAU's video-level BLEU improves from 0.301 to 0.413 and ROUGE from 0.117 to 0.367. Qwen2.5-VL-3B also benefits. The paper includes ablations of the components, number of retained groups, training objectives, and pruning strategies.

Critical strengths
  1. The paper addresses a concrete problem and improves results on two backbones. In Table 1 on page 6, Holmes-VAU's clip/event/video-level BLEU improves from 0.439/0.311/0.301 to 0.527/0.513/0.413. Qwen2.5-VL also improves at all three levels, providing some evidence of transferability.

  2. The ablations cover the main components. Table 3 on page 7 includes a control without the text prior. Tables 4–6 on page 9 examine the number of retained groups, pruning, and training objectives. These experiments support the combined design.

Critical weaknesses
  1. The central semantic grouping procedure is underspecified.

    Equation (5) on page 4 defines how to select the highest-scoring correspondence within an existing group, but not how the groups are formed, whether a clip can belong to multiple groups, or which groups are retained when there are more than four semantic types. The proposed “translation” uses predefined text-prototype matching, while training uses anomaly-category CE and BCE to distinguish informative evidence from normal or weak content. The text and supervision priors therefore need to be specified.

    If occurrences of the same event type are merged into one group, retaining a single representative clip may discard repeated events. The handling of events outside the text bank is also unclear. Please provide grouping pseudocode, the complete text bank, label sources, and training splits, and explain these cases. Page 6 explicitly states that spatial templates are used only during training; their construction and the final MLLM prompts should also be provided.

  2. Answer metrics do not directly establish better retention of critical evidence.

    BLEU, CIDEr, METEOR, and ROUGE in Table 1 measure agreement with reference answers, not whether selected frames cover the critical event. The two examples in Figure 3 suggest that the method can help, but do not establish a general evidence-retention advantage.

    Without the text prior, Table 3 still reports BLEU of 0.524/0.504/0.401 across the three levels, so the gains cannot all be attributed to textual hints. A more direct test would fix the backbone and 12-frame budget and compare uniform sampling, anomaly-score sampling, and semantic redundancy merging in terms of anomaly-interval coverage, stratified by event duration. A blinded faithfulness assessment of sampled answers would further check whether the responses are supported by the actual frames. These tests would address the paper's central claim.

  3. The correspondence between quality improvements and acceleration results remains unclear.

    Table 1 on page 6 reports a reduction in sampling time per eight-frame unit from 177.92 to 49.74 ms. Table 5 on page 9 reports 3281 and 3045 ms for no pruning and learned pruning, without explaining how these workloads correspond. The main table's mean BLEU/METEOR is approximately 0.484/0.211, matching the no-pruning row in Table 5, whereas learned pruning yields 0.467/0.209. This numerical similarity does not establish configuration mixing, but the current presentation leaves it unclear whether the reported quality and speed are achieved together.

    Please report quality, sampling time, and end-to-end latency for the same checkpoint and inference configuration. The 7.8% improvement in Table 5 is consistent with a throughput definition, 3281/3045−13281/3045-1, whereas the latency reduction is 7.19%; the definition should be explicit. The abstract's 13.4%/20.4% should also be identified as absolute differences on a 100-point scale, corresponding to relative gains of approximately 38.25%/138.37%.

Questions for authors
  1. How are semantic groups constructed and selected? How does the method handle repeated events of the same type, events outside the text bank, and more than four semantic types?
  2. What are the sources of category labels, binary evidence labels, and spatial templates? Is the MLLM further trained? Please specify the training/validation/test scope and actual prompts.
  3. Does anomaly-interval coverage improve under the same 12-frame budget, particularly for brief events? Does the advantage remain without the text prior?
  4. Which pruning configurations and workloads are used in Tables 1 and 5? Can quality and end-to-end latency be reported under the same configuration, with explicit percentage definitions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.