acceptodds
Withdrawn from ICLR 2027

What Drives Anomaly Scores in VLM-Based Video Anomaly Detection?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) are increasingly used for video anomaly detection (VAD), yet it remains unclear how much of a VAD system's anomaly-scoring behavior is inherited from its pretrained VLM and how much is introduced or transformed by downstream VAD processing. For example, if a system assigns higher anomaly scores to segments containing running people than to otherwise comparable segments without running, is this preference already present in the pretrained VLM, or does it emerge after the VLM is incorporated into the VAD system? Final detection accuracy alone cannot distinguish these possibilities. We introduce a systematic framework for tracing anomaly-scoring behavior across the pretrained-to-VAD transition. We characterize scoring behavior through cue-score associations: the relative ordering of anomaly scores between matched cue-present and cue-absent segments, while matching anomaly status and observed context. This score-scale-independent comparison allows pretrained VLMs and their corresponding VAD systems to be analyzed on the same video content despite differences in architectures and scoring functions, while preserving each VAD method's native inference procedure. Across 11 open-weight VLMs, 7 representative VAD methods, and 3 video-anomaly benchmarks, we find that VAD processing selectively reshapes pretrained scoring behavior: cue-score associations can persist, attenuate, disappear, reverse, or emerge after downstream processing, with these transformations depending jointly on the VLM and VAD method. Intermediate analysis identifies where these transformations arise within VAD pipelines, while temporal analysis shows when downstream processing changes anomaly evidence before, during, and after annotated events. These findings reveal that anomaly scoring in VLM-based VAD reflects both behavior inherited from the pretrained model and systematic modifications introduced by downstream detection mechanisms, providing a complementary view of VAD systems beyond final detection accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
video anomaly detection, vision-language models, multimodal learning, anomaly detection, model behavior, representation bias, robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zBeuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how downstream VAD processing reshapes the anomaly-scoring behavior of pretrained VLMs. Through cue-level, stage-wise, and temporal analyses across multiple models and datasets, the authors examine which scoring behaviors are preserved or modified.

Critical strengths

The paper conducts extensive experiments across a diverse set of VLM backbones, VAD methods, and benchmark datasets. The evaluation covers multiple aspects of anomaly-scoring behavior, providing a systematic empirical analysis beyond conventional detection performance metrics.

Critical weaknesses

1. Unclear motivation and practical implications. The term "VLM" is overly broad, as this work specifically focuses on generative, GPT-like multimodal models. More importantly, although the authors investigate how anomaly-scoring behavior changes from pretrained VLMs to downstream VAD systems, the motivation behind this question remains insufficiently justified. It is unclear how answering it could benefit practical VAD development or contribute to theoretical understanding.

2. Limited informativeness of the findings. The main conclusions are largely descriptive and lack clear, generalizable insights. For instance, the finding that cue-score associations may be preserved, weakened, reversed, or newly introduced depending on the VLM and VAD method is rather inconclusive. Despite extensive experiments, the paper provides limited explanations of why these changes occur or what meaningful insights can be drawn from them.

3. Insufficient rigor in matched-pair construction. The matched-pair strategy only controls a few coarse contextual factors, such as anomaly status, people count, scene category, and motion level. It does not ensure that the target cue is the primary difference between paired segments, as other factors such as lighting, backgrounds, and additional semantic cues may vary substantially. These uncontrolled differences could confound the observed cue-score associations, undermining the reliability and interpretability of the conclusions.

Questions for authors
  1. What concrete theoretical or practical benefits can be gained from understanding how downstream VAD processing reshapes pretrained anomaly-scoring behavior?

  2. Can the authors identify any consistent and generalizable insights beyond the observation that different VLM-VAD combinations produce different scoring behaviors?

  3. How can the authors ensure that the observed cue-score associations are not confounded by uncontrolled differences between matched video segments?

Reviewer_gQrkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduce a study of how anomaly detection models change the use of information already available in a pretrained VLM

Critical strengths

I would say that the clearest contribution of the paper is the measurement framework. Same matched video pairs are evaluated, comparison of their score despite the different numerical score scales, and tracking the comparisons through intermediate stages accross time.

Critical weaknesses

Beyond the large amount of experiments peresented in the paper, the contribution of the paper to understand anomaly knowledge itself is very weak.

It not surprising that changes to prompting, visual input selection, scoring procedures, etc.. around VLM would produce different behaviour. And I would say it is also reasonable to expect thAt large scale multimodal pretraining to provide VLMs with knowledge relevant to anomaly detection.

Questions for authors
  • After such large scale experiments, what is the established rule for predicting which associations will change in pipeline that was not tested before?

  • After reading this paper, what can I now explain or predict that I could not explain or predict before? to me the answer is very limited.

Reviewer_T75jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how much of a VLM-based VAD system's anomaly-scoring behavior comes from the pretrained VLM itself, and how much is introduced or transformed by the downstream VAD processing. Behavior is measured through cue-score associations. For eight contextual cues (person, vehicle, running, night, indoor, street, dense people, high motion) and two event attributes (early event, long event), the authors compare matched cue-present and cue-absent segments. Pairs come from different videos and are matched on anomaly status and prespecified observed context (people category, scene, motion level); the cue under study is excluded from the matching variables. The association statistic D_c is the mean within-pair sign of the score difference, so it depends only on ordering and is invariant to strictly increasing score transformations. That property is what allows the pretrained reference — the original checkpoint probed with a fixed "Does this video contain an abnormal event? Answer Yes or No." prompt and scored by Yes/No log-odds — to be compared against VAD systems with different architectures, score scales, and temporal resolutions on the same video content. The downstream change is ΔD = D^VAD − D^ref. Each cell is then classified as preserved, attenuated, amplified, suppressed, introduced, or reversed, depending on the direction and detectability of the two associations. Two further analyses locate where changes arise (stage-wise D_c at intermediate pipeline outputs) and when anomaly evidence is assigned (event-aligned temporal profiles using within-video percentile ranks).

Contributions: (i) a scale-free framework for tracing scoring behavior across the pretrained-to-VAD transition; (ii) a large-scale empirical study over 11 open-weight VLMs, 7 VAD methods (reasoning, visual evidence selection, representation readout, backbone adaptation), 3 benchmarks, and 231 backbone-method-dataset configurations; (iii) the finding of "selective reshaping" — associations persist, attenuate, disappear, reverse, or emerge depending jointly on the VLM and the VAD method, with most cue-level change concentrated at the first method-specific stage; (iv) evidence that behavioral retention is only moderately and method-dependently related to detection performance, which argues for reporting behavior alongside AUC/AP.

Validation: test splits of UCF-Crime, XD-Violence, and MSAD; 10 s segments with eight uniformly sampled frames; cue labels kept only under three-VLM annotator agreement; inference by source-video cluster bootstrap (2,000 replicates, percentile 95% CIs, two-sided bootstrap tests) with Benjamini-Hochberg correction within each backbone-cue family, and detectability at adjusted q < 0.05.

Critical strengths

The paper asks what VLM-based VAD systems actually rely on, not just how well they detect. That question matters because it targets a gap AUC and AP cannot close: systems with similar detection performance can relate to their backbones in visibly different ways, and the paper shows exactly this on 231 configurations. The question, more than the framework, is what would make the paper worth reading.

The empirical design is well matched to the question and broad enough to be the most substantial evidence available on it. The rank-based statistic avoids comparing raw scores across models with different scales and semantics; the same matched pairs are reused for the reference, the final VAD score, and intermediate outputs, so differences reflect the scoring system rather than the content; source-video cluster bootstrap and multiple-comparison correction handle the dependence structure; and the cue-level, stage-wise, and temporal analyses give three views of one question. Even if the conclusions eventually have to be narrowed to the evaluated implementations, the measurements themselves remain usable.

Critical weaknesses

The fidelity of the evaluated implementations is unclear, and this bears directly on the main result. The main text states that each method's native inference procedure is preserved, but Appendix D.1 describes changes to input units, frame sampling, prompts, and auxiliary components, and explicitly notes that these implementations are not exact reproductions; several methods lack complete public code or prompts, and some reported detection results differ from those in the original papers. Without the full prompts, the reconstructed components, and the training and inference configurations, I cannot tell whether the reported behavior belongs to the original methods or to the variants implemented here. The contribution is an empirical characterization of named, published systems; if the ports are not faithful, the central claims describe this paper's implementations rather than the field's methods, and the conclusions should be limited accordingly.

Several headline claims are stronger than the design supports. The matched analysis compares cue-present and cue-absent clips from different videos, not the same video before and after a controlled cue intervention, so differences in event type, viewpoint, background, or other co-occurring cues can explain the score gap. The stage-wise analysis shows where a change first becomes observable, not which component caused it. Terms such as "drives," "inherited," "introduced," and "responsible for" go beyond what this supports. Relatedly, an association is labeled "introduced" when D^ref is not significant and D^VAD is significant, even if ΔD is not significant, whereas "suppressed" requires a detectable ΔD. On MSAD, six cells are classified as introduced despite no detectable ΔD, and Figure 2 does not distinguish these from cells with a directly significant ΔD. Since "selective reshaping" is defined by the distribution of these categories, a definition that inflates one category and a figure that conflates two evidence standards weaken the central claim at the exact point where it is stated.

Questions for authors
  1. What does the proposed framework add beyond prior cue-based analyses, particularly Zhang et al. [1]?

  2. How would Figure 2 and the main conclusions change if transformation categories required a significant ΔD?

  3. Can the authors compare their implementations with the official methods in terms of anomaly scores, D_c, and transformation categories?

  4. Can the cue-score relationship be tested through controlled cue insertion, removal, or masking within the same video?

  5. Do the main findings remain consistent on datasets with more diverse scenes and context-dependent anomalies, such as ShanghaiTech or NWPU Campus?

[1] M. Zhang et al., "Do LVLMs Truly Understand Video Anomalies? Revealing Hallucination via Co-Occurrence Patterns," Advances in Neural Information Processing Systems, vol. 38, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.