What Drives Anomaly Scores in VLM-Based Video Anomaly Detection?

Vision-language models (VLMs) are increasingly used for video anomaly detection (VAD), yet it remains unclear how much of a VAD system's anomaly-scoring behavior is inherited from its pretrained VLM and how much is introduced or transformed by downstream VAD processing. For example, if a system assigns higher anomaly scores to segments containing running people than to otherwise comparable segments without running, is this preference already present in the pretrained VLM, or does it emerge after the VLM is incorporated into the VAD system? Final detection accuracy alone cannot distinguish these possibilities. We introduce a systematic framework for tracing anomaly-scoring behavior across the pretrained-to-VAD transition. We characterize scoring behavior through cue-score associations: the relative ordering of anomaly scores between matched cue-present and cue-absent segments, while matching anomaly status and observed context. This score-scale-independent comparison allows pretrained VLMs and their corresponding VAD systems to be analyzed on the same video content despite differences in architectures and scoring functions, while preserving each VAD method's native inference procedure. Across 11 open-weight VLMs, 7 representative VAD methods, and 3 video-anomaly benchmarks, we find that VAD processing selectively reshapes pretrained scoring behavior: cue-score associations can persist, attenuate, disappear, reverse, or emerge after downstream processing, with these transformations depending jointly on the VLM and VAD method. Intermediate analysis identifies where these transformations arise within VAD pipelines, while temporal analysis shows when downstream processing changes anomaly evidence before, during, and after annotated events. These findings reveal that anomaly scoring in VLM-based VAD reflects both behavior inherited from the pretrained model and systematic modifications introduced by downstream detection mechanisms, providing a complementary view of VAD systems beyond final detection accuracy.