acceptodds
Under review as a conference paper at ICLR 2027

Prototype-driven Hierarchical Rule Induction and Matching for Efficient Video Anomaly Detection

Abstract

Video anomaly detection (VAD) aims to identify rare and unexpected events in video. Though training-free VAD using vision-language models (VLMs) has shown promise, existing pipelines rely on dense-frame semantic generation and reasoning, which is computationally expensive and not good at modeling context-dependent anomalies. To address these issues, we propose ProtoRule-VAD, an efficient framework that converts normal videos into reusable hierarchical semantic rules without anomalous supervision or model parameter updates. In the offline stage, scene-level and object-level normal prototypes are constructed from normal videos and used to induce contrastive hierarchical rules with a pretrained VLM, including scene-level, object-level, and condition-level rules. During inference, motion saliency and prototype-based visual novelty are combined to prioritize candidate segments. Condition-level rules then select relevant scene-level and object-level rules for selective rule matching and anomaly scoring. ProtoRule-VAD delivers competitive performance and interpretable rule-based evidence without requiring anomalous data and fine-tuning. Notably, our processing frame rate is over 500 FPS on one NVIDIA A6000 GPU, substantially improving online inference throughput over existing VLM-based VAD methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Anomaly Detection, Training-Free Learning, Vision-Language Models, Multimodal Reasoning, Prototype Learning, Rule Induction, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_aT4ZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work introduces ProtoRule‑VAD for training‑free video anomaly detection. It builds scene and object prototypes offline from normal videos, then leverages a frozen VLM to generate three tiers of hierarchical semantic rules. At inference time, motion saliency and prototype‑based novelty filter candidate segments, and condition‑level rules route relevant rules for anomaly scoring. The method produces interpretable outputs and achieves very high inference FPS with competitive AUC results on standard benchmarks. However, incremental novelty is limited, several design choices lack sufficient validation, and key failure modes are left unaddressed.

Critical strengths
  1. The framework works in training‑free settings. It requires no anomalous samples and does not fine‑tune multimodal foundation models.

  2. The paper includes comprehensive ablation studies for core modules. It also evaluates prototype representativeness and quality of the induced rule bank in the appendix.

Critical weaknesses
  1. The overall novelty is incremental. Prior work, such as AnomalyRuler, already builds normality rules from normal examples. The new contributions of this paper are not sufficiently distinct.

  2. Condition‑level rules only serve as a routing module and are excluded from anomaly scoring. The paper provides little experimental justification for this design decision.

  3. The candidate prescreening step filters out certain true anomalies before semantic matching. This recall limitation is only mentioned for future work without any attempted mitigation.

  4. Max aggregation for scene and object scores causes clear failure cases. Weak object‑level anomaly signals can override consistent normal evidence from the scene branch. The authors do not test alternative fusion operators.

  5. The whole rule bank depends on VLM generation. Though rule quality is statistically summarized, no ablation quantifies performance drop caused by noisy or flawed generated rules.

  6. There is no cross‑domain experiment. The authors acknowledge that domain shift, occlusion, and low resolution hurt performance but provide no supporting empirical results.

Questions for authors
  1. Some generated rules contain artifacts or noise. How much AUC gain can you get after removing those low‑quality rules manually? Ks and Ko are manually set hyperparameters. How do they affect detection performance, rule‑bank size, and online FPS when varied?
  2. See the above weakness of other questions
Reviewer_HV5NOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProtoRule-VAD, a training-free video anomaly detection (VAD) method based on a pretrained vision-language model (VLM). The key idea is to use normal videos to construct reusable hierarchical semantic rules for VAD. ProtoRule-VAD achieves 500 FPS inference speed.

Critical strengths

The proposed method achieves 500 FPS inference speed, which is much faster than existing VLM-based VAD approaches.

Critical weaknesses
  1. The main concern is the insufficient novelty. The idea of inducing rules from normal videos was proposed by AnomalyRuler. This paper is an incremental work on AnomalyRuler. The new hierarchical rule is more of an engineering development, lacking technological novelty and insight. The improvement over existing approaches is also marginal.

  2. Inference speed is one of the main advantages, but related analysis is incomplete. (1) Many baselines shown in Table 1 are not compared in Figure 3, particularly AnomalyRuler, an important prior work. (2) No ablation study on efficiency analysis. What design choices or components make the proposed method fast is not sufficiently demonstrated. (3) For efficiency analysis, comparison under different model sizes, configurations, or backbones is important. Each method in the figure should be a line rather than a point. EfficientNet [r1] is a classic example.

  3. The evaluation is not comprehensive. (1) The evaluation considers only three dated datasets: UCF-Crime, ShanghaiTech, and XD-Violence. There have been many more advanced and complicated datasets in the literature, such as Holmes-VAU (also cited in this paper) and UBnormal [r2] (used by AnomalyRuler). (2) Only one backbone VLM, Qwen2.5-VL-7B, is adopted. The generalizability to different VLMs is not demonstrated. (3) As a VLM-based method, reasoning is a key advantage. However, the reasoning quality is not quantitatively evaluated.

Given the above three main concerns, my rating is clear rejection. A more application-oriented venue may be more suitable for this work.

[r1] Tan, Mingxing, and Quoc Le. "Efficientnet: Rethinking model scaling for convolutional neural networks." ICML, 2019.

[r2] Acsintoae, Andra, et al. "Ubnormal: New benchmark for supervised open-set video anomaly detection." CVPR, 2022.

Questions for authors

See weaknesses.

(Minor) In many places, there is no space between the period "." and the next word.

Reviewer_rGeQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ProtoRule-VAD, a video anomaly detection (VAD) framework that aims to improve online inference efficiency while avoiding anomalous supervision or VLM fine-tuning. In the offline stage, the method extracts scene-level and object-level features from normal training videos using CLIP and YOLOv8n, clusters them via K-Means into normal prototypes, and prompts a frozen VLM (Qwen2.5-VL-7B) to induce scene-level, object-level, and condition-level rules. During online inference, segments are first prescreened by fusing rankings of pixel-wise motion saliency and prototype-based visual novelty to select top-M candidates. Subsequently, condition-level rules are used to route relevant scene and object rules, and anomaly scores are computed as the margin between visual-text similarities to potential-violation rules and normal rules. Experiments are reported on UCF-Crime, ShanghaiTech, and XD-Violence, claiming competitive frame-level AUC and an inference throughput of 562 FPS.

Critical strengths

Offloading the heavy VLM semantic generation process to the offline stage and achieving fast online retrieval by caching text embeddings is a sound engineering approach to address the inference latency bottleneck of multimodal large language models in video anomaly detection.

Critical weaknesses

1.Lack of novelty. The proposed method essentially performs a naive engineering integration of existing, off-the-shelf modules (K-Means prototype clustering, YOLOv8n object detection, CLIP vision-text embeddings, and prompt generation using pretrained VLMs). 2.In the actual generation results for the ShanghaiTech person category shown in Appendix B.3 (Listing 5), all 10 input images are from the normal training set, but the 6 normal rules output by the VLM are all mechanically repetitive. 3.The paper claims an online inference speed of 562 FPS (only 1.78 ms per frame) on a single A6000 GPU. However, according to Eq. (4) and Algorithm 2 (Lines 2–6), before filtering out the candidate set, the system must run YOLOv8n to detect foreground objects for each sliding window, and feed both the global frame and all object crops into CLIP (ViT-B/32) to extract features for computing visual novelty. In videos containing dense pedestrians, performing per-window detection and multiple CLIP encodings makes it impossible to achieve 562 FPS. Moreover, the paper completely omits the specific value of the candidate set size M, and Table 1 misses experimental data for comparison methods on ShanghaiTech. 4.Existing training-free rule/retrieval methods (such as AnomalyRuler and Flashback) are highly homogenized. 5.Equations (7)–(8) fail to define how to aggregate the similarities of multiple rules within the routed subset (whether to take the maximum or the mean).

Questions for authors

1.Appendix B.4 explicitly states that the anomaly texts generated by condition-level rules are not used for anomaly scoring. Why invoke the VLM to generate, encode, and cache these 4,590 texts? 2.Across the three datasets, what is the exact value or percentage set for the candidate set size M in Eq. (5) (a fixed number or a percentage)? Please provide a detailed breakdown of the per-frame latency (in milliseconds) for: (a) video decoding and pixel difference calculation, (b) YOLOv8n object detection across all sliding windows, (c) CLIP ViT-B/32 feature extraction for global frames and object crops across all sliding windows, and (d) rule matching. Were the test set features pre-extracted offline when measuring the 562 FPS in Figure 3? 3.While the motivation of offline rule induction to accelerate online inference is commendable, the paper suffers from severe internal data and logical contradictions (generating 4,590 condition-level rules accounting for 92.7% of the total while claiming they do not participate in scoring), unconvincing efficiency claims (asserting 562 FPS while running YOLOv8n and multi-object CLIP encoding on every sliding window, completely omitting the candidate set size M), and incomplete baseline comparison tables (Table 1). The authors are requested to address these issues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.