acceptodds
Under review as a conference paper at ICLR 2027

A VLM Answer Is Not an Anomaly Score: Rank Compression Across Image and Video Anomaly Detection

Abstract

Anomaly detection aims to identify observations that deviate from normal patterns. Recent work has increasingly used pretrained vision–language models (VLMs) for training-free image and video anomaly detection without task-specific retraining. Anomaly detection is commonly evaluated by how well anomaly scores rank anomalous images or video frames above normal ones. Generative VLMs, however, assign probabilities to possible answers and then decode a single answer. This decoding step may discard ordering information. We call this loss decoded-answer rank compression. To isolate this effect, we compare two ways of scoring the same VLM output: one uses only the decoded answer, while the other computes a probability-weighted score over all possible answers. Probability-weighted scoring consistently outperforms decoded-answer scoring across image and video anomaly detection benchmarks, using different VLMs and answer scales. The mean gains range from 7.66 to 19.95 points on the primary metrics. Most of this gap comes from decoded-answer ties. Breaking decoded-answer ties with answer probabilities recovers at least 95% of the average performance gap on every benchmark. We further investigate how these ties affect evaluation. We find that evaluating tied scores one input at a time makes the reported AUROC depend on input order, and that linear PR interpolation can inflate reported performance. These results reveal that both how VLM outputs are scored and how discrete anomaly scores are evaluated affect reported anomaly-detection performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image anomaly detection, Video anomaly detection, Vision-language models, Evaluation metrics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JAwAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the "decoded-answer rank compression" problem in unsupervised anomaly detection. It compares Decoded-Answer Scoring (DEC) with Probability-Weighted Scoring (PROB). Experiments conducted on four benchmark datasets, four VLMs, and seven answer scales show that using PROB consistently leads to performance improvements. It is demonstrated that PROB can retain the ranking information in the output distribution of visual language models (VLMs).

Critical strengths
  • This paper reveals a frequently overlooked issue in anomaly detection based on VLM; there is a mismatch between discrete output and continuous anomaly ranking metrics.
  • Extensive comparative experiments and detailed ablation studies effectively demonstrated and evaluated the impact of the scoring strategy.
Critical weaknesses
  • The PROB method relies on the logarithmic likelihood values of the sequence, and this metric may be affected by the change in the length of the answer tokens. This effect is particularly evident in the numerical rating scale.
  • The experiment failed to clearly determine whether the improvement in video detection performance was due to the improvement in "internal anomaly localization" or the enhancement of "cross-video ranking" ability. Moreover, the computational cost brought about by evaluating a large number of candidate answers has not been quantified.
  • The proposed probability-weighted scoring method in this paper is conceptually more intuitive and heuristic. The paper should further clarify the innovations and discuss the related works on "anomaly unawareness" and "illusion", including AA-CLIP (CVPR 2025), Alert-CLIP (CVPR 2026), and VAD-DPO (NeurIPS 2025), etc.
  • Additionally, there is an error in the citation of VADTree. It is incorrectly labeled as published in 2026, but in fact, this achievement was published in NeurIPS 2025.
Questions for authors
  • How does PROB perform in cases involving multiple token numerical answers?
  • Can the authors separate within-video from cross-video ranking improvements and report the inference latency and memory overhead across different answer scales?
  • How does rank compression relate to limitations in underlying anomaly perception, such as anomaly-unawareness or co-occurrence hallucinations? Can improved probability-based ranking compensate for systematic semantic biases? Additionally, could the authors verify the accuracy of the references, particularly the publication year of VADTree (NeurIPS 2025)?
Reviewer_xZMgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors investigate how converting generative VLM outputs into anomaly scores affects image and video anomaly detection. They compare decoded-answer scoring (DEC), which uses the most probable answer, with probability-weighted scoring (PROB), which uses the expected anomaly value over the answer distribution. The results show that PROB consistently improves anomaly ranking, and the authors further analyze the effects of score ties and evaluation metric choices.

Critical strengths

Clear Presentation and Systematic Evaluation

The paper is well organized, and the comparison between DEC and PROB is clearly explained and carefully controlled. The authors evaluate four image and video anomaly detection benchmarks, four VLM families, and seven answer scales. Additional experiments examine model size, prompt variations, and evaluation metrics, providing consistent evidence for the reported performance differences.

Critical weaknesses

1. Limited Scientific Insights and Generalizability.

Although information loss from discretizing continuous probabilities is well established, the paper provides useful empirical evidence of its impact on VLM-based anomaly detection. However, the factors governing the observed rank compression remain insufficiently explained. For example, DEC uses only a small fraction of the 91 possible scores (Sec. 5.3), but the causes of this score concentration are not clearly identified. Understanding these factors would help establish when similarly severe rank compression should be expected beyond the evaluated settings.

2. Insufficient Motivation and Practical Significance.

Training-free methods such as WinCLIP [1] can already produce continuous anomaly scores without generative VLMs. Although the authors highlight the natural-language capabilities of generative VLMs (lines 41, 42), the paper does not establish how these capabilities contribute to the studied scoring problem. This leaves the distinctive motivation for focusing on generative VLMs insufficiently justified.

Furthermore, existing generative VLM-based anomaly detection methods employ various scoring strategies, including numerical outputs and additional score processing [2, 3, 4]. Although the paper evaluates numerical answer scales and temporal smoothing, it remains unclear how closely its DEC setting reflects existing detection pipelines or whether the reported improvements translate to those pipelines. This limits the practical significance of the findings.

[1] Jeong, J., et al. (2023). WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation. CVPR.

[2] Ye, M., et al. (2025). VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models. CVPR.

[3] Ahn, S., et al. (2026). AnyAnomaly: Zero-Shot Customizable Video Anomaly Detection with LVLM. WACV.

[4] Zanella, L., et al. (2024). Harnessing Large Language Models for Training-free Video Anomaly Detection. CVPR.

Questions for authors

1. Scientific Insights and Generalizability.

What factors determine the extent of rank compression in generative VLMs? In particular, why does DEC use only a small fraction of the available scores even with finer answer scales (Sec 5.3), and under what conditions should similar behavior be expected?

2. Motivation and Practical Significance.

Given that training-free methods such as WinCLIP [1] already produce continuous anomaly scores, what distinctive value does the generative VLM setting offer for studying this problem? To what extent does decoded-answer rank compression affect existing generative VLM-based anomaly detection pipelines [2, 3, 4], particularly those using numerical outputs or additional score processing?

Reviewer_XFbhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether using a VLM’s final answer as an anomaly score loses useful information. It compares decoded-answer scoring with probability-weighted scoring, called PROB. Both scores use the same model output probabilities. The results show that PROB gives better anomaly rankings across several models and benchmarks. The paper also examines how ties and different metric calculations affect the results. Although the authors have made good efforts, some issues still need to be addressed.

Critical strengths
  1. The paper makes a clear comparison between the two scoring rules. It keeps the model, input, prompt, and answer probabilities fixed. The experiment that only breaks ties also helps show how much of the gain comes from ordering samples with the same answer.

  2. The experiments cover several VLMs, answer scales, and image and video benchmarks. The authors also test different prompts and smoothing settings. The evaluation-code analysis is useful, as it shows that the same scores can give different method rankings under different metric calculations.

Critical weaknesses
  1. The added value over existing work is not clear enough. Previous studies have already used answer probabilities to obtain finer scores and reduce ties. The paper needs to better explain what new knowledge it adds and show how much the scoring change helps existing anomaly-detection methods.

  2. The candidate-answer scores may differ from actual model outputs. DEC chooses the answer with the highest complete-string probability from a fixed set. This is not always the answer produced by normal decoding. The current checks do not fully cover numerical answers, different token lengths, or outputs outside the candidate set.

Questions for authors
  1. In the Introduction and Related Work sections, the authors should explain the differences from G-Eval and Beyond Yes and No more clearly. They should state which findings are new for anomaly detection.

  2. The authors should add a test using a representative published anomaly-detection method. They should keep the original pipeline and replace only its scoring step with PROB. Reporting performance, inference time, and model calls would help show the practical benefit.

  3. In Section 3 and Appendix A.1, the authors should compare DEC with actual generated answers on more answer scales. They should report how often the two agree and how much probability is left outside the candidate set.

  4. The authors should test different answer formats and token lengths. For example, the same numerical value may be written in different ways. This would help show whether the results depend on the chosen answer strings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.