acceptodds
Withdrawn from ICLR 2027

Which Way Is Broken? Language-Free Zero-Shot Anomaly Detection with Frozen Vision Foundation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Zero-shot anomaly detection (ZSAD) aims to identify and localize unseen defects without access to reference target samples. Prior approaches rely heavily on Vision-Language Models (VLMs) by matching visual patch tokens with hand-crafted or learnable text prompts. However, this language-grounded setup creates a fundamental bottleneck: visual defects are open-ended and fine-grained, making them difficult to describe accurately in text. Recent language-free methods avoid text, but they change the vision model itself. We show that neither is needed. Frozen vision models naturally separate normal and defective patches along a direction that is shared across datasets. We learn a few such directions for normal and defective patches directly on the frozen features, and score each patch by how much it is closer to the corresponding set. During training, each direction is treated as a small spread of nearby directions rather than a single point. Since defects on unseen products point in slightly different directions, such a direction is more likely to transfer. The method uses no text, changes no model weights, and needs very little training data. Across nine benchmarks in strict zero-shot settings, it matches or exceeds language-based and vision-only methods on AUROC, and improves F1-max by 6.8 points at the pixel level and 1.6 points at the image level. It also performs well in the data-constrained regime, trained on only two images per class, it already matches or outperforms prior methods trained on the full data. It works with six different vision backbones and benefits directly from stronger ones. The source code will be publicly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
zero-shot anomaly detection, anomaly segmentation, vision foundation models, geometrical representations, hyperspherical embeddings

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YPWbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new method for zero-shot anomaly detection (unfortunately, the method is not named). The method is based on the analysis that abnormal features and normal features have different “angles”/”directions” in the latent space. The model learns these angles during training. To increase diversity, these angles are also perturbed using a vMF sampler. The model is evaluated on standard industrial datasets.

Critical strengths
  • The analysis showcasing the separation of anomalous and normal features is well thought out
  • The idea behind this method is simple and works well
Critical weaknesses

Major Weaknesses:

  • AnomalyVFM outperforms the method when using the RADIO backbone. I do not see a reason to use only DINOv3 as a backbone; the main goal is to improve zero-shot anomaly detection overall, not just some backbones. What is the performance of the proposed method using the RADIOv2.5 ViT-L backbone? The supplementary material has only C-RADIOv3 ViT-L results.
  • The method is evaluated only on industrial datasets, neglecting medical ones.

Minor Weaknesses:

  • As far as I know, the maximum is non-differentiable. Is the residual alignment score (Equation 10) derived differently during training? Or how do you bypass this non-differentiability?
Questions for authors
  • How does the method perform with RADIOv2.5 ViT-L backbone in comparison to AnomalyVFM with the same backbone?
  • How does the method perform on medical datasets?
  • How do you make Equation 10 differentiable?
Reviewer_ZZjLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a text-encoder-free zero-shot anomaly detection method that replaces textual prompts with normal and anomalous feature directions in a frozen visual foundation model. The method learns directional distributions across multiple feature layers and employs the von Mises-Fisher distribution to model these directions and their uncertainty. Pixel-level and image-level anomaly scores are determined by the relative alignment of features with anomalous and normal directions.

Critical strengths
  1. The proposed method does not rely on a text encoder and can be adapted to various frozen visual backbones. It achieves unified pixel-level and image-level scoring using a small number of learnable directions, resulting in low parameter and deployment costs.

  2. The experiments cover nine industrial and medical benchmarks, multiple visual backbones, few-shot efficiency, and several ablation studies.

Critical weaknesses
  1. The evidence supporting the claim that frozen visual models naturally separate normal and anomalous features along shared directions is circular. The oracle direction is directly defined using the mean features of normal and anomalous samples based on their labels in each dataset, so projecting features along this direction will naturally reveal separation.

  2. The references contain serious hallucinations:

  • Thomas Defard, Aleksandr Setkov, Angel Angelov, Chethan Li, and Abhinav Dhall. Padim: A Patch Distribution Modeling Framework for Anomaly Detection and Localization. In International Conference on Pattern Recognition (ICPR), pp. 475–489, 2021.
  • Jiawei Yu, Ye Zheng, Xiang Wang, Wei Li, Yida Shen, and Chunhua Zhao. Fastflow: Unsupervised Anomaly Detection and Localization via 2d Normalizing Flows. arXiv preprint arXiv:2111.07677, 2021.
Questions for authors

See weaknesses.

Reviewer_xTCpOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a language-free zero-shot anomaly detection method using frozen vision foundation models. It learns normal and anomalous reference directions in the visual feature space and employs von Mises–Fisher (vMF) sampling to improve generalization. Anomaly scores are computed through cosine similarity differences for pixel-level localization and image-level detection. Experiments on nine benchmarks demonstrate competitive performance

Critical strengths
  1. The proposed method is simple, and the experiments cover a wide range of benchmarks.

  2. The paper is well-organized, clearly written, and easy to follow.

Critical weaknesses
  1. The evidence supporting the existence of shared anomaly directions is mainly based on visualizations. While these results provide some support for the observation, they are insufficient to justify the strong claim that such directions consistently generalize across different categories, datasets, and backbones.

  2. Is a set of only five directions sufficient to capture the variations in normal appearances and anomaly patterns across different categories? Do these learned directions correspond to category-specific clusters, or do they simply form a decision boundary over a mixture of categories?

  3. The authors claim that their method learns anomaly directions that generalize across different object categories. However, if some categories contain significantly more training samples than others, the learned directions may be biased toward those dominant categories rather than capturing truly category-independent anomaly patterns.

  4. The paper may contain references with fabricated author information.

Questions for authors

See Weaknesses.

  1. The evidence supporting the existence of shared anomaly directions is mainly based on visualizations. While these results provide some support for the observation, they are insufficient to justify the strong claim that such directions consistently generalize across different categories, datasets, and backbones.

  2. Is a set of only five directions sufficient to capture the variations in normal appearances and anomaly patterns across different categories? Do these learned directions correspond to category-specific clusters, or do they simply form a decision boundary over a mixture of categories?

  3. The authors claim that their method learns anomaly directions that generalize across different object categories. However, if some categories contain significantly more training samples than others, the learned directions may be biased toward those dominant categories rather than capturing truly category-independent anomaly patterns. Have the authors considered the impact of category imbalance on directional learning?

  4. The paper may contain references with fabricated author information.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.