acceptodds
Withdrawn from ICLR 2027

Rethinking Readouts in CLIP-Based Zero-Shot Anomaly Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

CLIP-based zero-shot anomaly detectors build task-specific decision rules on top of frozen vision-language representations. However, a strong representation does not guarantee a strong decision: useful anomaly evidence may already be encoded in the representation but not fully used by the readout, i.e., the mechanism that turns the representation into an anomaly score. We study this gap in CLIP-based zero-shot anomaly detection. For a canonical binary CLIP readout, the anomaly score depends on a single decision direction. We show that substantial predictive information remains outside this direction, that the remaining information is structured rather than a by-product of high dimensionality, and that the useful directions vary across anomaly contexts. We then ask whether this unused information can be recovered with only a simple change to the detector. We keep the original detector frozen and add a lightweight, image-conditioned reader that makes only a small bounded correction to the existing decision pathway, using auxiliary anomaly data for training. Despite its limited capacity, this intervention improves a diverse set of CLIP-based anomaly detectors. Our results suggest that improving the representation is only part of the problem in zero-shot anomaly detection. It is also important to ask how much task-relevant information the final readout actually uses, and whether information left unused by the original decision can be recovered.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
clip, anomalydetection, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7D3GOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper explores how influential feature noise is on downstream zero-shot performance. To do so, they explore how much a decision direction influences the overall performance. To do so, they train a small detector that fixes this and evaluate it on MVTec AD.

Critical strengths
  • Thinking about how to reduce noise in the CLS token and the patch features makes sense.
Critical weaknesses

Major Weaknesses:

  • The method is tested only on one dataset (MVTec AD). This is far below the usual number of datasets used in zero-shot anomaly detection. Typically, we use ~7 industrial datasets and ~5 medical datasets. Due to testing on only one dataset, the conclusion cannot be conclusive.
  • It is highly unlikely that the introduction does not contain a single reference. On top of this, the whole paper is poorly written and hard to understand.
  • A lot of things are presumed without any introduction. For instance, the test on 48 defect classes is just described with the results, and simply stated that the explanation is in the supplementary. You have half a page left; use it to describe things more effectively.

Minor Weaknesses:

  • Many mathematical symbols are not defined but are instead assumed to be known. This decreases the readability substantially.
Questions for authors

How does the proposed method work on other datasets? See VisualAD or AnomalyVFM for a list of datasets.

Overall, the biggest problem with this paper is the presentation. I do not see a way to improve this without a substantial rewrite. As a result, it is highly unlikely that I will change my mind.

Reviewer_oei2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ProtoDeCo separates anomaly ranking from target-side normality measurement. A fixed detector selects low-scoring target images, while frozen CLIP features form category-specific normal prototypes. Prototype distances calibrate image scores without changing pixel predictions. Matched feature-space comparisons and transductive controls evaluate the approach across industrial and medical datasets.

Critical strengths
  1. A clear and useful research question with targeted analysis. The paper distinguishes information encoded in frozen CLIP representations from information actually used by a detector's decision rule. The orthogonal decomposition, structured-versus-random subspace comparison, and context-dependence analysis provide more insight than an accuracy-only evaluation.

  2. Well-motivated controlled interventions. The matched static/dynamic comparison, parallel-versus-orthogonal margin counterfactual, and feature-swapping experiment test specific implications of the proposed explanation. In particular, the orthogonal-only counterfactual retaining the observed gain is a useful link between the analysis and the AnomalyCLIP intervention.

Critical weaknesses
  1. The diagnostic representation is not the same information source read by the practical intervention. The geometric analysis considers a global CLIP image embedding and a canonical binary text direction, whereas the dynamic reader is conditioned on host-specific spatial/patch features and intervenes at different states across hosts. The orthogonal-margin counterfactual shows where the score change acts for AnomalyCLIP, but does not establish that the reader's improvement comes specifically from information already present in the analyzed global orthogonal component rather than additional patch-level cues. This distinction is central to the paper's proposed mechanism.
  2. The evidence for deployable cross-domain gains is narrower than the strong diagnostic results might suggest. The near-oracle residual-probe results rely on target-domain anomaly labels and are therefore not evidence of zero-shot transfer. The deployed reader is additionally trained on Real-IAD-SV, while the native hosts use different pretraining/source histories; the static comparison controls image conditioning but not every alternative use of the extra supervision. The principal zero-shot evaluation uses only MVTec AD, and the average gain is modest (+0.66 AUROC points), with no gain on VisualAD and a small, variable improvement on VCP-CLIP. It is therefore difficult to judge how general the readout-gap remedy is across unseen target domains.
Questions for authors

See Critical Weaknesses.

Reviewer_Po5SOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces the concept of the Representation-Readout Gap for addressing the prevailing phenomenon of "heavy representation, light readout" in current CLIP-based zero-shot anomaly detection (ZSAD). The authors find that the classic binary CLIP readout mechanism utilizes only a single decision direction, resulting in a large amount of structured, task-relevant predictive information being left behind in the orthogonal residual space. To this end, the paper designs a lightweight image-conditioned residual reader. While keeping the pre-trained host models completely frozen, it successfully recovers this hidden information with the aid of auxiliary anomaly training data and validates its effectiveness across multiple mainstream ZSAD models.

Critical strengths

S1.The paper takes a refreshing path by scientifically decoupling representation quality from the utilization rate of the readout mechanism, offering a novel perspective for understanding the information utilization efficiency of foundational models in downstream tasks.

S2.Through the geometric orthogonal decomposition in Equations (3) to (9), combined with the Orthogonal Oracle and low-rank PCA analysis, the paper robustly demonstrates that the unread information is not only objectively existent but also low-rank and highly structured.

Critical weaknesses

W1.The authors propose decomposing visual features into parallel and orthogonal components, claiming that the orthogonal component drives performance. This is counterintuitive and lacks clear physical intuition. Furthermore, the actual performance improvement brought by this design is marginal (e.g., a mere 0.4% absolute increase in image-level AUROC from 91.55% to 91.95%).

W2.The proposed lightweight reader architecture shares strong similarities with existing designs (such as VisualAD), yet the current version lacks adequate analysis and discussion regarding these connections. The average gains on MVTec AD are relatively limited, with zero improvement over certain baselines (e.g., VisualAD), which undermines its overall novelty and practical impact.

W3.The training of the proposed method heavily relies on the auxiliary anomaly dataset Real-IAD-SV, which deviates from standard zero-shot AD evaluation protocols. Moreover, the evaluation is restricted solely to MVTec AD, lacking validation on broader industrial benchmarks or cross-domain datasets to prove generalizability.

Questions for authors

As listed in Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.