acceptodds
Under review as a conference paper at ICLR 2027

0-CLIP: Training-free Anomaly Detection Hidden in CLIP's Projection

Abstract

CLIP-based zero-shot anomaly detection commonly improves dense discriminability through prompts or adapters trained on auxiliary data. However, reliable transfer to unseen domains remains a practical challenge. This motivates a closer examination of how much anomaly information is already present in CLIP itself. Our preliminary experiments show that CLIP's frozen visual projector contains useful dense anomaly structure, but its patch-level scores tend to rank normal and anomalous regions in the wrong order. Based on this observation, we introduce 0-CLIP, a training-free method that converts into a patch-to-text anomaly operator. The construction reverses the normal–abnormal score polarity, forms a compact operator from the principal singular directions of , and selectively recovers lower-energy directions using a separation criterion computed on the first unlabeled test image of each category. Without any labeled data, the resulting operator yields both pixel-level anomaly maps and image-level anomaly scores. Across industrial and medical benchmarks, 0-CLIP achieves performance competitive with training-based methods, and ablation studies show the effectiveness of the proposed method. These results suggest that substantial dense anomaly discriminability is already latent in CLIP's frozen visual projector and can be exposed without training. Our code is available in the Supplementary Material.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
CLIP, Zero-shot, Anomaly detection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nuAqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new approach to continual zero-shot anomaly detection, called 0-CLIP. The main hypothesis is that CLIP already has normality and abnormality baked into its projection layer. To check this, they construct a “new” projection layer from the existing one by performing an SVD and gradually building it using the first test sample. The method is evaluated across multiple standard industrial and medical datasets, on which it achieves good results.

Critical strengths
  • The overall idea is quite simple
  • The paper is well written and easy to follow.
Critical weaknesses

Major Weaknesses:

  • The method is not compared to the current state of the art for continual zero-shot anomaly detection, ReMem [1]. This method outperforms it on the two datasets they test. It is unclear how it works on others. Since they have released the code publicly, I do not see a reason why 0-CLIP shouldn’t be compared to it. This might overturn the claim about SOTA results.
  • The method works only with the final-layer features of the vision encoder, neglecting information from earlier layers.

Minor Weaknesses:

  • How does the method work when using a 0-shot method that also fine-tunes the projection head? For example, AdaCLIP?
  • How are the std’s for the 5 runs calculated? The way it is calculated might underestimate the actual performance jumps based on the first image in the test stream.
  • I would mention the latest language-free zero-shot methods [2,3] in the related work, as they also achieve better results on some datasets.

Remarks:

  • A mistake in the citations: DRAEM: A discriminatively trained model for surface anomaly detection -> DRÆM – A discriminatively trained reconstruction embedding for surface anomaly detection

[1] Yi, L., Chen, Z., Wu, G., Ding, J., Wang, X., & Ning, Z. (2026). ReMem: A Dynamic Memory Evolution Detector for Zero-Shot Anomaly Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings (pp. 7697-7705).

[2] Fučka, M., et al. (2026). AnomalyVFM--Transforming Vision Foundation Models into Zero-Shot Anomaly Detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 35555-35566).

[3] Hou, Y., et al. (2026). VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21346-21356).

Questions for authors
  • How does 0-CLIP perform in comparison to ReMem on datasets besides MVTec AD and VisA?
  • Could the method be expanded to work with earlier layers as well?
  • How does the method work when applied to a 0-shot model that tunes the projection head, e.g. AdaCLIP?
  • How are the Standard Deviations for the multiple seed runs calculated?
6 comments

W1/Q1. How does 0-CLIP perform in comparison to ReMem on datasets besides MVTec AD and VisA?

Thank you for pointing us to ReMem. In short, ReMem is stronger on MVTec-AD and VisA, while 0-CLIP$^\dagger$ is higher on most other datasets and on average.

We ran its official code, here are results (pixel P-AUROC / PRO; image I-AUROC / I-AP):

DatasetReMem pixel0-CLIP$^\dagger$ pixelReMem image0-CLIP$^\dagger$ image
MVTec-AD98.4 / 95.597.8 / 94.098.7 / 99.598.2 / 99.3
VisA98.2 / 94.798.9 / 93.794.3 / 94.592.9 / 93.7
BTAD97.1 / 74.398.0 / 83.993.6 / 97.396.5 / 97.9
MPDD85.0 / 79.497.5 / 93.571.1 / 74.673.3 / 77.5
DTD-Syn98.3 / 95.498.3 / 95.297.6 / 99.298.0 / 99.3
SDD91.3 / 66.093.5 / 77.868.2 / 41.285.6 / 76.2
DAGM96.4 / 91.098.2 / 96.499.4 / 97.499.6 / 96.7
KSDD298.3 / 93.398.5 / 96.289.0 / 74.896.4 / 91.8
Real-IAD91.5 / 69.397.4 / 88.663.3 / 57.280.8 / 79.0
Brain MRI89.7 / 64.296.9 / 81.260.9 / 85.374.0 / 91.1
Liver CT96.3 / 82.097.5 / 92.0––
Retina RESC93.3 / 75.795.1 / 81.7––
TN3K71.7 / 36.780.0 / 57.9––
Br35H––93.7 / 88.789.5 / 83.8
Retina OCT––64.6 / 85.888.1 / 95.9
Average92.7 / 78.396.0 / 87.182.9 / 83.089.4 / 90.2

Two differences in setting between 0-CLIP and ReMem:

  • ReMem uses a DINOv2 backbone, while all methods in our tables use CLIP.
  • ReMem sees the whole test set in advance and scores it several times, while our online variant uses only past images.

Revision: Added as Table 8 (Appendix E.3).

W2/Q2. Could the method be expanded to work with earlier layers as well?

Thank you for your question. 0-CLIP already uses earlier layers: it takes patch tokens from layers 12, 16, 20 and 24 (Eq. (1) and Sec. 4.1) and builds a separate operator for each layer.

We compared this default with the final layer alone (P-AUROC / PRO / I-AUROC / I-AP):

LayersMVTec-ADVisA
12/16/20/24 (default)95.2 / 88.1 / 94.7 / 97.395.3 / 86.1 / 85.1 / 87.0
final layer (24 only)92.2 / 82.6 / 93.6 / 96.892.1 / 82.7 / 82.9 / 85.7

Using only the final layer lowers all eight metrics. Combining multiple layers is therefore highly beneficial and indispensable to 0-CLIP.

Revision: Sec. 3.1 now states explicitly that 0-CLIP constructs these operators for each layer.

W3/Q3. How does the method work when applied to a 0-shot model that tunes the projection head, e.g. AdaCLIP?

Using AdaCLIP's official code and weights, we applied our construction to its trained patch heads and kept everything else unchanged (P-AUROC / PRO / I-AUROC / I-AP):

AdaCLIP patch headsMVTec-ADVisA
Original86.8 / 33.8 / 92.0 / 96.495.1 / 71.3 / 83.0 / 84.9
With our construction43.9 / 12.8 / 76.8 / 88.257.3 / 32.1 / 62.9 / 68.9

The second row is our run of the official code with only the patch heads changed. AdaCLIP's heads are new layers trained on labeled data, and this training has already aligned their polarity and singular-value weights with AD. Flipping and reweighting them therefore undoes what training learned. 0-CLIP is meant to replace this training.

Revision: Added as Table 11 (Appendix E.6) and noted in the Discussion (Sec. 5).

W4/Q4. How are the Standard Deviations for the multiple seed runs calculated?

Each seed shuffles the test stream of every class, which changes the calibration image. For each seed we average the metrics over classes, and report the mean and standard deviation over the 5 seeds (Sec. 4.1 and Appendix E.7). Below we also compute the standard deviation over the 5 seeds for each class first and then average over classes (mean ± dataset std / per-class std; identical on 1-class datasets):

Dataset (#cls)SettingP-AUROCPROI-AUROCI-AP
MVTec-AD (15)0-CLIP$^\dagger$97.83 ± 0.25 / 0.4693.95 ± 0.21 / 0.5798.22 ± 0.12 / 0.6099.34 ± 0.07 / 0.26
0-CLIP94.91 ± 0.20 / 0.4787.30 ± 0.47 / 0.6694.44 ± 0.32 / 0.6997.35 ± 0.21 / 0.48
VisA (12)0-CLIP$^\dagger$98.90 ± 0.03 / 0.0893.73 ± 0.23 / 0.4792.92 ± 0.31 / 0.6793.73 ± 0.23 / 0.58
0-CLIP95.32 ± 0.10 / 0.1986.16 ± 0.04 / 0.2685.29 ± 0.16 / 0.3386.98 ± 0.11 / 0.27
BTAD (3)0-CLIP$^\dagger$98.03 ± 0.03 / 0.0983.89 ± 0.78 / 0.9996.54 ± 0.36 / 0.6097.90 ± 0.30 / 0.42
0-CLIP96.48 ± 0.23 / 0.2681.74 ± 0.57 / 0.6694.98 ± 0.64 / 1.1594.59 ± 0.35 / 0.65
MPDD (6)0-CLIP$^\dagger$97.47 ± 0.36 / 0.5193.54 ± 0.68 / 1.0273.27 ± 2.12 / 5.1877.48 ± 1.60 / 3.32
0-CLIP96.17 ± 0.07 / 0.1191.10 ± 0.19 / 0.2876.79 ± 0.26 / 0.7081.84 ± 0.13 / 0.44
DTD-Syn (12)0-CLIP$^\dagger$98.33 ± 0.01 / 0.1495.19 ± 0.10 / 0.3498.03 ± 0.14 / 0.4799.25 ± 0.03 / 0.21
0-CLIP97.99 ± 0.05 / 0.1093.99 ± 0.17 / 0.2798.01 ± 0.05 / 0.2499.28 ± 0.01 / 0.07
SDD (1)0-CLIP$^\dagger$93.51 ± 0.2777.84 ± 0.6485.56 ± 0.8976.24 ± 1.98
0-CLIP91.88 ± 0.2477.97 ± 0.2684.52 ± 0.1678.41 ± 0.29
DAGM (10)0-CLIP$^\dagger$98.19 ± 0.03 / 0.1396.38 ± 0.07 / 0.1999.56 ± 0.08 / 0.0896.72 ± 0.23 / 0.29
0-CLIP96.91 ± 0.05 / 0.0894.08 ± 0.12 / 0.1799.53 ± 0.02 / 0.0798.30 ± 0.12 / 0.24
KSDD2 (1)0-CLIP$^\dagger$98.52 ± 0.2996.17 ± 0.3296.42 ± 0.1891.78 ± 0.23
0-CLIP97.28 ± 0.3094.81 ± 0.2595.48 ± 0.0989.57 ± 0.15
Real-IAD (30)0-CLIP$^\dagger$97.38 ± 0.05 / 0.1488.59 ± 0.24 / 0.5880.84 ± 0.06 / 0.3978.99 ± 0.11 / 0.48
0-CLIP95.34 ± 0.05 / 0.1287.37 ± 0.08 / 0.2976.29 ± 0.12 / 0.2174.23 ± 0.11 / 0.25
Liver CT (1)0-CLIP$^\dagger$97.53 ± 0.1392.01 ± 0.54––
0-CLIP95.85 ± 0.0790.87 ± 0.17––
Brain MRI (1)0-CLIP$^\dagger$96.88 ± 0.0681.16 ± 0.3673.96 ± 0.7191.13 ± 0.14
0-CLIP93.74 ± 0.0676.28 ± 0.1880.42 ± 0.2194.72 ± 0.05
Retina RESC (1)0-CLIP$^\dagger$95.11 ± 0.2481.66 ± 0.80––
0-CLIP87.36 ± 0.1459.01 ± 0.63––
TN3K (1)0-CLIP$^\dagger$80.01 ± 0.4557.87 ± 0.34––
0-CLIP82.44 ± 0.1852.99 ± 1.04––
Br35H (1)0-CLIP$^\dagger$––89.49 ± 0.7383.75 ± 0.79
0-CLIP––96.06 ± 0.0594.59 ± 0.05
Retina OCT (1)0-CLIP$^\dagger$––88.07 ± 1.3295.92 ± 0.45
0-CLIP––79.29 ± 0.1691.99 ± 0.06

For offline 0-CLIP, even the per-class std stays small across datasets (mostly below 0.7%), showing minimal sensitivity to the calibration image. Larger online fluctuations such as on MPDD come from stream order in RareCLIP's memory.

We also tested the two extreme cases of the first image directly. In each seed's shuffled test stream, we calibrated every class once on its first normal image and once on its first anomalous image, and then scored the whole stream as usual.

DatasetCalibration imageP-AUROCPROI-AUROCI-AP
MVTec-ADFirst image (default)94.91 ± 0.20 / 0.4787.30 ± 0.47 / 0.6694.44 ± 0.32 / 0.6997.35 ± 0.21 / 0.48
Normal94.92 ± 0.14 / 0.5087.41 ± 0.28 / 0.5294.40 ± 0.33 / 0.6597.34 ± 0.23 / 0.48
Anomalous94.91 ± 0.20 / 0.4787.28 ± 0.47 / 0.6694.41 ± 0.33 / 0.6997.33 ± 0.22 / 0.49
VisAFirst image (default)95.32 ± 0.10 / 0.1986.16 ± 0.04 / 0.2685.29 ± 0.16 / 0.3386.98 ± 0.11 / 0.27
Normal95.25 ± 0.11 / 0.1986.08 ± 0.09 / 0.2585.25 ± 0.20 / 0.3886.96 ± 0.14 / 0.31
Anomalous95.33 ± 0.11 / 0.1686.19 ± 0.09 / 0.2585.31 ± 0.16 / 0.3087.00 ± 0.10 / 0.23

In both cases, all metrics stay within 0.15 points of the default. Eq. (7) averages prototype contrast over all patches of the calibration image. Even in an anomalous image, most patches are normal, so defects barely affect which directions are selected.

Revision: The calibration-image comparison is added as Table 13 (Appendix E.8) and referenced in the ablation study (Sec. 4.3).

Reviewer_R6mYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a training-free, zero-shot anomaly detection approach for CLIP, which specifically leverages CLIP's visual projector to extract anomaly patch-level representations.

The authors propose to extract patch level information from selected CLIP visual encoder layers by using CLIP's visual projector on their respective patch tokens and by applying thin SVD to retain more important directions. Then, they propose to invert the polarity of the projector, as they empirically find that normal-abnormal scores tend to be inverted. Finally, they inversely reweight the most important directions to balance their contributions, and they use a single-image calibration step to recover lower-energy directions useful for the task. The resulting per-layer anomaly scores obtained by using the calibrated projectors on the selected layers' patch tokens are then aggregated to get both localized and global anomaly scores.

Authors also propose an additional online inference mode that leverages RareCLIP (a prior work) to build a memory from previous images to better identify anomalies.

Critical strengths
  1. The paper is overall well written and easy to read.

  2. The fact that CLIP’s pretrained projection alone contains useful information for localized anomaly detection appears both novel and useful to me, as avoiding to tune different model components and just leveraging one can be useful for practical applications.

  3. The empirical finding of the need for polarity inversion is very interesting and may motivate further research using this approach, in the context of anomaly detection.

  4. The proposed method is training-free. Still, its performance is competitive with methods that require auxiliary training labeled data, which is also a very practical strong point.

Critical weaknesses
  1. The finding of needing polarity inversion is quite important in the paper, but it is supported only empirically and lacks a principled or intuitive explanation. Since this operation is core to the proposed method, the authors should explain better why this happens.

  2. The analysis regarding the number of lower-energy directions to be scanned may also require more clarification: the authors set the percentage of these directions (pep_e) to 0.1, but if the recovery objective is reliable, there should be no need to limit their number. Also, in Figure 5, it seems that average performance across metrics is maximized at pp = 0.8, meaning that, when prp_r is set to 0.6 (the optimal hyperparameter used in the experiments), pep_e should be larger that 0.1 (the other optimal hyperparameter) if my understanding is correct.

Nonetheless, the aforementioned flaws are not major and do not compromise the overall quality of the approach, hence my judgement stays broadly positive.

Questions for authors
  1. What motivates the polarity inversion beyond working well empirically? An intuitive explanation, not only based on results or observations, would better help clarify this point.
  2. Could the authors clarify why it is necessary to limit the number of lower-energy directions to be scanned? Why all the remaining directions cannot be swept using the Sep objective?
3 comments

Q1. Why is polarity inversion needed?

We appreciate this question. CLIP trains P\mathbf{P} only on the global [CLS] token to match image captions and never constrains individual patch tokens. As prior studies (CLIP Surgery, CLIP-AD) observed, projecting raw ViT patch tokens directly through P\mathbf{P} produces inverted similarity maps: for example, given an image of a cat, [CLS] has high cosine similarity to the prompt "cat", whereas the cat patches themselves get lower similarity to "cat" than background patches. While prior training-free works modify ViT attention layers to fix this inside the encoder, negating P\mathbf{P} directly flips the sign of every patch-to-text cosine similarity at the output without changing the encoder.

The same inversion occurs in anomaly detection, where each patch is scored by its cosine similarity to the "abnormal" prompt minus that to the "normal" prompt (Eq. (2)). Averaged over all test images of the 13 datasets with pixel masks, this difference under P\mathbf{P} is:

Similarity ("abnormal" − "normal") under P\mathbf{P}Layer 12Layer 16Layer 20Layer 24
Normal patches−0.005−0.004−0.010−0.003
Anomalous patches−0.012−0.018−0.027−0.008

On all 13 datasets and in all four layers, anomalous patches score lower than normal ones under P\mathbf{P}. Under −P-\mathbf{P}, every sign flips, so anomalous patches score higher than normal ones.

Revision: Added as Table 5 (Appendix D).

Q2. Why limit the number of scanned directions?

Thank you for your question. Scanning slightly further (ρe=0.2\rho_e=0.2) helps on both datasets (MVTec-AD, VisA), whereas scanning too far hurts (P-AUROC / PRO / I-AUROC / I-AP):

ρe\rho_eMVTec-ADVisA
0 (no recovery)92.5 / 82.9 / 94.1 / 97.291.1 / 81.0 / 84.1 / 86.3
0.1 (default)95.2 / 88.1 / 94.7 / 97.395.3 / 86.1 / 85.1 / 87.0
0.295.7 / 89.5 / 95.1 / 97.596.5 / 87.8 / 85.6 / 87.5
0.394.8 / 81.0 / 94.3 / 97.096.9 / 85.6 / 84.0 / 86.8
0.4 (all remaining)92.3 / 69.1 / 93.0 / 96.595.7 / 77.9 / 82.3 / 85.4

Although every candidate uses the same selection rule, scanning too far into the tail hurts for two reasons:

  1. Label-free selection is imperfect, and tail candidates are weaker. Because Eq. (7) uses no anomaly labels, testing many extra tail candidates inevitably admits some false acceptances. As shown below, candidate quality peaks in the default window (ρe=0.1\rho_e=0.1, directions 461–536), where recovery clearly separates helpful directions from harmful ones, but drops back in the remaining tail (537–768).
  2. Inverse weighting (1/σk1/\sigma_k) amplifies false acceptances in the tail. While σk\sigma_k stays moderate across the default window, it shrinks toward zero in the tail. Because each accepted direction is weighted by 1/σk1/\sigma_k, tail directions receive disproportionately large weights, so even a few false acceptances heavily distort the operator. Conversely, under CLIP's original weights σk\sigma_k, directions with near-zero σk\sigma_k would barely affect the operator.

We verify both points across all test images of the 13 datasets with pixel masks by projecting every patch feature onto each of the 768 singular directions of P\mathbf{P} alone, using the response as the anomaly score, and computing patch AUROC:

Role in 0-CLIPDirections of P\mathbf{P}Countσk\sigma_k rangeSingle-direction AUROCDirections with AUROC > 50
Retained (ρr=0.6\rho_r=0.6)1–4604601.24–0.4751.253%
Candidate pool (ρe=0.1\rho_e=0.1)461–536 (all, before recovery)760.46–0.4454.159%
↳ RecoveredSelected from 461–53642 (avg.)0.46–0.4459.571%
↳ RejectedExcluded from 461–53634 (avg.)0.46–0.4447.544%
Unused tail (ρe>0.1\rho_e > 0.1)537–7682320.44–0.0051.853%

We therefore use a single conservative default (ρe=0.1\rho_e=0.1) across all 15 datasets without per-dataset tuning.

Revision: The single-direction analysis is added as Table 9 (Appendix E.4).


Reviewer_R6mYOpenReview ↗

Response to the Authors

Thank you for the detailed response and for the new results. The new analysis on pep_e mostly addresses my concern on limiting the number of scanned directions.

However, my main concern about providing an explanation about the need of polarity inversion stays mostly unresolved.

I understand that CLIP is not trained to align the individual patch tokens with text embeddings. However, I still lack an explanation about why this particularly results into an inversion. Why should CLIP specifically produce inverted scores, and not just noisy, non-inverted scores for instance? The authors did not answer this question.

For this reason, I am keeping my score of Weak Acceptance.

Thank you for the follow-up. Below is a small experiment that points to a possible cause.

The global ([CLS]) token gets all of its image content as an attention-weighted sum of the patch tokens' attention readouts, i.e., the patch tokens passed through the layer's value and output projections. A patch token and its readout in the next layer have negative cosine similarity for over 98% of patches in every setting (mean −0.23 to −0.51). As no attention layer follows the last layer, layer 23 replaces 24 (11 replaces 12 for ViT-B/16):

BackboneDatasetLayers1st2nd3rd4th
ViT-L/14-336 (paper)MVTec-AD12/16/20/2399.6% (−0.51)99.5% (−0.28)99.4% (−0.37)100.0% (−0.37)
ViT-L/14-336 (paper)VisA12/16/20/2399.6% (−0.51)99.5% (−0.28)99.5% (−0.37)100.0% (−0.38)
ViT-B/16MVTec-AD6/8/10/11100.0% (−0.38)99.2% (−0.24)99.3% (−0.28)98.2% (−0.39)
ViT-B/16VisA6/8/10/11100.0% (−0.38)99.2% (−0.23)99.2% (−0.28)98.2% (−0.40)

The global token thus receives patch content with a largely reversed sign. Because P\mathbf{P} is trained only on the global token, it is fitted to this reversed content; applied directly to patch tokens, it would then give inverted rather than random scores.

Consistent with this, feeding the attention readouts instead of the patch tokens to the original +P+\mathbf{P} removes the inversion: P-AUROC rises from 14.4 / 15.9 (Table 3(a)) to 85.6 / 85.5 on MVTec-AD / VisA. It still trails −P-\mathbf{P} on patch tokens, however (both without truncation or recovery; P-AUROC / PRO / I-AUROC / I-AP):

OperatorMVTec-ADVisA
−P-\mathbf{P} on patch tokens (Table 3(a))85.7 / 84.2 / 92.7 / 96.784.1 / 79.3 / 84.2 / 86.7
+P+\mathbf{P} on attention readouts85.6 / 72.6 / 83.0 / 92.285.5 / 66.7 / 73.3 / 77.3

P-AUROC is similar, but PRO drops by about 12 points and the image-level metrics also drop, possibly because the readout, being a transformed patch token, loses part of the patch's local information.

Why CLIP's value and output projections produce readouts with negative cosine to the patch tokens remains unclear, we leave this to future work.

Reviewer_XUknOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that existing CLIP-based zero-shot anomaly detection methods often rely on auxiliary training data for patch-level alignment, which can limit generalization to unseen domains.They claim that CLIP's frozen visual projector already contains anomaly-relevant information that can be exploited without additional training. To this end, they propose 0-CLIP, which constructs a patch-to-text anomaly operator through polarity reversal, singular-direction re-weighting, and selective direction recovery using an unlabeled test image.

Critical strengths

Training-free approach with broad empirical evaluation.

The paper presents a training-free approach that constructs an anomaly detection operator from CLIP's frozen projector using spectral decomposition. It demonstrates promising results across diverse industrial and medical benchmarks, although the comparison with relevant state-of-the-art methods could be more comprehensive.

Critical weaknesses

1. Limited Explanation of the Spectral Transformation.

The proposed spectral transformations achieve promising results, and Table 3 provides useful ablations. However, the relationship between the projector's singular directions and anomaly-relevant information remains insufficiently explained. In particular, it is unclear why inverse singular-value weighting should consistently improve anomaly discrimination rather than amplify directions containing irrelevant variation. A more direct analysis of the selected directions and their responses to normal and anomalous patches would strengthen the conceptual justification of the method.

2. Insufficient justification of the direction-recovery objective.

The direction-recovery criterion in Eq. (7) maximizes the average abnormal–normal text similarity margin over patches from a single unlabeled calibration image. However, increasing this margin does not necessarily improve the separation between normal and anomalous patches, particularly when the calibration image contains only normal regions.

Although Table 3(c) compares alternative objectives and demonstrates the empirical benefit of the proposed criterion, it remains unclear how the recovered directions depend on the anomaly content of the calibration image. An analysis separating normal-only and anomaly-containing calibration images would help establish the reliability of this design choice.

Additional concern (Reference accuracy).

Some references contain incorrect paper titles. For example, DRAEM (Zavrtanik et al., 2021) is cited with "Reconstruction Model" instead of "Reconstruction Embedding." Additionally, some papers are cited as arXiv preprints despite having been formally published, suggesting that the bibliographic information could be updated.

Questions for authors

1. Interpretation of the Spectral Directions. Could the authors explain why inverse singular-value weighting improves anomaly discrimination? In particular, how do the retained and recovered directions differ in their responses to normal and anomalous patches?

2. Reliability of Unlabeled Calibration. How sensitive is the direction-recovery procedure to whether the first unlabeled calibration image is normal or anomalous? Could the authors report the performance separately for these two conditions?

3. Evaluation of Small Anomalies. Could the authors additionally report pixel-level AUPR and F1-max? Given the patch-based nature of CLIP features, these metrics would help assess fine-grained localization performance, particularly for small anomalies where false positives may significantly affect precision.

Minor comments on presentation.

Some figures could benefit from clearer explanations and annotations. In particular, Figure 1 could more clearly distinguish the training and inference procedures of the compared methods. Figures 3 and 4 would also benefit from more explicit descriptions of the text anchors, heatmap colors, and plot axes to improve readability.

4 comments

W1/Q1. Why does 1/σk1/\sigma_k help, and how do retained and recovered directions respond to normal and anomalous patches?

Thank you for this question. In short, σk\sigma_k measures how strongly CLIP uses a direction for global image–caption matching, not how useful it is for anomaly detection:

  • Why 1/σk1/\sigma_k helps: In P\mathbf{P}, the top 5% of directions (1–38) carry the largest weights σk\sigma_k yet slightly hurt detection (mean AUROC < 50), whereas the remaining retained directions (39–460) help on average. Replacing σk\sigma_k with 1/σk1/\sigma_k downweights this harmful leading group and upweights the helpful remainder. Because we already drop the tail where σk\sigma_k is near zero, 1/σk1/\sigma_k stays moderate and does not amplify noise.
  • Retained vs. recovered: Recovered directions (from 461–536) separate normal and anomalous patches much more clearly than retained ones, and recovery filters out candidates that hurt detection.

We verify both points first per direction on all test images of the 13 datasets with pixel masks, and then as a combined operator on MVTec-AD and VisA.

1. Individual directions. For each of the 768 singular directions of P\mathbf{P}, we score patches by their projection onto that direction alone and compute patch AUROC:

Role in 0-CLIPDirections of P\mathbf{P}Countσk\sigma_k rangeSingle-direction AUROCDirections with AUROC > 50
Retained (top 5%)1–881.24–0.6548.846%
Retained (top 5%)9–38300.64–0.6048.948%
Retained (rest)39–76380.60–0.5851.352%
Retained (rest)77–2301540.58–0.5351.453%
Retained (rest)231–4602300.53–0.4751.553%
Candidate pool461–536 (all, before recovery)760.46–0.4454.159%
↳ RecoveredSelected from 461–53642 (avg.)0.46–0.4459.571%
↳ RejectedExcluded from 461–53634 (avg.)0.46–0.4447.544%
Unused tail537–7682320.44–0.0051.853%

Grouping all 768 directions into 12 equal bins shows the overall trend across the spectrum:

Directions1–6465–128129–192193–256257–320321–384385–448449–512513–576577–640641–704705–768
σk\sigma_k1.24–0.580.58–0.560.56–0.540.54–0.520.52–0.500.50–0.490.49–0.470.47–0.450.45–0.420.42–0.080.07–0.020.02–0.00
Mean AUROC49.850.652.150.651.851.852.253.151.451.752.552.7

This confirms both points: among retained directions, only the top 5% average below 50; recovered directions score well above retained ones, whereas rejected candidates fall below chance.

2. Combined operator. We then combine directions into a single operator as in 0-CLIP, varying only the direction set or weights (P-AUROC / PRO on MVTec-AD and VisA, seed 0):

Direction setWeightsMVTec-ADVisA
Retained + recoveredσk\sigma_k94.0 / 84.991.7 / 82.0
Retained + recoveredequal94.8 / 87.193.9 / 84.4
Retained + recovered (0-CLIP)1/σk1/\sigma_k95.2 / 87.895.4 / 86.2
↳ drop top 5%σk\sigma_k95.1 / 87.494.1 / 84.0
↳ drop top 5%1/σk1/\sigma_k95.4 / 88.096.3 / 86.9
Retained only1/σk1/\sigma_k92.5 / 82.991.1 / 81.0
Retained + random*1/σk1/\sigma_k93.6 / 85.692.6 / 83.0
Retained + rejected1/σk1/\sigma_k89.2 / 79.883.1 / 75.5

*Count-matched from 461–536.

  • Downweighting the harmful leading group: Under σk\sigma_k weights, dropping the top 5% brings a large gain, confirming that these dominant directions drag the operator down. Rather than adding a separate cutoff hyperparameter, 1/σk1/\sigma_k achieves an even larger gain smoothly.
  • Upweighting the helpful remainder: Even without the top 5%, 1/σk1/\sigma_k still beats σk\sigma_k on both metrics by giving larger weights to the more informative lower-σk\sigma_k directions. Together, the two effects explain the steady gain across σk→equal→1/σk\sigma_k \to \text{equal} \to 1/\sigma_k (consistent with Table 3(b)).
  • Effect of recovery: Adding recovered directions substantially improves over retained ones alone and beats count-matched random candidates, whereas adding rejected ones hurts.

In summary, 0-CLIP turns CLIP's P\mathbf{P} into an effective anomaly operator by combining many weak directions: 1/σk1/\sigma_k rebalances the spectrum and recovery adds informative ones from a single unlabeled image, with no masks or training.

Revision: Sec. 4.3 now states this conclusion, with the single-direction and combined-operator results in Table 9 (Appendix E.4).


W2/Q2. Does recovery depend on whether the calibration image is normal or anomalous?

Thank you. It doesn't. In each seed's shuffled test stream, we calibrated every class once on its first normal image and once on its first anomalous image, then scored the entire stream as usual (5 seeds mean):

DatasetCalibration imageP-AUROCPROI-AUROCI-AP
MVTec-ADFirst image (default)94.91±0.2087.30±0.4794.44±0.3297.35±0.21
MVTec-ADNormal94.92±0.1487.41±0.2894.40±0.3397.34±0.23
MVTec-ADAnomalous94.91±0.2087.28±0.4794.41±0.3397.33±0.22
VisAFirst image (default)95.32±0.1086.16±0.0485.29±0.1686.98±0.11
VisANormal95.25±0.1186.08±0.0985.25±0.2086.96±0.14
VisAAnomalous95.33±0.1186.19±0.0985.31±0.1687.00±0.10

Recovery keeps a candidate direction only if adding it raises the calibration patches' average cosine margin between the abnormal and normal prompts (Eq. (7)). Even on a normal image, directions carrying irrelevant variation or the wrong polarity reduce this cosine margin and get filtered out, matching W1 where recovered directions reach 59.5% AUROC versus 47.5% for rejected ones. Moreover, because this margin averages over all patches and most patches are normal even in an anomalous image, defects barely affect which directions are selected.

Revision: Added as Table 13 (Appendix E.8) and referenced in the ablation study (Sec. 4.3).


Q3. Pixel AUPR and F1-max.

Thank you for the suggestion. Below are pixel AUPR / F1-max (%) across all 13 datasets with pixel masks (†^\dagger online; ‡^\ddagger labeled memory bank):

DatasetAnomCLIPAA-CLIPRareCLIP$^\dagger$MRAD$^\ddagger$FB-CLIPWinCLIP0-CLIP0-CLIP$^\dagger$
MVTec-AD34.5 / 39.146.9 / 47.966.6 / 64.643.3 / 45.344.6 / 46.018.1 / 24.941.8 / 47.165.7 / 64.0
VisA21.3 / 28.323.9 / 29.047.4 / 51.025.6 / 32.128.4 / 35.45.4 / 9.123.6 / 30.447.0 / 50.7
BTAD45.5 / 49.752.5 / 54.863.2 / 61.543.7 / 47.746.5 / 50.88.4 / 12.747.9 / 52.663.7 / 62.1
MPDD28.9 / 34.226.4 / 28.536.1 / 38.034.8 / 39.330.5 / 33.913.4 / 15.027.4 / 34.034.0 / 36.0
DTD-Syn63.8 / 63.162.5 / 61.079.8 / 74.662.3 / 61.763.8 / 60.815.9 / 23.556.0 / 56.978.8 / 73.7
SDD41.0 / 42.517.6 / 33.828.4 / 35.538.6 / 42.431.8 / 38.18.6 / 14.642.3 / 44.641.8 / 46.0
DAGM56.5 / 58.955.2 / 56.959.9 / 60.553.7 / 55.855.0 / 55.78.3 / 15.060.7 / 62.068.6 / 67.6
KSDD257.0 / 59.863.8 / 63.168.1 / 66.460.8 / 60.866.3 / 65.315.5 / 23.644.9 / 52.867.0 / 66.0
Real-IAD28.0 / 35.329.8 / 35.433.0 / 39.326.5 / 33.828.3 / 34.33.5 / 8.126.0 / 33.733.7 / 40.3
Liver CT3.6 / 6.911.1 / 20.36.9 / 13.25.7 / 10.57.4 / 13.66.1 / 12.84.5 / 8.87.0 / 13.7
Brain MRI57.1 / 53.141.9 / 43.243.1 / 52.053.5 / 50.751.4 / 49.611.9 / 20.331.1 / 36.741.6 / 51.8
Retina RESC48.4 / 47.164.6 / 58.459.9 / 55.746.2 / 44.651.8 / 48.717.1 / 24.522.5 / 28.354.7 / 52.7
TN3K45.7 / 47.836.5 / 41.540.0 / 45.442.3 / 43.739.0 / 42.222.8 / 32.036.4 / 42.634.9 / 41.5
Average40.9 / 43.541.0 / 44.148.6 / 50.641.3 / 43.741.9 / 44.211.9 / 18.135.8 / 40.849.1 / 51.2

Overall, 0-CLIP$^\dagger$ achieves the highest average AUPR and F1-max (49.1 / 51.2), while offline 0-CLIP triples training-free WinCLIP in AUPR (35.8 vs. 11.9). Together with its high PRO (82.6, Table 1), offline 0-CLIP recalls small defects well, while its lower AUPR comes from mild background responses that supervised baselines suppress with ground-truth mask training and 0-CLIP$^\dagger$ suppresses with unlabeled online rarity.

Revision: Added as Table 7 (Appendix E.2) and referenced in Sec. 4.2.


Figures & References.

Thank you for the careful reading. Revision:

  • Fig. 1: Clearly separates training from inference.
  • Fig. 3: Explicitly defines ta\mathbf{t}_a in the caption as the averaged CLIP text embedding of the "abnormal" prompts.
  • Fig. 4: Added a colorbar and y-axis.
  • References: Corrected the DRAEM title and updated arXiv preprints to their published venue versions.

Thank you for the detailed rebuttal and additional experiments. My main concerns have been partially addressed, so I have updated my recommendation to Weak Accept.

A few concerns remain:

  1. I agree with R6mY that the necessity of polarity inversion remains insufficiently explained.
  2. It would be helpful to include category-wise quantitative results.
  3. Although Figure 4 has been revised, it remains difficult to read without substantial zooming (around 500%). Further improvements would enhance the presentation.

Thank you for reading our rebuttal carefully. Below we address the remaining points:

1. Necessity of polarity inversion.

As we also shared in our follow-up response to Reviewer R6mY, below is a small experiment that points to a possible cause.

The global ([CLS]) token gets all of its image content as an attention-weighted sum of the patch tokens' attention readouts, i.e., the patch tokens passed through the layer's value and output projections. A patch token and its readout in the next layer have negative cosine similarity for over 98% of patches in every setting (mean −0.23 to −0.51). As no attention layer follows the last layer, layer 23 replaces 24 (11 replaces 12 for ViT-B/16):

BackboneDatasetLayers1st2nd3rd4th
ViT-L/14-336 (paper)MVTec-AD12/16/20/2399.6% (−0.51)99.5% (−0.28)99.4% (−0.37)100.0% (−0.37)
ViT-L/14-336 (paper)VisA12/16/20/2399.6% (−0.51)99.5% (−0.28)99.5% (−0.37)100.0% (−0.38)
ViT-B/16MVTec-AD6/8/10/11100.0% (−0.38)99.2% (−0.24)99.3% (−0.28)98.2% (−0.39)
ViT-B/16VisA6/8/10/11100.0% (−0.38)99.2% (−0.23)99.2% (−0.28)98.2% (−0.40)

The global token thus receives patch content with a largely reversed sign. Because P\mathbf{P} is trained only on the global token, it is fitted to this reversed content; applied directly to patch tokens, it would then give inverted rather than random scores.

Consistent with this, feeding the attention readouts instead of the patch tokens to the original +P+\mathbf{P} removes the inversion: P-AUROC rises from 14.4 / 15.9 (Table 3(a)) to 85.6 / 85.5 on MVTec-AD / VisA. It still trails −P-\mathbf{P} on patch tokens, however (both without truncation or recovery; P-AUROC / PRO / I-AUROC / I-AP):

OperatorMVTec-ADVisA
−P-\mathbf{P} on patch tokens (Table 3(a))85.7 / 84.2 / 92.7 / 96.784.1 / 79.3 / 84.2 / 86.7
+P+\mathbf{P} on attention readouts85.6 / 72.6 / 83.0 / 92.285.5 / 66.7 / 73.3 / 77.3

P-AUROC is similar, but PRO drops by about 12 points and the image-level metrics also drop, possibly because the readout, being a transformed patch token, loses part of the patch's local information.

Why CLIP's value and output projections produce readouts with negative cosine to the patch tokens remains unclear, we leave this to future work.

2. Category-wise quantitative results.

Thank you for the suggestion. We have added the five-seed per-category results (mean ± std across P-AUROC, PRO, I-AUROC, and I-AP for both offline 0-CLIP and online 0-CLIP$^\dagger$) on MVTec-AD (Table 14) and VisA (Table 15) in Appendix E.9 of the revised manuscript.

3. Readability of Figure 4.

Thank you for pointing this out. We have further simplified Figure 4 in the revised manuscript by keeping a single visual example in the top row, making the anomaly maps, colorbar, and plot labels readable at normal viewing scale.

Comments on the paper

1 comment

Dear Reviewers, ACs, SACs and PCs,

We sincerely appreciate the time and effort you have dedicated to reviewing and handling our manuscript, and we thank the ICLR 2027 Program Committee for organizing the review process.

We propose 0-CLIP, which turns CLIP's frozen visual projector directly into patch-to-text operators for zero-shot anomaly detection via polarity reversal, spectral reweighting, and single-image calibration. As the reviewers highlighted, the paper is well-written and easy to follow (R2, R3), leveraging the frozen projector alone is novel and useful (R2), and the method is simple (R3), training-free (R1, R2), competitive with auxiliary-trained methods (R2), and broadly evaluated across industrial and medical benchmarks (R1, R3).

We greatly appreciate your constructive feedback. In response, we have carefully revised the manuscript as follows (with new results in the appendix and brief pointers in the main text):

  • Spectral directions (R1, R2). A single-direction analysis on 13 datasets and a combined-operator analysis on MVTec-AD and VisA explain why 1/σk1/\sigma_k weighting and direction recovery help, and why the scan range is limited (Table 9, Appendix E.4; Sec. 4.3).
  • Polarity inversion (R2). Under CLIP's original P\mathbf{P}, anomalous patches have a lower mean prototype margin than normal ones on all 13 datasets and four layers (Table 5, Appendix D).
  • Calibration image (R1, R3). Calibrating on a normal or an anomalous image changes all metrics on MVTec-AD and VisA by less than 0.15 points (Table 13, Appendix E.8; Sec. 4.3).
  • ReMem (R3). Comparison on 15 datasets, where 0-CLIP$^\dagger$ is higher on average (Table 8, Appendix E.3).
  • Pixel AUPR and F1-max (R1). Results on 13 datasets (Table 7, Appendix E.2; Sec. 4.2).
  • AdaCLIP (R3). Applying our construction to its trained patch heads lowers performance, as they already have the correct polarity (Table 11, Appendix E.6; Sec. 5).
  • Clarifications. Per-layer operators in Sec. 3.1 (R3); AnomalyVFM, VisualAD and FoundAD in the Introduction (R3); clearer Figs. 1, 3 and 4 (R1); corrected references (R1, R3).

We sincerely hope these updates address your concerns and help better convey the findings of 0-CLIP to the ICLR community. We look forward to further discussion.

Thank you very much,
The Authors


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.