acceptodds
Under review as a conference paper at ICLR 2027

Beyond Cosine Similarity: Uncovering Anomaly Information in CLIP's Raw Embedding Geometry

Abstract

Extensive experiments following the image-text cosine similarity pre-training paradigm of Contrastive Language-Image Pre-training (CLIP) indicate that CLIP excels at image classification rather than fine-grained anomaly detection; consequently, a prevailing consensus holds that CLIP is anomaly-agnostic. Existing studies have concentrated on downstream training to better adapt CLIP for anomaly detection tasks; in other words, image-text cosine similarity is the computational foundation underlying CLIP-based anomaly detection. In this paper, we investigate the directional information about anomalies in CLIP's raw embedding space and the limitations of cosine similarity with the idea of computing geodesics distance for anomaly shifts. We investigate the raw representation of CLIP and find that anomaly information is already encapsulated in CLIP's raw embedding space, which is mainly formed by class-dependent directional variation. The key reason why CLIP cannot recognize the normality/abnormality in images is that the now-ubiquitous cosine similarity ignores the anisotropy of the normal/abnormal distribution. We demonstrate this through controlled geometric analysis, and use a top-down statistical method as the simplest parameter-free update to uncover the latent anomaly information, making this process interpretable. In extensive experiments, CLIP's latent anomaly information substantially improves AD performance over image-text cosine similarity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
CLIP, Embedding Geometry, Anomaly Detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_kKCXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper explores the abnormal information in the original embedding space of CLIP, apart from the traditional cosine similarity of text and image. This paper models the normal embeddings under class conditions using Gaussian distribution and uses Mahalanobis distance for anomaly scoring. Additionally, through geometric analysis based on displacement decomposition and stretch ratio, this paper explains the advantages of anisotropic metrics.

Critical strengths
  • This paper explores the abnormal discrimination information contained in the pre-trained CLIP representations from the perspective of statistical geometry, providing an alternative solution to the previous methods based on prompts or feature enhancement.
  • The proposed Mahalanobis scoring method does not require the update of the backbone network and achieves satisfactory results on multiple benchmarks..
Critical weaknesses
  • Insufficient comparison with strong few-shot baselines. This method estimates the class conditional statistics based on normal reference samples. However, the previous baseline methods usually rely on cosine similarity, which means the supervisory information is inconsistent. Therefore, the comparison might not be fair.
  • The ablation experiments show that the performance improvement mainly stems from anisotropic weighting rather than radial information.
  • This paper proposes that anomalies are amplified in the low-variance direction, but the reported stretch ratios in the experiments are less than 1, which instead indicates that the model is more inclined towards the high-variance direction.
Questions for authors
  • Under the same level of supervision, how does this method perform compared to the strong baseline model?
  • If the CLIP features are normalized by L2 and the covariance is re-estimated to calculate the Mahalanobis distance, what kind of performance will be obtained?
  • How can the "tight-direction" assumption be explained in relation to the observed phenomenon of "stretch ratios less than 1"?
  • I cannot agree that the estimated covariance measure is regarded as the "intrinsic geometric structure" of CLIP. Why is it not only a statistical measure for a specific task?
Reviewer_bCXNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper revisits the common view that CLIP is relatively insensitive to visual anomalies and argues that anomaly information is already encoded in CLIP’s raw, unnormalized embedding space. The paper attributes the weakness of conventional CLIP-based anomaly detection partly to cosine similarity, which discards class-dependent anisotropic structure through L2 normalization. Thus, the paperp proposes to model normal embeddings with a Gaussian distribution and use the inverse covariance as a class-specific local metric, leading to a Mahalanobis-distance anomaly score. Experiments on industrial and medical anomaly-detection datasets show substantial improvements over cosine similarity and several existing CLIP-based methods.

Critical strengths
  • The propose method is simple, straightforward, and interpretable.
  • The proposed method demonstrates that substantial anomaly information can be extracted directly from frozen CLIP embeddings without optimizing additional model parameters.
  • The paper evaluates on a relatively broad collection of industrial and medical datasets, and it openly reports failure cases such as BrainMRI, where the Gaussian/local-metric assumptions appear less appropriate.
Critical weaknesses
  • Potentially unfair experimental comparison
    • The proposed method estimates class-specific statistics using normal training images, while several methods in Table 3 are primarily zero-shot approaches. It is therefore difficult to determine whether the gains come from the proposed metric or from access to additional normal data. Comparisons with few-shot methods under matched normal-sample budgets would provide a more meaningful evaluation.
  • Insufficient comparison with recent and closely related work
    • The evaluation omits several relevant few-shot/training-free anomaly detection methods. In particular, CLIP-Core [A] is highly relevant because it also performs few-shot CLIP-based anomaly detection and is already discussed in the paper (but not experimentally compared against the proposed method in the paper). Furthermore, ANoCo [B] provides a recent training-free alternative that explicitly models the structure of the normal feature manifold, which has not been discussed or compared against in the paper. Other recent CLIP-based few-shot methods such as DE-CLIP [C], and AnoPLe [D] are also not discussed or compared against. Comparisons under matched normal-sample budgets seem necessary to establish competitiveness relative to the current state-of-the-art methods.
  • Unclear novelty relative to established covariance-based anomaly detection
    • In terms of the algorithm in the proposed method, the proposed method estimates the mean and covariance of normal frozen features and uses Mahalanobis distance as an anomaly score. This general formulation is well established: Lee et al. [E] use class-conditional Gaussian feature distributions with Mahalanobis scoring for abnormal/OOD detection; Rippel et al. [F] fit a multivariate Gaussian to pretrained deep features and use Mahalanobis distance for visual anomaly detection; and PaDiM [G] similarly models patch-level pretrained features with multivariate Gaussian distributions for anomaly detection and localization. Given these prior works, together with the more directly related CLIP-Core [A], it is unclear how much of the contribution is a new detection method versus a new interpretation of why covariance-aware scoring works particularly well in CLIP’s raw embedding space.
  • The geometric interpretation appears stronger than what is theoretically established
    • The paper assumes that the local metric is Gc=Σc−1G_c=\Sigma_c^{-1}, after which the connection between the geodesic and Mahalanobis distance follows naturally. However, it is not clear that this metric is actually induced by CLIP’s contrastive objective rather than being a useful statistical choice. Stronger theoretical or empirical evidence seems necessary to support the claim about CLIP’s intrinsic embedding geometry.
  • Limited evidence that the observation is specific to CLIP
    • Since the proposed method only uses frozen image embeddings and normal-class statistics, similar improvements may also arise for supervised or self-supervised visual representations. Comparisons with non-CLIP backbones using the same covariance-based scoring would help determine whether the identified phenomenon is specifically associated with CLIP’s contrastive embedding geometry.

[A] Xu et al., CLIP-Core: A Few-Sample Anomaly Detection Method for Surface Defects. IEEE Access 2026.

[B] Seo et al., Anomaly as Non-Conformity via Training-Free Graph Laplacian Energy Minimization. CVPR 2026.

[C] Zhang et al., DE-CLIP: Few-Shot Anomaly Detection via Difference-Guided Embedding Editing. ACL 2026.

[D] Lee et al., Bidirectional Multimodal Prompt Learning with Scale-Aware Training for Few-Shot Multi-Class Anomaly Detection. CVPR 2026.

[E] Lee et al., A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. NeurIPS 2018.

[F] Rippel et al., Modeling the Distribution of Normal Data in Pre-Trained Deep Features for Anomaly Detection. ICPR 2020.

[G] Defard et al., PaDiM: a Patch Distribution Modeling Framework for Anomaly Detection and Localization. ICPR 2020, International Workshop on Industrial Machine Learning.

Questions for authors
  • Could the authors clarify the supervision/data regime for the methods in Table 3 and provide comparisons under the same number of normal training samples?
  • Could the authors compare against recent and closely related few-shot/training-free methods, particularly CLIP-Core [A] and ANoCo [B]? Comparisons with recent CLIP-based methods such as DE-CLIP [C] and AnoPLe [D] would also help establish the competitiveness of the proposed approach.
  • Given prior methods that already fit Gaussian distributions to pretrained features and use Mahalanobis distance for anomaly detection [E–G], what is the main novelty of the proposed method beyond applying this established formulation to raw CLIP features?
  • What evidence establishes that Gc=Σc−1G_c=\Sigma_c^{-1} represents the intrinsic local metric induced by CLIP training rather than simply a useful covariance-based metric for anomaly detection?
  • Would the same Mahalanobis scoring mechanism provide similar gains when applied to non-CLIP pretrained visual representations?

For more details, please refer to the weakness section above.

Reviewer_XCT6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why CLIP performs poorly on anomaly detection when using cosine similarity. The authors argue that CLIP already contains anomaly information in its raw embeddings, but cosine similarity fails to capture the anisotropic feature distribution. They use Mahalanobis distance with class-specific covariance to detect anomalies without additional model training. The authors also provide geometric analysis and experiments on 11 datasets. The results show strong improvements on several benchmarks.

Critical strengths
  1. The paper studies CLIP anomaly detection from a different perspective. Instead of learning new prompts or features, the authors explore the geometry of existing embeddings. This direction is interesting and has some practical value.

2.The authors evaluate the method on 11 datasets from industrial and medical domains. The method achieves strong results on MVTec, VisA and OCT2017. The improvements over the cosine similarity baselines are quite large on several datasets.

Critical weaknesses
  1. The novelty of the method is quite limited. Mahalanobis distance with class-specific covariance is already a well-known approach for anomaly detection. The authors also mention CLIP-Core, which already uses Mahalanobis distance on CLIP features. However, they dont provide direct comparisons with this method. The paper needs to explain more clearly what is actually new beyond applying this existing metric to raw CLIP embeddings.

  2. The baseline comparisons are not fully fair. The proposed method uses normal training images to estimate the mean and covariance, while several competing CLIP methods in Table 3 target zero-shot anomaly detection. These methods dont have access to the same normal samples. The authors should compare against strong few-shot and normal-only anomaly detection methods under the same training data setting. Otherwise, the reported improvements are hard to judge.

  3. The geometric theory is not very convincing. The authors assume a constant metric and directly define it using the inverse covariance. This basically leads to the standard Mahalanobis distance. It doesnt really prove that this metric captures the intrinsic geometry learned by CLIP. The authors also dont directly verify the Gaussian assumption on the embedding distributions. The poor result on BrainMRI further questions how general these assumptions are. The current theory feels more like a reinterpretation of a standard statistical method.

Questions for authors

Please address the weaknesses.

Reviewer_qej9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that CLIP is not intrinsically anomaly-agnostic: its apparent failure on anomaly detection (AD) is an artifact of the image-text cosine-similarity scorer used throughout the CLIP-AD literature. Normal/abnormal information is claimed to already be present in the raw (unnormalized) image embedding, which L2 normalization destroys by projecting onto an isotropic hypersphere, and to be recoverable by a class-conditional Mahalanobis distance.

Method: per class, fit a multivariate Gaussian to NN normal training images (ridge- regularized covariance) and score by Mahalanobis distance to the class mean — no training, no prompts, frozen encoder. The stated motivation is Riemannian: contrastive loss "curves" the embedding space, but under a constant metric the geodesic distance collapses to Mahalanobis (Appendix A). A second analysis defines a stretch ratio on the covariance eigenbasis, concluding that anomalies lie in "loose" (high-variance) directions.

Validation: image-level AUROC/AP/F1-max on 11 datasets (8 industrial, 3 medical), three CLIP backbones, a few-shot sweep, a ridge-strength table, and a radial/anisotropic ablation.

Critical strengths

(1) A clean, well-controlled ablation that isolates the source of the gain. Table 6 toggles the radial displacement and the class-conditional precision matrix independently, and the result is unambiguous: anisotropy contributes +8.86/+10.64 AUROC (MVTec/VisA) while the radial component contributes <0.2<0.2 points on every metric. This matters because it rules out the most obvious alternative explanation — that the gain comes from the embedding norm, which L2 normalization discards — and it converts a vague geometric narrative into a falsifiable, tested claim. This is the single strongest piece of evidence in the paper and it should be foregrounded rather than buried in Section 4.3.

(2) A practically attractive, training-free recipe with unusually broad coverage. Fitting a per-class Gaussian and computing a Mahalanobis score requires no training, no prompt engineering, and no gradient access; the authors report running everything on a single laptop GPU. Combined with results on 11 datasets spanning industrial inspection and medical imaging, plus a few-shot sweep down to N=4N=4, this is a genuinely useful data point for practitioners who want a strong, cheap baseline before investing in prompt-learning methods. This is consequential because it suggests the field's default zero-shot prompt ensemble may be leaving a large amount of performance on the table for a much smaller engineering cost.

Critical weaknesses

(1) The theoretical core does not support the paper's central claim, and the microscopic mechanism is internally contradictory.

  • The geodesic derivation is vacuous. Appendix A assumes a constant metric G(x)≡GcG(x)\equiv G_c, then derives that the geodesic is a straight line and the distance is the Mahalanobis distance. But that is what a constant-metric space is: an Rm\mathbb{R}^m with a fixed quadratic form. The derivation therefore does not "reduce" a geodesic to a Mahalanobis distance — it assumes the flatness that makes them equal. The paper's premise (contrastive loss curves the space, so geodesics are curved) is the exact negation of its own assumption. If the metric is constant, there is no curvature and no geodesic content, and the correct statement is the plain one: "we model the normal class as a Gaussian and use its Mahalanobis distance." Nothing in the method or experiments tests whether G(x)G(x) actually varies. This is consequential because the geodesic framing is the paper's stated conceptual contribution (Contribution 2), and it is doing no work.
  • The stated mechanism is contradicted by the reported result. Section 3.4 hypothesizes (p. 5) that "the anomalous directions of different foreground objects tend to fall on tight directions, enabling the Mahalanobis distance to assign larger weights γk\gamma_k." The experiments then find the opposite: r(δ)<1r(\delta)<1 across all backbones (Table 4), i.e. anomaly displacements concentrate in loose, high-variance directions. This is not reconciled anywhere. Worse, the consequence runs backwards: if anomalies live in low-γ\gamma directions, Mahalanobis assigns them smaller weights γkck2\gamma_k c_k^2 — it suppresses the anomaly signal relative to Euclidean distance. So the paper's own measurement undercuts the sentence immediately following it ("Mahalanobis ... effectively suppresses normal variations while amplifying anomalous signals"). The explanation that would actually be consistent — normal noise also concentrates in loose directions, so down-weighting them removes noise — is never stated or tested.
  • No control for the stretch ratio. For any Gaussian, deviations from the mean concentrate in high-variance directions, so normal samples will also exhibit r<1r<1. The paper reports r(δ)r(\delta) only for anomaly displacements and compares against the isotropic baseline r=1r=1. Without r(δ)r(\delta) for normal test samples, the Table 4 finding may distinguish nothing at all.
  • The Gaussian assumption is never tested. The "maximum entropy" and "predictive power test" justifications are not evidence: Mahalanobis scoring is optimal for any elliptical distribution, so good AD performance cannot confirm Gaussianity. A direct test (e.g., Mardia's kurtosis/skewness) or a robustness check against heavy tails is absent.

All of this is consequential because it means the paper's interpretive contribution — why the method works — is not established, leaving only "Mahalanobis scores work well on CLIP features."

(2) The contribution is not adequately distinguished from prior work, and the headline comparisons are confounded in the authors' favour.

  • Novelty. Class-conditional Gaussian + Mahalanobis on frozen pretrained features is a long-standing recipe: PaDiM (patch-level Gaussians + Mahalanobis), Gaussian-AD (multivariate Gaussian on pretrained deep features), Mahalanobis++ (feature normalization before MD), and — most directly — CLIP-Core (Xu et al., 2026), which the paper cites and which fits a normal memory bank on CLIP patch features and scores with Mahalanobis. Whitened CLIP (Betser et al., 2025), also cited, is essentially the whitening transform Σ−1/2\Sigma^{-1/2} that underlies the same metric. None of these is compared numerically anywhere in the paper. Without that comparison, the claim that this is a new way to "release" latent anomaly information is not substantiated.
  • The headline numbers rest on a strawman. Section 4.2 advertises "+53.1% on BTAD and +44.8% on OCT2017," which are the gains over the image-text prompt baseline (41.7→94.8 and 51.0→95.8). But the paper's own image-only cosine control in the same table gives 91.5 and 89.9, so the gain attributable to the proposed metric is +3.3 and +5.9. The advertised figures are against a baseline (zero-shot prompts "a photo of {class} / defective {class}") that no contemporary CLIP-AD method uses. This also undercuts the paper's motivating premise: the image-only column (87.4 MVTec, 95.2 DAGM, 94.2 DTD) shows CLIP's raw embeddings are not anomaly-agnostic under a trivial linear scorer, which means the failure being diagnosed belongs to text prompting, not to cosine similarity or anisotropy per se.
  • Few-shot vs. zero-shot. The proposed method requires NN labeled normal training images per class, whereas WinCLIP, AnomalyCLIP, and AdaCLIP in Table 3 are zero-shot. Comparing a few-shot method to zero-shot methods and reporting it as competitive is not a like-for-like comparison; the few-shot counterparts (WinCLIP+, CLIP-Core) are not included. And even on these terms the method loses on 5 of 11 datasets — DAGM (97.4 vs. Crane 99.2), KSDD (86.2 vs. 97.8), KSDD2 (93.6 vs. AdaCLIP 95.6), BrainMRI (84.5 vs. 95.4), and HeadCT (89.7 vs. 95.3) — while Table 3 only discusses the wins. Notably, on HeadCT the proposed method is also worse than the image-text cosine baseline in Table 2 (89.7 vs. 91.8), which is never mentioned.
  • No variance reported on Tables 2 and 3, despite claims of robustness; Table 6 reports std, so the machinery exists.
  • Only image-level evaluation. The CLIP-AD literature the paper positions against is predominantly evaluated on pixel-level localization (AUROC/PRO). Reporting only image-level AUROC makes the method incomparable to its baselines and flatters a first-moment-plus-covariance scorer, which is much easier at image level.
Questions for authors
  1. Stretch ratio control. What is r(δ)r(\delta) for normal test samples? If it is also well below 1 — as it must be for Gaussian data — then Table 4's finding is not anomaly-specific. Please report it.
  2. Direction of the mechanism. Section 3.4 hypothesizes anomalies fall on tight directions but measures loose ones, and Mahalanobis down-weights loose directions. Please state which is the paper's position and reconcile it with the claim that Mahalanobis amplifies the anomaly signal.
  3. Estimation with N≪mN\ll m. With m=768m=768 and NN in the tens-to-hundreds, Σ^craw\hat\Sigma^{\text{raw}}_c is rank-deficient and Σ^c−1\hat\Sigma_c^{-1} amplifies the regularization. How much of the gain survives proper shrinkage (Ledoit–Wolf) or PCA-whitening at matched effective rank? And why is λ=0.01\lambda=0.01 chosen when Table 1 shows λ=0.001\lambda=0.001 is better (96.68 vs. 96.39)?
  4. Is anything CLIP-specific? The method uses only the image covariance; the text embeddings that the introduction says define the "T+/T− axis" play no role in Eq. 5. Does the same procedure work on a non-CLIP frozen backbone (e.g., DINOv2, ResNet)? If so, what in the analysis depends on CLIP's geometry?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.