acceptodds
Under review as a conference paper at ICLR 2027

CrossCLIP: Bidirectional Cross-Modal Interaction with Instance-Adaptive Frequency Modulation for Anomaly Detection

Abstract

Universal anomaly detection aims to identify anomalies in unseen object categories and domains without target-specific training data. Recent methods leverage pretrained vision-language models, such as CLIP, to align visual and textual representations of normality and abnormality. However, they typically adapt the two modalities independently or rely on unidirectional interaction, limiting the use of complementary cross-modal information. To address this limitation, we propose CrossCLIP, which enables bidirectional cross-modal interaction through three adapters: a Vision-Conditioned Textual Adapter (VCTA), a Text-Conditioned Spectral Adapter (TCSA), and a Textual Reference Adapter (TRA). VCTA incorporates image information into text prompts to generate instance-specific normality and abnormality anchors, which in turn guide TCSA to perform instance-adaptive frequency modulation for capturing transferable anomalous patterns across diverse objects and domains. TRA further generates a text-grounded normality reference to complement normal-image-based comparison in zero- and few-shot settings. Extensive experiments on diverse industrial and medical datasets demonstrate that CrossCLIP consistently outperforms the state-of-the-art universal anomaly detection methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Frequency Domain Learning, CLIP, Anomaly Detection

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_t9sLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CrossCLIP to address the insufficient interaction between visual and textual adaptation in existing CLIP-based anomaly detection methods. Specifically, VCTA conditions normal and anomalous text prototypes on the query image, TCSA uses visual features and text prototypes to modulate local and global spectra, and TRA supplements the memory of real normal support samples with virtual normal patches generated through text projection, thereby supporting both zero-shot and few-shot inference.

Critical strengths
  1. The paper presents a fairly comprehensive experimental evaluation, covering 11 industrial and medical benchmarks, including zero-shot and few-shot settings, and reports the mean and standard deviation over multiple runs.
  2. VCTA, TCSA, and TRA are organized around the same cross-modal interaction objective. Component ablation studies show that the full model outperforms individual modules on most metrics, with particularly notable improvements in pixel-level AP
Critical weaknesses
  1. TCSA applies arbitrary learned phases to real-valued DFT coefficients and then directly performs the inverse transform, yet does not explain how Hermitian conjugate symmetry is preserved or how complex-valued outputs are handled. Moreover, performing a 1D FFT along the channel dimension of <CLS> lacks invariance to channel ordering and clear frequency semantics.

  2. The effect of bidirectional cross-modal interaction has not been sufficiently demonstrated. VCTA’s anchors are themselves generated from global visual features, while TCSA receives both these same global features and the anchors.

  3. The full model underperforms certain single-module or two-module configurations on several individual datasets or metrics, indicating that interactions between modules are unstable. The paper emphasizes only average improvements and lacks explanations for categories with degraded performance and negative interactions.

  4. There are numerous incorrect cross-references in the paper, such as “Appendix ??” and “Fig ??”.

Questions for authors
  1. How is it ensured that the inverse FFT produces real-valued outputs after phase modulation?

  2. Why does the full model perform worse on some datasets and metrics?

Reviewer_K1emOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses universal anomaly detection across unseen industrial and medical domains and proposes CrossCLIP, a framework that enables bidirectional vision-language interaction. Specifically, it introduces a Vision-Conditioned Textual Adapter (VCTA) for instance-specific textual anchors, a Text-Conditioned Spectral Adapter (TCSA) for adaptive frequency modulation, and a Textual Reference Adapter (TRA) for integrating text-generated and visual normality references. Experiments on 11 benchmarks demonstrate competitive zero-shot and few-shot anomaly detection and localization performance.

Critical strengths
  1. Interesting cross-modal adaptation framework. The paper explores bidirectional vision-language interaction and instance-adaptive frequency modulation for universal anomaly detection. The integration of textual and visual references provides a useful approach to supporting both zero-shot and few-shot settings.

  2. Comprehensive experimental evaluation. The method is evaluated on 11 industrial and medical benchmarks, with competitive results across multiple metrics. The ablation studies provide useful evidence for the effectiveness of the proposed components, particularly in anomaly localization.

Critical weaknesses
  1. Limited methodological novelty. CrossCLIP mainly combines existing visual-conditioned prompt learning, frequency-domain adaptation, and reference-based anomaly detection techniques. The distinctions from AdaCLIP, FE-CLIP, and AdaptCLIP remain insufficiently clear. The authors should better highlight the fundamental methodological novelty beyond integrating these existing designs.

  2. Insufficient justification of frequency-domain modulation. The proposed TCSA applies a 1D FFT to global CLIP embeddings, whose feature dimensions do not have an obvious frequency ordering. Furthermore, the complex phase modulation in Eq. (12) raises questions about conjugate symmetry and real-valued feature reconstruction. The authors should clarify these implementation details and provide stronger justification for the spectral design.

  3. Unclear independent contribution of bidirectional interaction. Although the complete framework achieves competitive performance, Appendix Table 10 shows that textual guidance only improves the average Image-level AUROC by 1.4 points and Pixel-level AP by 3.4 points over independent adaptation. The authors should provide stronger evidence that bidirectional interaction, rather than the individual adapters, constitutes the primary source of improvement.

  4. Generalization claims require further clarification. The improvements are not consistent across all benchmarks. For example, CrossCLIP underperforms AdaptCLIP on MVTec Image-level AUROC in Table 1 and on several VisA few-shot localization metrics. Moreover, the substantial gains on medical datasets mainly involve relatively large lesions. Further experiments on small anomalies and more diverse medical modalities would strengthen the generalization claims.

  5. Missing important related work. The paper overlooks UniMMAD: Unified Multi-Modal and Multi-Class Anomaly Detection via MoE-Driven Feature Decompression (CVPR 2026), which investigates unified anomaly detection across heterogeneous modalities and domains through adaptive feature modeling. Although its task setting differs from CLIP-based zero-/few-shot anomaly detection, it is relevant to the broader motivation of cross-modal adaptation and domain generalization. The authors should discuss this connection and clarify the differences between these approaches.

Questions for authors

See the weaknesses part.

Reviewer_D7P6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CrossCLIP, a framework for zero- and few-shot visual anomaly detection on unseen categories and domains. The work is motivated by the limited cross-modal interaction in existing CLIP adaptation methods and the potential of frequency-aware feature adaptation to improve generalization. With a frozen CLIP backbone, CrossCLIP introduces three adapters: VCTA generates image-conditioned normal and abnormal text anchors; TCSA uses visual and textual information to modulate frequency amplitudes and phases; and TRA combines learned and text-generated normal references with available support features.

Critical strengths

The proposed adaptation is well motivated by the mismatch between CLIP’s general image–text alignment objective and the fine-grained discrimination between normality and abnormality required for anomaly detection. VCTA and TCSA offer a coherent response: visual context informs the normal and abnormal textual anchors, which in turn guide visual feature adaptation. This reciprocal conditioning provides a reasonable design for adapting CLIP to anomaly detection when target-category training data are unavailable or scarce.

Critical weaknesses
  1. Insufficient analysis of the frequency modulation mechanism. A central contribution is the claim that jointly conditioning frequency modulation on visual features and textual anchors captures transferable anomaly-related features. However, the visualizations in Figures 10–13 do not establish that the suppressed components are irrelevant or that the retained components correspond to anomalies. Both normal textures and subtle defects can contain high-frequency information, so the observed spectral attenuation could reflect generic smoothing rather than selective enhancement of anomaly-related features. Controlled comparisons against simple low-pass filtering and visual-only modulation, together with quantitative analysis across unseen categories, would help substantiate the proposed mechanism and its claimed transferability.

  2. Insufficient comparison with recent language-free approaches. Recent methods such as UniADet, AnomalyVFM, and VisualAD demonstrate competitive anomaly detection without a language branch in the detector. The paper does not adequately position CrossCLIP against these alternatives or establish what additional benefit its textual branch and cross-modal interactions provide. This is particularly relevant because bidirectional cross-modal adaptation is a central contribution. Comparisons with recent vision-only methods, accounting for differences in backbones and auxiliary training data, would strengthen the evaluation. A controlled comparison against learned visual normal/abnormal prototypes would further clarify whether textual conditioning provides meaningful gains in accuracy or generalization that justify its additional complexity.

Questions for authors
  1. Clarification of the frequency modulation mechanism. What evidence shows that the proposed modulation selectively preserves anomaly-related information, rather than primarily smoothing the features? In particular, can the authors clarify whether joint visual–textual conditioning provides benefits beyond simple low-pass filtering or visual-only modulation, and how these benefits support the claimed transferability across categories?

  2. Clarification of the advantage over language-free approaches. What specific advantage does retaining CLIP’s language branch provide over recent language-free methods such as UniADet, AnomalyVFM, and VisualAD? Under comparable visual backbones and auxiliary training data, is there evidence that textual conditioning improves detection or generalization beyond what learned visual normal/abnormal prototypes can achieve?

  3. Clarification of scope and terminology. What does “universal anomaly detection” denote in this paper beyond the standard zero-shot and few-shot anomaly detection settings? Please clarify whether it implies any additional capability or evaluation requirement. If it is simply an umbrella term for these settings, I suggest using “zero- and few-shot anomaly detection” consistently to describe the demonstrated scope more precisely. This is a terminology suggestion, not a request to introduce a new task.

Reviewer_uTCCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets universal (zero- and few-shot) visual anomaly detection with a frozen CLIP backbone. Its premise is that existing CLIP-based detectors adapt the visual and textual branches either independently or through unidirectional interaction, leaving complementary cross-modal information unused. The proposed fix lets the two modalities condition each other throughout adaptation, via three adapters.

The Vision-Conditioned Textual Adapter keeps learnable normal/abnormal context tokens and adds a per-image residual from the query's global CLIP feature, gated so that training starts from object-agnostic prompts, yielding instance-specific anchors. The Text- Conditioned Spectral Adapter takes those anchors plus the global feature and uses small MLPs to predict per-band amplitude and phase multipliers over a frequency decomposition of the patch and global features. The Textual Reference Adapter builds a text-grounded virtual reference set from learnable prototypes plus a projection of the normal anchor, and blends it with real support patches via soft nearest-neighbour alignment and a mixing weight that vanishes when no supports are given. A regime-consistency KL loss distils the support-conditioned prediction into the zero-shot one, so one trained model serves any number of supports. Final scores average the textual map, the spectrally adapted visual map, and the reference-adapter maps across layers.

Validation: zero-shot on five industrial and six medical benchmarks, few-shot at 1/2/4 shots, ablations over the three adapters and the spectral-adapter design choices, cross-modal guidance, band count and reduction ratio, a compute-cost table, and a backbone comparison. The authors claim best averages in both regimes, with the largest gains in localization.

Critical strengths

(1) The medical localization results are large, consistent, and not attributable to a scoring artifact. CrossCLIP improves average pixel-level AUROC/AP on the four colonoscopy benchmarks from 84.8/44.2 (AdaptCLIP) to 90.8/65.9, and in the 1-shot setting average pixel AP goes from 50.8 to 77.7. Those margins are far too large to be seed noise, and they are consistent across all three shot settings and both metrics — unusual in a literature where gains are typically 1–2 points. This is consequential because pixel-level AP is the metric that actually matters for clinical deployment, and because the paper offers a coherent explanation (polyps and lesions are large, low-frequency structures) that a reader can test.

(2) A single trained model serves zero- and any-shot inference without recalibration. The mixing weight that is pinned to zero when no supports exist, plus the KL objective that forces both regimes onto a common output scale, is a genuine practical contribution: most few-shot detectors need a separate model or at least a re-tuned threshold per shot count. This is consequential for deployment realism, which the paper does not emphasize enough.

Critical weaknesses

(1) The ablation does not support the paper's central mechanism, and in places contradicts it.

The claim is that the three adapters are mutually reinforcing — specifically that VCTA's anchors steer TCSA, which is why the joint gain exceeds the sum of parts. Table 5 does not show this. On MPDD image-level AUROC the single-adapter rows are VCTA 72.6, TCSA 77.3, TRA 75.3, and the full model 79.5 — a joint gain of about +2.2 over the best single component, not the "clear margin" the text asserts. On BTAD image AUROC, the TRA-only row (86.8) is higher than the VCTA+TRA row (86.0) and the TCSA+TRA row is 83.0 — adding components makes things worse. On DTD, VCTA-only image AP is 98.9 while the full model is 99.1 and TCSA-only is 99.1. The only large joint gain is pixel-level AP on BTAD (+21.1), and the text generalises from that single cell. Table 6 is used to split the credit between instance conditioning and frequency modulation, but the split does not hold up. On MPDD image-level AUROC the conditioning-only row (80.8) beats the full model (79.5). On BTAD image-level AP the frequency-only row (93.1) beats the full model (92.6). On DTD pixel-level AP frequency-only (98.7) vs full (99.1) is within noise. More fundamentally, the two toggles are not cleanly separable: turning frequency modulation off changes the parameter count, the receptive field, and the residual structure simultaneously, so the 2×2 cannot attribute gains to "conditioning" and "frequency" independently the way the text claims. There is no ablation isolating TRA's textual grounding from TRA's mere presence: the ✗✗✓ row adds a whole reference-comparison branch, and the paper attributes the gain to the text. A natural control — TRA with the MLP-of-anchor branch removed, leaving only prototypes — is missing, and it is exactly the control that would support "textual reference" as opposed to "extra learnable reference." (2) On the canonical benchmark the method loses, and the "zero-shot" framing is generous.

MVTec-AD is the standard reference point for this literature and is listed first in every table. On image-level AUROC CrossCLIP scores 91.9 against AdaptCLIP's 93.5 and AnomalyCLIP's 91.6, i.e. it is not the best method on MVTec, and on image-level AP it is 96.6 vs AdaptCLIP's 96.7. The abstract's claim of consistently outperforming the state of the art and the text's "best average" are technically true only because of the MPDD and BTAD cells. Reporting only domain averages hides this, and the paper never acknowledges that its own strongest baseline wins on the most-used dataset. In the few-shot tables the picture is similar: VisA 4-shot image AUROC is 93.1 for both AdaptCLIP and CrossCLIP (a tie presented as a win), and VisA pixel AP is worse for CrossCLIP in all three shot settings (e.g. 40.0 vs 41.8 at 4-shot). The paper acknowledges the VisA pixel-AP point only in the Appendix limitations. The "zero-shot" setting is cross-dataset transfer: the model is trained on MVTec-AD or VisA and evaluated elsewhere. The medical results in particular come from a model trained on industrial data — which makes them impressive — but the two zero-shot medical image-level datasets are Br35H and Covid, and the Covid column is degenerate (APRIL-GAN scores 31.9 AUROC). Averaging over a two-dataset pool where one is near-random inflates the reported medical image-level average substantially. Per-dataset effect sizes, not domain averages, should carry the claim. Implementation details contain an inconsistency that affects fair comparison: the backbone is stated as "ViT-L/14@336px" but inputs are "resized to 518×518". The paper asserts that all competing methods share "the same backbone, input resolution and patch-level feature extraction except for WinCLIP" — but the re-implemented baselines (marked †) would need the same 518 input to make that true, and this is not stated for them. (3) Novelty relative to the cited work is not established. VCTA is conceptually very close to AdaCLIP's dynamic injection of query visual context into textual prompt embeddings, which the paper cites and benchmarks against; TCSA applies instance-adaptive frequency masks of the kind introduced by DFF (also cited) with the condition replaced by text anchors; and the whole framework is explicitly built on AdaptCLIP ("Following AdaptCLIP, we use a similar fusion strategy"). The contribution is therefore the combination, and the paper does not argue why the combination is more than the sum of already-published parts, nor does it show what happens if only VCTA is grafted onto AdaptCLIP (Table 10's "Guided" vs "Independent" comparison is close but is not the same experiment, since both arms include TCSA).

Questions for authors

MVTec. Why does CrossCLIP underperform AdaptCLIP on MVTec image-level AUROC (91.9 vs 93.5) and AP (96.6 vs 96.7)? Is this a real limitation of the frequency branch on texture/object categories, and does it hold per category? Ablation. Please add the missing TRA control (prototypes only, no anchor MLP) and a VCTA-on-AdaptCLIP-only arm, and report the Table 5 ablation with the full model's row marked. Can you reconcile the non-monotone cells (BTAD image AUROC: VCTA+TRA 86.0 vs TRA-only 86.8)? Frequency selection. Which bands does TCSA actually amplify, and does the selected band depend on the textual condition or only on the image? A plot of learned per-band gain against the anchor would directly test the "text steers modulation" claim. Eq. 9. Please correct the band-partition formula and define u′u', v′v'.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.