acceptodds
Under review as a conference paper at ICLR 2027

AnchorCLIP: Enhancing CLIP Robustness through Interpretability

Abstract

Pretrained Vision-language models such as CLIP show impressive zero-shot performance on clean images. However, they are extremely vulnerable to common image corruptions, such as Gaussian noise, Pixelation or Motion blur. Existing approaches to improve robustness focus mostly on adversarial fine tuning but they have crucial limitations. They are ineffective for severe image corruptions, involve fine tuning the entire CLIP encoder, and are largely black box. In this paper, we leverage CLIP’s semantic interpretability to enforce robustness. We show that enforcing semantic consistency in concept space across corruptions significantly improves robustness. Our proposed method AnchorCLIP, uses a lightweight adapter to align the corrupted image embeddings to the interpretable latent space of clean CLIP embeddings, effectively “anchoring” the noisy representations to their semantic meanings. Extensive experiments indicate that AnchorCLIP outperforms existing adversarial robustness and interpretability based methods on widely used benchmark datasets, while maintaining performance on clean data. Distinct from previous work, AnchorCLIP is based on interpretability, is lightweight, and consistently outperforms strong baselines for severe image corruptions. By using interpretability as a mechanism for robustness instead of just post-hoc explanations, AnchorCLIP provides a scalable path toward vision-language systems that are both inherently robust and transparent.

Primary area:
interpretability and explainable AI
Keywords:
CLIP, robustness, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_NJmaOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on the zero-shot classification robustness of CLIP under common image corruptions, where the goal is to maintain accuracy on corrupted images without task-specific labels. To address this issue, the authors proposed AnchorCLIP, a lightweight adapter framework with frozen CLIP encoders. AnchorCLIP is trained once on clean images with synthetic corruptions and uses intermediate-layer class tokens, a concept dictionary, and interpretability-based losses to align corrupted-image embeddings with clean semantic concepts. The authors validated the method on CIFAR-10C, CIFAR-100C, and ImageNet-C, and the results showed that AnchorCLIP achieved higher accuracy than CLIP, TeCoA, and FARE under most corruptions while largely preserving clean-image performance.

Critical strengths
  1. The method design is practical. It freezes both CLIP encoders and trains only a lightweight adapter. Inference needs a single forward pass, with no clean reference image and no test-time optimization, so it avoids full-encoder fine-tuning and TTA overhead and is easy to plug into existing CLIP pipelines.

  2. The paper reports a useful negative result. In the supplementary experiments, SpLiCE and COLA both have lower accuracy than CLIP under common corruptions, showing that these concept/subspace methods do not transfer directly to corruption settings. This is a useful reference for future work on semantic constraints.

Critical weaknesses
  1. The robustness comparison is not well aligned with the task. The main robustness baselines, TeCoA and FARE, are trained for adversarial attacks rather than common corruptions. Reporting corruption accuracy therefore shows only transfer, not a direct comparison under the same threat model. As a result, the results do not establish competitiveness against corruption-specific methods.

  2. The interpretability claim is not sufficiently supported. The paper offers only qualitative concept visualizations, with no faithfulness or intervention tests. even in Figure 8, a boot image yields “jewelry”. Thus, “inherently explainable” and “not a Black Box” lack support.

  3. Table 14 compares only two single losses and two symmetric mixtures (0.7/0.3 and 0.3/0.7), with no 0.5/0.5 or finer grid, and reports results on only four corruptions of CIFAR-100-C. It therefore shows only that the two losses work better together, not that 0.7/0.3 is an optimal or robust weighting.

Questions for authors

See the Critical Weaknesses

Reviewer_CtnFOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

AnchorCLIP aims to improve CLIP classification under common image corruptions, such as noise and blur. It freezes the CLIP encoder and trains an adapter that combines intermediate features with the final image embedding. Two objectives, concept-distribution matching and semantic reconstruction, align corrupted representations with clean-image semantics.

Critical strengths

The paper addresses a relevant problem. Noise, blur, and compression artifacts affect visual recognition.

The method improves CLIP average classification accuracy on several benchmarks.

Critical weaknesses
  1. The dictionary and standardization module basically just follow SpLICE design, adapter also references existing intermediate-layer feature scheme, even with OLS reconstruction and KL matching added, whole method is still stitching existing components together. paper also doesn't explain what new academic insight this work brings, novelty not enough.

  2. My main concern is: does performance gain come from concept-related training objectives, or just because adapter architecture itself is stronger? AnchorCLIP uses intermediate-layer features and also Transformer and MLP, but compared task baseline only uses a single linear layer, no control baseline with same architecture and same input, this kind of performance attribution not convincing.

  3. Fig. 7 crossword image gets associated with keyboard and typing, totally unrelated concepts; in Fig. 8 jewelry still ranks as top concept. so-called semantic recovery effect not stable at all, just listing a few top-ranked words also can't prove these concepts really explain model's classification decisions, interpretability evidence too weak.

  4. performance gains on synthetic corruption benchmarks do have value, but autonomous driving and medical deployment scenarios never actually tested, authors' description of application scenarios already goes beyond experimental scope. also parameter count, GPU memory usage, end-to-end latency not fully compared, hard to evaluate how much benefit this method brings after actual deployment.

  5. Fig. 2 structure proportion, lines and layout all feel awkward, two core losses stuffed into same generic module, each one's computation process not clear enough. Fig. 1 and appendix concept visualization labels too small, too much whitespace, these figures almost no help for understanding method's logic.

  6. On OxfordPets dataset, mean accuracy only goes from 44.86% to 45.42%, improvement pretty small, and under several corruption types performance actually drops. five repeated runs only cover CIFAR-10, can't prove whether this small transfer gain is reliable, cross-dataset transfer results still need more thorough stability analysis.

Questions for authors
  1. What happens when the adapter, intermediate inputs, and training protocol are kept unchanged, but only MSE or cosine alignment is used? How much does the concept objective add over that baseline?

  2. How should readers interpret the semantic shifts in Figures 7 and 8? Can quantitative evaluation or concept interventions test whether these concepts explain the final predictions?

  3. Are results available on captured real-world degradations? How many parameters, how much memory, and how much inference latency does the method add to CLIP?

1 comment

The authors provided no rebuttal. So I maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.