acceptodds
Withdrawn from ICLR 2027

Adversarial Bottleneck Explanations for Single-Pair CLIP-like Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Explaining a frozen vision–language model at the level of one image–text pair requires attributing a matching score without retraining or auxiliary examples. We study a bottleneck approach that separates feature selection from attribution readout. The Adversarial Bottleneck Method (ABM) searches over bounded intermediate-feature gates using the paired cosine score and assigns importance from a final-state Gaussian KL score. This removes the explicit compression–relevance coefficient used by M2IB while retaining a configurable perturbation path. We derive a finite-step bound for the projected gate formulation that accounts for boundary clipping and provides an explicit sufficient condition for one-step surrogate ascent. The bound does not imply optimal compression or attribution faithfulness. Reported experiments on Conceptual Captions, ImageNet, and Flickr8k show favorable image-side Confidence Drop point estimates under the inherited evaluation protocol, with complementary ROAD, component, and AltCLIP results. The submitted text wrapper has unresolved validity defects, and the historical measurements lack an exact run-to-implementation binding; they do not yet validate the reference formulation. ABM is a method for inspecting a model's matching score, with theoretical support for its update geometry rather than for causal or human-semantic explanations.

Primary area:
interpretability and explainable AI
Keywords:
explainable AI, vision-language models, CLIP, feature attribution, information bottleneck, projected gradient optimization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_3u4XOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the interpretability of frozen CLIP-style vision-language models at the level of individual image-text pairs, proposing an Adversarial Bottleneck Method (ABM) designed to identify intermediate features that significantly influence the image-text matching score. The authors introduce continuous gating variables to the intermediate features of the vision or text encoders, constructing an information bottleneck through interpolation between the original features and Gaussian noise. Subsequently, the gating variables are optimized using projected gradient sign updates, with the image-text cosine similarity serving as the search objective. Unlike traditional gradient-integration-based attribution methods, ABM constructs a Gaussian channel based on the final gating state and calculates feature importance via KL divergence, thereby generating image heatmaps and text token attribution scores. The paper explicitly notes discrepancies between the submitted implementation and the theoretical reference algorithm, acknowledges known flaws in the text attribution code, and admits that historical experimental results cannot be traced back to the exact implementation version. These issues severely limit the credibility of the current experimental conclusions, and the paper's overall novelty is limited.

Critical strengths
  1. The authors attempt to decouple feature selection from importance readout and employ direct gating optimization to avoid the gradient attenuation potentially caused by Sigmoid parameterization; this design approach is sound.
  2. The experiments cover multiple datasets and attribution metrics.
Critical weaknesses
  1. The paper acknowledges substantial discrepancies between the theoretical algorithm and its actual implementation, rendering the theoretical analysis insufficient to directly support the primary experimental conclusions.
  2. There are clear errors in the implementation of the attribution method.
  3. The method’s novelty is limited, and it fails to sufficiently distinguish itself from existing information bottleneck and adversarial attribution methods. The key components of ABM—including feature noise interpolation, gradient sign updates, projection constraints, and feature scoring based on Gaussian KL divergence—are closely related to existing information bottleneck attribution or gradient-based feature perturbation techniques.
Questions for authors
  1. State the consistency between the theoretical algorithm and the experimental implementation.
  2. Regarding the errors in embedding parameter passing and batch dimension indexing identified in Section 3.3, can the authors provide the corrected implementation and re-run the text attribution experiments?
Reviewer_7yv2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes the Adversarial Bottleneck Method (ABM) to explain why a frozen vision-language model (CLIP) matches a given image and caption. ABM blends internal features with noise and uses gradient steps to keep the features that preserve image-caption similarity, producing importance maps for the image and the caption. Unlike M2IB, it needs no setting to balance relevance against compression. The authors prove a guarantee on when each step does not lower the similarity, and report better image-side results than M2IB and other baselines on CLIP and AltCLIP across three datasets.

Critical strengths
  • The paper tries to tackle a relevant research problem of explaining matches of VLM text-image pairs.
  • The writing, although hard to comprehend, is honest about many weaknesses and errors of the paper.
Critical weaknesses
  • The paper is completely incomprehensible. The abstract packs jargon and technical terms to the extend that it becomes unreadable. The introduction is not any better in this regard. There is no real motivation to the research problem, it continues to discuss related works with the assumption that the reader is well aware of the key references and terms from prior work, and the contributions are not clear because there is not enough context for the reader. To illustrate the issue, the following terms are not defined or explained anywhere in the text (excerpt, the list goes on):
    • final-state Gaussian KL score
    • perturbation path
    • one-step surrogate ascent
    • alignment surrogate
    • full-retention initialization
    • regularized plug-in readout
    • noise-adapted search trajectory
    • compression–relevance coefficient
    • deterministic narrowing
    • run-to-implementation binding
  • The authors admit to bugs and issues in the code that produced the results:
    • The abstract admits that their own implementation and results do not validate the presented mathematical formulation: "The submitted text wrapper has unresolved validity defects, and the historical measurements lack an exact run-to-implementation binding; they do not yet validate the reference formulation."
    • Sec 3.3. reads "The submitted text wrapper passes a text embedding to an argument intended to hold the image embedding. In addition, the statements lamb[0]=1 and lamb[-1]=1 index the batch axis; for a batch of one they overwrite the entire gate, rather than selecting boundary tokens. These facts preclude treating this wrapper as a verified implementation of pair-conditioned token attribution." The authors essentially admit to two bugs and conclude that their own implementation is not to be trusted. By this statement alone, none of the results of the paper can be trusted either.
    • Several other parts in the text mention discrepancies between the theoretical formulation and the implementation.
  • The experimental results, apart from not trustworthy, are also not convincing, as directly stated by the authors (quotes):
    • No statistical signifigance: "The main confidence and ROAD tables contain single-run point estimates"
    • The differences on ROAD are very small (Tab. 3, 1.8867 vs. 1.8631) : "the small aggregate difference is not an established ranking"
    • Hyperparameters are chosen on the test set: "Layer and iteration defaults were selected from the reported evaluation sweep, not an independent held-out tuning set" for their method but not for the competitor: "M2IB uses its recommended default rather than matched tuning"
    • Results for NIB are missing: "The present tables contain M2IB and Grad-ECLIP but no verified matched NIB result"
  • The AI use statement reads: "Generative AI tools assisted with manuscript refinement, language clarity and readability, and LaTeX formatting. The authors are responsible for the final manuscript." While it is not entirely clear, the quotes above indicate that AI may have been used beyond manuscript refinement, e.g., for composing parts of the paper based on the code, and the raised issues were not properly addressed by the authors before submission.

As conclusion, the authors write: "The main unresolved issues are execution correspondence, matched-budget text evaluation, close-comparator coverage, and statistical uncertainty."
which roughly translates to: the code doesn’t match the method described in the paper, the caption results aren’t fair, results for the closest competitor (NIB) are missing, and there’s no measure of how reliable the reported single-run results are.

Questions for authors

The paper requires a complete rewrite to make it easier to read. Please define technical terms when they are first used in the paper and clearly relate them to the mathematical formations.
As already stated in the paper, the bugs need to be fixed and results rerun, including multiple runs with statistical significance tests.
A comparison with NIB need to be included and the benefit of ABM over NIB needs to be clearly shown.

Reviewer_tz8BOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes ABM, an attribution method for the matching score of a single image-text pair in a frozen CLIP-like model. The method optimizes gates on intermediate features to increase the pair's cosine similarity, then computes an attribution map from the gates using a Gaussian KL score. It removes the trade-off coefficient and sigmoid parameterization used in M2IB. The paper gives a one-step bound for the projected update and reports Confidence Drop/Increase and ROAD with CLIP ViT-B/32 on Conceptual Captions, ImageNet, and Flickr8k, with additional AltCLIP results.

Critical strengths
  1. The problem is practically relevant. Explaining an individual image-text match without retraining or extra examples is useful for inspecting retrieval errors. Directly optimizing the gates is a simple alternative to the relevance-compression objective in M2IB. The reported image results are also promising, with lower Confidence Drop than M2IB on all three datasets in Table 2, although their validity needs to be resolved.
  2. The authors clearly disclose the limitations of the implementation and evaluation. In particular, they distinguish the analyzed algorithm from the submitted implementation and acknowledge that the text results do not establish valid pair-conditioned attribution. This helps readers identify which claims still need evidence.
Critical weaknesses
  1. The experiments do not sufficiently support the effectiveness of ABM. The central idea is to obtain better image-text attribution by directly optimizing feature gates and then applying a KL readout. The reported image metrics are promising, but Section 4.1 explicitly states that these historical results do not yet validate either algorithmic variant. The text results are also retained only as archival records, rather than evidence of valid pair-conditioned attribution. These limitations leave the main comparisons and component ablations without a firm basis for concluding that ABM improves attribution quality.

  2. The comparisons do not clearly establish the benefits of the main design choices. ABM's layer and iteration count were selected from the evaluation sweep, while M2IB uses its recommended default rather than matched tuning. This makes it difficult to separate the effect of the method from the effect of configuration selection. Taking the historical values in Table 8 as example, the one-step setting already has lower image Confidence Drop than the reported M2IB default on all three datasets. A direct comparison with this simpler setting would help show what additional search steps contribute. The role of the KL readout also needs to be tested separately, since the one-step setting still uses it. A comparison with NIB would further clarify the contribution relative to closely related bottleneck methods. Proposition 3.1 concerns a local gate update; it does not establish attribution quality and therefore cannot substitute for this experimental evidence.

Questions for authors
  1. What experimental evidence can establish that ABM provides valid pair-conditioned attribution for both images and text, given the limitations stated in Section 4.1?
  2. Under the same evaluation protocol, what does multi-step search add over the one-step setting? With the final gates fixed, does the KL readout improve attribution quality over simpler ways of aggregating the retained features?
  3. Can ABM, M2IB, and NIB be compared with the same tuning budget on a separate validation split? Results over multiple seeds, with uncertainty estimated from paired image-text examples, would help determine whether the improvements are reliable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.