- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the interpretability of frozen CLIP-style vision-language models at the level of individual image-text pairs, proposing an Adversarial Bottleneck Method (ABM) designed to identify intermediate features that significantly influence the image-text matching score. The authors introduce continuous gating variables to the intermediate features of the vision or text encoders, constructing an information bottleneck through interpolation between the original features and Gaussian noise. Subsequently, the gating variables are optimized using projected gradient sign updates, with the image-text cosine similarity serving as the search objective. Unlike traditional gradient-integration-based attribution methods, ABM constructs a Gaussian channel based on the final gating state and calculates feature importance via KL divergence, thereby generating image heatmaps and text token attribution scores. The paper explicitly notes discrepancies between the submitted implementation and the theoretical reference algorithm, acknowledges known flaws in the text attribution code, and admits that historical experimental results cannot be traced back to the exact implementation version. These issues severely limit the credibility of the current experimental conclusions, and the paper's overall novelty is limited.
Critical strengths
- The authors attempt to decouple feature selection from importance readout and employ direct gating optimization to avoid the gradient attenuation potentially caused by Sigmoid parameterization; this design approach is sound.
- The experiments cover multiple datasets and attribution metrics.
Critical weaknesses
- The paper acknowledges substantial discrepancies between the theoretical algorithm and its actual implementation, rendering the theoretical analysis insufficient to directly support the primary experimental conclusions.
- There are clear errors in the implementation of the attribution method.
- The method’s novelty is limited, and it fails to sufficiently distinguish itself from existing information bottleneck and adversarial attribution methods. The key components of ABM—including feature noise interpolation, gradient sign updates, projection constraints, and feature scoring based on Gaussian KL divergence—are closely related to existing information bottleneck attribution or gradient-based feature perturbation techniques.
Questions for authors
- State the consistency between the theoretical algorithm and the experimental implementation.
- Regarding the errors in embedding parameter passing and batch dimension indexing identified in Section 3.3, can the authors provide the corrected implementation and re-run the text attribution experiments?