- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a double factorization to decompose image features into concepts, and decompose concepts into training image weights. This leads to a double attention mechanism, one with the concepts attending to the training images producing refined concepts, and one with the training images attending to the refined concepts. The objective is to reconstruct the training images by maximizing the cosine similarity to follow CLIP's direction. Specifically, given the features of CLIP image encoder of n images Z (shape nxd) the authors learn a C matrix (with softmax constraint) of shape (n x k) where k are the number of concepts. C shows which training samples construct each concept. By performing W = C^T.Z the authors attend over the training images, and produce a refined concept dictionary of shape (k x d). The authors also learn another matrix B of shape (n x k) also softmax constrained which shows how refined dictionary reconstructs each individual embedding, and by performing B.W they get the reconstructed Z features. The objective is to minimize the cosine similarity between the reconstruction and original training image features.
Critical strengths
- The decomposition links learned concepts to their contributing training samples, which I think is novel and can aid the data attribution field. Different for other concept works, the method offers a mechanism for tracing learned concepts back to their contributing samples
- The paper provides theoretical analysis
Critical weaknesses
- As this is an incremental work on feature factorization, the authors should compare with other methods in the literature. Does their method provide an advantage over other methods? I do not see where the method shines. I dont see any evidence that the proposed method provides better interpretability. The authors compared only with CRAFT, but this is not a CLIP-based approach, and it is unclear how the authors adapted it to CLIP. Some existing works on interpreting CLIP with concepts are [R1 - R5]. These works are not even discussed in the related work. These works are even better because one can identify via text what each concept represents. On the other hand, the authors method needs manual inspection to get what this concept means.
- In my opinion, the approach could be viewed as two single-layer transformers cross-attending to each other. The first transformers with queries as learable concepts and keys-values as image embeddings, producing a refined concept dictionary. The second transformer has queries as the image embeddings with keys-values coming from the refined concept dictionary from the first transformer. The only difference is that only the weight matrices for attention are learned. So essentially this is like adding some learnable layers on top of CLIP, like an adaptor, to adapt CLIP to other applications (here interpretability). This also means the authors should compare with CLIP adaptations for interpretability. Some of the works are [R6, R7, and R8]. For example, R6 trains a concept-based surrogate to reproduce CLIP's representation similarities and provide concept explanations and R7 learns a projection from CLIP embeddings into a human-interpretable concept space using alignment and reconstruction losses. These works highly overlap with the proposed method.
- The fidelity and faithfulness of the reconstructed features is not proven with linear classifier training. Essentially you can change the whole distribution of CLIP, train a classifier on the new distribution and still get good results. I was expecting to see the zero-shot accuracy between the original CLIP and the reconstructed CLIP features. This is missing. This also means the insertion/deletion experiments cannot be trusted because they reply on the newly-trained classifier to extract confidence values, not the original zero-shot classifier (which is the text encoder of CLIP).
- How does the method scale to large datasets? The authors only use a subset of ImageNet (100 images) but this is far from realistic, and the computation will increase for every sample as the method needs to learn a new matrix for each test sample. This is important are we are currently in the large-data regime.
- Im not sure why gradients are used to identify the most important concepts. This disconnects the method from the way the authors select concepts. In my opnion the authors should evaluate by selecting the most important concepts from their learned matrix B, which identifies how strongly each concept participates in reconstructing a particular image. It measures concept activation. This is the correct way to evaluate this and select concepts. However, I do understand that this could lead to problems, because background concepts of the image (unrelated to a class) might not lead to a high class confidence, but still has a high reconstruction. Therefore, the authors can adapt the insertion/deletion metric but measure the reconsutrction of the features rather than the confidence metric, as concepts according to B are inserted/deleted. This correctly evaluates B. B-based ranking evaluates the importance of concepts to the learned reconstruction and should be reported. Essentially, another version of E1 where you gradually insert/delete concepts according to B, and measure this against the original CLIP embedding.
- Interpretability is only useful if we can actually do something with it. Can you for example improve CLIP zero shot accuracy with the extracted concepts? In the recent few years tons of papers are our about analyzing and understanding models, but no one is actually doing anything with those findings. They are just archived and forgotten. The real value comes when you have a practical application of your approach such as improving classification, segmentation...etc
[R1] Interpreting and Analysing CLIP’s Zero-Shot Image Classification via Mutual Knowledge [R2] Interpreting CLIP with Sparse Linear Concept Embeddings [R3] Matryoshka Sparse Autoencoders [R4] Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery [R5] Interpreting CLIP's Image Representation via Text-Based Decomposition [R6] EXPLAIN-R: Advancing Interpretability of CLIP Representations with Concept Surrogate Model [R7] Explaining CLIP Zero-shot Predictions Through Concepts [R8] Interpreting CLIP with Hierarchical Sparse Autoencoders
Questions for authors
I think the paper is interesting but it currently has several problems: 1) no comparisons with similar works and evidence that the proposed method has better interpretability, 2) evaluation is wrong, as the authors train their own linear classifier rather than using the original CLIP zero-shot classifier (which is the CLIP text encoder), and 3) some other evaluations missing. Additionally it is unclear how the method can scale to large datasets. I would also like to see a practical application of the method rather than just interpretability.