- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents a concept-based attribution method that is able to explain visual machine learning models in a post-hoc fashion. The method uses perturbation testing to evaluate the importance of individual concepts and thus does not need access to model weights. The model can be applied to different types of models, including not only classification models but also segmentation and captioning models. The approach is evaluated on two image datasets and different ML tasks.
Justification of my rating: the work is generally technically sound with minor flaws. However, the degree of novelty in the approach is rather limited and thus I vote for a "weak reject".
Critical strengths
Concrete strengths of the paper are:
- The introduction of a method for post-hoc explainability that is applicable to generally arbitrary ML models and tasks, as long as an attribution score can be defined
- The presentation of the paper is very clear and the experiments are comprehensive (including many details in the appendix)
- Interesting insights on the PRUE segmentation model are gained by the explanation method
- The approach can be easily applied in practice
- Code is provided with the submission, so reproducibility is given
Critical weaknesses
The main weakness of the paper is that the approach does not represent a fundametally new idea towards (post-hoc) explainability. The authors combine several existing concepts, such as perturbation testing, concept prediction via VLMs. Furthermore, although the approach does not rely on pre- labeled contcepts (which is a plus), it requires for each concept to be tested to have a dataset available that captures this concept visually. This represents a limitation that is not discussed in the paper. It is briefly mentioned in line 212, but not recognized as a general limitation.
Further weaknesses include:
- The approach very much relies on the output of the VLM. If the VLM captures the concept well, the results may be promising but if the concept is not well captured or ambiguous it is questionable, how well the approach works. Furthermore, the VLM may introduce its bias into the analysis and thereby into the explanation.
- The compared baseline methods TCAV and PCBM are from 2018 and 2023, respectively and not fully up to date.
- The empirical method for finding the best filling strategy is not motivated and little elaborated. It may even bias results, because different methods are mixed. From the approach one could derive the implicit and underlying hypothese that different concepts require different filling strategies, which is from my perspective not valid. The filling strategy should rather be designed with theoretical considerations in mind, such as reducing the amount of artifacts introduced by it to avoid biasing the model.
Questions for authors
- Why can PCBM not be applied on individual images to obtain local explanations? Please elaborate?
- The authors define a threshold (depending on the standard deviation of a fitted Gaussian distribution) to select how many blocks to occlude for a given concept (see Eq. 1). It is questionable, if this threshold works for concepts of different spatial coverage. It seems to work for concepts with small spatial coverage, such as Eye Glasses, but what about concepts like "Sky", which are spatially much more distributed? An empirical evaluation of concepts with different spatial coverage would be important to show that Eq. 1 generally works.
- The authors state: "For stable attribution scores, measuring Δy is sufficient." If this is stated this way, it needs to at least empirically demonstrated.
- Please add some qualitative results of attribution scores / maps for TCAV and PCBM
- Lines 400 and 401: is really the "attribution score" used or the "gender score" defined above?
- Paragraph 403-408: Maybe the bias observed is in the data and not (only) in the model? Was that verified? Maybe there are many more images with men and tie and this leads to more captions with this pair, so not the model is biased but the data.