acceptodds
Withdrawn from ICLR 2027

Hide and Seek: Hide Concepts to Seek Explanations for Black Box Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interpreting machine learning model predictions is a central challenge, particularly for black-box vision models. Concept-based attribution methods interpret models using human-understandable concepts. Existing concept-based methods can only interpret classification and regression models. They require access to model weights or large concept-annotated datasets, and most methods cannot provide image-level attribution. We propose Hide and Seek (HnS), a new method for concept-level attribution that does not require access to model weights or labeled concept datasets. HnS can interpret a wide range of vision models. Given a concept and image set, HnS retrieves concept-aligned images using a vision-language model, localizes the concept within each image, masks the localized region, and measures attribution from the change in the black-box model's output between the original and masked images. This enables faithful attribution by directly measuring the model’s behavior while also enabling local attributions. We validated HnS on CelebA and COCO-Stuff. Compared to existing methods, HnS showed better agreement with ground truth attributions. We demonstrated the applicability of HnS on diverse tasks, including image captioning and satellite image segmentation, where no existing concept attribution method can be used. We provided our anonymous code in the supplementary material.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, concept-interpretability, explainability, concept based attribution, AI Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_g5PeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a concept-based attribution method that is able to explain visual machine learning models in a post-hoc fashion. The method uses perturbation testing to evaluate the importance of individual concepts and thus does not need access to model weights. The model can be applied to different types of models, including not only classification models but also segmentation and captioning models. The approach is evaluated on two image datasets and different ML tasks.

Justification of my rating: the work is generally technically sound with minor flaws. However, the degree of novelty in the approach is rather limited and thus I vote for a "weak reject".

Critical strengths

Concrete strengths of the paper are:

  • The introduction of a method for post-hoc explainability that is applicable to generally arbitrary ML models and tasks, as long as an attribution score can be defined
  • The presentation of the paper is very clear and the experiments are comprehensive (including many details in the appendix)
  • Interesting insights on the PRUE segmentation model are gained by the explanation method
  • The approach can be easily applied in practice
  • Code is provided with the submission, so reproducibility is given
Critical weaknesses

The main weakness of the paper is that the approach does not represent a fundametally new idea towards (post-hoc) explainability. The authors combine several existing concepts, such as perturbation testing, concept prediction via VLMs. Furthermore, although the approach does not rely on pre- labeled contcepts (which is a plus), it requires for each concept to be tested to have a dataset available that captures this concept visually. This represents a limitation that is not discussed in the paper. It is briefly mentioned in line 212, but not recognized as a general limitation.

Further weaknesses include:

  • The approach very much relies on the output of the VLM. If the VLM captures the concept well, the results may be promising but if the concept is not well captured or ambiguous it is questionable, how well the approach works. Furthermore, the VLM may introduce its bias into the analysis and thereby into the explanation.
  • The compared baseline methods TCAV and PCBM are from 2018 and 2023, respectively and not fully up to date.
  • The empirical method for finding the best filling strategy is not motivated and little elaborated. It may even bias results, because different methods are mixed. From the approach one could derive the implicit and underlying hypothese that different concepts require different filling strategies, which is from my perspective not valid. The filling strategy should rather be designed with theoretical considerations in mind, such as reducing the amount of artifacts introduced by it to avoid biasing the model.
Questions for authors
  • Why can PCBM not be applied on individual images to obtain local explanations? Please elaborate?
  • The authors define a threshold (depending on the standard deviation of a fitted Gaussian distribution) to select how many blocks to occlude for a given concept (see Eq. 1). It is questionable, if this threshold works for concepts of different spatial coverage. It seems to work for concepts with small spatial coverage, such as Eye Glasses, but what about concepts like "Sky", which are spatially much more distributed? An empirical evaluation of concepts with different spatial coverage would be important to show that Eq. 1 generally works.
  • The authors state: "For stable attribution scores, measuring Δy is sufficient." If this is stated this way, it needs to at least empirically demonstrated.
  • Please add some qualitative results of attribution scores / maps for TCAV and PCBM
  • Lines 400 and 401: is really the "attribution score" used or the "gender score" defined above?
  • Paragraph 403-408: Maybe the bias observed is in the data and not (only) in the model? Was that verified? Maybe there are many more images with men and tie and this leads to more captions with this pair, so not the model is biased but the data.
Reviewer_pMt4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Hide and Seek (HnS), a method for computing concept attributions, i.e., for computing how important specific pre-defined concepts are for predicting specific outputs. To not rely on a concept-annotated dataset, the paper uses vision-language aligned models to identify image regions in a probing dataset that corresponds to the pre-defined concept. After identifying the image regions in which the concept is present, the paper masks those regions to assess how much the deletion of the concept reduces the output prediction. The paper argues that large drops indicate larger importance while minor drops indicate lower importance. The paper additionally proposes modality correction and orthogonalisation to improve the approach. In different experiments, the paper demonstrates that the proposed method outperforms TCAF and PCBM, and shows the effectiveness of the method in two case-studies.

Critical strengths
  • The paper is for most parts (except W1) well written and easy to follow.
  • The method seems to be a novel combination of established XAI ideas, with method specific adjustments and improvements (e.g., the modality correction and orthogonalisation). Further, the method seems to be fairly straightforward to apply which I consider an advantage.
  • The task of concept attribution is interesting.
Critical weaknesses

W1 The paper overclaims. In my opinion, the paper has several overclaims that are not substantiated by the results or prior work. Some examples include:

  • L. 013 "Existing concept-based methods can only interpret classification and regression models": Sparse autoencoders, for example, can be considered a concept-based method that is agnostic to the downstream task. Additionally, please refer to W6.
  • L. 021 "This [HnS] enables faithful attribution": Faithfulness is only evaluated with an approach very similar to HnS, as also acknowledged in L. 287; the experiment in Appendix G is not convincingly addressing this concern. Further, there are two theoretical arguments against faithfulness: (1) the masking introduces out-of-distribution artefacts (see W2) and (2) combinations of concepts are not considered (see W3).
  • L. 067 "It [HnS] is also the fastest method to measure attributions": Maybe among the tested baselines, but certainly not among all attribution methods.
  • L. 334 "HnS is better than TCAV at image retrieval": Only four qualitative examples are provided and it is unclear how these examples have been selected. Further, the interpretation of these results is not particularly convincing:
    • In Fig. 2 c) I do not see an apple in the center image of the top-3 for HnS (unlike stated in L. 339).
    • In Fig. 2 c) each of the top-3 images for TCAV contains black areas (e.g., black print on t-shirt in the right image; unlike stated in L. 339-340)

W2 Out-of-distribution issues are not discussed and addressed. The paper is based on the premise that removing important concepts leads to a drop in output confidence. However, a lot of prior work has established that deleting image regions for assessing feature importance comes with various limitations such as out-of-distribution issues (Hooker et al., 2019 - "A benchmark for interpretability methods in deep neural networks") or information leakage (Rong et al., 2023 - "A consistent and efficient evaluation strategy for attribution methods").

W3 Combinations of concepts are not discussed and addressed. Masking a concept could result in a low output drop even when the concept is important if the model has learned to consider that concept OR another present concept. I think such cases are currently not handled or discussed.

W4 The paper is a combination of well-established ideas which is not sufficiently clear. The paper proposes a novel method that, however, consists of various well-established ideas (at least in similar form). For example, CLIP-Dissect (Oikarinen et al., 2023) and Discover-then-Name (Rao et al., 2024) use vision-language aligned models for naming/finding concepts; and Explain-Any-Concept (Sun et al., 2023) and various evaluation approaches (e.g. FunnyBirds (Hesse et al., 2023)) perform concept interventions to assess a concept's importance. While the specific combination of steps appears to be novel, from my perspective, these and other similarities should be highlighted and discussed to properly place the paper in the context of related work.

W5 The experimental section lacks depth and breadth. The paper only compares to two baseline methods, i.e. TCAV and PCBM. Since there are quite a few methods that at least would be comparable to components of the proposed method, a more thorough comparison would improve the manuscript. Further, the analysis of the method lacks a bit depth. Sec. 4.1 reports a metric that is very close to the actual approach, Sec. 4.2 is a small-scale and qualitative image retrieval study. Sec. 5 rather presents case-studies, which are interesting but lack a comparison to related work.

W6 The framing about classification and regression is not fully clear. The paper argues that existing methods "can only interpret classification and regression models" (L. 014) while the proposed method is more task agnostic. To illustrate this, the paper considers the task of image captioning. However, the output of the studied model is processed in a way that the model can be interpreted as a regression model (the sum of all male-associated words) which should also be compatible with existing approaches.

Questions for authors
  • Can you please substantiate the highlighted claims of the paper?
  • Could OOD issues and information leakage interfere with the faithfulness of the attribution scores? If not, why?
  • How does the paper fit in the context of the wider XAI landscape? Especially approaches focusing on concept annotation with vision-language models and approaches doing concept interventions.
  • Could you please elaborate on the claim that existing methods can only interpret classification and regression models and that your method does not have this issue?
Reviewer_MZV8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Hide and Seek, a method for post-hoc concept based explainability, presented as the first concept-based method providing both local and global explanations for a wide variety of vision models without accessing internal weights and without a concept set. Instead of resorting to a dataset of concept images, it uses a VLM encoder to codify a textual concept into a vector that, after a processing step, is used as reference. By using an image dataset, the method determines a subset of it relevant to the concept, and measures its contribution as the difference between the output probability when the concept is present and the probability when it is occluded in the image. The method is compared against two methods (TCAV, PCBM) with two datasets (CelebA, COCO-Stuff), showing improvements with respect to state-of-the-art methods, and analyzed on two real-world vision tasks (image captioning and field segmentation).

Critical strengths
  1. Support on design choices

Design choices and hyper parameters such as the embedding preprocessing method, the VLM model, the number of relevant concept images K, the alpha controlling the threshold, the prompt, the images in the concept set, and the occlusion method are all supported by ablation studies.

Critical weaknesses
  1. Dependency on the VLM zero-shot capabilities

In addition to the limitations highlighted by the authors, the use of an external VLM imposes a dependency on its zero-shot capabilities. When considering domain-specific tasks on which a VLM has partial support, the explanations may reflect this. Furthermore, concept isolation is presented as a problem for similarly worded concepts (e.g., dog and canine), but imprecisions in the VLM or different words mapped to undistinguishable vectors could return misleading and unfaithful explanations.

  1. Image patching

The method splits an image into L patches, which are encoded using a VLM, and compared with each other to derive a threshold which is used to define concept presence or absence in a patch. The chosen value of L is not specified in the paper, and its selection is meaningful in a similar manner to parameter K (i.e., the number of positive images), as it defines explanation quality and compute time.

Questions for authors

1 - What value of L was used for the experiments, and which was the resolution of patches for each dataset?

2 - Is it possible that patches of low resolution introduce artifacts when extracting embedding vectors?

3 - With respect to weakness 1, could the authors provide more information about failure modes observed in the performed experiments?

Minor notes

  • Lines 287-288 and 484 should be revised in the final version of the manuscript

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.