acceptodds
Under review as a conference paper at ICLR 2027

Beyond Concept Accuracy: Spatial Grounding in Concept Bottleneck Models

Abstract

Concept Bottleneck Models (CBMs) aim to make predictions interpretable by decomposing reasoning into human-interpretable concepts and downstream decisions. While existing CBM research has extensively studied whether task predictions faithfully follow concept representations, it has paid less attention to how concept predictions depend on spatially localized information. Consequently, concept accuracy alone provides limited insight into the spatial evidence supporting a concept. To this end, we introduce a spatial grounding framework for concept representations that characterizes concept support through two complementary properties: spatial sufficiency and spatial necessity. Spatial sufficiency measures whether a region contains enough information to preserve a concept prediction, while spatial necessity measures whether removing that region disrupts the prediction. Together, these measures characterize how concept predictions depend on spatially localized information within the concept-generating representation. Across CUB-200-2011 and RIVAL-10, we find that concept and task accuracy does not determine spatial grounding; comparable predictive performance yields substantially different grounding behavior. Grounding depends on both the concept bottleneck and backbone representation, while joint concept-task optimization do not substantially change grounding. Finally, we show that attribution-based localization and spatial grounding capture distinct properties of concept representations. These findings establish spatial grounding as a complementary dimension for evaluating the interpretability and faithfulness of visual CBMs.

Primary area:
interpretability and explainable AI
Keywords:
concept bottleneck, interpretability, spatial grounding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_h5p7OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies spatial grounding in CBMs, arguing that concept prediction accuracy alone is insufficient to establish whether a model relies on the intended visual evidence. The authors propose an evaluation framework based on spatial sufficiency and necessity, together with a grounding score and a Spatial Concept Influence Matrix. Through experiments on CUB and RIVAL 10, they demonstrate that models with similar concept accuracy may show different spatial grounding properties.

Critical strengths
  1. The paper studies an important and relevant question concerning the faithfulness and interpretability of CBMs.

  2. The distinction between concept accuracy and spatial grounding is meaningful to the community and provides an interesting perspective on evaluating concept representations.

Critical weaknesses

1. Limited novelty and contribution

While the motivation is interesting, I find the methodological contribution to be relatively incremental. The proposed framework mainly builds on existing ideas of sufficiency, necessity, and perturbation based evaluation. In particular, the grounding score is defined as

G(R,ci)=suff⁡(R,ci)⋅nec⁡(R,ci),G(R,c_i) = \operatorname{suff}(R,c_i)\cdot\operatorname{nec}(R,c_i),

which is a rather simple combination of the two proposed measures.

The main finding, that accurate concept predictions do not necessarily imply reliance on the intended spatial evidence, is valuable but also somewhat expected given existing work on concept faithfulness and localization. Similar concerns have already been raised in the literature [7, 8], including empirical evidence that CBMs may predict concepts correctly while relying on irrelevant or spuriously correlated features.

I am therefore not fully convinced that the paper provides enough new insights beyond an additional empirical characterization of known limitations. I think the authors should explain more clearly what new understanding their framework provides compared to previous studies of concept locality and faithfulness.

2. Limited experimental scope and generalizability

Although several architectures and backbones are evaluated, the experiments are restricted to two visual attribute datasets, CUB and RIVAL 10. I find this somewhat limited, especially since the authors present spatial grounding as a complementary dimension for evaluating the interpretability and faithfulness of visual CBMs more generally (Sections 1 and 3).

Some of the analyses are also quite narrow. For example, the comparison of training strategies is conducted with only one CBM and backbone configuration. It is not clear whether the conclusion that training strategy has little influence on grounding would hold across other architectures.

I would have liked to see experiments on additional datasets and possibly different settings, to better understand how general the findings are. Evaluating several existing architectures is useful, but I do not think this alone provides sufficient evidence for the general conclusions of the paper.

Another concern is the dependence on the selected spatial regions. Since the metrics are computed over a region R⊆ΩR\subseteq\Omega, their values may depend on the spatial coverage

ρ(R)=∣R∣∣Ω∣.\rho(R)=\frac{|R|}{|\Omega|}.

The authors acknowledge this issue, but I think a more systematic evaluation of how region size and mask quality affect the results would strengthen the experimental analysis.

3. Insufficient validation of the proposed metrics

I think the proposed metrics require further validation. In particular, the authors perform interventions on the intermediate feature representation F=h(x)F=h(x) by applying a spatial mask MRM_R, such that

FR=MR⊙F.F_R=M_R\odot F.

While this avoids directly masking the input image, setting features outside RR to zero may still produce representations that are outside the distribution seen during training. I am not convinced that performing these interventions in feature space necessarily avoids the distribution shift problem, as suggested by the authors.

I also have some concerns about the mathematical formulation of the proposed measures. For example, spatial sufficiency is defined as

\frac{|f_i(F_R)-f_i(F_{\emptyset})|} {|f_i(F)-f_i(F_{\emptyset})|+\epsilon},$$ with the result clipped to $[0,1]$. Due to the absolute value in the numerator, a region may receive a high sufficiency score even when the prediction changes in the opposite direction relative to the fully masked baseline. For example, if $$f_i(F)=0.9,\quad f_i(F_{\emptyset})=0.5,\quad f_i(F_R)=0.1,$$ the sufficiency score is approximately $1$, although the prediction changes from $0.9$ to $0.1$. This seems inconsistent with the intended interpretation of sufficiency as preserving the original concept prediction. A similar concern applies to the interpretation of the necessity score. I think these issues deserve more discussion and ideally some controlled experiments. For example, comparing different masking baselines, testing the sensitivity to region size, or validating the metrics on synthetic settings where the relevant spatial evidence is known. Overall, I would like to see stronger evidence that the proposed scores measure the intended notion of spatial grounding rather than being affected by the particular intervention procedure. ### 4. Missing relevant related work I also think the related work section could be improved. There are several relevant papers studying spatially grounded or locality aware CBMs that are not discussed. For example, SALF CBM [1], SEG MIL CBM [2], Locality aware CBM [3], DCBM [4], and CHILI [5] all investigate different aspects of spatial concept representations, concept localization, or the relationship between concepts and visual evidence. I understand that these works are not necessarily trying to solve the exact same problem, but I think they are relevant to the motivation of this paper. It would be helpful to better explain how the proposed evaluation framework relates to these approaches, and what limitations of existing methods it addresses. Some of these methods could also be interesting additions to the experimental comparison. In addition, the proposed sufficiency and necessity measures should be better connected to existing literature on faithfulness evaluation, such as ERASER [6], which already studies related ideas through sufficiency and comprehensiveness. While the setting is different, the underlying idea of measuring how a prediction changes when evidence is retained or removed is closely related. I think a more complete discussion of these works would help clarify what is actually novel in the proposed formulation, and better position the paper within the existing literature. ### 5. Limited practical implications The paper introduces an evaluation framework, but I find that the practical implications remain somewhat unclear. Beyond showing that different models achieve different grounding scores, it is not clear how these measurements can be used to improve CBMs or inform their design. For example, if a model achieves high concept accuracy but low spatial grounding, what should we do differently when training or selecting that model? It would be interesting to see whether optimizing for higher grounding actually improves concept faithfulness, or whether the proposed scores correlate with improved robustness to spurious features. I understand that the goal of the paper is primarily evaluation rather than proposing a new training method. Still, I think showing at least one practical use case for the framework would make the contribution more convincing. ### 6. Reproducibility and code availability As far as I could tell, the authors have not released the code for the proposed framework or experiments. Given that the main contribution is an evaluation methodology, I think making the implementation available is particularly important for reproducibility. ## References [1] [Show and Tell: Visually Explainable Deep Neural Nets via Spatially Aware Concept Bottleneck Models](https://openaccess.thecvf.com/content/CVPR2025/html/Benou_Show_and_Tell_Visually_Explainable_Deep_Neural_Nets_via_Spatially-Aware_CVPR_2025_paper.html). Itay Benou and Tammy Riklin Raviv. CVPR, 2025. [2] [From Segments to Concepts: Interpretable Image Classification via Concept Guided Segmentation](https://arxiv.org/abs/2510.04180). Ran Eisenberg et al. arXiv, 2025. [3] [Locality aware Concept Bottleneck Model](https://arxiv.org/abs/2508.14562). Sujin Jeon et al. arXiv, 2025. [4] [DCBM: Data Efficient Visual Concept Bottleneck Models](https://proceedings.mlr.press/v267/prasse25a.html). Katharina Prasse et al. ICML, 2025. [5] [Enhancing Concept Localization in CLIP based Concept Bottleneck Models](https://arxiv.org/abs/2510.07115). Rémi Kazmierczak et al. TMLR, 2026. [6] [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://aclanthology.org/2020.acl-main.408/). Jay DeYoung et al. ACL, 2020. [7] [Do Concept Bottleneck Models Learn as Intended?](https://arxiv.org/abs/2105.04289). Andrei Margeloiu et al. arXiv, 2021. [8] [Do Concept Bottleneck Models Respect Localities?](https://arxiv.org/abs/2401.01259). Naveen Raman et al. arXiv, 2024.
Questions for authors
  1. Regarding Weakness 3, could the authors clarify why zero masking in the intermediate representation is expected to avoid distribution shift? Have alternative masking strategies been considered?

  2. Regarding Weakness 3, how do the authors justify the use of absolute differences in the sufficiency and necessity definitions, particularly when the masked prediction changes in the opposite direction?

  3. Regarding Weakness 2, is there additional evidence supporting the generality of the conclusions across datasets and training configurations beyond those currently evaluated?

  4. Regarding Weakness 4, could the authors clarify how the proposed framework differs from existing spatially grounded CBM approaches, particularly SALF CBM and SEG MIL CBM, and what additional insights it provides?

4 comments

We thank the reviewer for the detailed feedback. We appreciate the recognition of the importance of the problem and agree that the relationship to prior work, the interpretation of the proposed metrics, and their practical value should be clarified. We address the concerns below.

Novelty and distinction from prior work. We agree that our framework builds on established ideas of sufficiency, necessity, and perturbation-based evaluation. Our contribution is not to introduce these principles independently, but to adapt and combine them to characterize the spatial support of concept predictions within the representation consumed by the concept predictor.

The key distinction from concept locality is the joint evaluation of spatial sufficiency and necessity. Locality or localization identifies regions associated with a concept, but does not necessarily establish whether the region alone retains the concept evidence or whether the prediction depends on that region. These properties are not equivalent. For example, a region can be sufficient but not necessary when similar evidence is redundantly encoded elsewhere. Conversely, a region can be necessary but insufficient when evidence is distributed across multiple regions. The resulting support regimes provide information that cannot be obtained from localization or concept accuracy alone. We strengthen the discussion of Margeloiu et al. and Raman et al. and clarify that our framework complements prior concept-faithfulness and locality analyses by explicitly characterizing these spatial support regimes.

More broadly, our framework reveals that concept support is multi-dimensional. Models with similar concept accuracy or locality can exhibit fundamentally different support regimes, such as high sufficiency/low necessity or low sufficiency/high necessity, which are not visible under existing evaluations.

Experimental scope and generalizability. We acknowledge that the evaluation is limited to CUB and RIVAL-10 and that broader validation across datasets would strengthen the conclusions. Nevertheless, our experiments cover multiple bottleneck architectures, backbone families, and training strategies. In particular, sequential and joint training produce similar grounding scores on both CUB and RIVAL-10 (Appendix Table 9). We have therefore revised the manuscript to present these observations as evidence within the evaluated settings, rather than as general conclusions that necessarily extend to all CBM configurations.

Regarding spatial coverage, our evaluation includes regions derived from ground-truth masks, attribution maps, random regions, and object masks. These comparisons use controlled spatial coverage, and region size is reported in Table 3(B). We agree, however, that this does not constitute a systematic study of mask quality, such as sensitivity to annotation noise, mask erosion, or dilation. We clarified this limitation and qualify the conclusions accordingly.

We would be grateful if the reviewer could indicate specific datasets, CBM variants, or evaluation settings they believe are most critical and currently missing, as this would help us prioritize the most informative extensions in the revision and future work.

Validation of the proposed metrics.

  • Distribution shift from feature masking: We agree that feature-space interventions do not eliminate distribution shift entirely. Our claim is more limited: intervening on the representation consumed by the concept predictor isolates the contribution of spatial evidence while avoiding the substantial low-level perturbations introduced by image-space masking. We therefore view feature masking as a pragmatic intervention strategy rather than a guarantee of in-distribution evaluation.

Despite these limitations, the proposed metrics exhibit consistent behavior across controlled evaluations. They assign substantially higher grounding to ground-truth concept regions than to random regions and reliably distinguish architectures designed to promote spatially aligned concepts from standard CBMs (Table 3). Although this cannot provide a formal guarantee that the metrics perfectly capture grounding, it offers empirical evidence that they measure meaningful properties of spatial concept support rather than artifacts introduced by the intervention mechanism.

  • Absolute difference in the metrics: We agree that absolute differences do not encode directionality. Our intention was to measure the magnitude of dependence on a region rather than strict prediction preservation. Under this interpretation, sufficiency quantifies how much concept evidence is retained relative to the fully masked baseline, irrespective of direction. We acknowledge that direction-aware variants are possible and may provide additional insight. Importantly, our conclusions are derived from aggregate trends across concepts and models rather than isolated sign-flip cases.

Relationship to SALF-CBM and SEG-MIL-CBM. We agree that these approaches are relevant and will expand the related work discussion. SALF-CBM and SEG-MIL-CBM investigate spatially informed concept learning and localization through their respective architectural and learning strategies. Our work addresses a complementary question: given a trained CBM, does the spatial region associated with a concept actually support the prediction made by its concept predictor?

Our framework is an evaluation methodology rather than another spatially constrained architecture. It can therefore be applied across different CBM designs to examine whether their learned representations exhibit sufficient, necessary, redundant, or distributed spatial evidence. We also discussed DCBM, Locality-aware CBM, CHILI, and ERASER to better position the work in relation to spatial concept learning and faithfulness evaluation in the related work section of the revised version.

Practical implications. Our framework is intended as a diagnostic tool, not as a claim that maximizing the grounding score necessarily improves robustness or faithfulness. A high concept accuracy combined with low grounding indicates that correct predictions do not necessarily rely on the intended spatial evidence. The separate sufficiency and necessity scores further help identify whether this results from insufficient regional evidence or redundant/distributed evidence. Such diagnoses can inform model selection and motivate different architectural interventions. We will clarify these practical implications while acknowledging that establishing whether grounding optimization improves robustness requires further experiments.

Reproducibility. We agree that code availability is important for an evaluation framework. To preserve anonymity during double-blind review, we cannot publicly release the repository at this stage. We commit to releasing the implementation and experimental code upon acceptance and providing the repository link in the final version.

Thank you for the detailed response and the revised version.

What has been addressed. The related work is much improved, especially the new discussion of spatially-aware CBMs and ERASER. The clarifications on clipping and standard deviations, and the more careful wording of the main findings, are also helpful. The explanation given to Reviewer eVPK about where the interventions are applied would be worth adding to the main text.

What remains.

  1. Validation of the metric. Table 3 does not convince me that the scores measure grounding rather than the intervention itself. Random regions reach almost the same sufficiency as the annotated part masks, and several other regions (object masks, saliency maps) give higher grounding than the part masks. A controlled setting where the evidence location is known, and an alternative masking baseline, would help.

  2. The formulation. Equations 2 and 3 still use the absolute value ∣fi(FR)−fi(F∅)∣|f_i(F_R) - f_i(F_\emptyset)|, so the direction of the change is ignored. Appendix C.1 notes that for SL-CBM, nonzero necessity occurs only when sufficiency≈1\text{sufficiency} \approx 1, so grounding=sufficiency⋅necessity\text{grounding} = \text{sufficiency} \cdot \text{necessity} effectively reduces to necessity. This suggests the scores may depend strongly on how the concept head aggregates spatial features, which deserves discussion. In addition, according to the Figure 1 caption, the distributed regime (high necessity, low sufficiency) used to motivate the joint formulation is never observed.

  3. Distribution shift. The response acknowledges that feature masking does not remove distribution shift. However, the paper still states that it avoids it (l. 090-091) and now criticizes input perturbations for the same issue.

  4. Scope, practical use and code. No new experiments were added. The training-scheme comparison covers only the standard CBM (Table 9), and I could not find the mask-quality limitation mentioned in the response. No practical use case is shown, and an anonymized code repository would be compatible with double-blind review.

Overall, the revision improves the presentation and makes the claims more careful, and I thank the authors for engaging openly. However, my main technical concerns remain, so I will keep my score.

Thank you for the careful reading and for the detailed follow-up. We appreciate the recognition that the revised manuscript clarifies the relationship to prior work and improves the presentation of the framework.

Validation of the metric. We agree that the proposed scores are defined through interventions and therefore quantify the effect of retaining or removing spatial evidence on concept prediction. However, this is intentional rather than incidental. We define a concept as spatially grounded when the corresponding concept prediction is supported by the spatial evidence that is semantically associated with that concept. Under this definition, sufficiency and necessity provide an operational test of whether the concept predictor relies on the expected spatial evidence.

In our primary evaluation, ground-truth concept masks provide a controlled reference for where concept evidence is expected to reside. Spatial grounding therefore measures whether the concept predictor utilizes the corresponding spatial feature information in a manner aligned with human annotations. We agree that stronger validation through synthetic settings with known causal evidence or alternative intervention baselines would further strengthen the study.

However, we respectfully disagree that the current results suggest the metrics are dominated by the intervention procedure itself. For example, in Table 3(B), annotated part masks achieve substantially higher sufficiency than random regions (0.921 vs. 0.867), despite comparable spatial coverage. The proposed scores consistently rank semantically meaningful regions above random regions and distinguish architectures designed to encourage spatial alignment from standard CBMs, providing empirical evidence that they capture meaningful properties of spatial support rather than arbitrary intervention artifacts.

Our framework intentionally reveals cases where alternative regions (e.g., object masks or attribution-derived regions) achieve higher grounding than the annotated concept region. We do not interpret this as a failure of the metric; rather, it indicates that the model relies on spatial evidence that differs from the evidence designated by the human annotation. This is precisely the type of model–human misalignment that the proposed grounding analysis is intended to expose.

The formulation. Indeed, our intent was to measure the magnitude of spatial dependence rather than directional agreement with the original prediction. We agree that this choice warrants further discussion, particularly for architectures such as SL-CBM and in settings where necessity dominates the combined grounding score. It is indeed related to the design of concept predictor, especially SL-CBM derives concept predictions through cross-attention between spatial features and concept embeddings from a concept bank. As a result, concept evidence can be aggregated from multiple spatial locations, which naturally leads to high sufficiency but comparatively low necessity. We view this behavior as an informative property of the model rather than a failure of the metric, as it reveals how concept evidence is distributed within the representation. We have therefore revised the manuscript to clarify the interpretation and limitations of the current formulation and note that direction-aware variants are an interesting direction for future work.

While the distributed regime (high necessity, low sufficiency) is not observed in our current experiments, the redundant regime (high sufficiency, low necessity) is consistently observed across multiple architectures and datasets. This observation itself motivates the joint formulation, as sufficiency and necessity capture complementary aspects of concept support and reveal forms of evidence redundancy that would not be visible from either measure alone.

Distribution shift. Our claim is not that feature-space interventions eliminate distribution shift, but rather that they avoid some of the low-level perturbations introduced by image-space masking and propagated through the entire backbone, while allowing interventions to be performed directly on the representation consumed by the concept predictor. We have revised the manuscript accordingly to clarify that feature-space masking should be viewed as a pragmatic intervention strategy rather than a guarantee of in-distribution evaluation.

Scope and reproducibility. We agree that broader empirical validation would strengthen the conclusions and have revised the discussion to better qualify the scope of our findings. We will also release the implementation upon acceptance.

We sincerely appreciate the detailed feedback. The discussion has helped us substantially improve both the presentation of the framework and the clarity of its assumptions and limitations.

Reviewer_eVPKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper looks at spatial grounding in concept-based models, specifically looking at whether regions of an input are responsible for a specific concept prediction. They develop a set of metrics to assess this, including definitions for spatial concept influence (and a SCIM matrix), spatial grounding, and concept purity. They then run experiments with a variety of concept-based models to identify when models have impure concepts/patterns in their spatial grounding. They identify that grounding depends on the bottleneck + backbone, with custom-made CBMs (e.g., SL-CBM) performing well.

Critical strengths
  1. Interesting new metrics - The authors introduce a set of new metrics that seem rigorous and well-motivated. These metrics help define what spatial grounding look like, and whether concept bottleneck models truly learn the concepts that they set out to learn. The assessment of these metrics in practice is also interesting, with the experiments covering a fairly representative range of outcomes.
Critical weaknesses
  1. Unclear differentiation from prior work - My biggest worry is that it's unclear why we need new metrics to cover spatial grounding. Prior work (e.g., Margeliou et al. and Raman et al.) cover similar metrics, and moreover, the SL-CBM which was designed to tackle some of these issues also performs well here. Intuitively, spatial grounding is somewhat similar to concept locality, and so I'm not sure why exactly these new metrics cover this. Section 4.4 starts to get at some of this, but it doesn't fully explain why these new metrics are needed or how they differ.

Maybe more generally, I think the new metrics of this work are clear, and I appreciate the writing as well. I'm just not sure whether this changes my understanding of concept-based models, and whether these metrics are really necessary.

Questions for authors
  1. On page 6, the authors mention that grounding is not the same as accuracy; however, the set of models they test have similar values of grounding and similar values of accuracy (both range in a fairly narrow spectrum). It seems that only for SL-CBM the grounding really increases significantly; are there examples (outside of SL-CBM), with pairs of similarly accurate models actually differing heavily in grounding?
  2. Is the practical recommendation then to just use SL-CBM? What's the new insight practitioners should have based on this?
  3. How does the experiments with grounding/masking differ from some of the masking experiments in Raman et al.?
  4. When is purity a desirable property? I ask because for datasets like CUB, many of the concepts do overlap, so I'm not sure if purity would be appropriate there.
8 comments

We thank the reviewer for the thoughtful feedback. We appreciate the positive assessment of the clarity of the metrics and agree that their necessity and practical implications need to be articulated more clearly. We address the concerns below.

Differentiation from concept locality and prior work. We agree that spatial grounding is closely related to concept locality, and we do not intend to position it as an entirely separate problem. However, locality and grounding address complementary questions. Locality asks whether concept information or an explanation can be associated with a particular image region. Our framework asks whether that region functionally supports the concept prediction in the spatial representation consumed by the concept predictor.

The distinction becomes clearer by considering sufficiency and necessity jointly. A region can be sufficient because it retains enough evidence to reproduce a concept prediction, while not being necessary because similar evidence is available elsewhere. Conversely, a region can be necessary while being insufficient on its own if concept evidence is distributed across multiple locations. A localization score alone cannot distinguish these cases.

Thus, our contribution is not simply another localization metric, but an evaluation of the functional structure of spatial evidence. We will strengthen Section 4.4 to explain why these two complementary measurements provide information beyond localization and concept accuracy, and better position the work in relation to Margeloiu et al. and Raman et al.

Examples of similarly accurate models with different grounding. We agree that the differences among most bottleneck variants in Table 2(A) are relatively small in terms of their final grounding scores. We will avoid overstating this result. However, the component measures reveal differences that the aggregate score alone can obscure.

For example, on CUB with the same frozen ResNet-18 backbone, VLG-CBM and SL-CBM have identical concept accuracy (93.8%), but substantially different support profiles. VLG-CBM has sufficiency 0.35 and necessity 0.29, whereas SL-CBM has sufficiency 0.92 and necessity 0.08. Thus, SL-CBM makes the concept prediction highly recoverable from the target region, but removing that region has little effect, indicating redundant evidence elsewhere. This is a qualitatively different support regime, despite the identical concept accuracy.

We will make this example more prominent and clarify that the important distinction is not always a large difference in the aggregate grounding score, but also the different spatial support patterns captured by its two components.

Is the practical recommendation simply to use SL-CBM? No. Our results do not establish SL-CBM as universally preferable. In fact, the results illustrate why a single localization-oriented design choice may not be sufficient. On CUB, SL-CBM achieves very high sufficiency (0.92) but low necessity (0.08), resulting in a grounding score of 0.08. On RIVAL-10, it achieves higher grounding (0.27), with sufficiency 0.946 and necessity 0.270. The effect of the same bottleneck design therefore differs across datasets.

The practical value of our framework is diagnostic: it helps practitioners understand whether a concept prediction is supported by its intended region, whether evidence is redundant, or whether it is distributed. This can inform model selection and subsequent architectural or training decisions. For example, high sufficiency combined with low necessity suggests that encouraging localization alone may not remove redundant evidence. We do not claim that optimizing the proposed score necessarily improves robustness or faithfulness; establishing such relationships is an important direction for future work.

Difference from masking experiments in Raman et al. We agree that our intervention-based evaluation is related to the locality analyses in Raman et al. We will clarify this relationship rather than present the approaches as unrelated.

The key distinction is the evaluation objective and the way the interventions are interpreted. Raman et al. investigate concept locality through input perturbations and examine how concept predictions respond to changes in image regions. Our framework intervenes directly on the spatial feature representation consumed by the concept predictor. This allows us to evaluate two complementary conditions: retaining only a region and removing that region while retaining its complement. We then quantify sufficiency and necessity relative to a common fully masked reference.

Consequently, our framework characterizes not only whether a region influences a concept prediction, but also whether it is sufficient and necessary for the prediction. We will make this distinction explicit and acknowledge the conceptual connection to prior locality analyses.

When is spatial purity desirable? We agree that purity should not be interpreted as a universally desirable property. In CUB, concepts such as wing color and wing pattern can legitimately share the same spatial region. In such cases, a region associated with one concept may also influence another, resulting in lower purity even when both concepts are semantically grounded.

Purity is therefore useful when the goal is to assess spatial specificity or disentanglement between concepts, particularly when concepts are expected to have distinct spatial support. It is less appropriate as an optimization target when concepts naturally overlap. We will clarify that purity is a diagnostic measure of cross-concept spatial influence, rather than a requirement for a good CBM. Spatial grounding and purity capture complementary properties: the former measures support for a concept from its associated region, while the latter measures how specific that region's influence is to the concept.

What changed our understanding of CBMs? More broadly, our framework reveals that spatial support itself is multi-dimensional. Models with similar localization quality or concept accuracy can exhibit fundamentally different support profiles, such as high sufficiency/low necessity or low sufficiency/high necessity. These distinctions are not visible through concept accuracy or locality metrics alone. We therefore view the primary contribution not only as a new metric, but as evidence that concept predictions can rely on spatial evidence in qualitatively different ways despite appearing similar under traditional evaluations.

Reviewer_eVPKOpenReview ↗

Rebuttal Reply

Thank you for your rebuttal. I'm still confused on what exactly the difference is between locality and the metric you propose. You mentioned considering sufficiency and necessity; what exactly does locality not cover that this new metric does cover? Locality covers, from my understanding, whether particular regions of an image impact a particular concept (e.g., the masking experiment in Raman et al.). Their experiments masking different parts and measuring the impact on other parts gets at both the necessity and sufficiency I thought, so I wasn't sure where this new framework you propose fits in.

Reviewer_D6vYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper focuses on explainable AI (XAI), and introduced a spatial grounding framework for concept representations in Concept Bottleneck Models (CBMs).

Critical strengths
  • The paper is easy to read.
  • Detailed experimental results are conducted.
Critical weaknesses
  • Spatial information has been investigated a lot in existing work in XAI, including the ones in CBMs. The work is quite incremental and lacks novelty.
  • Spatial areas/regions are used to ground the concepts. However, it’s even not clear if all concepts can be linked to spatial areas/regions. In this sense, the motivation could be mistaken.
  • On page 3, the second paragraph, it says “Our objective is different:”. It sounds not different in fact, at least there is a close relationship.
  • For example, in Eqn (1), it’s not clear how to choose the region R, and it’s very time consuming to do so. The shape of R is not clear. In the manuscript, it seems only continuous region of R is considered, which may not make sense always.
  • On page 4 below Eqn (3), it says “We clip the results to [0, 1]”. It’s not clear if doing so is reasonable, and it’s not clear either what would happen if not doing so.
  • From the experimental results, it seems the method doesn’t work well, e.g. “Grounding != Accuracy”. It sounds plausible, but it doesn’t give any useful interpretation from the neural architectures.
  • The standard deviation (if this is what the authors mean) in e.g. Table 2 is too large.
  • There are also lots of relevant XAI methods expect CBMs, but they are not considered in the manuscript for comparison.
Questions for authors

See above the weaknesses.

4 comments

We thank the reviewer for the thoughtful and constructive feedback. We agree that the original manuscript did not clearly communicate the central contribution of the work, its distinction from existing localization and locality-based analyses, or the rationale behind several key design decisions in our framework. These comments helped us identify important points that were insufficiently explained in the submission. We have substantially revised the manuscript accordingly, including a clearer positioning of spatial grounding within the CBM literature, a more explicit discussion of the motivation and practical implications, and additional clarification of our metric design. We believe these revisions have considerably improved the clarity and overall presentation of the paper. We respond to each concern in detail below.

Novelty. We agree that the original presentation did not clearly distinguish spatial grounding from existing localization and locality analyses. However, our contribution is not another localization method. We introduce spatial grounding as a representation-level property that measures whether concept predictions are supported by the spatial feature regions that humans would expect to provide the relevant evidence. Unlike concept accuracy, which asks whether a concept is predicted correctly, and localization/locality methods, which ask where a concept can be associated in the input image, our framework asks whether the concept representation is functionally supported, both sufficiently and necessarily, by the corresponding spatial features from which it is computed. To quantify this property, we measure sufficiency and necessity of concept regions and introduce purity to assess whether concept predictions rely on information from unrelated spatial regions. Thus, we view spatial grounding as a new evaluation dimension for CBMs, complementary to accuracy, localization, and concept leakage, together with principled definitions and metrics for measuring this form of human-aligned concept faithfulness.

We improved the introduction section accordingly in the revised version. Importantly, localization methods answer whether a concept can be associated with a region, whereas grounding asks whether information from that feature region is actually used to form the concept representation.

Why feature-space grounding matters. A key distinction from prior locality work is that we study grounding in the spatial feature representation immediately preceding concept prediction, rather than in the input image. Concept predictors operate on these spatial features, making grounding a property of the learned representation rather than a post-hoc explanation. We define a concept as spatially grounded when its prediction is supported by the semantically relevant spatial evidence encoded in this representation. Accordingly, a grounded region should be both sufficient (retaining it preserves the concept prediction) and necessary (removing it substantially degrades the prediction). These complementary requirements motivate our formulation and jointly operationalize the notion of spatial grounding.

This distinction is important because a concept can be correctly localized yet still rely on information outside the intended region. For example, a concept may localize to a bird's wing while its activation remains partially driven by contextual cues elsewhere in the image. As shown in Figure 1, all images are evaluated using ground-truth concept annotations and therefore share perfect concept localization. Despite this, they exhibit markedly different spatial grounding scores. This highlights our central observation: localization identifies where a concept is present, whereas spatial grounding evaluates whether the concept representation actually relies on that evidence. Thus, localization does not imply grounding. Our framework is designed to measure this representation-level notion of concept faithfulness. We improved the introduction section accordingly in the revised version.

Not all concepts are spatially localizable. We agree that not all concepts admit a unique spatial grounding. Our framework is intended for visual concepts with identifiable spatial manifestations, such as object parts and visual attributes. More broadly, low grounding scores can reveal concepts whose evidence is inherently distributed rather than spatially localized.

Mathematical definition of region. We agree that the manuscript did not clearly explain how R is defined. In our primary evaluation, R is obtained directly from human-provided concept annotations, making ground-truth segmentation masks the natural choice because our goal is to assess whether the spatial support of a concept in the learned representation aligns with human-understood concept evidence. We further validate the framework using multiple alternative sources of R (Section 4.3, Table 3), demonstrating that the formulation is not tied to a particular region generator. Importantly, our framework does not assume that concepts are supported by a single contiguous region. Since grounding is quantified through complementary sufficiency and necessity measurements, evidence may be distributed, overlapping, or redundant across multiple locations. Consequently, a low grounding score should not be interpreted as a model failure; rather, it indicates that the concept prediction cannot be fully attributed to the intended spatial evidence and may instead rely on distributed or auxiliary information.

Clipping. Values above 1 arise when masking increases the relative concept evidence compared to the original representation, for example by suppressing competing evidence. Since our goal is to quantify whether the region contains sufficient (or necessary) evidence for the concept, rather than whether masking amplifies the score, any value above 1 (or below 0) already indicates complete sufficiency (or necessity) and is therefore saturated at 1 (or 0). We added the explanation to this choice in the method section in the revised version.

Grounding!=Accuracy. We agree that "Grounding ≠ Accuracy" alone is not sufficient insight. The practical implication is that concept accuracy cannot be used as a proxy for concept faithfulness. This is particularly relevant for CBMs, where concept predictions are often interpreted as evidence that the model has learned the intended concept. Two CBMs with similar concept accuracy can rely on substantially different spatial evidence (e.g. Table 2 (A) RIVAL-10). Therefore, model selection based solely on concept prediction performance may favor models whose concept representations are less aligned with the intended visual evidence. Our framework makes this hidden difference measurable.

Standard deviations. To clarify, the reported standard deviations are computed over concept-image pairs rather than across training runs. Therefore they quantify dataset heterogeneity and should not be interpreted as uncertainty of the reported mean performance. We revised the table caption and experimental setting to avoid this misunderstanding.

Existing XAI. Our goal is not to compare prediction performance against generic XAI methods, since they do not produce concept representations. Instead, they serve as alternative region generators that allow us to test whether high localization quality translates into functional concept support under our grounding metrics.

Many thanks for the detailed responses. I would like to commend the effort the authors have made. However, for the work in the manuscript, it has serious fundamental issues, e.g., the novelty and its practicality. The work is focused on CBMs and is incremental based on CBMs. The extra explanation the work provides except for CBMs is limited and is hard to obtain since the spatial area itself is hard to define and select. Moreover, the spatial features in deep neural networks are generally global and local, it is impractical to be (directly) used to do spatial grounding as the work introduced. I sincerely wish the work to be accepted in the near future, but for the current version here, I am reluctant to suggest acceptance.

Thank you for the thoughtful discussion and for carefully reading the revised manuscript. We appreciate the reviewer's recognition of our efforts and understand the concerns regarding novelty and practical impact.

We respectfully view the contribution differently. Our objective is not to introduce a new spatially-aware CBM architecture, nor to improve concept prediction performance, but to introduce spatial grounding as a complementary evaluation dimension for concept representations. In particular, we study whether concept predictions are supported by the spatial feature evidence from which they are formed, rather than whether concepts can simply be localized or predicted accurately.

We agree that modern deep representations contain both local and global information. Indeed, this observation motivates our framework. Rather than requiring concept information to remain perfectly localized throughout the network, we explicitly allow information aggregation within the backbone and evaluate whether the final concept predictor relies on the spatial evidence that humans would expect to support the concept. In our view, this makes spatial grounding a complementary notion to locality rather than a replacement for it.

We also agree that the scope of the current empirical study is limited and that stronger validation would further strengthen the work. Following the reviewers' feedback, we will revise the manuscript to better qualify the scope of the conclusions, clarify the assumptions underlying the framework, and more clearly position the contribution relative to prior work on locality, faithfulness, and spatially-aware CBMs.

Regardless of the final outcome, we sincerely appreciate the feedback and discussion, which helped us substantially improve the presentation, positioning, and limitations discussion of the work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.