WSG-CBM: Weakly Supervised Grounding for Concept Bottleneck Models

Concept Bottleneck Models (CBMs) improve interpretability by introducing human-understandable concepts as an intermediate representation between inputs and predictions. However, accurate concept prediction does not necessarily imply faithful concept reasoning. A model may correctly predict a concept while relying on visual regions that are semantically unrelated to that concept, causing a gap between concept correctness and concept grounding. We argue that concept grounding is a missing ingredient in concept bottleneck learning. To address this challenge, we formulate concept grounding as a weakly supervised concept-specific spatial evidence recovery problem. We propose the Weakly Supervised Grounded Concept Bottleneck Model (WSG-CBM), which learns concept-specific spatial evidence using only image-level concept supervision. To recover meaningful concept evidence under weak supervision, we further introduce grounding objectives based on evidence concentration, disentanglement, and reliance. Unlike existing grounded concept learning approaches, WSG-CBM requires neither concept localization annotations, segmentation masks, bounding boxes, nor external vision-language supervision. Experiments demonstrate that WSG-CBM consistently provides localized and faithful spatial evidence while maintaining competitive concept and classification performance. Our results show that concept grounding can emerge from concept supervision alone and highlight the importance of grounding as a first-class objective for faithful concept-based reasoning.