- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper tries to answer when the CBMs can provide faithful compact explanations for the predictions. The authors first identify the problem and then proposed two simple but effective stratgies that encourage CBMs to admit compact faitful explanations. Extensive experiments on both regular and VLM-based CBMs confirm that both constraints on the structure of the activation space and the structure of the inference layer's weights are needed.
Critical strengths
- the paper targets a real and important question in XAI. The propositions and conclusions are well supported by experiments and mathematical proofs.
- The formalization and the presentation of the paper is clear and easy to follow.
Critical weaknesses
- the conclusion of faithful explanations cannot be compact is not so surprising with unbounded activations. Proposition 1 shows that with unbounded activations and a linear head, any free concept with a non-zero weight difference can be pushed to infinity to flip the prediction. This follows almost directly from the definition, and it is the known reason why prior work on AXps for linear classifiers assumes bounded features.
- The group lasso gains are confounded with feature removal and accuracy loss. Column-wise group lasso zeroes out entire concepts, which is equivalent to training a CBM with a smaller bottleneck. Because explanation size is reported as a percentage of the original , improvements may largely reflect how many concepts were removed. The paper reports neither the number of surviving concepts nor a comparison against simply training on a smaller vocabulary.
Questions for authors
- The problem and the motivation for the study seem reasonable. However, the example in Figure 1 appears to be more of a hypothetical case than a real-world one, or too artificial. Can you provide some real-world failure examples to support the unfaithfulness of heuristic explanations?
- when you apply group lasso, how are the groups defined for each dataset? How many concepts survive group lasso in each setting?
3 comments
Reply to reviewer NpZx part 2
how are groups defined for glasso? How many concepts survive it?
One group per concept activation (Sec 4.2, l 266): the column of inference-layer weights attached to that activation, across all classes. Zeroing a group means no class uses that activation. This is the same for all datasets.
The number of activations used by at least one class drops substantially with glasso:
- Derm: dense 27 -> glasso 4
- WBC: 31 -> 6
- CUB: 230 -> 43 (some are all zero in the data, dense learns not to use them).
Comparison against a smaller vocabulary
By Prop 1, the gain comes precisely from “having fewer concepts”: optimizing loss + glasso dynamically constructs a smaller decision-focused vocabulary. The question of whether it outperforms offline feature selection rests on the choice of feature selection method. We’ll clarify this in the text.
I thank the authors for their detailed response and the additional results. The real-world counterexamples address my concern about Figure 1. However, I still feel that the proposed remedies, probabilistic concepts and group lasso, are well-established techniques, and that the gains from group lasso largely stem from removing concepts, as the reported numbers of surviving concepts suggest. I will therefore maintain my current rating.
Reply to reviewer NpZx
Thank you for appreciating how the paper addresses “a real and important question”, that our theory and conclusions are “well supported by experiments and mathematical proofs”, and that our manuscript is “clear and easy to follow”.
We address your other comments below. Please let us know if anything remains unclear, we are keen to engage.
The less obvious part is getting to Prop 1: one has to first consider the faithfulness of CBM explanations (a gap in the literature), apply formal explainability to CBMs (another gap), and identify the compactness-faithfulness trade-off (a third gap). Our goal is precisely to make the CBM community aware of these issues and to combat the implicit but widespread tendency to equate concept bottlenecks with reliable interpretation.
Note also that the main text focuses on Prop 1 precisely because it's intuitive. Our results in Sec B on bounded activations are less straightforward.
As noted in Sec 6 (line 442), FXAI algorithms for linear models assume activations are bounded, compatibly with Prop 1. But we go one step further: we deduce that if one needs compact AXps, they should constrain the model in terms of activations and/or weights. To the best of our knowledge, the FXAI literature has not proposed anything similar, and definitely not for CBMs.
We ensure a fair comparison between sparse and dense CBMs by "maximizing λ under the constraint that its label F1 is at most 10% worse than the dense CBM's on the validation split." (Sec 5.2, l 376; Sec D.5, l 1629)
Note that "turning off" concepts is precisely how glasso helps: by Prop 1, faithful explanations must cover every activation with a non-zero weight difference, so the number of activations still in use is what drives AXp size. We agree that for VLM-CBMs the gain comes from this.
Below we report Tab 4 with absolute numbers (avg # activations in the explanation) instead of percentages, for TopK-Act and EA+:
The message of Sec 5.2 is unchanged. Enet only partially works: it shrinks explanations of CBMs, but on VLM-CBMs it only helps EA+ (and it inflates TopK-Wei/Prod explanations in three cases, see Tab 4). Glasso shrinks explanations w.r.t. the dense model (except TopK-Act on VLM-CBMs, where it misbehaves everywhere), loses to enet only in two cases by a small margin (WBC/EA+: 2.5 vs 2.3; CUB/TopK-Act: 44.1 vs 39.9), and is essential for VLM-CBMs (EA+: 35.4 vs 167.0 on LF-CBM, 66.3 vs 287.2 on VLG-CBM, 640.7 vs 4682.9 on LaBo).
We’ll report the full table in the supplement.
Yes, Fig 1 is meant as an illustration. Below we report real counterexamples (failures) for TopK-prod (weights x activation) heuristics, computed on dense probabilistic CBMs, focusing on correctly predicted test inputs. In each case, we keep the top-k concepts fixed and change a single concept not in the explanation, flipping the prediction:
EA+ explanations of the same points (41, 7 and 10 concepts) include the changed concept, and are faithful by construction. We will add these examples to the appendix.