acceptodds
Under review as a conference paper at ICLR 2027

When are Concept Bottleneck Model Explanations Faithful and Compact?

Abstract

Concept bottleneck models (CBMs) are neural classifiers that allow to explain their decisions via high-level concepts, potentially enabling understanding, steering and debugging. However, their explanations are often derived heuristically. Building on formal explainability – we argue they should also be faithful, i.e., not misreport which concepts actually matter. We show that, for widespread CBM architectures, including recent VLM-based variants, faithful explanations must include all concepts in the bottleneck, compromising interpretability when this is large. This result applies to both heuristic and faithful-by-construction formal explanations. To ensure compact faithful explanations exist, we suggest i) modeling concepts probabilistically as binary or categorical random variables (rather than logits), and ii) employing per-concept training-time sparsification via group lasso (rather than regular elastic net). We also extend algorithms from formal explainability to CBMs, and show they outperform heuristics in terms of guarantees and explanation size. Overall, our work warns against naıve interpretability claims and provides formal conditions and practical strategies for encouraging CBMs to be as interpretable as advertised.

Primary area:
interpretability and explainable AI
Keywords:
Concept-bottleneck Models, Explainable AI, Formal Explainability, Probabilistic Modeling, Sparsity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NpZxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tries to answer when the CBMs can provide faithful compact explanations for the predictions. The authors first identify the problem and then proposed two simple but effective stratgies that encourage CBMs to admit compact faitful explanations. Extensive experiments on both regular and VLM-based CBMs confirm that both constraints on the structure of the activation space S\mathcal S and the structure of the inference layer's weights WW are needed.

Critical strengths
  • the paper targets a real and important question in XAI. The propositions and conclusions are well supported by experiments and mathematical proofs.
  • The formalization and the presentation of the paper is clear and easy to follow.
Critical weaknesses
  • the conclusion of faithful explanations cannot be compact is not so surprising with unbounded activations. Proposition 1 shows that with unbounded activations and a linear head, any free concept with a non-zero weight difference can be pushed to infinity to flip the prediction. This follows almost directly from the definition, and it is the known reason why prior work on AXps for linear classifiers assumes bounded features.
  • The group lasso gains are confounded with feature removal and accuracy loss. Column-wise group lasso zeroes out entire concepts, which is equivalent to training a CBM with a smaller bottleneck. Because explanation size is reported as a percentage of the original bb, improvements may largely reflect how many concepts were removed. The paper reports neither the number of surviving concepts nor a comparison against simply training on a smaller vocabulary.
Questions for authors
  • The problem and the motivation for the study seem reasonable. However, the example in Figure 1 appears to be more of a hypothetical case than a real-world one, or too artificial. Can you provide some real-world failure examples to support the unfaithfulness of heuristic explanations?
  • when you apply group lasso, how are the groups defined for each dataset? How many concepts survive group lasso in each setting?
3 comments
AuthorsOpenReview ↗

Reply to reviewer NpZx

Thank you for appreciating how the paper addresses “a real and important question”, that our theory and conclusions are “well supported by experiments and mathematical proofs”, and that our manuscript is “clear and easy to follow”.

We address your other comments below. Please let us know if anything remains unclear, we are keen to engage.

the fact that faithful explanations cannot be compact is not so surprising with unbounded activations

The less obvious part is getting to Prop 1: one has to first consider the faithfulness of CBM explanations (a gap in the literature), apply formal explainability to CBMs (another gap), and identify the compactness-faithfulness trade-off (a third gap). Our goal is precisely to make the CBM community aware of these issues and to combat the implicit but widespread tendency to equate concept bottlenecks with reliable interpretation.

Note also that the main text focuses on Prop 1 precisely because it's intuitive. Our results in Sec B on bounded activations are less straightforward.

It is the known reason why prior work on AXps for linear classifiers assumes bounded features

As noted in Sec 6 (line 442), FXAI algorithms for linear models assume activations are bounded, compatibly with Prop 1. But we go one step further: we deduce that if one needs compact AXps, they should constrain the model in terms of activations and/or weights. To the best of our knowledge, the FXAI literature has not proposed anything similar, and definitely not for CBMs.

The group lasso gains are confounded with feature removal and accuracy loss

We ensure a fair comparison between sparse and dense CBMs by "maximizing λ\lambda under the constraint that its label F1 is at most 10% worse than the dense CBM's on the validation split." (Sec 5.2, l 376; Sec D.5, l 1629)

Note that "turning off" concepts is precisely how glasso helps: by Prop 1, faithful explanations must cover every activation with a non-zero weight difference, so the number of activations still in use is what drives AXp size. We agree that for VLM-CBMs the gain comes from this.

explanation size as a percentage of the bottleneck

Below we report Tab 4 with absolute numbers (avg # activations in the explanation) instead of percentages, for TopK-Act and EA+:

Dataset (b)LayerTopK-ActEA+
Derm7pt - CBM (28)dense9.58.3
enet7.72.6
glasso7.72.0
WBCAtt - CBM (31)dense10.58.2
enet9.32.3
glasso8.72.5
CUB - CBM (312)dense51.247.3
enet39.931.7
glasso44.123.7
CUB - LF-CBM (202)dense201.9200.9
enet202.0167.0
glasso201.435.4
CUB - VLG-CBM (419)dense418.5416.1
enet419.0287.2
glasso418.066.3
CUB - LaBo (10,000)dense9974.89944.8
enet9989.34682.9
glasso9989.1640.7

The message of Sec 5.2 is unchanged. Enet only partially works: it shrinks explanations of CBMs, but on VLM-CBMs it only helps EA+ (and it inflates TopK-Wei/Prod explanations in three cases, see Tab 4). Glasso shrinks explanations w.r.t. the dense model (except TopK-Act on VLM-CBMs, where it misbehaves everywhere), loses to enet only in two cases by a small margin (WBC/EA+: 2.5 vs 2.3; CUB/TopK-Act: 44.1 vs 39.9), and is essential for VLM-CBMs (EA+: 35.4 vs 167.0 on LF-CBM, 66.3 vs 287.2 on VLG-CBM, 640.7 vs 4682.9 on LaBo).

We’ll report the full table in the supplement.

Fig 1 vs. real-world failures for heuristic explanations

Yes, Fig 1 is meant as an illustration. Below we report real counterexamples (failures) for TopK-prod (weights x activation) heuristics, computed on dense probabilistic CBMs, focusing on correctly predicted test inputs. In each case, we keep the top-k concepts fixed and change a single concept not in the explanation, flipping the prediction:

  • CUB (k = 10): a Brown Pelican, explained by solid tail, plain head, white forehead/nape/crown, long bill, very large size, etc. Changing wing color from grey to white yields White Pelican.
  • Derm7pt (k = 5): a nevus, explained by absent vascular structures, absent streaks, typical pigment network and dots/globules. Changing regression structures from absent to present yields melanoma.
  • WBCAtt (k = 5): an eosinophil, explained by dense chromatin, big cell, light blue cytoplasm, round red granules. Changing nucleus shape from segmented-bilobed to unsegmented-indented yields monocyte.

EA+ explanations of the same points (41, 7 and 10 concepts) include the changed concept, and are faithful by construction. We will add these examples to the appendix.

AuthorsOpenReview ↗

Reply to reviewer NpZx part 2

how are groups defined for glasso? How many concepts survive it?

One group per concept activation (Sec 4.2, l 266): the column of inference-layer weights attached to that activation, across all classes. Zeroing a group means no class uses that activation. This is the same for all datasets.

The number of activations used by at least one class drops substantially with glasso:

  • Derm: dense 27 -> glasso 4
  • WBC: 31 -> 6
  • CUB: 230 -> 43 (some are all zero in the data, dense learns not to use them).

Comparison against a smaller vocabulary

By Prop 1, the gain comes precisely from “having fewer concepts”: optimizing loss + λ\lambda glasso dynamically constructs a smaller decision-focused vocabulary. The question of whether it outperforms offline feature selection rests on the choice of feature selection method. We’ll clarify this in the text.

I thank the authors for their detailed response and the additional results. The real-world counterexamples address my concern about Figure 1. However, I still feel that the proposed remedies, probabilistic concepts and group lasso, are well-established techniques, and that the gains from group lasso largely stem from removing concepts, as the reported numbers of surviving concepts suggest. I will therefore maintain my current rating.

Reviewer_y2DtOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at concept bottleneck models (CBMs) and what makes a good explanation. The authors identify two features: faithfulness and compactness. Here, faithfulness refers to the explanations essentially uniquely identifying the relevant label, while compactness is using the simplest or fewest number of concepts necessary. The authors demonstrate a tension between a faithful and a compact representation, then show why this is difficult with standard CBMs. The solution is twofold: introduce L1 regularization and add probabilistic logits (e.g., through a sigmoid). They demonstrate in principle why this performs better, and also through experiments on standard CBM datasets (e.g., CUB).

Critical strengths
  1. **Elegant contrast between faithfulness and compactness ** - I like the way that the paper builds up to Proposition 1, and in general, there's little work on theory in the CBM space, so it's nice to see rigorous theory here. The Proposition demonstrates why we can't achieve both faithfulness and compactness simultaneously, and how a larger output space makes it harder to guarantee faithfulness.
  2. Nice formalization of explanation quality - The paper presents a nice formalization of different properties that "good" explanations should have. Faithfulness and compactness are both natural desiderata, and the paper does a good job defining and delineating both. The theory intuitively follows from this, and in general, I think this presents a nice perspective that adds to the literature/debate surrounding how and when to use CBMs.
Critical weaknesses
  1. Unclear how poor baselines do - My interpretation of Table 4 and Table 5 is that the biggest improvement in performance comes from using L1 regularization, with smaller gains for using probabilistic CBMs. My worry is that the introduction of L1 regularization as a way to learn sparse concepts is something that is fairly widely known in the machine learning literature. I think the problem tackled is important and relevant for CBMs, but in some sense the main part of the solution is also somewhat obvious (use L1 regularization). Maybe I'm missing something, but it seems that everything else is minor/seems secondary to this.
  2. Lack of theoretical Guarantees for the main method - The authors do a good job of establishing bounds earlier with Proposition 1 and the tension between faithfulness and compactness. However, the actual method introduced in the paper, in Section 4.2, has little in the way of theoretical guarantees, and in general, little discussion of the methods. It might be that theoretical guarantees aren't possible, but even a small example to show why this might help would be useful. For example, I know that [1] produces guarantees in a similar space about concept selection, so something similar might also be possible here?

[1] Raman, Naveen, Stephanie Milani, and Fei Fang. "Selecting Decision-Relevant Concepts in Reinforcement Learning." arXiv preprint arXiv:2604.04808 (2026).

Questions for authors
  1. One of the requirements for faithfulness is that the explanation uniquely identifies. Yet for many models, the explanation is based on logits or the embedding (e.g., CEMs); are you assuming that explanations are based on rounded values?
  2. Qualitatively, which concepts are used less by the EA+ method; that is, when we have a smaller budget on the number of concepts, which concepts are ignored or not used?
  3. What exactly is the novelty or new method of the paper, and is it primarily adding L1 regression?
  4. If the last layer was a shallow MLP (rather than always linear) as real CBMs are trained, does that change Proposition 1?

I think generally, I'm on the fence about this paper, and these questions can help me better understandthe contributions.

3 comments
AuthorsOpenReview ↗

Reply to reviewer y2Dt

Thank you for appreciating the paper's "elegant contrast between faithfulness and compactness", its "nice formalization of explanation quality", its “rigorous theory” addressing a gap in the CBM literature, and the perspective it adds to the debate surrounding CBM usage.

We hope the discussion below addresses all your concerns and are happy to further engage towards raising the score to full acceptance.

What exactly is the novelty?

Novelties:

  • Significant problem: we show that, despite being geared for interpretability & interaction, typical CBMs are not compactly faithfully explainable - a clear gap in the literature, as you noted
  • Insights: Prop 1 / Sec B pinpoint the two root causes: concept activations and the inference layer weights, which have to be adequately constrained. CBMs do not constrain them properly (Sec 4.1) because the literature is unaware of any trade-off between faithfulness and compactness (Def 1, Prop 1)
  • Solutions: Sec 4.2 suggest two simple strategies (group lasso (glasso) and probabilistic activations) that target these causes directly; no CBM uses glasso, few use probabilities
  • Algorithms: an exact algorithm for CBMs that finds more compact faithful explanations with guarantees, whereas existing strategies are heuristic in nature

Glasso and probabilistic modeling of concept activations are not advanced technology, but Prop 1, Sec B and our results show that they are what CBMs need

the main part of the solution (L1 penalty) is obvious & widely known

Note that group lasso is not L1:

  • Glasso is a sum of per-activation L2 terms (Sec 4.2, l 266): it imposes structured sparsity so as to ``turn off'' each activation across all classes; doing so circumvents Prop 1.
  • L1 is Lasso: it sets as many entries of the weight matrix to zero but imposes no structure; it is insufficient to avoid Prop 1. Our results show that enet (L1 + L2), which is close to lasso (L1), underperforms compared to glasso, see Tab 4, Fig 4 & 5.

Second, glasso only becomes natural once aware of Prop 1. In order to get Prop 1, one has to first consider the faithfulness of CBM explanations (a gap in the literature), apply formal explainability to CBMs (another gap), and identify the compactness-faithfulness trade-off (a third gap).

No CBM architecture uses glasso (Sec 4.1, Tab 1).

the methods introduced in Sec 4.2 have little theoretical guarantees

They are backed by Prop 1 and Sec B. Roughly speaking, Prop 1 says that a CBM with (a) dense inference layer weights and (b) unbounded activations does not admit compact faithful explanations. Our solutions target (a) and (b):

  • Glasso targets (a): it explicitly pushes the weights between each concept jj to any label y∈[m]y \in [m] to 0; in turn, this pushes δwj\delta w^j to 0.
  • Bounded/probabilistic activations target (b). Both prevent Prop 1 from applying. Prop 2 and 3 in Sec B precisely detail how restricting the concept activation space reduces the set of valid counterexamples, as we discuss in Sec B.3.

We will use the extra page to move the proof of Prop 1 to the main text (to aid intuition), and to report this discussion.

What about [1]?

Thanks for the pointer. [1] selects a global concept set for RL policies and bounds the resulting value loss. We instead study per-instance faithful explanations of a fixed CBM.

The analogue of their guarantee in our setting is Prop 1.

Tab 4/5 suggest glasso is better than probabilistic activations

They are complementary. Each tackles a different condition in Prop 1.

Our results show that:

  • Dense models: probabilities do most of the work. Dense CUB, EA+ size: ReLU 57.5% vs prob 15.1% of the bottleneck.
  • VLM-CBMs: activations are already bounded ([-1, 1]), so glasso is what helps.
  • Both together: with glasso, ReLU vs prob = 14.0% vs 8.1% (WBC), 21.3% vs 7.3% (Derm).
  • The simplex ablation from 5zeB (prob with vs without (b)) also shows probabilistic modeling matters beyond bounding.

Explanations are based on logits, do you assume they are rounded?

Def 1 applies to continuous activations: if, despite fixing all logits in E\cal E, one can find or construct a counterexample yielding a different class, then E\cal E is not faithful. There is no need for rounding.

CEMs

Concept embedding models are out of scope. Def 1 extends naturally (fix embeddings rather than logits), but the analysis of Prop 1, 3 and 4 changes. We will consider them in future work and note this limitation in the paper.

AuthorsOpenReview ↗

Reply to reviewer y2Dt part 2

Which concepts does lowering the budget drop? Especially for EA+.

EA+ and TopK* take no budget: they return a faithful explanation of whatever size is needed.

As for which concepts are left out, Prop 2-3 give a general rule: a concept can be omitted when its weight difference with every alternative class (times how far its activation can move against the prediction) is small; concepts with large ∣δw∣|\delta w| and spare room to move must be kept. By Prop 1, for unbounded logits no concept with δw≠0\delta w \ne 0 can be omitted.

Prop 1 for MLP inference layers

Prop 1 works for linear inference layers, which are standard in CBMs owing to the ease of extracting explanations from them.

The core idea translates to MLPs. Intuitively, if a concept activation can be manipulated to make the MLP transfer probability mass from the prediction to an alternative class, then faithful explanations have to cover it. If all concept activations have this property, only explanations covering them all are faithful. But we prefer to avoid imprecise claims and leave finding the exact conditions for MLPs to future work.

Reviewer_y2DtOpenReview ↗

Thank you for the rebuttal

Thank you for the rebuttal, and I think it makes clear what Glasso is. I appreciate the in-depth answer, and the clarifications on both CEM and [1]. I do still think that the use of L2 activations here is quite natural/not super novel, but I do see the importance of the problem (and the simplicity of the solution can be a good thing). I'll maintain my score of a 3, which advocates for weak acceptance.

Reviewer_5zeBOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates when concept bottleneck models (CBMs) can provide faithful and compact explanations. The paper introduces abductive explanations (AXps) to formally define faithfulness. An explanation is considered faithful if it contains a selected group of concepts and their activations so that the prediction can't change if those are fixed. However, this would require selecting a large set of concepts, up to the entire bottleneck. The paper then shows that architectural changes such as using probabilistic concept activations and group lasso regularization can make explanations more compact.

Critical strengths

The paper studies an underexplored limitation of how CBMs provide explanations, i.e., that the selected concepts (usually the top contributing ones) may not be sufficient to explain a prediction faithfully, for example, because the absence of another concept could also be necessary for the prediction.

Critical weaknesses
  1. The benefits of the proposed architectural changes are not supported by adequate evidence. Regarding using probabilistic activations, I am concerned that the measured benefits do not come from probabilities but mainly from bounding the activation space. Before recommending using probabilistic activations, I would suggest that the authors do these two sanity checks: (1) compute min and max activations for each concept logit in the training set and use these as bounds for the explanations, then evaluate how often these bounds are violated on the test set and whether this actually results in unfaithful explanations. If it does happen, then that would be evidence that we actually need to bound the activations. (2) If we actually see violations and need to bound the activations, do we really need probabilities? I would suggest checking whether capping logits only at test time or during training can provide the same benefits. One additional observation I have about probabilities is that they might even be misleading, as applying a sigmoid to logits that are learned via a similarity score (and not calibrated via presence/absence labels) does not produce a probability that a concept is present, since the resulting value depends on the scale of such logits. Regarding group lasso, this is essentially reducing the number of concepts globally, and that reduction can come with a substantial cost in accuracy if the concepts are necessary for prediction (e.g., in the VLG-CBM example). Also, the accuracy comparisons do not control for decision-level sparsity (i.e., what the VLG-CBM paper defines as NEC or MCBM paper defines as NCC). Since sparsity and accuracy tend to be inversely correlated, a fair accuracy comparison should control for the first.

  2. VLM-based CBMs do not use their original architectures. The paper claims to evaluate LF-CBM, VLG-CBM and LaBo, however for all these CBMs, the paper uses the same CLIP-based architecture where concept activations are computed as cosine similarities and fed to the classifier. In this way, the experiments essentially only evaluate LaBo-style architecture but with different concept vocabularies rather then the CBMs it claims to evaluate. This is problematic because the paper investigates architectural changes and it is not tested whether they actually generalize to the LF-CBM or VLG-CBM.

Questions for authors

N/A

5 comments
AuthorsOpenReview ↗

Reply to reviewer 5zeB

We would like to point out this is not a paper about achieving the new SoTA performance with a novel CBM variant, but about pushing the boundaries of the current understanding on how these models work and when they can fail us.

We stress that our proposed architectural changes are just one part of our contributions. Before that, we provide theoretical arguments and a formal explainability framework to show the limitations of not only current CBM architectures but also current heuristics to extract “interpretable” explanations.

As we answer the reviewers’ questions below, this provides a strong foundation to support our architectural changes. Please let us know if anything remains unclear, we are keen to engage.

The proposed changes are not supported by adequate evidence

Evidence:

  • Theory: Prop 1 shows that if a CBM's concept activation space and inference layer weights are not appropriately constrained, compact faithful explanations do not exist. Hence, we propose to impose suitable constraints on CBMs.
  • Experiments: our results for regular CBMs (with logits, ReLUs and probabilistic activations), VLM-CBMs (with cosine similarity activations) and Selective CBMs on four datasets (one with four concept vocabularies) indicate that our solutions (probabilistic activations, group lasso, exact FXAI algorithms) yield more compact faithful explanations, whereas unbounded logits and dense models struggle (Sec 4: Fig 2, Tab 4; Sec D.2-D.5: Fig 3, 4, 5, 6).

Sanity check 1: guarantees wrt training min-max box

For CBMs with logits, we compute the per-activation [min, max] box on training data. Test points outside it are 6.8% (Derm), 0.5% (WBC), 5.3% (CUB). Results:

  • Many EA+ explanations certified against the training box are invalidated by activation values observed on the test set: 34% (Derm), 17% (WBC) and 89% (CUB) for dense CBMs, 98% for group lasso (glasso) on CUB;
  • Moreover, 80-100% are unfaithful against the actual (unbounded) logit space.

As expected, "empirical boxes" like this provide no guarantees against counterexamples outside the training range (e.g. human interventions); see Sec 7 and C.2 for a discussion.

Benefits come from bounding the activations not probabilities Sanity check 2: is capping logits only enough?

We suggest using probabilities precisely as they are a principled way of bounding the concept activations (see Sec 4.2, l 256).

Probabilities constrain (a) activations of all concepts to [0,1] and (b) activations of one-hot encoded categorical concepts to sum to one (e.g. p(red)+p(blue)=1p(red) + p(blue) = 1).

To check whether (a) alone is enough, we took our probabilistic CBMs and computed explanations against mere capping to [0,1][0, 1] (a) and the full probabilistic constraints (a) + (b) on the same test points as in the paper. Below we report the average size of faithful explanations (in # activations) for TopK-Act and EA+:

Dataset (# act)LayerTopK-ActEA+
Derm7pt (28)dense9.5 -> 21.28.3 -> 18.0
glasso7.7 -> 9.12.0 -> 2.3
WBCAtt (31)dense10.5 -> 27.68.2 -> 19.6
glasso8.7 -> 23.42.5 -> 3.5
CUB (312)dense51.2 -> 182.647.3 -> 155.0
glasso44.1 -> 97.823.7 -> 27.2

The arrows show (a) + (b) -> (a) only. Result: ignoring (b) always enlarges faithful explanations, often by 2-4x (e.g., dense CUB, EA+: 47.3 to 155.0). Hence, (b) is very beneficial, as we claimed. The gap shrinks with glasso, consistently with the two strategies being complementary. We will add these results to the supplement.

CelebA has no categorical concepts so (b) doesn't apply.

probabilities may be misleading if applied to cosine similarity activations

Agreed. We do not propose using probabilistic activations for our VLM-CBMs with cosine similarity activations, as these are already bounded to [-1, 1]. Furthermore, their LLM-based vocabularies do not have categorical concepts, so they don’t benefit from (b) above. For them we only evaluate glasso (Sec 5.2).

We will clarify this in the text.

Glasso has an accuracy cost a fair accuracy comparison should control for sparsity

We stressed glasso's cost in Benefits and Limitations, l 275. Tab 5 reports its drop in label F1.

We ensure a fair comparison between sparse and dense CBMs by "maximizing λ\lambda under the constraint that its label F1 is at most 10%10\% worse than the dense CBM's on the validation split." (Sec 5.2, l 376; Sec D.5, l 1629)

Even under this constraint, glasso can be worth it. E.g., VLG-CBM: label F1 drops 0.626 -> 0.576, but faithful explanations shrink from 99.4% to 15.9% of the bottleneck.

AuthorsOpenReview ↗

Reply to reviewer 5zeB, part 2

the comparisons do not control for decision-focused sparsity (NEC)

Sparsity is the mechanism through which our strategy acts (penalty -> sparsity -> explanation size), so controlling for it would block the effect we measure. We therefore control for accuracy instead.

Moreover, NEC/NCC measure per-class sparsity, whereas faithful explanations must rule out all rival classes and thus depend on the number of concepts used by any class (Prop 1). We evaluated this by comparing enet vs glasso-regularized CBMs on CUB using the EA+ algorithm: enet yields NEC 8.3, glasso only 43. Yet, enet faithful explanations are larger (31.7 vs 23.7 activations).

The paper only evaluates LaBo-style architectures

In Sec 4.2 we noted that existing VLM-CBMs with a backbone (LF-CBM, VLG-CBM) effectively use unbounded logits, hence by Prop 1 they cannot be compactly faithfully explained (l 234). Evaluating them would replicate the results in Fig 2, where logits are pathological. We evaluate LaBo-style architectures precisely because they are not pathological: they employ bounded cosine similarity activations, eluding Prop 1.

We will rename LF-CBM and VLG-CBM to LaBo-LF and LaBo-VLG for clarity.

  • Regarding sanity check 1, I am not sure I understand how the percentage of invalidated explanations was computed. If only 0.5–6.8% of test samples fall outside the training activation bounds, shouldn't the explanations be guaranteed to remain faithful for the remaining test samples? In particular, I find it difficult to reconcile the 5.3% of out-of-bounds test samples on CUB with the reported 89% of invalidated explanations. Could the authors clarify what exactly was measured and how these percentages were obtained?

  • Regarding sanity check 2, I appreciate the additional experiment. However, this is not the baseline I was suggesting. My suggestion was to compare probabilistic CBMs against CBMs using logits clipped to empirical bounds, either during training or only at test time, evaluating both classification accuracy and explanation compactness. Correct me if I am wrong, but your experiment seems to investigate the benefits of imposing mutual-exclusivity constraints on categorical concepts, which is a different question.

  • Regarding probabilities for VLM-CBMs, the authors are correct here. I withdraw this criticism and encourage them to clarify this distinction in the paper. However, methods such as VLG-CBM and M-CBM do use concept presence/absence labels to train their bottlenecks in a multi-label fashion with sigmoid and binary cross-entropy. This is also why I encouraged the authors to evaluate the original architecture.

  • Regarding group lasso, my concern remains that group lasso essentially reduces the number of concepts in the bottleneck, so the reduction in faithful explanation size is expected. It seems somewhat trivial to me that removing concepts, potentially at the cost of classification accuracy, results in smaller explanations.

  • Regarding decision-level sparsity, I buy the clarification and I encourage the authors to include a comparison in the paper between NEC/NCC and AXps.

  • Regarding VLM-CBM architectures, I appreciate the clarification and agree that renaming the models improves clarity, but it does not resolve my concern that the original architectures are not evaluated. For example, VLG-CBM already learns concept-presence predictions using multi-label supervision, so its sigmoid outputs could be used to train the final classifier without requiring cosine similarity activations.

I am increasing my score, as I appreciate the authors rebuttal and the additional experiments. However, my overall assessment of the paper remains unchanged for the most part. In my opinion, there is a potentially valuable theoretical contribution, but the paper is not yet ready for publication and would benefit a lot from another revision addressing the remaining concerns. I therefore still recommend rejection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.