- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work introduces Matryoshka Concept Bottleneck Models (MCBMs). MCBMs decrease the cardinality of the concept set involved in predictions. The authors use feature-selection based on minimum redundancy maximum relevance (mRMR) so as to order the concepts in terms of informativeness, and then train multiple classification heads. Each of these heads uses only a subset of the sorted concept set, thus leading to increased interpretability due to including only a subset of the whole available concept set, i.e. a sparser concept set.
Critical strengths
The motivation of the work is sound and novel to the CBM literature. I believe that CBMs would benefit from the idea of using concept subsets. More specifically:
- Using only a subset of concepts per prediction, is, to the best of my knowledge, a new idea on CBMs. These subsets can help with interpretability and annotation efficiency of CBMs, especially in datasets with high concept set cardinality, since the end-user will have to read (or annotate) a fewer number of concepts.
- The mRMR-based feature selection idea is used before training, thus being possible to integrate to other CBM (or CEM) architectures.
Critical weaknesses
Clarity: The main weakness of this work is the clarity of the text. In most cases, terminology outside of the machine learning field and specifically the CBM literature is used, making the text very hard to understand. I strongly suggest that the authors polish their text.
Experimental results: I also struggle with the experimental section. The authors selected to run single-seed studies on two out of three datasets. Also, for the dataset that has been run for multiple seeds, in Fig. 2b, the authors don't report multi-seed error bars, and in Table 2 they do not include standard deviations. Furthermore, the experiments include "unrun comparisons" (Line 279) in Table 1.
I think it would have also been useful to test MCBMs on other CBM architectures such as SCBM.
Finally, the authors claim to have used the test set during development time as seen in: "The test sets have been evaluated during development, so the results describe the reported configurations rather than a fresh confirmation study." (Lines 469-470) and "The test set has been observed in earlier project experiments; the present comparisons are not a fresh holdout study." (Lines 603-604)
On top of that, the authors did not include any supplementary material and there isn't a reproducibility statement included.
All of the above, make me skeptical about the validity of the presented experimental results.
Typos
- Lines 097-098: "... while stochastic CBMs model concept dependencies...", please capitalize Stochastic, as seen in the rest of the text.
- Figure 1: unless I am mistaken, in the equation for it should write instead of . Also, shouldn't the index on the heads, go "1, 2, 3, N" instead of "1, 2, n, N"?
- Lines 306-307: "On CUB, moving from 112 to 64 concepts changes accuracy by −0.092 ± 0.132 percentage points across paired seeds." I think it is not percentage points.
Questions for authors
- Maybe I missed it, but given that one task head is trained per concept subset, wouldn't this lead to training different task heads? Can you provide results on the computational efficiency of MCBMs, including the hardware used for them?
- How would the concept subset selection of mRMR and multiple task heads look like in a Concept Embedding Model [1], since embeddings of concepts are used instead of the concepts themselves?
[1] Espinosa Zarlenga, Mateo, et al. "Concept embedding models: Beyond the accuracy-explainability trade-off." Advances in neural information processing systems 35 (2022): 21400-21413.