- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a method for analyzing parts of a pretrained network that are relevant for concept detection. The method itself is an adaptation of the work of Savarese et al. (NeurIPS 2020). The basic idea of the method is to learn a binary mask on activations that can recover the networks activations for positive concept samples () and certain specificity on negative samples (). Furthermore, the authors perform an analysis using their method on multiple datasets and architectures. An emphasis is put on investigating the results of the method wrt. to multiple metrics (sufficiency/necessity of the resulting masks, diversity of the masks, etc.).
The authors derive some interesting insights from their analysis, which I would boldly summarize as (1) yes, we can find relevant structures in a network for concept detection (this is not surprising, but a validation of their method) and (2) these structure are not unique, there are multiple solutions.
Critical strengths
- The paper was indeed an interesting read and the presented method is reasonable for the research questions at hand.
- The method itself is well explained (despite some minor issues I listed below)
- I like the analysis part of the paper (although the later ends were difficult to understand I have to confess), the used metrics are reasonable, well motivated and nicely explained.
- The main result for me is that indeed there is a multitude of masks that work reasonably well, therefore, the aim of some research directions to identify THE relevant parts of a network is questionable given the results of this paper. Therefore, I learned something interesting from the paper. This is also nicely summarized in the Conclusions section of the paper.
- I like the distinction of the metrics between metrics that have been used in the loss (directly or indirectly) and metrics that were not optimized.
- I appreciate the implementation details given in appendix C.
Critical weaknesses
- The main problem of the paper is in my opinion the lack of novelty. We have seen the importance of subnetworks in earlier work already (Lottery Ticket Hypothesis) and the fact that concepts can be explained by multiple subnetworks equally has been also shown. The paper is not over-claiming something in general, it is just the question whether it reaches the novelty bar.
- The authors use the former classification head of the network (i.e. the softmax output for the classes in ) and to define a relation between original target classes () and concept labels () are required. This is done (as far as I got it) by using the set of examples with labeled concepts and building subsets which contain all classes that share examples with a concept (positively labeled example). All of the classes in are treated equal - isn't this a problem for the loss? How does the class distribution look like in these sets? A single mislabeled example could change the whole analysis.
Questions for authors
- Eq. (3) - the set is used but never formally defined. I can guess the meaning of it from the sentence "landing on the classes related to the positive set" or search the appendix (and find it in L661), however, a formal definition or an extension of the sentence would be reasonable in the main part of the paper.
- Eq. (4) shows an additional dependency of on . I guess this is on purpose, since in contrast to Eq. (3) the binary mask is enforced? This part might need a few sentence of explanation to be added.
- The authors introduce a scalar offset into the relaxation of the binary mask parameters. An ablation experiment would be interesting to show the effect that the authors described.
- I wonder how the results of the analysis change when more sparsity is enforced.
- Very minor comment: equation numbers before eq. (1) would not hurt
- Why are the ViT results only given in the appendix and on a smaller scale? Was this due to computational limitations?
4 comments
Thanks for the detailed answers
I would like to thank the authors for their detailed answers, the fact that concepts are defined solely by classes is something that I missed. It is now clearly defined in the paper and I also appreciate that the ViT results moved in the main part.
Results of the promised experiments (Q3, Q4)
Dear Reviewer 9Nas,
Thank you for the follow-up. We are glad that the class-level definition of concepts and the ViT results are now clear. As promised, here are the results of the ablation (Q3) and the sweep (Q4).
Q3 ( Ablation) We trained masks with on four cells, with 5 seeds each and all other settings fixed.
| Cell | ||||
|---|---|---|---|---|
| GTSRB / DenseNet-121 | collapse | 4 | 1 | 1 |
| AWA2 / DenseNet-121 | collapse | 2 | 1 | 1 |
| GTSRB / EfficientNetV2 | 22 | 5 | 3 | 1 |
| GTSRB / ViT-B/16 | 9 | 1 | 1 | 1 |
(Epochs to 95% of the final feature cosine similarity, mean of 5 seeds; collapse = no recovery, all masks fail. Default , EfficientNetV2 (9 epochs).)
Without , every masked weight starts at half its value (), and early training has to recover the dense features themselves, not only their scale as our earlier reply said. At , EfficientNetV2 and the ViT need 22 and 9 epochs to do so, and on both DenseNets the sparsity penalty removes weights faster than feature matching can restore them, so all 10 masks collapse. Increasing gives the mask more time to recover. On GTSRB/DenseNet-121, starts even further from the dense features than after one epoch (cosine 0.25 vs. 0.31) but recovers within 4 epochs. For , validity no longer depends on (kept AUC 0.997, specificity 0.999, accuracy on positives 0.92 except one seed at 0.87).
In addition, does change which weights are selected. In every cell, masks with the same seed but a different (1 vs. the default) agree less ( 0.45–0.66) than masks with the same but different seeds ( 0.61–0.88, within this ablation). So is a source of multiplicity besides data order, which our Limitations listed as untested. We will add these results, with training curves, as a new appendix.
Q4 ( sweep). On the same four cells and seeds, we also trained masks with , with all other settings fixed.
| Cell | |||||
|---|---|---|---|---|---|
| GTSRB / DenseNet-121 | 5 | 5 | 5 | 1 | 0 |
| AWA2 / DenseNet-121 | 5 | 5 | 4 | 0 | 0 |
| GTSRB / EfficientNetV2 | 5 | 5 | 0 | 0 | 0 |
| GTSRB / ViT-B/16 | 5 | 5 | 4 | 0 | 0 |
(Masks out of 5 that pass the health check of Appendix D.7. Bold marks the paper . For the DenseNets the paper uses (GTSRB) and (AWA2), which give the same counts as .)
In every cell, as increases, mask density falls by 1.5 to 3 orders of magnitude over the range. Above the paper , masks start losing the concept-positive images, about one decade higher on the DenseNets and two decades higher on EfficientNetV2 and the ViT.
The same also behaves differently per backbone. At most DenseNet masks are still valid (9 of 10) and no EfficientNetV2 mask is, which is why we set per architecture. For EfficientNetV2 and the ViT, the paper () is conservative, since also passes and gives masks 2.5× and 7× smaller. Within the valid range, multiplicity hardly changes. between seeds moves by at most 0.05 within a cell while density changes up to 26×, and the seeds still make the same concept decisions (). So sets how many weights a mask keeps, but not how much the seeds disagree. The full table and figure will go in the same appendix.
Thank you again for engaging with us and providing your valuable feedback. Should there remain any unresolved concerns or uncertainty, please let us know. We would also like to kindly ask whether there are concrete concerns that are holding you back from reconsidering the scores. We are eager and open to further discussions to ensure clarity.
Thanks for these results
Thank you very much for these additional results and related sensitivity analysis.
Initial Comments
We thank the reviewer for the careful reading and the detailed questions, which helped us tighten the method section. To avoid a possible ambiguity in our wording: the masks act on the weights of the frozen network, not on its activations. The loss compares activations (the pre-classifier features), but the mask decides which weights are kept.
W1. Novelty. We agree that subnetwork discovery is established, and our method builds directly on LTH, CS and weight masking (Csordás et al., 2021; Lepori et al., 2023). What changes is what the mask is asked to find. These works train the mask to solve a task the model was trained on, so the subnetwork can be checked against that task. Human-defined concepts have no such task, so our mask instead reproduces the network's own features on concept-positive inputs, which lets us look for concepts the model was never trained to predict. We also agree that subnetworks have been shown not to be unique for trained behaviors (Frankle et al., 2020; Gorbett & Whitley, 2023; Bayazit et al., 2024; Méloux et al., 2025). We extend this to concepts and add a test that changes the conclusion. The weights shared by all seeds often fail to detect the concept (AUC ≥ 0.8 in only 21 of 57 CNN cells), and how much the masks overlap does not predict whether they work. A stable overlap therefore cannot be read as the concept's circuit (General Comment G3).
W2. The class set Yc. Yc is not built from the examples that carry the concept. It comes from the dataset's class–concept annotation table, and a class belongs to Yc if the table annotates it with c (Section 3.1; General Comment G5). A single mislabeled image therefore cannot add a class to Yc. On GTSRB the table follows from the traffic-sign specification and is exact; on AWA2 it is a crowd-sourced class-level judgment, which is one reason we picked both datasets. Treating the classes in Yc equally is intended. The sufficiency term does not use Yc at all, while the specificity term counts probability mass on any concept class as a false detection, whichever class it is. The class distribution matters when the positives are concentrated in a few classes, because the mask could then reproduce the dominant class instead of the concept. We therefore scale the specificity weight by purity, the share of the most frequent class among the positives (App. B). Table 3 reports the number of classes and the purity of every concept.
Q1. Yc is now defined formally in Section 3.1, before the loss.
Q2. The dependency on w⊙m(c) was missing from the specificity term in the submitted Eq. 3; both terms depend on the mask. The specificity term (now Eq. 5) and the detection score (now Eq. 6) use the same masked network. During training the mask is the soft relaxation of Eq. 2, and at test time it is binarized, which Section 3.1 now states explicitly.
Q3. The logits start at 0, so without δ every weight would be multiplied by σ(0) = 0.5 at initialization, and early training would be spent restoring the feature scale rather than selecting weights. With δ = 2, weights start at about 0.88 of their value. We are running a small ablation with δ ∈ {0, 1, 2, 3} and will post the results here.
Q4. A larger sparsity weight λ pushes the mask to keep fewer weights, and we expect it to eventually cost concept detection. Our grid already spans densities from 0.14% to 34% across backbones, and κ corrects for density, but we have not varied λ within a cell. We are running a sweep over λ from 10⁻⁹ to 10⁻⁵ on a subset of cells and will report how validity and multiplicity change.
Q5. All equations are now numbered.
Q6. Yes, the reason was compute. The ViT results are now in the main text, at the same depth as the CNNs with 30 seeds per cell, for 12 concepts so far (General Comment G1).