acceptodds
Under review as a conference paper at ICLR 2027

Characterizing Weight-Level Concept Representations in Vision models

Abstract

Concept-based explanations reveal which human-understandable concepts a network encodes, but mostly locate them in activation space, as directions or probes, rather than in the parameters that compute them. Mechanistic interpretability computation in weights and circuits, but typically for behaviors the model was trained to produce, such as a task, subtask or output class. We bring the two together: starting from human-defined concepts the model was never trained to predict, we learn sparse binary masks over the weights of frozen vision models that isolate the parameters needed to detect each concept. We first show that these masks are valid concept detectors, and then test whether this parameter-level structure is unique. Our main finding is that it is not: masks trained with different random seeds are equally valid but select different weights. The weights they all share often fail to detect the concept on their own, and ablating each mask changes the network's class predictions differently. A single mask is thus enough to detect a concept, but not to claim which weights compute it; such claims require several independently trained masks and a direct test of what they share.

Primary area:
interpretability and explainable AI
Keywords:
Concept-based explanation, Mechanistic interpretability, Model pruning / sparse subnetworks, explainability, Weight-level attribution, Superposition

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9NasOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a method for analyzing parts of a pretrained network that are relevant for concept detection. The method itself is an adaptation of the work of Savarese et al. (NeurIPS 2020). The basic idea of the method is to learn a binary mask on activations that can recover the networks activations for positive concept samples (LsuffL_{\text{suff}}) and certain specificity on negative samples (LspecL_{\text{spec}}). Furthermore, the authors perform an analysis using their method on multiple datasets and architectures. An emphasis is put on investigating the results of the method wrt. to multiple metrics (sufficiency/necessity of the resulting masks, diversity of the masks, etc.).

The authors derive some interesting insights from their analysis, which I would boldly summarize as (1) yes, we can find relevant structures in a network for concept detection (this is not surprising, but a validation of their method) and (2) these structure are not unique, there are multiple solutions.

Critical strengths
  • The paper was indeed an interesting read and the presented method is reasonable for the research questions at hand.
  • The method itself is well explained (despite some minor issues I listed below)
  • I like the analysis part of the paper (although the later ends were difficult to understand I have to confess), the used metrics are reasonable, well motivated and nicely explained.
  • The main result for me is that indeed there is a multitude of masks that work reasonably well, therefore, the aim of some research directions to identify THE relevant parts of a network is questionable given the results of this paper. Therefore, I learned something interesting from the paper. This is also nicely summarized in the Conclusions section of the paper.
  • I like the distinction of the metrics between metrics that have been used in the loss (directly or indirectly) and metrics that were not optimized.
  • I appreciate the implementation details given in appendix C.
Critical weaknesses
  1. The main problem of the paper is in my opinion the lack of novelty. We have seen the importance of subnetworks in earlier work already (Lottery Ticket Hypothesis) and the fact that concepts can be explained by multiple subnetworks equally has been also shown. The paper is not over-claiming something in general, it is just the question whether it reaches the novelty bar.
  2. The authors use the former classification head of the network (i.e. the softmax output for the classes in YY) and to define LspecL_{\text{spec}} a relation between original target classes (YY) and concept labels (CC) are required. This is done (as far as I got it) by using the set of examples with labeled concepts and building subsets YcY_c which contain all classes that share examples with a concept cc (positively labeled example). All of the classes in YcY_c are treated equal - isn't this a problem for the loss? How does the class distribution look like in these sets? A single mislabeled example could change the whole analysis.
Questions for authors
  1. Eq. (3) - the set YcY_c is used but never formally defined. I can guess the meaning of it from the sentence "landing on the classes related to the positive set" or search the appendix (and find it in L661), however, a formal definition or an extension of the sentence would be reasonable in the main part of the paper.
  2. Eq. (4) shows an additional dependency of pθp_\theta on w⊙m(c)w \odot m^{(c)}. I guess this is on purpose, since in contrast to Eq. (3) the binary mask is enforced? This part might need a few sentence of explanation to be added.
  3. The authors introduce a scalar offset δ\delta into the relaxation of the binary mask parameters. An ablation experiment would be interesting to show the effect that the authors described.
  4. I wonder how the results of the analysis change when more sparsity is enforced.
  5. Very minor comment: equation numbers before eq. (1) would not hurt
  6. Why are the ViT results only given in the appendix and on a smaller scale? Was this due to computational limitations?
4 comments
AuthorsOpenReview ↗

Initial Comments

We thank the reviewer for the careful reading and the detailed questions, which helped us tighten the method section. To avoid a possible ambiguity in our wording: the masks act on the weights of the frozen network, not on its activations. The loss compares activations (the pre-classifier features), but the mask decides which weights are kept.

W1. Novelty. We agree that subnetwork discovery is established, and our method builds directly on LTH, CS and weight masking (Csordás et al., 2021; Lepori et al., 2023). What changes is what the mask is asked to find. These works train the mask to solve a task the model was trained on, so the subnetwork can be checked against that task. Human-defined concepts have no such task, so our mask instead reproduces the network's own features on concept-positive inputs, which lets us look for concepts the model was never trained to predict. We also agree that subnetworks have been shown not to be unique for trained behaviors (Frankle et al., 2020; Gorbett & Whitley, 2023; Bayazit et al., 2024; Méloux et al., 2025). We extend this to concepts and add a test that changes the conclusion. The weights shared by all seeds often fail to detect the concept (AUC ≥ 0.8 in only 21 of 57 CNN cells), and how much the masks overlap does not predict whether they work. A stable overlap therefore cannot be read as the concept's circuit (General Comment G3).

W2. The class set Yc\mathcal{Y}_c. Yc\mathcal{Y}_c is not built from the examples that carry the concept. It comes from the dataset's class–concept annotation table, and a class belongs to Yc\mathcal{Y}_c if the table annotates it with cc (Section 3.1; General Comment G5). A single mislabeled image therefore cannot add a class to Yc\mathcal{Y}_c. On GTSRB the table follows from the traffic-sign specification and is exact; on AWA2 it is a crowd-sourced class-level judgment, which is one reason we picked both datasets. Treating the classes in Yc\mathcal{Y}_c equally is intended. The sufficiency term does not use Yc\mathcal{Y}_c at all, while the specificity term counts probability mass on any concept class as a false detection, whichever class it is. The class distribution matters when the positives are concentrated in a few classes, because the mask could then reproduce the dominant class instead of the concept. We therefore scale the specificity weight by purity, the share of the most frequent class among the positives (App. B). Table 3 reports the number of classes and the purity of every concept.

Q1. Yc\mathcal{Y}_c is now defined formally in Section 3.1, before the loss.

Q2. The dependency on w⊙m(c)w \odot m^{(c)} was missing from the specificity term in the submitted Eq. 3; both terms depend on the mask. The specificity term (now Eq. 5) and the detection score (now Eq. 6) use the same masked network. During training the mask is the soft relaxation of Eq. 2, and at test time it is binarized, which Section 3.1 now states explicitly.

Q3. The logits start at 0, so without δ every weight would be multiplied by σ(0) = 0.5 at initialization, and early training would be spent restoring the feature scale rather than selecting weights. With δ = 2, weights start at about 0.88 of their value. We are running a small ablation with δ ∈ {0, 1, 2, 3} and will post the results here.

Q4. A larger sparsity weight λ pushes the mask to keep fewer weights, and we expect it to eventually cost concept detection. Our grid already spans densities from 0.14% to 34% across backbones, and κ corrects for density, but we have not varied λ within a cell. We are running a sweep over λ from 10⁻⁹ to 10⁻⁵ on a subset of cells and will report how validity and multiplicity change.

Q5. All equations are now numbered.

Q6. Yes, the reason was compute. The ViT results are now in the main text, at the same depth as the CNNs with 30 seeds per cell, for 12 concepts so far (General Comment G1).

Reviewer_9NasOpenReview ↗

Thanks for the detailed answers

I would like to thank the authors for their detailed answers, the fact that concepts are defined solely by classes is something that I missed. It is now clearly defined in the paper and I also appreciate that the ViT results moved in the main part.

AuthorsOpenReview ↗

Results of the promised experiments (Q3, Q4)

Dear Reviewer 9Nas,

Thank you for the follow-up. We are glad that the class-level definition of concepts and the ViT results are now clear. As promised, here are the results of the δ\delta ablation (Q3) and the λ\lambda sweep (Q4).

Q3 (δ\delta Ablation) We trained masks with δ∈{0,1,2,3}\delta \in \{0, 1, 2, 3\} on four cells, with 5 seeds each and all other settings fixed.

Cellδ=0\delta = 0δ=1\delta = 1δ=2\delta = 2δ=3\delta = 3
GTSRB / DenseNet-121collapse411
AWA2 / DenseNet-121collapse211
GTSRB / EfficientNetV222531
GTSRB / ViT-B/169111

(Epochs to 95% of the final feature cosine similarity, mean of 5 seeds; collapse = no recovery, all masks fail. Default δ=2\delta = 2, EfficientNetV2 0.50.5 (9 epochs).)

Without δ\delta, every masked weight starts at half its value (σ(0)=0.5\sigma(0) = 0.5), and early training has to recover the dense features themselves, not only their scale as our earlier reply said. At δ=0\delta = 0, EfficientNetV2 and the ViT need 22 and 9 epochs to do so, and on both DenseNets the sparsity penalty removes weights faster than feature matching can restore them, so all 10 masks collapse. Increasing δ\delta gives the mask more time to recover. On GTSRB/DenseNet-121, δ=1\delta = 1 starts even further from the dense features than δ=0\delta = 0 after one epoch (cosine 0.25 vs. 0.31) but recovers within 4 epochs. For δ≥1\delta \geq 1, validity no longer depends on δ\delta (kept AUC ≥\geq 0.997, specificity ≥\geq 0.999, accuracy on positives ≥\geq 0.92 except one seed at 0.87).

In addition, δ\delta does change which weights are selected. In every cell, masks with the same seed but a different δ\delta (1 vs. the default) agree less (κ\kappa 0.45–0.66) than masks with the same δ\delta but different seeds (κ\kappa 0.61–0.88, within this ablation). So δ\delta is a source of multiplicity besides data order, which our Limitations listed as untested. We will add these results, with training curves, as a new appendix.

Q4 (λ\lambda sweep). On the same four cells and seeds, we also trained masks with λ∈{10−9,10−8,10−7,10−6,10−5}\lambda \in \{10^{-9}, 10^{-8}, 10^{-7}, 10^{-6}, 10^{-5}\}, with all other settings fixed.

Cell10−910^{-9}10−810^{-8}10−710^{-7}10−610^{-6}10−510^{-5}
GTSRB / DenseNet-12155510
AWA2 / DenseNet-12155400
GTSRB / EfficientNetV255000
GTSRB / ViT-B/1655400

(Masks out of 5 that pass the health check of Appendix D.7. Bold marks the paper λ\lambda. For the DenseNets the paper uses 3⋅10−73\cdot10^{-7} (GTSRB) and 2⋅10−72\cdot10^{-7} (AWA2), which give the same counts as 10−710^{-7}.)

In every cell, as λ\lambda increases, mask density falls by 1.5 to 3 orders of magnitude over the range. Above the paper λ\lambda, masks start losing the concept-positive images, about one decade higher on the DenseNets and two decades higher on EfficientNetV2 and the ViT.

The same λ\lambda also behaves differently per backbone. At 10−710^{-7} most DenseNet masks are still valid (9 of 10) and no EfficientNetV2 mask is, which is why we set λ\lambda per architecture. For EfficientNetV2 and the ViT, the paper λ\lambda (10−910^{-9}) is conservative, since 10−810^{-8} also passes and gives masks 2.5× and 7× smaller. Within the valid range, multiplicity hardly changes. κ\kappa between seeds moves by at most 0.05 within a cell while density changes up to 26×, and the seeds still make the same concept decisions (RCconcept≤1.003RC_\text{concept} \leq 1.003). So λ\lambda sets how many weights a mask keeps, but not how much the seeds disagree. The full table and figure will go in the same appendix.


Thank you again for engaging with us and providing your valuable feedback. Should there remain any unresolved concerns or uncertainty, please let us know. We would also like to kindly ask whether there are concrete concerns that are holding you back from reconsidering the scores. We are eager and open to further discussions to ensure clarity.

Reviewer_9NasOpenReview ↗

Thanks for these results

Thank you very much for these additional results and related sensitivity analysis.

Reviewer_2bBLOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method to obtain concept-based masks for weights of a trained model, mainly CNNs. It explores whether such masks are valid, and how unique they are. Across a series of analyses across three architectures and 19 concepts from two datasets, it claims that while the masks are valid but not unique, so one cannot attribute specific weights to a concept from one run.

Critical strengths
  1. The general problem is interesting and potentially useful. Since concept and circuit extraction methods are popularly used to understand large models, an analysis along the lines this paper proposes can be a useful sanity check to avoid incorrect conclusions from a specific circuit.
Critical weaknesses
  1. A major concern for me is the utility of this study. The paper only evaluates small models, that are CNNs, on two small-scale datasets, and only 19 concepts across those two datasets. It is unclear to me how useful or transferable any of the findings would be to modern models that are being used. Finding subnetworks or concept masks would particularly make sense for large scale models, so the experimental setup is a significant limitation. There are some ViT experiments provided, but it is unclear why they are only in the appendix and not a part of the main analyses.
  2. It is unclear what to take away from the experimental evaluation. The main takeaway, that masks are valid but not unique, could be interesting, but the vast majority of experimental analyses and result figures show a fine-grained set of results that are hard to parse or understand what to take away from them. For example, Figure 1 and Figure 2 have several subfigures that measure various metrics that are defined in this paper, and the main takeaway in most seems to be the some configurations perform well and others perform worse. What high level understanding can we gain from this? How should one use the results in this paper if attempting to apply this in one's own setup? I am also unclear on the justifications of some of the metrics. For example, in L249, why is necessity evaluated by inverting the full mask? Isn't it possible that some parts of the mask are necessary while others are not? Also, how is it claimed that the masks are valid (L377) while also claiming that necessity is poor (L311-316)?
Questions for authors

Clarifications on the questions in the weaknesses would help, but unless I have significantly misunderstood the paper, it is unlikely to affect my assessment. I would be happy to reconsider if I have misunderstood, however.

4 comments
AuthorsOpenReview ↗

Initial comments

We thank the reviewer for the critical reading. We agree that our analysis can help prevent incorrect conclusions from being drawn from a single circuit, and we acknowledge the concerns about clarity and about generalization to more modern models. The feedback has helped us improve the paper substantially.

We believe a central concern, how masks can be valid while necessity is poor, comes from our submitted text not clearly separating two questions: whether a mask is a valid detector, and whether the concept is computed only in its weights. We address it first, because it changes how the results read.

Valid but not necessary (W2). Section 3.3 now sorts the five criteria by the question they answer. Sufficiency, specificity and sparsity are part of the training objective and ask whether a mask is a valid detector. Masks pass them in nearly every cell. Necessity and redundancy are measured afterwards and ask whether the concept is computed only in the mask's weights. A valid mask has no reason to pass them, and should not if several valid masks exist. Poor necessity is in fact what our main finding predicts. In most cells, removing one mask's weights does no more damage than removing random weights of the same magnitude, and a linear probe still finds the concept in what remains (median gain 0.29 AUC; Fig. 1b,c). The weights of one mask are enough to detect the concept, but they are not the only weights that can.

Why invert the full mask? (W2). Ablating the whole subnetwork follows prior mask-based analyses (Csordás et al., 2021; Lepori et al., 2023; Bayazit et al., 2024). Csordás et al. (2021) also report that inverting masks across hidden layers can leave task performance largely intact, which they attribute to an ensemble of co-existing subnetworks that can each solve the task; our necessity and redundancy results show an analogous redundancy for visual concepts. We agree that parts of a mask to be necessary while others are not. Such parts must be kept by every valid mask, which is why we compute the agreed core. This set, formed by the weights that all 30 seeds keep (Section 5), is often much smaller than a single mask (8–70% of its weights). Ablating the core alone is the direct test of partial necessity, and we plan to add it to the final version.

What is new, and how to use it (W2). The main new result concerns the weights that all seeds share. Even when masks from different seeds overlap strongly, the shared weights often fail to detect the concept (AUC ≥ 0.8 in only 21 of 57 CNN cells), and how much the masks overlap does not predict whether they work (within-group ρ = −0.05; Fig. 2b). Intersecting masks, or keeping only the weights that are stable across runs, is therefore not a safe way to make a concept circuit trustworthy. We restructured the results around findings like this one (General Comment G2). Each figure now opens with its finding, and the text separates what holds in every backbone from what does not. Masks are valid detectors in nearly every cell, no two seeds keep the same weights, and masks of the same concept always overlap more than masks of different concepts. Whether the shared weights still work, however, depends on the backbone and the data. In practice, one mask is enough to detect or monitor a concept. Attributing a concept to specific weights, or pruning or editing them, needs several masks and a direct test of the weights they share; Section 6 gives four recommendations for this (General Comment G4).

Scale (W1). The ViT-B/16 results are now in the main text, run at the same depth as the CNNs with 30 seeds per cell (General Comment G1). With 4–12× more maskable weights than our CNNs, the ViT corroborates the main findings. We cannot test billion-parameter models, and the Limitations now say so. Within our budget we chose depth over breadth, because claims about multiplicity need many runs per cell: 2,070 masks in total, each compared against matched nulls. The two datasets were chosen for their concept annotations rather than their size (Section 3.2). General Comment G4 explains why we expect larger models to admit more equivalent masks, not fewer.

We hope this clarifies the points the reviewer raised, and we are happy to answer further questions.

AuthorsOpenReview ↗

Follow-up: validity vs. necessity

Dear Reviewer 2bBL,

As the discussion period ends on 18 November, we wanted to check whether our response addressed your concerns.

One of your questions was how masks can be valid while necessity is poor. Our submitted text did not clearly separate two questions, and the revised Section 3.3 does. Sufficiency, specificity and sparsity test whether a mask is a valid detector, and necessity and redundancy test whether the concept is computed only in that mask's weights. Since our results show that several valid masks exist, poor necessity is what we should expect.

The revision also moves the ViT-B/16 results into the main text at 30 seeds per cell (General Comment G1), restructures Figures 1–2 so each states one finding (General Comment G2), and adds practical recommendations in Section 6 (General Comment G4).

If any of these points remain unconvincing, we would be glad to discuss them before the deadline.

AuthorsOpenReview ↗

Core ablation results: partial necessity (App. M)

Dear Reviewer 2bBL,

In our first reply we agreed that some parts of a mask could be necessary while others are not, and promised to test this directly by ablating the core, i.e., the weights kept by all 30 masks of a cell. The results are now in Appendix M, for all 76 cells (including the 19 ViT cells).

The core is a large part of a mask (median 33% of its weights), and removing it does about as much damage as removing the whole mask (median |AUC − 0.5| of 0.062 vs. 0.065). When we compare this ablation with removing the same number of weights at random from a single trained mask, the core is more destructive than at least 90% of draws in only 15 of 76 cells (25 against magnitude-matched random weights). After removing the core, a refit linear probe still recovers a median 0.30 AUC, so the concept stays linearly decodable without it.

Thus, in a few cells the core holds weights that detection needs, but in most, the weights every seed agrees on are no more necessary than other weights of the same mask, which means that a mask is not a small indispensable kernel plus redundant padding. This supports the paper's central claim that weight-level attributions need several masks and a direct test of what they share.

Thank you again for pushing on this point, which led to a stronger analysis. We would be glad to clarify anything before the deadline.

Thank you very much for the detailed responses and revisions. I am happy to increase my score.

Reviewer_gE45OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies human-understandable concepts in the weight-space of a model. Specifically, they learn sparse binary masks over the weights of frozen vision models to identify parameters supporting the concept. First, they test and validate that these masks are valid concept detectors, then, using multiple seeds, they test the uniqueness of the parameter-level structure recovered. The main finding is that different seeds recover valid but often different parameters supporting the same concept detector.

Critical strengths
  • Soundness.
    • The study is built upon a rigorous methodology that has been visibly thought through. The criteria are well-motivated and adequate. I particularly like the addition of Redundancy and controls, and starting from a more controlled dataset to a more realistic one.
  • Important research question and convincing and insightful results
Critical weaknesses
  • The paper is generally clear, but given its richness, it would benefit from (a) better articulating contributions (e.g., in the introduction), (b) possibly a table in the appendix summarizing all the concepts introduced.
  • I believe the goal of Table 1 is to position the work, yet the works cited as closest in the text (e.g., Lepori et al. 2023) do not appear there, nor do any other weight-based methods.
Questions for authors
  • I believe some work (e.g., Zhou et al. "Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask". 2020) shows that masking can be thought of as a training procedure, which can build classification performance into untrained models. This implicitly suggests that the masks recover concepts that support this classification. What results do the authors expect if they ran their pipeline on an untrained network?
  • Clarification
    • Why are the ViT results in the appendix and not in the main text?
    • How are the positive and negative sets selected to train the masks?
3 comments
AuthorsOpenReview ↗

Initial clarifications

We thank the reviewer for the positive assessment of the methodology and for the constructive suggestions.

Weaknesses. Section 1 now includes an explicit list of contributions, and all 19 concepts are summarized in a single table (Table 3, App. C), referenced from Section 3.2 (General Comment G2). Table 1 now includes the weight-masking methods (Csordás et al., 2021; Lepori et al., 2023), and Section 2 discusses the work on non-unique sub-networks (General Comment G3).

Q1: Untrained network. This is a fair concern. Zhou et al. (2019) show that a mask alone can turn a randomly initialized network into a working classifier. Because they train masks on the labels of the task they aim to isolate, accuracy alone cannot help distinguish whether the subnetwork was discovered or built. Our masks are not trained on class labels. Our objective (Eqs. 4–5) only asks the mask to match the dense network behavior on concept-positive inputs and to keep concept-negative input predictions away from the concept's classes. Based on the information carried by the loss alone, the most our procedure could build is a binary predictor of concept-positive vs. concept-negative inputs, and any class information has to come from the trained network.

We ran the suggested experiment on three concepts (GTSRB shape_round and border_color_red, AwA2 stripes), using a randomly initialized ResNet-18, and a trained model with the same configuration as reference. On random weights, the masks found a concept positive vs concept negative discriminator for shape_round, while for the other 2 concepts it stayed at chance. Detection AUC alone therefore cannot rule out that the discriminator is built by a mask. Our method allows us to test a property the loss never optimizes: exact-class accuracy on concept-positive inputs. In all three cases the masks collapse on accuracy: 0.00–0.01, whereas masks on trained backbones keep it at 0.93–0.99, close to the dense network. The class structure our masks preserve is therefore inherited from the trained network, not built by the mask. We are aware this is a first check on three concepts; we plan to extend it to more concepts and backbones, with more seeds, in the final version.

Clarifications. The ViT results are now in the main text, with 30 seeds for 12 concepts (Section 3.2, Figs. 1–2; General Comment G1). The positive and negative sets are now defined formally in Section 3.1, with batching details in App. D.4 (General Comment G5).

AuthorsOpenReview ↗

Follow-up: revision and untrained-network control

Dear Reviewer gE45,

Thank you again for your positive review. As the discussion period ends on 18 November, we wanted to check whether our response and the revised PDF address your points:

  • The introduction now lists the contributions, all 19 concepts are in one table (Table 3), and Table 1 includes the weight-masking methods.
  • The ViT results are now in the main text at 30 seeds.
  • We also ran the untrained-network control you suggested: on a randomly initialized ResNet-18, masks keep exact-class accuracy at 0.00–0.01, against 0.93–0.99 on trained backbones, so the class structure comes from the trained network rather than being built by the mask.

If anything remains unclear, we are happy to clarify. If the revision addresses your concerns, we would be grateful if you would consider reflecting this in your score or confidence.

Thank you to the authors for their detailed responses and the effort put into the revision.

I appreciate the improvements to the presentation and the additional ViT experiments. I'm also glad the authors explored the untrained-network control I suggested, and I found the results interesting.

Overall, I think this is a well-conducted study with interesting findings, and I feel that my concerns have been addressed. However, I'm not familiar enough with the literature on weight-level interpretability to confidently judge the extent of its novelty. While my assessment remains positive, I prefer to keep my original score of weak acceptance.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Addressing common issues

We thank all reviewers for their careful reading and constructive feedback, which have led to substantial improvements to the paper. We have uploaded a revised version in which text changed in the main paper is shown in blue. Below, we address the points raised by more than one reviewer, and we refer to them from the individual replies.

G1. Vision Transformer results (gE45, 2bBL, 9Nas)

ViT-B/16 has 4–12× more maskable weights than our CNNs (Table 4), so by the deadline we had completed only 10 seeds for 11 concepts. Given that the main figures were already information-dense, we were concerned that adding results at a different scale would harm clarity, so we placed these results in the Appendix. We agree this undersold them, and since the reviews came in we have focused our compute on closing this gap. The revised version runs the ViT at the same depth as the CNNs (30 seeds) for each of 12 concepts (8 GTSRB, 4 AWA2), with the same magnitude-matched nulls. The ViT is now part of all main figures and of Sections 4–5, and its results corroborates the main findings of the paper.

G2. Readability and presentation (gE45, 2bBL, 9Nas)

We agree that the results sections were hard to follow, and we have revised them:

  • Contributions. The introduction now ends with an explicit list of contributions (gE45).
  • Figures organized around findings. We redesigned the main figures so that each supports one finding, stated in bold at the start of its caption, and covers all four architectures. We also changed what is plotted. Instead of raw scores, most of which sat near the top of the axis, each panel now shows the comparison that carries the finding: for example, the margin of the kept AUC over the best magnitude-matched null (Fig. 1a), or κ\kappa against core AUC with group differences removed (Fig. 2b). Per-cell breakdowns are in the appendix.
  • Interpreting variation across configurations. 2bBL noted that the figures mainly show that some configurations perform better than others. We have revised the text to better distinguish what varies with the configuration from what holds everywhere. For instance, in every checked cell, more weights are shared between masks of the same concept than with masks of other concepts on the same backbone (Fig. 2a), so seeds disagree on weights, but not arbitrarily. How far a single mask can be trusted, however, depends on the backbone and the data. This dependence is itself a finding, and it motivates the recommendations in G4.
  • Concept overview. As requested by gE45, we have gathered all the concepts introduced in the paper in a single table (Table 3, App. C), referenced from Section 3.2.
  • Notation and text. Yc\mathcal{Y}_c, Dc+D^+_c and Dc−D^-_c are now defined formally in Section 3.1, all equations are numbered (9Nas), and Sections 4 and 5 were rewritten so that each paragraph states its result first.

G3. Positioning against other weight-based subnetwork techniques (gE45, 2bBL, 9Nas)

A general concern has been the positioning and novelty of our technique against LTH, CS and the work of Lepori et al. While our underlying machinery directly builds upon theirs, we change what a mask is asked to do or find. In these works, masks are trained to solve a task or a sub-task the model was trained on, so the recovered subnetwork can be checked against that task. Differently, our work starts from human-defined concepts, which the network was never trained to predict, and we replace the task loss with a concept-conditional objective (Eqs. 3–5). The goal is to have the mask reproduce the dense network's own pre-classifier features on concept-positive images, while being penalized when concept-negative images land on the concept's classes. The masked network then detects the concept through the frozen head, without probes or extra labels.

We agree with 9Nas that subnetworks are known not to be unique for trained behaviors (Frankle et al., 2020; Gorbett & Whitley, 2023; Bayazit et al., 2024; Méloux et al., 2025). We show that the same holds for human-defined concepts, which the network was not trained to predict. We test this against nulls matched in size and weight magnitude, and we also test whether the weights shared by all seeds still detect the concept. Our results show that shared weights often fail on their own (AUC ≥0.8\geq 0.8 in only 21 of 57 CNN cells), and how much masks overlap does not predict whether they work (within-group ρ=−0.05\rho = -0.05). A stable overlap between masks therefore cannot be read as the concept's circuit. We revised the paper to make this positioning explicit. The first contribution in the introduction now states the extension from trained tasks to concepts, Related Work discusses the multiplicity literature above, and Table 1 now includes the weight-masking methods and a multiplicity column (gE45).

G4. Utility and practical use (2bBL)

2bBL asked how the findings transfer to modern models and how to apply them. We see two implications:

  1. Detection and monitoring. Our objective needs only class-level concept annotations; it needs no labelled sub-task and no probe. Any classifier trained on such a dataset can therefore be turned into a concept detector. For this use, any single valid mask suffices, since masks from different seeds make nearly the same concept decision (Rashomon capacity ≤1.095\leq 1.095 out of 2 in every CNN cell; Fig. 2c).
  2. Attribution, pruning and editing. Masks discovered by sparsification can support three further uses: claiming that a set of weights computes the concept, as in an audit; pruning a network or unlearning an undesired concept; and editing a given concept by changing its weights. Our results show that a single mask is not enough for any of them. Different seeds name different weights, and the weights they share often fail to detect the concept. Removing a mask's weights breaks the frozen readout, but the concept stays linearly decodable from the rest of the network (Fig. 1c). Masks that agree on the concept also change the class predictions differently (Fig. 2c). Section 6 therefore recommends four checks to run before using a mask for any of these purposes.

Our findings hold across three CNNs and ViT-B/16 (86M parameters). Since our results are consistent with concepts spread across redundant, shared weights, we expect larger models to admit more equivalent masks rather than fewer. Testing this is future work, which we now state in the Limitations. The Introduction now states these uses explicitly.

G5. How concept sets are constructed (gE45, 9Nas)

Concept labels are inherited from class-level annotations, which Section 3.1 now defines formally. Yc\mathcal{Y}_c is the set of classes annotated with concept cc in the dataset's class–concept table. Dc+D^+_c contains all training images of those classes, and Dc−D^-_c the rest. Because Yc\mathcal{Y}_c comes from the annotation table rather than from individual images, a single mislabeled image cannot add a class to it. Positive and negative sets usually differ in size, so each epoch we draw balanced batches by subsampling the larger side. On GTSRB, concepts follow deterministically from the sign specification and are exact per sign. AWA2 annotations are crowd-sourced class-level judgments, and we include both datasets to see which findings survive noisier concepts.

AuthorsOpenReview ↗

Final revision

We have uploaded the final revision. Blue text marks the changes that answer the reviewers' comments; Appendices L and M are new and fully marked.

  • ViT-B/16 at full scale (G1). The ViT now covers all 19 concepts with 30 seeds per cell (it covered 12 when we first posted G1), so every main figure and section uses 76 cells and 2,280 masks. The same findings are still valid: the kept network beats the best magnitude-matched null in 19/19 ViT cells, ablation is not more destructive than random in most cells, and the core detects the concept (AUC ≥ 0.8) in only 7 of 19 cells.
  • New Appendix L (9Nas, Q3–Q4): ablation of the mask offset δ\delta and sweep of the sparsity weight λ\lambda, done by training 175 additional masks.
  • New Appendix M (2bBL): ablation of the core, the direct test of partial necessity we promised (see our reply to 2bBL).
  • Correction to G3. The multiplicity column of Table 1 was already in the submission, what we added was the weight-masking rows.
  • Corrections to our replies to 9Nas. For δ≥1\delta \geq 1, kept AUC and specificity do not depend on δ\delta, but accuracy on positives does slightly (on AWA2/DenseNet-121, 1 of 5 masks passes the health check at δ=1\delta = 1, against 4 of 5 at the default). On the ViT, most masks fail only at λ=10−6\lambda = 10^{-6}, three decades above the paper λ\lambda rather than two. Appendix L also shows that changing the software and hardware environment changes a mask about as much as changing the seed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.