- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper casts neuron identification as Empirical Risk Minimization: the probing dataset plays the role of the training set, the candidate concepts the hypothesis class, and the similarity metric the objective. On this basis it derives PAC-style bounds on the fidelity of the selected concept, instantiates them for accuracy, AUROC, IoU, recall and precision (arguing that AUROC and recall degrade for rare concepts), and proposes Bootstrap Explanation (BE), which runs identification on K bootstrap resamples of the probing set and returns the concepts whose selection frequency reaches a threshold t. Evaluation is on synthetic data and on Broden with CLIP-Dissect on ResNet-50.
Critical strengths
- A principled formulation of neuron identification. Casting neuron identification as ERM gives a coherent way to analyze existing explanation methods, and the resulting fidelity bounds offer useful insight into how sample size, concept rarity, and metric choice affect reliability.
- Explicit representation of explanation uncertainty. BE moves beyond single-concept assignment by returning a set of plausible explanations. The synthetic and Broden experiments suggest that it improves empirical coverage when probing data are limited, which makes it a promising direction for more reliable neuron interpretation.
- A combination of theoretical and empirical analysis. The paper combines formal analysis, controlled synthetic experiments, and real-data evaluation, which helps connect the proposed statistical framework to practical neuron identification.
Critical weaknesses
- The stated sample-size guideline is hard to reconcile with the Broden results. Sec. 3.2 states that with |C| = 1,000 at 95% confidence and ε = 0.05, |Dprobe| ≈ 2,100 suffices, calling it "highly practical." The Broden experiments use ~31,652 probe images and still report large disagreement. The paper does not discuss how the two statements fit together; if the bound concerns the fidelity gap rather than the identity of the selected concept, saying so explicitly would resolve the apparent conflict.
- The validation scope is narrower than the claims. BE is evaluated only with CLIP-Dissect under soft-WPMI and cosine similarity on Broden; NetDissect appears only in qualitative examples; AUROC-based identification, which the theory singles out as unreliable for rare concepts, is absent from the BE evaluation. The only baseline is single-concept assignment, so the benefit of the bootstrap aggregation relative to a simpler margin-based prediction set is not isolated.
- The empirical evaluation is relatively narrow. The quantitative evaluation rests mainly on CLIP-Dissect with a specific backbone, layer, and concept dataset. The synthetic experiments provide controlled evidence, but the paper gives limited quantitative evidence that its conclusions generalize across models, layers, identification methods, and concept distributions.
- The effect of the binarization thresholds is not analyzed. For binary metrics, the bounds depend on the thresholds τ_f and τ_c. The paper notes that τ_f is "specified by the identification method" but does not analyze how the choice of threshold affects the guarantee.
Questions for authors
- Is there any result or experiment showing how closely the bootstrap coverage approaches the idealized bound of Thm. 4.1?
- Table F.1 shows bootstrap time growing from 64.7s to 348.7s when the concept set expands from 1,198 to 20,000 concepts, while activation computation grows only slightly. How should practitioners balance K against this cost in large concept-set settings?
6 comments
Rebuttal to Reviewer MZGJ (2/4)
Weakness 2b: "The only baseline is single-concept assignment, so the benefit of the bootstrap aggregation relative to a simpler margin-based prediction set is not isolated."
A2b: Thank you for the suggestion. We have added a comparison with margin-based and other prediction sets of the same mean size to App. K.5 (Fig. K.2 and Table K.4) of the revised paper, and repeated it for NetDissect and AUROC in App. K.6 (Table K.5).
Setup. Same as our Broden evaluation in Sec. 4.3 with all probe images, plus the NetDissect and AUROC (m = 100) settings of A2a. Each baseline returns the top concepts of the probe ranking and differs only in how many concepts each neuron gets: global top-k (the same number for every neuron), margin (all concepts within m of the best probe score), and scaled margin (m times the standard deviation of the neuron's probe scores). Each baseline is matched to BE's mean set size at t = 95:
| Similarity | Mean |S| | Single | Global top-k | Margin | Scaled margin | BE |
|---|---|---|---|---|---|---|
| soft-WPMI | 4.86 | 0.504 | 0.780 | 0.750 | 0.775 | 0.808 |
| Cosine | 1.63 | 0.834 | 0.901 | 0.951 | 0.948 | 0.956 |
| IoU (NetDissect) | 2.36 | 0.716 | 0.882 | 0.871 | 0.902 | 0.912 |
| AUROC | 2.22 | 0.717 | 0.886 | 0.913 | 0.892 | 0.932 |
Table R2: Coverage at BE's mean set size (K = 100, t = 95) on Broden with all probe images.
From Table R2 and Fig. K.2, we can see:
- Bootstrap aggregation helps beyond set size. At the same mean size, BE has the highest coverage in every setting, 0.5 to 2.8 points above the best baseline.
- Margin-based sets are not reliable across metrics. The margin set falls below global top-k under soft-WPMI and IoU but is close to BE under cosine, so whether gaps in probe scores reflect uncertainty depends on the metric. BE's bootstrap frequencies give the best set under all four metrics.
Weakness 3: "The empirical evaluation is relatively narrow. The quantitative evaluation rests mainly on CLIP-Dissect with a specific backbone, layer, and concept dataset. The synthetic experiments provide controlled evidence, but the paper gives limited quantitative evidence that its conclusions generalize across models, layers, identification methods, and concept distributions."
A3: Thank you for the suggestion. We have extended the quantitative evaluation along each axis you list, changing one component at a time from our Broden evaluation in Sec. 4.3 (K = 100, t = 95):
- Models and layers: ResNet-50 layer 3, ViT-B/16 (CLS token of the last layer), and ConvNeXt-base (last stage), in App. K.4 (Table K.3).
- Concept distributions: the ImageNet validation set as the probing dataset (App. K.4), and AUROC concept sets of increasing rarity (A2a, App. K.6).
- Identification methods: NetDissect and AUROC-based identification (A2a, App. K.6).
| Model | Layer | Probe set | soft-WPMI: BE / single / best baseline | Cosine: BE / single / best baseline |
|---|---|---|---|---|
| ResNet-50 | layer 4 | Broden | 0.81 / 0.50 / 0.78 | 0.96 / 0.83 / 0.95 |
| ResNet-50 | layer 3 | Broden | 0.78 / 0.46 / 0.77 | 0.95 / 0.80 / 0.94 |
| ViT-B/16 | last (CLS) | Broden | 0.64 / 0.33 / 0.65 | 0.97 / 0.89 / 0.96 |
| ConvNeXt-base | last | Broden | 0.67 / 0.32 / 0.67 | 0.98 / 0.88 / 0.97 |
| ResNet-50 | layer 4 | ImageNet val | 0.83 / 0.52 / 0.80 | 0.96 / 0.76 / 0.94 |
Table R3: Coverage of the held-out top-1 concept by BE, single-concept assignment, and the best size-matched baseline (global top-k, margin, or scaled margin at BE's mean set size), across models, layers, and probing datasets (CLIP-Dissect). Full results are in App. K.4 (Table K.3).
From Table R3 and Table R1, we can see: BE improves coverage over single-concept assignment in every setting, and beats the best baseline in 9 of 10 settings.
Rebuttal to Reviewer MZGJ (3/4)
Weakness 4: "The effect of the binarization thresholds is not analyzed. For binary metrics, the bounds depend on the thresholds τ_f and τ_c. The paper notes that τ_f is "specified by the identification method" but does not analyze how the choice of threshold affects the guarantee."
A4: To clarify, "Thresholds do not affect the theoretical guarantee". The thresholds τ_f, τ_c are part of how the neuron and concept functions are defined. Once they are fixed, the binarized functions f̃(x) = 1[f(x) > τ_f] and c̃(x) = 1[c(x) > τ_c] are fixed functions of x, and the probing samples remain i.i.d. Theorem 3.2 and Corollary 3.3 therefore hold unchanged for every choice of threshold.
We support this empirically (App. K.3, Table K.1) on Broden (ResNet-50 layer 4, 764 concepts with ground-truth labels), varying one threshold at a time and measuring how often BE and single-concept assignment cover the held-out top-1 concept:
| Setting | BE coverage (acc. / IoU) | Single-concept coverage (acc. / IoU) | BE mean |S| (acc. / IoU) |
|---|---|---|---|
| Default: τ_f = 95th pct., τ_c = label present | 0.81 / 0.94 | 0.48 / 0.73 | 6.7 / 2.1 |
| Vary τ_f | |||
| 80th pct. | 0.91 / 0.93 | 0.68 / 0.75 | 2.9 / 1.9 |
| 90th pct. | 0.87 / 0.94 | 0.58 / 0.76 | 4.2 / 1.9 |
| 99th pct. | 0.49 / 0.89 | 0.25 / 0.62 | 21.2 / 3.2 |
| 99.5th pct. | 0.30 / 0.86 | 0.14 / 0.58 | 27.2 / 4.4 |
| Vary τ_c | |||
| Mask area > 1% (697 concepts) | 0.82 / 0.95 | 0.49 / 0.77 | 6.0 / 1.8 |
| Mask area > 5% (537 concepts) | 0.82 / 0.96 | 0.51 / 0.79 | 5.8 / 1.8 |
| Mask area > 10% (445 concepts) | 0.84 / 0.96 | 0.53 / 0.78 | 5.5 / 1.8 |
Table R4: Coverage of the held-out top-1 concept by BE (t = 0.95K) and by single-concept assignment under different thresholds, with BE's mean prediction-set size.
BE improves coverage over single-concept assignment under every threshold, so its benefit does not depend on a particular threshold choice.
Q1: "Is there any result or experiment showing how closely the bootstrap coverage approaches the idealized bound of Thm. 4.1?"
A-Q1: Thank you for the question. We compare BE with the idealized version of Thm. 4.1 (each of the K trials uses a fresh probing set) in the controlled synthetic setting of Fig. 3(c, d) (|C| = 1,000, s* = 0.9, K = 100, t = 0.95K, 2,000 repetitions), for two margins Δ. We evaluate the bound of Thm. 4.1 with the true single-trial error p, estimated by Monte Carlo:
| Δ | |D_probe| | Single trial (1 − p) | Thm. 4.1 bound | Ideal BE | BE |
|---|---|---|---|---|---|
| 0.05 | 300 | 0.247 | 1.000 | 1.000 | 0.721 |
| 0.05 | 500 | 0.525 | 1.000 | 1.000 | 0.906 |
| 0.05 | 800 | 0.823 | 1.000 | 1.000 | 0.985 |
| 0.05 | 1,500 | 0.993 | 1.000 | 1.000 | 1.000 |
| 0.02 | 800 | 0.079 | 0.808 | 1.000 | 0.466 |
| 0.02 | 1,500 | 0.222 | 1.000 | 1.000 | 0.690 |
| 0.02 | 3,000 | 0.585 | 1.000 | 1.000 | 0.931 |
| 0.02 | 6,000 | 0.944 | 1.000 | 1.000 | 0.996 |
Table R5: Coverage of the optimal concept by a single trial, ideal BE, and BE, with the bound of Thm. 4.1.
From Table R5, we can see:
- Ideal BE always satisfies the bound, as Thm. 4.1 states.
- BE approaches the bound as the probe set grows. It falls below the bound when the single-trial error is large, because bootstrap resamples reuse the same probing set and are therefore not independent; the gap closes once a single trial is moderately reliable (1 − p ≥ 0.5), where BE reaches 0.91 to 1.00.
Rebuttal to Reviewer MZGJ (1/4)
We thank the reviewer for taking the time to review and we are glad that the reviewer recognizes our principled ERM formulation of neuron identification, BE's explicit representation of explanation uncertainty, and the combination of theoretical and empirical analysis! We address your concerns and questions below.
Weakness 1: "The stated sample-size guideline is hard to reconcile with the Broden results. Sec. 3.2 states that with |C| = 1,000 at 95% confidence and ε = 0.05, |Dprobe| ≈ 2,100 suffices, calling it "highly practical." The Broden experiments use ~31,652 probe images and still report large disagreement. The paper does not discuss how the two statements fit together; if the bound concerns the fidelity gap rather than the identity of the selected concept, saying so explicitly would resolve the apparent conflict."
A1: Thank you for the suggestion. The reviewer is correct that the bound in Corollary 3.3 concerns the fidelity gap. Note that recovering the identity of the selected concept (Corollary 3.4) additionally depends on the margin between the optimal concept and the other concepts, which varies with the target neuron and the similarity function. A small margin leads to the lower agreement in the Broden experiment, and we have added a note in Sec. 3.2 of the revised paper to clarify this, with supporting analysis in App. K.2.
We further clarify that the practical value of the sample-size guideline lies in its scaling law. Although it does not directly give the concept agreement rate in practice, it shows that the error r decreases as 1/√|D_probe|. This gives users a practical guideline: they can start by labeling a small set and estimate the required probe size from the scaling rule.
Weakness 2a: "The validation scope is narrower than the claims. BE is evaluated only with CLIP-Dissect under soft-WPMI and cosine similarity on Broden; NetDissect appears only in qualitative examples; AUROC-based identification, which the theory singles out as unreliable for rare concepts, is absent from the BE evaluation."
A2a: Thank you for the suggestion. Following your suggestion, we evaluate BE quantitatively with NetDissect and AUROC-based identification, and have added the results to App. K.6 (Table K.5) of the revised paper.
Setup. Same as our Broden evaluation in Sec. 4.3 (ResNet-50 layer 4, the same probe/test split, K = 100, t = 95), with two other identification methods: (1) NetDissect, using pixel-level IoU with the Broden segmentation masks over 542 concepts; and (2) AUROC as in INVERT, using image-level Broden labels. To test rare concepts, we keep the AUROC concepts with at least m ∈ {100, 20, 5} positive images in each half (277, 549, and 1,014 concepts):
Table R1: Coverage on Broden with all probe images (2,048 neurons).
From Table R1, we can see: