acceptodds
Under review as a conference paper at ICLR 2027

Reliable Neuron Identification: A Statistical Learning Framework for Neuron Explanations

Abstract

A central goal of mechanistic interpretability is to reverse-engineer deep neural networks into human-understandable mechanisms. A foundational step in this pipeline is neuron identification, which maps internal neural representations to human-understandable concepts. However, existing methods for neuron identification largely operate as empirical heuristics, making it unclear when the resulting neuron explanations can be trusted. To bridge this critical gap, we establish the first theoretical foundation for neuron identification by uncovering a fundamental correspondence to statistical learning theory: for a given neuron, selecting the concept that best explains its activation behavior is equivalent to Empirical Risk Minimization (ERM) over a concept hypothesis class. This ERM formulation unifies popular identification methods (e.g., Network Dissection, CLIP-Dissect) under a common objective and casts explanation reliability as a classical generalization problem. Leveraging this connection, we derive PAC-style fidelity bounds that characterize how explanation reliability depends on probing sample size and concept class complexity, and reveal a critical limitation of commonly used metrics: AUROC- and recall-based identification can become substantially less reliable for rare concepts. Finally, to capture uncertainty in neuron explanations, we introduce Bootstrap Explanation (BE), a principled procedure that produces concept prediction sets, with a provable coverage guarantee for its idealized variant. By connecting interpretability with statistical learning theory, our work provides a principled foundation for reliable neuron identification.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, neuron explanation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MZGJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper casts neuron identification as Empirical Risk Minimization: the probing dataset plays the role of the training set, the candidate concepts the hypothesis class, and the similarity metric the objective. On this basis it derives PAC-style bounds on the fidelity of the selected concept, instantiates them for accuracy, AUROC, IoU, recall and precision (arguing that AUROC and recall degrade for rare concepts), and proposes Bootstrap Explanation (BE), which runs identification on K bootstrap resamples of the probing set and returns the concepts whose selection frequency reaches a threshold t. Evaluation is on synthetic data and on Broden with CLIP-Dissect on ResNet-50.

Critical strengths
  • A principled formulation of neuron identification. Casting neuron identification as ERM gives a coherent way to analyze existing explanation methods, and the resulting fidelity bounds offer useful insight into how sample size, concept rarity, and metric choice affect reliability.
  • Explicit representation of explanation uncertainty. BE moves beyond single-concept assignment by returning a set of plausible explanations. The synthetic and Broden experiments suggest that it improves empirical coverage when probing data are limited, which makes it a promising direction for more reliable neuron interpretation.
  • A combination of theoretical and empirical analysis. The paper combines formal analysis, controlled synthetic experiments, and real-data evaluation, which helps connect the proposed statistical framework to practical neuron identification.
Critical weaknesses
  • The stated sample-size guideline is hard to reconcile with the Broden results. Sec. 3.2 states that with |C| = 1,000 at 95% confidence and ε = 0.05, |Dprobe| ≈ 2,100 suffices, calling it "highly practical." The Broden experiments use ~31,652 probe images and still report large disagreement. The paper does not discuss how the two statements fit together; if the bound concerns the fidelity gap rather than the identity of the selected concept, saying so explicitly would resolve the apparent conflict.
  • The validation scope is narrower than the claims. BE is evaluated only with CLIP-Dissect under soft-WPMI and cosine similarity on Broden; NetDissect appears only in qualitative examples; AUROC-based identification, which the theory singles out as unreliable for rare concepts, is absent from the BE evaluation. The only baseline is single-concept assignment, so the benefit of the bootstrap aggregation relative to a simpler margin-based prediction set is not isolated.
  • The empirical evaluation is relatively narrow. The quantitative evaluation rests mainly on CLIP-Dissect with a specific backbone, layer, and concept dataset. The synthetic experiments provide controlled evidence, but the paper gives limited quantitative evidence that its conclusions generalize across models, layers, identification methods, and concept distributions.
  • The effect of the binarization thresholds is not analyzed. For binary metrics, the bounds depend on the thresholds τ_f and τ_c. The paper notes that τ_f is "specified by the identification method" but does not analyze how the choice of threshold affects the guarantee.
Questions for authors
  • Is there any result or experiment showing how closely the bootstrap coverage approaches the idealized bound of Thm. 4.1?
  • Table F.1 shows bootstrap time growing from 64.7s to 348.7s when the concept set expands from 1,198 to 20,000 concepts, while activation computation grows only slightly. How should practitioners balance K against this cost in large concept-set settings?
6 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer MZGJ (1/4)

We thank the reviewer for taking the time to review and we are glad that the reviewer recognizes our principled ERM formulation of neuron identification, BE's explicit representation of explanation uncertainty, and the combination of theoretical and empirical analysis! We address your concerns and questions below.


Weakness 1: "The stated sample-size guideline is hard to reconcile with the Broden results. Sec. 3.2 states that with |C| = 1,000 at 95% confidence and ε = 0.05, |Dprobe| ≈ 2,100 suffices, calling it "highly practical." The Broden experiments use ~31,652 probe images and still report large disagreement. The paper does not discuss how the two statements fit together; if the bound concerns the fidelity gap rather than the identity of the selected concept, saying so explicitly would resolve the apparent conflict."

A1: Thank you for the suggestion. The reviewer is correct that the bound in Corollary 3.3 concerns the fidelity gap. Note that recovering the identity of the selected concept (Corollary 3.4) additionally depends on the margin between the optimal concept and the other concepts, which varies with the target neuron and the similarity function. A small margin leads to the lower agreement in the Broden experiment, and we have added a note in Sec. 3.2 of the revised paper to clarify this, with supporting analysis in App. K.2.

We further clarify that the practical value of the sample-size guideline lies in its scaling law. Although it does not directly give the concept agreement rate in practice, it shows that the error r decreases as 1/√|D_probe|. This gives users a practical guideline: they can start by labeling a small set and estimate the required probe size from the scaling rule.


Weakness 2a: "The validation scope is narrower than the claims. BE is evaluated only with CLIP-Dissect under soft-WPMI and cosine similarity on Broden; NetDissect appears only in qualitative examples; AUROC-based identification, which the theory singles out as unreliable for rare concepts, is absent from the BE evaluation."

A2a: Thank you for the suggestion. Following your suggestion, we evaluate BE quantitatively with NetDissect and AUROC-based identification, and have added the results to App. K.6 (Table K.5) of the revised paper.

Setup. Same as our Broden evaluation in Sec. 4.3 (ResNet-50 layer 4, the same probe/test split, K = 100, t = 95), with two other identification methods: (1) NetDissect, using pixel-level IoU with the Broden segmentation masks over 542 concepts; and (2) AUROC as in INVERT, using image-level Broden labels. To test rare concepts, we keep the AUROC concepts with at least m ∈ {100, 20, 5} positive images in each half (277, 549, and 1,014 concepts):

SimilarityConcepts keptSingleBEMean |S|
IoU (NetDissect)-0.7160.9122.36
AUROCm = 1000.7170.9322.22
AUROCm = 200.5630.8783.52
AUROCm = 50.2560.70416.60

Table R1: Coverage on Broden with all probe images (2,048 neurons).

From Table R1, we can see:

  1. BE generalizes to other identification methods. It raises coverage from 0.716 to 0.912 with NetDissect and from 0.717 to 0.932 with AUROC (m = 100), using about two concepts per neuron.
  2. AUROC degrades for rare concepts, as Table 1 predicts. Admitting rarer concepts lowers single-concept coverage from 0.717 to 0.256; with m = 5, it is 0.229 for neurons whose reference concept has frequency below 0.1%, against 0.456 for the others. BE reports this uncertainty through its set size, which grows from 2.2 to 16.6 concepts.
AuthorsOpenReview ↗

Rebuttal to Reviewer MZGJ (2/4)

Weakness 2b: "The only baseline is single-concept assignment, so the benefit of the bootstrap aggregation relative to a simpler margin-based prediction set is not isolated."

A2b: Thank you for the suggestion. We have added a comparison with margin-based and other prediction sets of the same mean size to App. K.5 (Fig. K.2 and Table K.4) of the revised paper, and repeated it for NetDissect and AUROC in App. K.6 (Table K.5).

Setup. Same as our Broden evaluation in Sec. 4.3 with all probe images, plus the NetDissect and AUROC (m = 100) settings of A2a. Each baseline returns the top concepts of the probe ranking and differs only in how many concepts each neuron gets: global top-k (the same number for every neuron), margin (all concepts within m of the best probe score), and scaled margin (m times the standard deviation of the neuron's probe scores). Each baseline is matched to BE's mean set size at t = 95:

SimilarityMean |S|SingleGlobal top-kMarginScaled marginBE
soft-WPMI4.860.5040.7800.7500.7750.808
Cosine1.630.8340.9010.9510.9480.956
IoU (NetDissect)2.360.7160.8820.8710.9020.912
AUROC2.220.7170.8860.9130.8920.932

Table R2: Coverage at BE's mean set size (K = 100, t = 95) on Broden with all probe images.

From Table R2 and Fig. K.2, we can see:

  1. Bootstrap aggregation helps beyond set size. At the same mean size, BE has the highest coverage in every setting, 0.5 to 2.8 points above the best baseline.
  2. Margin-based sets are not reliable across metrics. The margin set falls below global top-k under soft-WPMI and IoU but is close to BE under cosine, so whether gaps in probe scores reflect uncertainty depends on the metric. BE's bootstrap frequencies give the best set under all four metrics.

Weakness 3: "The empirical evaluation is relatively narrow. The quantitative evaluation rests mainly on CLIP-Dissect with a specific backbone, layer, and concept dataset. The synthetic experiments provide controlled evidence, but the paper gives limited quantitative evidence that its conclusions generalize across models, layers, identification methods, and concept distributions."

A3: Thank you for the suggestion. We have extended the quantitative evaluation along each axis you list, changing one component at a time from our Broden evaluation in Sec. 4.3 (K = 100, t = 95):

  • Models and layers: ResNet-50 layer 3, ViT-B/16 (CLS token of the last layer), and ConvNeXt-base (last stage), in App. K.4 (Table K.3).
  • Concept distributions: the ImageNet validation set as the probing dataset (App. K.4), and AUROC concept sets of increasing rarity (A2a, App. K.6).
  • Identification methods: NetDissect and AUROC-based identification (A2a, App. K.6).
ModelLayerProbe setsoft-WPMI: BE / single / best baselineCosine: BE / single / best baseline
ResNet-50layer 4Broden0.81 / 0.50 / 0.780.96 / 0.83 / 0.95
ResNet-50layer 3Broden0.78 / 0.46 / 0.770.95 / 0.80 / 0.94
ViT-B/16last (CLS)Broden0.64 / 0.33 / 0.650.97 / 0.89 / 0.96
ConvNeXt-baselastBroden0.67 / 0.32 / 0.670.98 / 0.88 / 0.97
ResNet-50layer 4ImageNet val0.83 / 0.52 / 0.800.96 / 0.76 / 0.94

Table R3: Coverage of the held-out top-1 concept by BE, single-concept assignment, and the best size-matched baseline (global top-k, margin, or scaled margin at BE's mean set size), across models, layers, and probing datasets (CLIP-Dissect). Full results are in App. K.4 (Table K.3).

From Table R3 and Table R1, we can see: BE improves coverage over single-concept assignment in every setting, and beats the best baseline in 9 of 10 settings.

AuthorsOpenReview ↗

Rebuttal to Reviewer MZGJ (3/4)

Weakness 4: "The effect of the binarization thresholds is not analyzed. For binary metrics, the bounds depend on the thresholds τ_f and τ_c. The paper notes that τ_f is "specified by the identification method" but does not analyze how the choice of threshold affects the guarantee."

A4: To clarify, "Thresholds do not affect the theoretical guarantee". The thresholds τ_f, τ_c are part of how the neuron and concept functions are defined. Once they are fixed, the binarized functions f̃(x) = 1[f(x) > τ_f] and c̃(x) = 1[c(x) > τ_c] are fixed functions of x, and the probing samples remain i.i.d. Theorem 3.2 and Corollary 3.3 therefore hold unchanged for every choice of threshold.

We support this empirically (App. K.3, Table K.1) on Broden (ResNet-50 layer 4, 764 concepts with ground-truth labels), varying one threshold at a time and measuring how often BE and single-concept assignment cover the held-out top-1 concept:

SettingBE coverage (acc. / IoU)Single-concept coverage (acc. / IoU)BE mean |S| (acc. / IoU)
Default: τ_f = 95th pct., τ_c = label present0.81 / 0.940.48 / 0.736.7 / 2.1
Vary τ_f
80th pct.0.91 / 0.930.68 / 0.752.9 / 1.9
90th pct.0.87 / 0.940.58 / 0.764.2 / 1.9
99th pct.0.49 / 0.890.25 / 0.6221.2 / 3.2
99.5th pct.0.30 / 0.860.14 / 0.5827.2 / 4.4
Vary τ_c
Mask area > 1% (697 concepts)0.82 / 0.950.49 / 0.776.0 / 1.8
Mask area > 5% (537 concepts)0.82 / 0.960.51 / 0.795.8 / 1.8
Mask area > 10% (445 concepts)0.84 / 0.960.53 / 0.785.5 / 1.8

Table R4: Coverage of the held-out top-1 concept by BE (t = 0.95K) and by single-concept assignment under different thresholds, with BE's mean prediction-set size.

BE improves coverage over single-concept assignment under every threshold, so its benefit does not depend on a particular threshold choice.


Q1: "Is there any result or experiment showing how closely the bootstrap coverage approaches the idealized bound of Thm. 4.1?"

A-Q1: Thank you for the question. We compare BE with the idealized version of Thm. 4.1 (each of the K trials uses a fresh probing set) in the controlled synthetic setting of Fig. 3(c, d) (|C| = 1,000, s* = 0.9, K = 100, t = 0.95K, 2,000 repetitions), for two margins Δ. We evaluate the bound of Thm. 4.1 with the true single-trial error p, estimated by Monte Carlo:

Δ|D_probe|Single trial (1 − p)Thm. 4.1 boundIdeal BEBE
0.053000.2471.0001.0000.721
0.055000.5251.0001.0000.906
0.058000.8231.0001.0000.985
0.051,5000.9931.0001.0001.000
0.028000.0790.8081.0000.466
0.021,5000.2221.0001.0000.690
0.023,0000.5851.0001.0000.931
0.026,0000.9441.0001.0000.996

Table R5: Coverage of the optimal concept by a single trial, ideal BE, and BE, with the bound of Thm. 4.1.

From Table R5, we can see:

  1. Ideal BE always satisfies the bound, as Thm. 4.1 states.
  2. BE approaches the bound as the probe set grows. It falls below the bound when the single-trial error is large, because bootstrap resamples reuse the same probing set and are therefore not independent; the gap closes once a single trial is moderately reliable (1 − p ≥ 0.5), where BE reaches 0.91 to 1.00.
Reviewer_GjCTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper draws a connection between neuron identification (i.e labeling) tasks and ERM used as the basis of learning these days. In neuron identification a probe dataset is used to assign the best name to a neuron, which the paper connects with how in training of models we use a probe dataset to find the best model (shown in Fig 1). This connection is then used to provide a PAC-style guarantee over the hyper-parameters used in neuron identification (similarity metric used, probe dataset size etc.). Lastly the paper proposes a Bootstrap Explanation method, where multiple probe dataset are used to provide a better labeling for the neuron. Instead of assigning a single label (e.g. ‘Cat’) to a neuron using the full probe dataset, the paper goes over frequency of names that get assigned over multiple probe subsets and then provides multiple candidate names (e.g. ‘0.72 x Cat’, ‘0.22 x Pet’, etc.). There are some quantitative results in Fig 3 showing that the proposed BE has a better coverage than the standard single concept labeling.

Critical strengths

S1) The paper is well-written and easy to follow. In particular Figures 1 and 2 helped quite a lot to understand the text. Fig 3 was also relatively easy to grasp.

S2) Automated Neuron/Concept labeling has become very standard these days and the labels are often trusted for downstream tasks (e.g. providing a language-level explanation to the user). Therefore, focusing on reliability guarantees for such labeling sounds quite interesting and relevant.

S3) The provided convergence rates are separated based on similarity metric (Table 1), which is quite sensible.

Critical weaknesses

W1) I appreciate Table 1 and find it quite interesting, where the paper provides a per-metric convergence rate based on the similarity metric being used. However, I find that the discussion falls short to only provide proofs and not connect it to a practical case. In particular, in lines 324-332, a hypothetical case is discussed that for a certain config (concept size, target confidence and epsilon), a dataset size of 2,100 would be enough. This is very nice to show, but it is not necessarily connected to a real-world use case. Suppose someone in the community wants to run a labeling method (e.g. CLIP-Dissect) on a set of concepts with a probe dataset. Could this method be used to directly show that the estimate rate would empirically hold in that case? E.g. showing the achieved labeling accuracy against what the rate would have recommended across different dataset sizes.

I find some empirical evidence like this to fully ground the proof into a design choice for future readers is missing.

W2) I think the proposed BE method of using multiple subsets to come up with a set of labels for a neuron rather than a single guess is quite sensible to me. However, the provided results for this are extremely limited. The only comparison to direct labeling is the “coverage” reported in Figure 3.

First, the only definition of coverage that I could find was in Line 850 in the appendix. This should not be the case, when this is the primary metric of the paper.

Second, as far as I understood, coverage is the same as Recall, where it evaluates if the correct name appears on the list. This makes the gains in Figure 3 quite trivial to me, as BE always would output a set labels rather than a single one, the Recall (whether the correct label appears in the set) would almost always increase, and beating the single-name baseline (orange curve) would seem quite trivial. Could the authors please justify if this is not the case?

In addition, providing more labels to the user can also come at a cost, especially if some of them turn out to be irrelevant, which is currently not being measured under “coverage”. What is missing is some sort of Precision metric, showing how precise the output set is. Also a user study where it is reported how much of the output set the users find relevant, and if the frequency-based ranking within the set agrees with their interpretation. For this one could for instance show the top activating images and patches for the neuron together with the output of the method. One could also provide the candidate words to the user, ask them to rank them on relevance, and measure the ranking correlation with the ranking the method provides.

Questions for authors

A few written in the weaknesses above.

5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer GjCT (1/2)

We thank the reviewer for taking the time tor review and we are glad that the reviewer finds our paper well-written and easy to follow, the focus on reliability guarantees for neuron labeling relevant, and the per-metric convergence rates sensible! We address your concerns and questions below.


Weakness 1: "I appreciate Table 1 and find it quite interesting, where the paper provides a per-metric convergence rate based on the similarity metric being used. However, I find that the discussion falls short to only provide proofs and not connect it to a practical case. In particular, in lines 324-332, a hypothetical case is discussed that for a certain config (concept size, target confidence and epsilon), a dataset size of 2,100 would be enough. This is very nice to show, but it is not necessarily connected to a real-world use case. Suppose someone in the community wants to run a labeling method (e.g. CLIP-Dissect) on a set of concepts with a probe dataset. Could this method be used to directly show that the estimate rate would empirically hold in that case? E.g. showing the achieved labeling accuracy against what the rate would have recommended across different dataset sizes."

"I find some empirical evidence like this to fully ground the proof into a design choice for future readers is missing."

A1: Thank you for the suggestion. Following your suggestion, we compare our bound with the fidelity gap achieved in a real-world Broden dataset, and add this experiment to App. K.1 (Fig. K.1) of the revised paper.

Setup. Same as our Broden evaluation in Sec. 4.3 (ResNet-50 layer 4, 2,048 neurons, 31,652 probe and 31,653 held-out images), except that we use the 764 ground-truth Broden concept labels with at least 20 positive images and binarize each neuron at its 95th percentile, so that the closed-form rates of Table 1 apply to accuracy and IoU. For nested probe sets of increasing size (From 500 to 31,652), we compare the bound of Corollary 3.3 (δ = 0.05) with the observed fidelity gap s(f, c*) − s(f, ĉ) on the held-out half.

From Fig. K.1, the bound holds for every neuron at every probe size. Its asymptotic trend holds on real data: the observed gap shrinks as 1/√|D_probe|, as the bound predicts. This trend gives practitioners a usable approach to estimate required sample size based on a small probing dataset. For example, users can estimate the gap at 500 images and use the scaling law to predict the gap at 31,652 images, or estimate the needed probing set size according to target gap.


Weakness 2: "I think the proposed BE method of using multiple subsets to come up with a set of labels for a neuron rather than a single guess is quite sensible to me. However, the provided results for this are extremely limited. The only comparison to direct labeling is the “coverage” reported in Figure 3."

Weakness 2a: "First, the only definition of coverage that I could find was in Line 850 in the appendix. This should not be the case, when this is the primary metric of the paper."

A2a: Thank you for pointing this out. We have moved the definition of coverage to the beginning of Sec. 4.3 in the revised paper.


Weakness 2b: "Second, as far as I understood, coverage is the same as Recall, where it evaluates if the correct name appears on the list. This makes the gains in Figure 3 quite trivial to me, as BE always would output a set labels rather than a single one, the Recall (whether the correct label appears in the set) would almost always increase, and beating the single-name baseline (orange curve) would seem quite trivial. Could the authors please justify if this is not the case?"

A2b: Thank you for raising this. We agree that a larger set can only increase coverage, so the fair test is whether BE covers more than simpler sets of the same size. We have added this comparison to App. K.5 (Fig. K.2 and Table K.4) of the revised paper.

Setup. Same as Fig. 3(a, b) with all probe images. Each baseline returns the top concepts of the probe ranking and differs only in how many concepts each neuron gets: global top-k (the same number for every neuron), margin (all concepts within m of the best probe score), and scaled margin (m times the standard deviation of the neuron's probe scores). Each baseline is matched to BE's mean set size:

MetricMean |S|SingleGlobal top-kMarginScaled marginBE
soft-WPMI4.860.5040.7800.7500.7750.808
Cosine1.630.8340.9010.9510.9480.956

Table R1: Coverage at BE's mean set size (t = 95, K = 100).

From Table R1 and Fig. K.2, we can see: BE's gain is not explained by set size alone. Under both metrics, BE has the highest coverage among sets of the same mean size.

AuthorsOpenReview ↗

Rebuttal to Reviewer GjCT (2/2)

Weakness 2c: "In addition, providing more labels to the user can also come at a cost, especially if some of them turn out to be irrelevant, which is currently not being measured under “coverage”. What is missing is some sort of Precision metric, showing how precise the output set is."

A2c: We agree that a large prediction set also comes at a cost. For BE, the threshold t directly controls this trade-off: a lower t returns smaller sets, and a higher t returns larger sets with higher coverage. Table R2 shows the resulting coverage–set-size curve, which is the BE curve of Fig. K.2 in the revised paper (same setting as Table R1):

t/Ksoft-WPMI: mean |S|soft-WPMI: coverageCosine: mean |S|Cosine: coverage
Single1.000.5041.000.834
0.51.460.5701.050.847
0.72.130.6541.200.895
0.93.790.7651.480.941
0.954.860.8081.630.956
0.996.550.8491.930.975

Table R2: Coverage and mean prediction-set size of BE for different thresholds (K = 100). "Single" is single-concept assignment.

In practice, users can flexibly decide the tradeoff by choosing different t, whose cost is negligible as it does not need to rerun the bootstrap.


Weakness 2d: "Also a user study where it is reported how much of the output set the users find relevant, and if the frequency-based ranking within the set agrees with their interpretation. For this one could for instance show the top activating images and patches for the neuron together with the output of the method. One could also provide the candidate words to the user, ask them to rank them on relevance, and measure the ranking correlation with the ranking the method provides."

A2d: Thank you for the suggestion. Following it, we ran a user study on Prolific.

Setup. We sample 80 neurons from our Broden evaluation in Sec. 4.3 (ResNet-50 layer 4, CLIP-Dissect with soft-WPMI, K = 100, t = 95), stratified by BE set size from 1 to 15. For each neuron, raters see its 12 top-activating test images together with its BE words (up to the 6 most frequent) and 2 random words from outside the neuron's top 200 concepts, in shuffled order, and judge whether each word describes the images (yes / no / unsure). Each neuron is rated by 3 of 12 raters, all of whom passed an attention check. A word's relevance is the share of yes answers; the overall average weights each set-size stratum by its share of neurons:

BE set sizeNeuronsBE words rated relevantRandom words rated relevant
1131.000.10
2130.880.04
3–4200.760.12
5–8200.580.12
9–15140.480.12
All (weighted)800.760.10

Table R3: Share of BE words and random foil words that raters judge relevant to the neuron's top-activating images.

From Table R3, we can see:

  1. Users find most of BE's output relevant: 76% of BE words (95% CI 72–79%), against 10% of random foils.
  2. Relevance decreases with set size: single-concept sets are always judged relevant, while about half of the words in sets of 9 to 15 are. This meets our expectation, as large sets indicate the neurons' concept is uncertain.

For the ranking, BE's frequency ranking agrees with users: in the 67 neurons with at least two shown words, the most frequent word is judged relevant 77% of the time, against 58% for the lower-ranked words, and relevance decreases with rank (0.77, 0.64, 0.57, 0.60, 0.49, 0.52 for ranks 1 to 6).


Summary

  1. W1 (bound vs. real-world labeling accuracy): On Broden, the bound holds for every neuron and probe size; although conservative, the observed gap follows its 1/√|D_probe| rate, so a small pilot run predicts the gap at larger probe sizes (App. K.1).
  2. W2a (coverage definition): Coverage is now defined at the start of Sec. 4.3.
  3. W2b (coverage vs. recall): BE's gain is not explained by set size alone: at the same mean set size, it matches or exceeds global top-k and margin-based sets by adapting the set size to each neuron (App. K.5).
  4. W2c (precision of the prediction set): The threshold t allows users to control the trade-off between coverage and set size, and coverage rises smoothly with set size (Table R2, Fig. K.2).
  5. W2d (user study): In a user study with 80 neurons, raters judge 76% of BE words relevant against 10% of random words, and BE's most frequent word is judged relevant more often than the others (77% vs. 58%).

We believe that we have addressed the reviewer's concerns. Please let us know if there are additional questions and we would be happy to discuss further.

Dear reviewer GjCT,

As the discussion period is going to the end, we would like to send a kind reminder to ensure we address all your concerns. If there are any remaining concerns, please let us know and we will be happy to address them.

Reviewer_Udu6OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates neuron identification — picking the concept maximizing similarity to a neuron's activations on a probing set — as ERM over the concept set, unifying methods like Network Dissection and CLIP-Dissect. From this view it derives PAC-style fidelity bounds via union bound over finite concept sets, instantiating convergence rates for accuracy, IoU, etc, and showing AUROC/recall degradation for rare concepts. The paper then proposes Bootstrap Explanation (BE), which bootstrap-resamples the probing set K times and returns frequent winners as a prediction set, with a binomial coverage guarantee for an idealized independent-trials variant. Simulations and Broden experiments on ResNet-50 show bounds track O(1/√n) behavior and BE improves coverage over single-concept assignment, especially with scarce probes.

Critical strengths
  • The paper shows a clean conceptual reframing that makes probing data as the training set and concepts as the hypothesis class — a useful lens for reasoning about reliability.
  • Moving from point estimates to prediction sets with an explicit coverage statement is the right direction for uncertainty-aware interpretability.
Critical weaknesses
  • Key realities of the pipeline sit outside the theory: concept labels c(x) are often model-estimated (CLIP/SigLIP) rather than ground truth, and thresholds τf, τc plus spatial pooling are fixed beforehand. How do label noise and threshold choice affect the fidelity bounds in practice?

  • For the 20k-word open vocabularies CLIP-Dissect actually uses, the log|C| penalty makes the bound quite loose. What probe size would the bound require for a non-vacuous guarantee there, and does that match the empirical regime in Fig.3?

  • Empirical scope is narrow relative to the generality claimed: only ResNet-50 layer-4 with CLIP-Dissect/NetDissect on Broden, despite claims covering SAE features, TCAV directions, and linear probes. It would be better to also demonstrate LLM features which would help assess broader utility.

Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer Udu6 (1/2)

We thank the reviewer taking the time to review and we are glad to hear that the reviewer recognize our ERM reframing is a useful lens for reasoning about reliability, and that moving from point estimates to prediction sets with explicit coverage is the right direction for uncertainty-aware interpretability! We address your concerns below.


Weakness 1: "Key realities of the pipeline sit outside the theory: concept labels c(x) are often model-estimated (CLIP/SigLIP) rather than ground truth, and thresholds τf, τc plus spatial pooling are fixed beforehand. How do label noise and threshold choice affect the fidelity bounds in practice?"

A1: To clarify, "Thresholds and pooling do not affect the theoretical guarantee". The thresholds τ_f, τ_c and the spatial pooling are part of how the neuron and concept functions are defined. Once they are fixed, the binarized functions f̃(x) = 1[f(x) > τ_f] and c̃(x) = 1[c(x) > τ_c] are fixed functions of x, and the probing samples remain i.i.d. Theorem 3.2 and Corollary 3.3 therefore hold unchanged for every choice of threshold and pooling.

For label noise, our results still hold for the observed label c′: sup over c of |ŝ(f, c′; D) − s(f, c′)| ≤ r(f, D, δ/|C|). The bound w.r.t. the true label introduces a new term depending on the magnitude of label noise (denoted as |s(f, c′) - s(f, c)|).

We support this empirically with a new experiment (App. K.3, Tables K.1 and K.2) on Broden (ResNet-50 layer 4, 2,048 neurons, 764 concepts with ground-truth image-level labels, 31,652 probe and 31,653 held-out images). We vary one variable at a time from the default setting and measure how often BE and single-concept assignment cover the held-out optimal concept (the held-out top-1 concept, as a proxy for the population optimum). The label-noise rows use the controlled synthetic setting of Fig. 3(c, d), flipping every concept's labels at rate η and measuring coverage of the clean c*:

SettingBE coverage (acc. / IoU)Single-concept coverage (acc. / IoU)BE mean |S| (acc. / IoU)
Default: τ_f = 95th pct., avg pooling, τ_c = label present0.81 / 0.940.48 / 0.736.7 / 2.1
Vary τ_f
80th pct.0.91 / 0.930.68 / 0.752.9 / 1.9
90th pct.0.87 / 0.940.58 / 0.764.2 / 1.9
99th pct.0.49 / 0.890.25 / 0.6221.2 / 3.2
99.5th pct.0.30 / 0.860.14 / 0.5827.2 / 4.4
Vary pooling
Avg from 7×7 maps0.81 / 0.940.50 / 0.746.5 / 2.1
Max from 7×7 maps0.69 / 0.930.37 / 0.7011.2 / 2.3
Vary τ_c
Mask area > 1% (697 concepts)0.82 / 0.950.49 / 0.776.0 / 1.8
Mask area > 5% (537 concepts)0.82 / 0.960.51 / 0.795.8 / 1.8
Mask area > 10% (445 concepts)0.84 / 0.960.53 / 0.785.5 / 1.8
Vary symmetric label noise (simulation, accuracy; n = 1,500 / 6,000)
No noise1.00 / 1.000.99 / 1.009.4 / 1.0
η = 0.10.96 / 1.000.72 / 1.0031.7 / 1.5
η = 0.20.73 / 1.000.27 / 0.9740.3 / 15.9
η = 0.30.39 / 0.890.06 / 0.5042.9 / 36.9

Table R1: Coverage of the held-out optimal concept by BE (t = 0.95K) and by single-concept assignment under different thresholds, pooling, and simulated label noise, with BE's mean prediction-set size.

BE improves coverage over single-concept assignment in every setting, so its benefit does not depend on a particular threshold or pooling choice and is robust to label noise.


Weakness 2: "For the 20k-word open vocabularies CLIP-Dissect actually uses, the log|C| penalty makes the bound quite loose. What probe size would the bound require for a non-vacuous guarantee there, and does that match the empirical regime in Fig.3?"

A2: We thank the reviewer for the question. To clarify, the impact of |C| is minimal as it grows only logarithmically. Concretely, using the example in Sec. 3.2 (accuracy, 1 − δ = 95%, ε = 0.05), Corollary 3.3 requires |D| ≥ log(2|C|/δ) / (2ε²) probe samples. For different concept-set sizes:

|C|Required |D|Relative to |C| = 1,000
1001,6590.78×
1,000 (example in Sec. 3.2)2,1191.00×
1,198 (Broden)2,1551.02×
20,000 (CLIP-Dissect vocabulary)2,7181.28×
100,0003,0401.43×
1,000,0003,5011.65×

Table R2: Probe size required by Corollary 3.3 for accuracy at 1 − δ = 95% and ε = 0.05.

Growing the vocabulary from 1,000 to 20,000 concepts raises the required probe size by only 28%, to about 2,700 images.

AuthorsOpenReview ↗

Rebuttal to Reviewer Udu6 (2/2)

Weakness 3: "Empirical scope is narrow relative to the generality claimed: only ResNet-50 layer-4 with CLIP-Dissect/NetDissect on Broden, despite claims covering SAE features, TCAV directions, and linear probes. It would be better to also demonstrate LLM features which would help assess broader utility."

A3: Thank you for the suggestion. To address your concern, we have extended the quantitative evaluation of BE beyond ResNet-50 layer 4 on Broden, changing one component at a time: the layer (ResNet-50 layer 3), the backbone (ViT-B/16 and ConvNeXt-base), and the probing dataset (the ImageNet validation set). All other settings follow our Broden evaluation (CLIP-Dissect with the 20,000-word concept set, the held-out top-1 concept as the reference, K = 100, t = 95). The results are added to App. K.4 (Table K.3) of the revised paper:

ModelLayerProbe setsoft-WPMI: BE / single (mean |S|)Cosine: BE / single (mean |S|)
ResNet-50layer 4Broden0.81 / 0.50 (4.9)0.96 / 0.83 (1.6)
ResNet-50layer 3Broden0.78 / 0.46 (6.0)0.95 / 0.80 (1.7)
ViT-B/16last (CLS)Broden0.64 / 0.33 (10.8)0.97 / 0.89 (1.4)
ConvNeXt-baselastBroden0.67 / 0.32 (9.3)0.98 / 0.88 (1.4)
ResNet-50layer 4ImageNet val0.83 / 0.52 (4.1)0.96 / 0.77 (1.9)

Table R3: Coverage of the held-out top-1 concept by BE and by single-concept assignment, with BE's mean prediction-set size, across models, layers, and probing datasets.

BE improves coverage over single-concept assignment in every setting, by 30 to 35 points under soft-WPMI and by 8 to 19 points under cosine similarity.

In addition, following your suggestion on the LLM features, we also apply BE to the 16k-width Gemma Scope SAE [1] on the layer-12 residual stream of Gemma-2-2B [2]. The latents are labeled with 660 token-level concepts (part of speech, dependency relation, morphological features, and lemma) from the Universal Dependencies English Web Treebank [3], using token-level IoU and resampling whole documents (App. K.7, Table K.6):

LatentsNumberSingleBEBE mean |S|
All13,1200.5450.7483.88
Best probe IoU ≥ 0.044,6980.8480.9421.56

Table R4: Coverage of the held-out top-1 concept by BE and by single-concept assignment for Gemma Scope latents of Gemma-2-2B.

BE also improves coverage on LLM features, by 9 points on latents with a meaningful concept match (Latent with best IoU ≥ 0.04) and by 20 points over all latents.


Summary

  1. W1 (label noise and threshold choice): Our guarantees hold for any fixed thresholds and pooling and for the observed labels, with an extra label-noise term with respect to the true labels; empirically, BE outperforms single-concept assignment under every threshold, pooling, and symmetric label-noise setting we tested.
  2. W2 (log|C| looseness for 20k vocabularies): The required probe size grows only logarithmically with |C|: going from 1,000 to 20,000 concepts raises it by 28%, to about 2,700 images.
  3. W3 (empirical scope, LLM features): New results on ResNet-50 layer 3, ViT-B/16, ConvNeXt-base, and the ImageNet validation set (App. K.4), and on Gemma Scope SAE latents of Gemma-2-2B (App. K.7) show that BE improves coverage over single-concept assignment in every setting, by 8 to 35 points.

We believe that we have addressed your concerns. Please let us know if there are additional questions and we would be happy to discuss further.


References

[1] T. Lieberum, S. Rajamanoharan, A. Conmy, L. Smith, N. Sonnerat, V. Varma, J. Kramár, A. Dragan, R. Shah, and N. Nanda. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2. BlackboxNLP Workshop, 2024.

[2] Gemma Team. Gemma 2: Improving Open Language Models at a Practical Size. arXiv:2408.00118, 2024.

[3] N. Silveira, T. Dozat, M.-C. de Marneffe, S. R. Bowman, M. Connor, J. Bauer, and C. D. Manning. A Gold Standard Dependency Corpus for English. LREC, 2014.

Dear reviewer Udu6,

As the discussion period is going to the end, we would like to send a kind reminder to ensure we address all your concerns. If there are any remaining concerns, please let us know and we will be happy to address them.

Thanks for the authors' clarification! The added experimental results and explanation further enhanced the manuscript, I'll keep my rating.

Comments on the paper

1 comment
Area_Chair_8MQPOpenReview ↗

Reviewer's responses to authors

Dear reviewers,

If not done already, could you please take a look and reply briefly to the "Official Comment(s) by Authors" / Rebuttals, possibly by an acknowledgement that you have seen it.

Feel free to respond in more detail, in particular, if you have further questions or opinions to the replies from the authors.

With best regards, an AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.