acceptodds
Under review as a conference paper at ICLR 2027

Coarse-to-Fine Neuron-Path Discovery for Interpretable Vision Transformers

Abstract

Understanding how Vision Transformers (ViTs) compute their predictions remains challenging: most explanations highlight salient patches but do not reveal cross-layer internal information flow. Neuron-path discovery addresses this by tracing a prediction to a sequence of influential FFN neurons across layers, yet existing methods are hard to interpret (index-only neuron IDs) and expensive due to per-layer search over thousands of neurons. We present Concept-Guided Coarse-to-Fine Neuron-Path Discovery, a framework that injects human-interpretable visual concepts into neuron-path search while preserving neuron-level grounding. We first construct a global concept bank from attention-selected patch embeddings and partition FFN neurons into concept-aligned groups shared across layers. We then perform coarse-to-fine path discovery: (i) search for an influential concept-level path in a low-dimensional group-gated attribution space, and (ii) refine each selected concept group to a representative neuron to obtain a final neuron path. This design reduces the search space without sacrificing causal relevance. Experiments on ImageNet with multiple ViT variants show that our method finds neuron paths that are more causally faithful under interventions than prior neuron-path baselines, while achieving 5× practical speedups. Moreover, the resulting concept paths expose coherent semantic trajectories across depth, enabling inspection of how visual evidence emerges and where depth-localized bottlenecks arise.

Primary area:
interpretability and explainable AI
Keywords:
Vision Transformer, Interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uAqzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The paper proposes Concept-Guided Coarse-to-Fine Neuron-Path Discovery, which injects human-interpretable visual concepts into neuron-path search while preserving neuron-level grounding. The method constructs a global concept bank from attention-selected patch embeddings, partitions FFN neurons into concept-aligned groups, searches for an influential concept-level path in a group-gated attribution space, and then refines each selected group to a representative neuron.

Critical strengths
  • The research question is interesting: explaining ViT predictions through cross-layer internal information flow is important.
  • The goal of connecting neuron-level paths with human-interpretable concepts is potentially useful.
Critical weaknesses
  • The paper is hard to follow. As an explainable AI paper, the method itself is not well explained. There are many math notations and definitions. Although the paper provides a notation table, it is still hard to understand the full pipeline. Some key details, such as how the concept bank is built and how concept-aligned groups are found, are mostly in the appendix. This makes the reader jump between the main paper and appendix many times.

  • The framework includes concept-bank construction, attention-based patch selection, activation-signature clustering, concept alignment, group-gated attribution, local/prefix-conditioned GG-JAS, greedy search, and neuron refinement. However, the paper does not clearly explain why each part is needed. Many design choices feel based on intuition. Although the paper has some sensitivity studies, it does not clearly test the importance of each main component. Thus, it is hard to know which parts really lead to the reported improvement.

  • The paper says these are semantic trajectories, but it does not clearly show how useful or reliable they are in practice. It is not clear what a model user should learn from outputs like blurred green background, long dog fur, or pointed animal ear across layers. There is no user study or systematic evaluation showing that humans can better understand, trust or use these explanations compared with existing methods.

  • The default method choice is confusing. The paper introduces prefix-conditioned scoring to capture cross-layer interactions. However, the prefix-free local version works better and is used as the default.

  • RAD/EAD measure accuracy changes after zeroing or doubling selected units. But this does not necessarily mean the discovered path is the true computation path of the model. In particular, doubling activations may create unnatural model states, so it is unclear how much the enhancement results support the claimed faithfulness.

Questions for authors

See weaknesses.

7 comments
AuthorsOpenReview ↗

Reply to Reviewer uAqz

We thank Reviewer uAqz for the constructive comments and suggestions. We have revised the manuscript to improve its clarity and conducted additional experiments to address the concerns.

W1: "The paper is hard to follow..."

We thank the reviewer for this feedback. We agree that the presentation could be clearer. We have redrawn several figures, particularly Figure 2 (method overview), and revised the manuscript to better explain the methodology and experimental results.

The revised overview more clearly separates the three stages of our framework: (1) offline concept-group construction, (2) coarse group scoring using GG-JAS, and (3) fine-grained neuron refinement. We have also clarified the corresponding mathematical definitions and implementation details.

W2: "...However, the paper does not clearly explain why each part is needed...."

We thank the reviewer for highlighting the need to validate the contribution of each component.

Additional component ablation. We conducted a systematic ablation study on ViT-L/32 and ViT-B/16, using 1,000 ImageNet validation images and three seeds per random control. We replaced each component with an alternative at its respective stage and recomputed all downstream selections.

The table reports the change in RAD (percentage points) relative to the full method, which achieves 7.8% on ViT-L/32 and 4.9% on ViT-B/16.

StageComponent replacementViT-L/32 ΔRADViT-B/16 ΔRAD
CoarseGG-JAS → Random group−7.9−4.8
GG-JAS → Activation ranking−8.3−4.4
Probability → Logit objective−1.8−1.6
PF → PC scoring−1.5PF > PC (Table 1)
FineMost-active → Random neuron−6.5−4.0
Most-active → JAS-based choice−4.4—
OfflineActivation clustering → Random groups−0.2—
Attention-based → Uniform patches−0.3—

The results clarify the contributions of the components:

1. Group scoring and neuron refinement drive intervention-based faithfulness. Replacing GG-JAS with random group selection reduces RAD by 7.9 / 4.8 percentage points (ViT-L/32 / ViT-B/16), while random neuron refinement reduces it by 6.5 / 4.0 points. These substantial drops support the importance of both stages. Replacing the probability objective with logits or the most-active neuron criterion with JAS-based refinement also reduces performance.

2. Activation-based grouping improves semantic interpretability. Replacing activation clustering with size-matched random groups changes RAD by only −0.2 points (not significant), but substantially reduces semantic quality. Specifically:

  • Within-group semantic coherence decreases from 0.215 to 0.037. We measure coherence by assigning each neuron's 25 most-activating patches to their nearest concepts and computing the mean pairwise similarity of concept distributions within groups, using normalized Jensen–Shannon divergence.
  • Image-specific grounding decreases from 35.8% to 33.1% under GPT-6 Luna and from 25.3% to 22.9% under Claude Haiku 5.5, corresponding to drops of 2.7 and 2.4 percentage points, respectively. Grounding measures the fraction of path concepts explicitly supported by the input image, evaluated on 1,000 ImageNet images.

3. Other design choices improve performance or efficiency. PF outperforms PC in intervention-based evaluation, supporting its selection as the default. Uniform patch selection has a limited effect on RAD (−0.3 points), indicating that attention-based patch selection primarily serves as a construction heuristic. Additional sensitivity analyses across the concept-bank size, patch budget, and selected layers are provided in Appendix I.4.

Together, these ablations show that coarse-to-fine attribution is the primary contributor to intervention-based faithfulness, while activation-based grouping and concept alignment support semantic interpretability. We will incorporate these results and clarify the role of each component in the revised manuscript.

W3: "The paper says these are semantic trajectories, but it does not clearly show how useful or reliable they are in practice."

We thank the reviewer for highlighting the need for stronger evidence of semantic reliability and practical interpretability.

Interpretation of concept paths. Our method associates selected neurons with concept labels and measures their influence through layer-wise interventions. This allows users to examine which visual cues are associated with a prediction and identify the layers where the corresponding neurons have substantial causal effects.

For example, in the Border Collie case, concepts such as blurred green background (L17), long dog fur (L21), and pointed animal ear (L23) highlight contextual and object-related cues. These observations can help identify potentially relevant evidence, including possible background reliance, for further investigation. However, the semantic validity of such labels requires independent evaluation.

Additional semantic evaluation. To systematically assess semantic reliability, we conducted two complementary evaluations on 1,000 correctly classified ImageNet validation images, using GPT-6 Luna and Claude Haiku 5.5 as independent VLM judges.

For each image, we constructed a concept path from the five layers with the largest single-neuron removal effects, preserving their layer order. Random control paths were sampled from the same concept bank at matching layer positions, excluding selected groups, across three random seeds.

We evaluated two aspects:

  1. Semantic explanation preference. Given the image, predicted class, and two concept paths, each judge selects the more informative, image-supported explanation, or responds tie or neither. We randomized A/B order and concealed method identities and causal-effect scores. Win rates are computed over decisive comparisons, excluding ties and neither responses.
  2. Image-specific grounding and class relevance. In separate evaluations, each judge assesses whether each concept is visually supported by the image (yes/no/uncertain) and relevant to the predicted class (yes/no). We report Grounding Rate, the fraction explicitly supported by the image, and Grounded Relevance Rate, the fraction both visually supported and class-relevant.

The results are summarized below.

JudgeMethodWin RateGrounding RateGrounded Relevance
GPT-6 LunaRandom path—23.23%12.18%
Discovered path76.22%35.68%21.30%
Claude Haiku 5.5Random path—13.28%10.28%
Discovered path76.93%26.02%20.84%

Both judges prefer the discovered paths in 76.22% and 76.93% of decisive pairwise comparisons, respectively. Compared with random paths, grounding improves by 12.45 and 12.74 percentage points, while grounded relevance improves by 9.12 and 10.56 points.

These results provide systematic evidence that the discovered paths convey more image-grounded and class-relevant information than random concept trajectories. While VLM-based evaluation does not replace a human user study, it offers quantitative evidence of semantic reliability beyond qualitative examples.

W4: "The default method choice is confusing."

We thank the reviewer for raising this concern regarding the choice between prefix-free (PF) and prefix-conditioned (PC) scoring.

We introduce PC to investigate prefix-dependent cross-layer interactions, where candidate groups are evaluated conditional on previously selected groups. In contrast, PF evaluates candidate groups independently at each layer. Under PF, greedy selection reduces to choosing the highest-scoring group at each layer.

Although PC explicitly incorporates the selected prefix, its greedy search also makes subsequent scores depend on earlier selections. This can introduce sensitivity to previous decisions and does not necessarily improve the final intervention effects. PF avoids this dependence and provides a simpler selection procedure.

Empirically, PF consistently achieves stronger intervention effects across all three ViT backbones (Table 1). In our additional component ablation on ViT-L/32, replacing PF with PC reduces RAD by 1.5 percentage points under otherwise matched settings.

We therefore adopt PF as the default based on its empirical performance, while retaining PC as an alternative for studying prefix-conditioned group selection. We will clarify this motivation and the distinction between the two strategies in the revised manuscript.

W5: "RAD/EAD measure accuracy changes after zeroing or doubling selected units. ...it is unclear how much the enhancement results support the claimed faithfulness."

We thank the reviewer for this important observation regarding the interpretation of intervention-based faithfulness.

Following NeuronPath (Wang et al., 2025) [1], we adopt RAD and EAD as intervention-based metrics for assessing the influence of selected neurons, maintaining consistency with prior work.

We agree that these metrics do not establish the complete or unique computational path of the model. Our objective is to identify compact sets of causally influential neurons across layers, rather than recover the model's full computational circuits.

Additional removal-based evidence. To strengthen the evaluation, we examine both RAD and target-logit margin changes under neuron removal. Our component ablations show that replacing GG-JAS with random group selection reduces RAD by 7.9 / 4.8 percentage points on ViT-L/32 / ViT-B/16, while random neuron refinement reduces it by 6.5 / 4.0 points.

We also evaluate the selected groups before neuron refinement. Removing these groups at every layer decreases classification accuracy by 11.7 / 6.4 percentage points more than removing the same number of randomly selected neurons on ViT-L/32 / ViT-B/16, respectively. This provides additional evidence that the coarse selection identifies influential neuron groups.

We acknowledge that doubling activations may induce unnatural internal states. Therefore, we treat removal-based evidence (RAD and target-logit margin drop) as the primary support for causal influence, with EAD serving as supplementary evidence.

These intervention results support the causal importance of the selected neurons under the evaluated perturbations, without establishing a complete computational circuit or the unique information-flow path. We will clarify this distinction and refine the corresponding claims in the revised manuscript.

Reference

[1] Wang, Y., et al. Discovering Influential Neuron Path in Vision Transformers. ICLR, 2025. https://arxiv.org/abs/2503.09046

We thank the reviewer again for the constructive feedback, which has helped us improve the presentation, component validation, and interpretation of our results.

Reviewer_jKo9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a coarse to fine neuron path discovery for ViT interpretability mechanism. More specifically, the authors bridges NeuroPath (which identifies single influential neuron within a layer) and concept based methods (which learns a set of concepts to explain the model), aiming to improve both low-level faithfulness and high-level interpretability, by first grouping activation-aligned neurons within a layer and search for the most influential group at each layer (coarse), then selecting the single neuron within that group (fine), finally visualizing the selected group and neuron path with the representative patches from the learned concepts. 3 ViT backbones are evaluated with the proposed method for faithfulness, efficiency and semantic interpretability. The experiments show that the proposed methods outperforms the baseline methods in faithfulness (measured by RAD and EAD) and efficiency (measured by wall clock and memory usage). Qualitative results for semantic interpretability is selectively shown in the appendix.

Critical strengths

S1: The writing is easy to follow, with clear definition of the proposed methods, the procedure, and evaluation methods.

S2: Strong evidence of improvements over baselines in terms of faithfulness and efficiency.

Critical weaknesses

W1: Arguments on semantic interpretability of the proposed method is insufficiently supported. The paper emphasizes concept-guided discovery and interpretable semantic path, but it is unclear how interpretable the results are. The main supporting evidence is Figure 4, where the authors claim "same-class semantic alignment across layers". First of all the text in that figure is barely readable. After zooming in, figure 4(a) does not seem to show a clear pattern of agreement in these classes, since the agreement score are generally below 0.6 except for one peak (in sports car). Another related concern is that within-class concept-selection consistency does not necessarily mean semantic correctness or information propagation.

W2: It is not very clear to me how much effect does the offline computed concept bank have on the overall results. The authors acknowledged that the proposed method depends on the quality of the learned concept bank. How much influence does it have? The answer to that will significantly affect the robustness of the proposed framework and hence its significance in contribution.

Questions for authors

Q1: the authors state that their method preserves attribution fidelity, how is fidelity defined in this work and is it measured?

Q2: In table 1 the PF+argmax variant consistently outperforms PC+argmax, can the authors elaborate a bit more on why prefix-free selection performs better?

Q3: Can the authors clarify how the concept bank is learned, and what is the role of that concept bank in the process of discovery of the path? Can the authors provide more quantitative or qualitative results on how does the quality of the concept bank affect the overall performance of the proposed method?

Q4: Can the authors provide more explanation on the within class concept selection patterns presented in Figure 4? How should the result be interpreted, and what are some clear pattern that we can draw from those results?

8 comments
AuthorsOpenReview ↗

Reply to Reviewer jKo9

We thank Reviewer jKo9 for the constructive feedback and helpful suggestions. We have also redrawn several figures to improve readability and presentation quality. We address the specific concerns below.

W1: "Arguments on semantic interpretability of the proposed method is insufficiently supported." "Another related concern is that within-class concept-selection consistency does not necessarily mean semantic correctness or information propagation." & Q4: "Can the authors provide more explanation on the within class concept selection patterns presented in Figure 4?"

We thank the reviewer for highlighting the need for stronger semantic validation and clarification of Figure 4.

Clarification of Figure 4. Figure 4 measures within-class concept-selection consistency, rather than semantic correctness or causal information propagation. Specifically, the alignment score (A_y(l)) represents the frequency of the most frequently selected concept at layer (l) among images of class (y). It measures how consistently a concept is selected across images of the same class.

The results reveal layer-dependent concept recurrence. For example, the sports car class exhibits stronger concept concentration at certain depths, while other layers show more diverse selections. These patterns characterize the consistency of concept selection but do not independently establish semantic correctness. We have improved Figure 4's readability and will clarify this interpretation in the revised manuscript.

Additional semantic evaluation. To directly address the reviewer's concern, we conducted two complementary evaluations on 1,000 correctly classified ImageNet validation images, using GPT-6 Luna and Claude Haiku 5.5 as separate VLM judges.

For each image, we constructed a concept path from the five layers with the largest single-neuron removal effects, preserving their layer order. Random control paths were sampled from the same concept bank at matching layer positions, excluding the selected groups, across three random seeds.

We evaluated two aspects:

  1. Semantic explanation preference. Given the image, predicted class, and two concept paths, each judge selects the more informative, image-supported explanation, or responds tie or neither. We randomized the A/B order and concealed method identities and causal-effect scores. Win rates are calculated over decisive comparisons, excluding ties and neither responses.
  2. Image-specific grounding and class relevance. In separate evaluations, each judge assesses whether each concept is visually supported by the image (yes/no/uncertain) and relevant to the predicted class (yes/no). We report Grounding Rate (fraction explicitly supported by the image) and Grounded Relevance Rate (fraction both visually supported and class-relevant).

The results are summarized below.

JudgeMethodWin RateGrounding RateGrounded Relevance
GPT-6 LunaRandom path—23.23%12.18%
Discovered path76.22%35.68%21.30%
Claude Haiku 5.5Random path—13.28%10.28%
Discovered path76.93%26.02%20.84%

Both judges prefer the discovered paths in 76.22% and 76.93% of decisive pairwise comparisons, respectively. Compared with random paths, grounding improves by 12.45 and 12.74 percentage points, while grounded relevance improves by 9.12 and 10.56 points.

These results provide additional quantitative evidence that the discovered paths convey more image-supported and class-relevant information than random concept trajectories. This evaluation directly addresses semantic validity beyond the within-class selection consistency illustrated in Figure 4.

W2: "It is not very clear to me how much effect does the offline computed concept bank have on the overall results." & Q3: "Can the authors clarify how the concept bank is learned, and what is the role of that concept bank in the process of discovery of the path?..."

We thank the reviewer for requesting further clarification of the concept bank and its contribution to path discovery.

Concept bank construction. As detailed in Appendices C–D, we select the eight patches with the highest [CLS] attention from each ImageNet validation image, yielding approximately 400k patches. Their last-layer embeddings are clustered into K = 1,024 concept prototypes using cosine k-means.

For each transformer layer, FFN neurons are clustered based on their activation signatures across these patches. Each group's mean embedding of its 128 most-activating patches is then matched to the nearest concept prototype, and groups assigned to the same concept are merged.

Role in path discovery. The concept bank provides two components:

  • Candidate grouping: It organizes neurons into concept-aligned groups that serve as the search units for coarse group scoring (GG-JAS).
  • Semantic labeling: It assigns concept labels to the selected groups, allowing the resulting neuron paths to be interpreted semantically.

GG-JAS ranks candidate groups by their contribution to the prediction, and fine-grained refinement identifies influential neurons within the selected groups. Thus, the bank defines the candidate structure and semantic labels, while the attribution and refinement stages drive causal selection.

Additional component ablation. To quantify the contribution of each component, we conducted ablations on ViT-L/32 and ViT-B/16, using 1,000 ImageNet validation images and three seeds per random control. We replaced individual components with alternatives at their respective stages and recomputed all downstream selections.

The following table reports changes in RAD (percentage points) relative to the full method, which achieves 7.8% on ViT-L/32 and 4.9% on ViT-B/16.

ComponentAblationViT-L/32 ΔRADViT-B/16 ΔRAD
Coarse scoringRandom group−7.9−4.8
Activation ranking−8.3−4.4
Fine refinementRandom neuron−6.5−4.0
JAS-based choice−4.4—
Concept groupingSize-matched random groups−0.2—
Patch selectionUniform patches, same budget−0.3—

The ablations reveal two complementary findings.

1. Causal faithfulness is primarily driven by group scoring and neuron refinement. Replacing GG-JAS with random group selection reduces RAD by 7.9 percentage points on ViT-L/32, while random neuron refinement reduces it by 6.5 points. In contrast, replacing activation clustering with size-matched random groups changes RAD by only −0.2 points, and uniform patch sampling by −0.3 points. Additional sensitivity analyses across the concept-bank size, patch budget, and selected layers also show stable performance.

2. Activation-based concept grouping improves semantic interpretability. We evaluated two semantic properties under the same grouping ablation:

  • Within-group semantic coherence: For each neuron, we assigned its 25 most-activating patches to their nearest concepts, obtaining a concept-response distribution. Coherence measures the average pairwise similarity of these distributions within a group using normalized Jensen–Shannon divergence. Replacing activation clustering with random groups reduces coherence from 0.215 to 0.037.
  • Image-specific grounding: Using the VLM evaluation protocol described above, we measured the fraction of path concepts explicitly supported by the image. Random grouping reduces grounding from 35.8% to 33.1% under GPT-6 Luna and from 25.3% to 22.9% under Claude Haiku 5.5, corresponding to drops of 2.7 and 2.4 percentage points, respectively.

These findings indicate that activation-based grouping improves the semantic coherence and image grounding of discovered paths, despite its limited effect on the measured intervention-based faithfulness. The separate random-path comparison in W1 further shows that the discovered paths are substantially more image-grounded than randomly sampled concept trajectories.

Appendices J–K additionally provide representative concept labels with their top-activating patches and complete concept paths for qualitative inspection.

We thank the reviewer again for the suggestions, which have helped us strengthen the empirical evaluation and clarify the role of the concept bank.

Q1 " how is fidelity defined in this work and is it measured?"

By attribution fidelity, we refer to the extent to which discovered neuron paths retain causal influence on model predictions under controlled interventions. We evaluate this using Removal Accuracy Deviation (RAD) and Enhancement Accuracy Deviation (EAD), with logit-margin deviations providing additional evidence. As shown in Table 1, our coarse-to-fine method achieves stronger intervention effects than neuron-level baselines, indicating that the reduced search space does not compromise intervention-based faithfulness. We will clarify this definition and use the more precise term intervention-based causal faithfulness in the revised manuscript.

Q2 "...can the authors elaborate a bit more on why prefix-free selection performs better" Thank the reviewer for giving this comment. PF scores candidate groups independently at each layer, whereas PC conditions group selection on the previously selected prefix. Although PC is designed to capture cross-layer interactions, its scores may be affected by accumulated prefix interactions and errors from earlier greedy selections, potentially weakening the attribution signal. In contrast, PF avoids this dependency and may provide more stable group rankings.

Empirically, PF + argmax consistently achieves higher RAD than PC + argmax across ViT-B/16 (4.36% vs. 3.91%), ViT-B/32 (5.59% vs. 5.39%), and ViT-L/32 (8.24% vs. 6.72%). These results motivate our choice of PF as the default strategy. We will clarify this empirical rationale in the revised manuscript.

Reviewer_CYWXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a coarse-to-fine procedure for finding influential FFN neuron paths in pretrained vision transformers. It clusters neurons by their activation patterns, assigns the groups to a shared visual concept bank, and scores groups through gated interventions. Each selected group is then reduced to its most active neuron. Experiments on ImageNet with three ViT backbones report stronger removal and enhancement effects than the evaluated neuron-path baselines, together with lower search time.

Critical strengths

Grouping neurons speeds up the search, and the selected paths have a larger effect on predictions than the tested baselines.

Critical weaknesses
  1. The attribution setup is inconsistent. Sections 3-4 and Appendix E turn off only the selected groups, leaving the others active. Appendix H instead gives an all-zero baseline. These are different interventions. It is unclear which one produced the reported results.

  2. The need for IG is unclear. Under Appendix E's definition, Eqs. (24)-(25) make GG-JAS equal to the target-logit drop after deleting the selected groups. This requires only one deletion forward pass per candidate once the original output is cached. The paper instead uses repeated forward/backward passes without showing what this extra cost adds.

  3. The concept labels are not validated for the explained image. These labels may describe reference patches, but that does not establish their meaning for this image. The removal tests show that the selected neurons matter; they do not verify the assigned labels.

Questions for authors
  1. Which baseline and interpolation path were actually used for Tables 1-2?
  2. How does direct deletion scoring compare with IG in selected paths, intervention effects, and runtime?
  3. How was the link between each selected neuron and its concept label checked, especially for the electric-ray example in Figure 13?
6 comments
AuthorsOpenReview ↗

Reply to reviewer CYWX

We first thank Reviewer CYWX for the constructive feedback.

W1 & Q1 — Attribution Setup. W1"The attribution setup is inconsistent..."&“Q1"1. Which baseline and interpolation path were actually used for Tables 1-2?". We thank the reviewer for identifying an inconsistency in our attribution formulation. The implementation underlying Tables 1–2 uses shared-path Integrated Gradients (IG) with target-class probability as the attribution objective. We will correct the mathematical definitions and explicitly document the interpolation paths for both PF and PC.

W2 & Q2 — Empirical Benefits of Shared-Path IG. W2"The need for IG is unclear...."&"1. How does direct deletion scoring compare with IG in selected paths, intervention effects, and runtime?" We agree that, under a candidate-specific deletion baseline, the summed IG attribution equals the corresponding output difference by completeness. However, our implementation computes per-group attributions along a shared interpolation path, which is generally not equivalent to individual deletion scoring.

We therefore compare shared-path IG (T = 20, target probability) against direct deletion (target-logit drop) on 1,000 ImageNet validation images, keeping the concept bank, neuron grouping, and argmax refinement fixed.

MethodRAD ↑EAD ↑Removal Margin Drop ↑Relative Speed ↑
PF + IG7.80%7.30%1.0294.25×
PF + Deletion5.60%5.30%0.7641.00×
PC + IG6.30%5.90%0.8582.21×
PC + Deletion4.90%5.20%0.6441.00×

Shared-path IG consistently achieves stronger intervention effects and faster discovery under both PF and PC. Compared with direct deletion, PF + IG improves RAD by 2.20 percentage points and achieves a 4.25× speedup, while PC + IG improves RAD by 1.40 percentage points with a 2.21× speedup. Removal-based margin drops also improve under both strategies.

These findings support the effectiveness and computational efficiency of our implemented shared-path IG procedure relative to the evaluated direct-deletion baseline. We will clarify its distinction from candidate-specific deletion attribution and revise the corresponding methodological part.

AuthorsOpenReview ↗

Reply to reviewer CYWX

W3 "The concept labels are not validated for the explained image.... they do not verify the assigned labels." &Q3 "How was the link between each selected neuron and its concept label checked, especially for the electric-ray example in Figure 13?"

We thank the reviewer for highlighting the need to validate the semantic associations of selected neurons.

Neuron–concept alignment. Each selected neuron inherits the concept label of its activation-aligned group, assigned through nearest-prototype matching with the global concept bank. This provides a group-level semantic association, while removal interventions independently measure the neuron's causal influence.

For Figure 13 (electric ray), influential neurons at L11, L17, and L21 are associated with overhead power lines, guitar pick tip, and glowing digital numerals, respectively. These labels reflect global concept assignments, and the corresponding neurons have substantial removal effects. We acknowledge that causal influence does not, by itself, verify whether these labels accurately describe the particular image.

Additional semantic evaluation. To systematically evaluate image-specific validity, we conducted two complementary evaluations on 1,000 correctly classified ImageNet validation images, using GPT-6 Luna and Claude Haiku 5.5 as independent VLM judges.

For each image, we constructed a concept path using the five layers with the largest single-neuron removal effects, preserving their layer order. Random control paths were sampled from the same concept bank at matching layer positions, excluding the selected groups, across three random seeds.

  1. Semantic explanation preference. Given the image, predicted class, and two concept paths, each judge selects the more informative, image-supported explanation, or responds tie or neither. We randomize A/B order and conceal method identities and causal-effect scores. Win rates are calculated over decisive comparisons, excluding ties and neither responses.
  2. Image-specific grounding and class relevance. In separate evaluations, each judge assesses whether each concept is visually supported by the image (yes/no/uncertain) and relevant to the predicted class (yes/no). We report Grounding Rate (fraction explicitly supported by the image) and Grounded Relevance Rate (fraction both visually supported and class-relevant).

The results are summarized below.

JudgeMethodWin RateGrounding RateGrounded Relevance
GPT-6 LunaRandom path—23.23%12.18%
Discovered path76.22%35.68%21.30%
Claude Haiku 5.5Random path—13.28%10.28%
Discovered path76.93%26.02%20.84%

Both judges consistently prefer the discovered paths in blind pairwise comparisons, with win rates of 76.22% and 76.93%. Compared with random paths, grounding improves by 12.45 and 12.74 percentage points, while grounded relevance improves by 9.12 and 10.56 points, respectively.

We further evaluated whether activation-based grouping supports coherent neuron–concept associations. Replacing activation clustering with size-matched random groups reduces within-group semantic coherence from 0.215 to 0.037. This metric measures the similarity of concept-response distributions among neurons within each group, using their top-activating patches. The random-group control also reduces image grounding by 2.7 and 2.4 percentage points under GPT-6 Luna and Claude Haiku 5.5, respectively.

Together, these results provide quantitative evidence that the discovered concept paths are more image-grounded and class-relevant than random trajectories, and that activation-based grouping improves semantic coherence. We will incorporate these evaluations and clarify the distinction between causal influence and semantic validity in the revised manuscript.

We thank the reviewer again for the suggestions, which have helped us improve the methodological clarity and empirical validation of our work.

AuthorsOpenReview ↗

Follow-up on Rebuttal and Revisions

Dear Reviewer CYWX,

Thank you for your valuable feedback.

We believe our responses and revisions address your concerns. We welcome any further discussion before Nov 18 AOE and would greatly appreciate your reconsideration of our score.

Thank you again for helping us improve our paper!

Best regards,
Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Remarks

Author Remarks

We sincerely thank all reviewers for their time and constructive feedback. We appreciate that the reviewers recognize:

  • The importance of the research problem, describing the research question as "interesting" [uAqz] and explaining ViT predictions through cross-layer information flow as "important" [uAqz].
  • The potential value of our approach, highlighting that "Grouping neurons speeds up the search" [CYWX] and that connecting neuron-level paths with human-interpretable concepts is "potentially useful" [uAqz].
  • The empirical improvements, noting "strong evidence of improvements over baselines in terms of faithfulness and efficiency" [jKo9] and that the selected paths have "a larger effect on predictions" [CYWX].

To address the reviewers' concerns, we conducted additional experiments and provided detailed clarifications:

  • Semantic interpretability [CYWX, jKo9, uAqz]: Conducted a systematic evaluation on 1,000 ImageNet images using two independent VLM judges (GPT-6 Luna and Claude Haiku 5.5). Compared with random concept paths from the same bank, our discovered paths are preferred in 76.22% and 76.93% of decisive comparisons. They also achieve higher image grounding (+12.45/+12.74 percentage points) and grounded class relevance (+9.12/+10.56 points), providing additional quantitative evidence for their semantic interpretability (General Response 1).
  • Component ablations [jKo9, uAqz]: Conducted comprehensive ablations on two ViT backbones and 1,000 ImageNet images, isolating the contributions of individual components. The results show that group-gated attribution and neuron refinement are critical for intervention-based faithfulness, while activation clustering and concept alignment primarily improve semantic coherence and image grounding (General Response 2).
  • Attribution formulation and efficiency [CYWX]: Clarified the implemented shared-path Integrated Gradients (IG) formulation and compared it with direct deletion scoring. On ViT-L/32, PF + IG improves RAD from 5.60% to 7.80% while achieving a 4.25× speedup, supporting both its effectiveness and computational efficiency.
  • Methodological clarity and evaluation scope [jKo9, uAqz]: Improved the presentation of the method pipeline and figures, clarified the empirical motivation for prefix-free (PF) scoring, and refined our definition of intervention-based causal faithfulness. We also clarified that our method identifies compact, causally influential neuron paths without claiming to recover complete computational circuits.

We will incorporate these additional experiments, analyses, and clarifications into the revised manuscript, together with improved figures and methodological descriptions.

We again thank the reviewers for their valuable feedback and the AC and SAC for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.