- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes Concept-Guided Coarse-to-Fine Neuron-Path Discovery, which injects human-interpretable visual concepts into neuron-path search while preserving neuron-level grounding. The method constructs a global concept bank from attention-selected patch embeddings, partitions FFN neurons into concept-aligned groups, searches for an influential concept-level path in a group-gated attribution space, and then refines each selected group to a representative neuron.
Critical strengths
- The research question is interesting: explaining ViT predictions through cross-layer internal information flow is important.
- The goal of connecting neuron-level paths with human-interpretable concepts is potentially useful.
Critical weaknesses
-
The paper is hard to follow. As an explainable AI paper, the method itself is not well explained. There are many math notations and definitions. Although the paper provides a notation table, it is still hard to understand the full pipeline. Some key details, such as how the concept bank is built and how concept-aligned groups are found, are mostly in the appendix. This makes the reader jump between the main paper and appendix many times.
-
The framework includes concept-bank construction, attention-based patch selection, activation-signature clustering, concept alignment, group-gated attribution, local/prefix-conditioned GG-JAS, greedy search, and neuron refinement. However, the paper does not clearly explain why each part is needed. Many design choices feel based on intuition. Although the paper has some sensitivity studies, it does not clearly test the importance of each main component. Thus, it is hard to know which parts really lead to the reported improvement.
-
The paper says these are semantic trajectories, but it does not clearly show how useful or reliable they are in practice. It is not clear what a model user should learn from outputs like blurred green background, long dog fur, or pointed animal ear across layers. There is no user study or systematic evaluation showing that humans can better understand, trust or use these explanations compared with existing methods.
-
The default method choice is confusing. The paper introduces prefix-conditioned scoring to capture cross-layer interactions. However, the prefix-free local version works better and is used as the default.
-
RAD/EAD measure accuracy changes after zeroing or doubling selected units. But this does not necessarily mean the discovered path is the true computation path of the model. In particular, doubling activations may create unnatural model states, so it is unclear how much the enhancement results support the claimed faithfulness.
Questions for authors
See weaknesses.
7 comments
W3: "The paper says these are semantic trajectories, but it does not clearly show how useful or reliable they are in practice."
We thank the reviewer for highlighting the need for stronger evidence of semantic reliability and practical interpretability.
Interpretation of concept paths. Our method associates selected neurons with concept labels and measures their influence through layer-wise interventions. This allows users to examine which visual cues are associated with a prediction and identify the layers where the corresponding neurons have substantial causal effects.
For example, in the Border Collie case, concepts such as blurred green background (L17), long dog fur (L21), and pointed animal ear (L23) highlight contextual and object-related cues. These observations can help identify potentially relevant evidence, including possible background reliance, for further investigation. However, the semantic validity of such labels requires independent evaluation.
Additional semantic evaluation. To systematically assess semantic reliability, we conducted two complementary evaluations on 1,000 correctly classified ImageNet validation images, using GPT-6 Luna and Claude Haiku 5.5 as independent VLM judges.
For each image, we constructed a concept path from the five layers with the largest single-neuron removal effects, preserving their layer order. Random control paths were sampled from the same concept bank at matching layer positions, excluding selected groups, across three random seeds.
We evaluated two aspects:
- Semantic explanation preference. Given the image, predicted class, and two concept paths, each judge selects the more informative, image-supported explanation, or responds tie or neither. We randomized A/B order and concealed method identities and causal-effect scores. Win rates are computed over decisive comparisons, excluding ties and neither responses.
- Image-specific grounding and class relevance. In separate evaluations, each judge assesses whether each concept is visually supported by the image (yes/no/uncertain) and relevant to the predicted class (yes/no). We report Grounding Rate, the fraction explicitly supported by the image, and Grounded Relevance Rate, the fraction both visually supported and class-relevant.
The results are summarized below.
| Judge | Method | Win Rate | Grounding Rate | Grounded Relevance |
|---|---|---|---|---|
| GPT-6 Luna | Random path | — | 23.23% | 12.18% |
| Discovered path | 76.22% | 35.68% | 21.30% | |
| Claude Haiku 5.5 | Random path | — | 13.28% | 10.28% |
| Discovered path | 76.93% | 26.02% | 20.84% |
Both judges prefer the discovered paths in 76.22% and 76.93% of decisive pairwise comparisons, respectively. Compared with random paths, grounding improves by 12.45 and 12.74 percentage points, while grounded relevance improves by 9.12 and 10.56 points.
These results provide systematic evidence that the discovered paths convey more image-grounded and class-relevant information than random concept trajectories. While VLM-based evaluation does not replace a human user study, it offers quantitative evidence of semantic reliability beyond qualitative examples.
W4: "The default method choice is confusing."
We thank the reviewer for raising this concern regarding the choice between prefix-free (PF) and prefix-conditioned (PC) scoring.
We introduce PC to investigate prefix-dependent cross-layer interactions, where candidate groups are evaluated conditional on previously selected groups. In contrast, PF evaluates candidate groups independently at each layer. Under PF, greedy selection reduces to choosing the highest-scoring group at each layer.
Although PC explicitly incorporates the selected prefix, its greedy search also makes subsequent scores depend on earlier selections. This can introduce sensitivity to previous decisions and does not necessarily improve the final intervention effects. PF avoids this dependence and provides a simpler selection procedure.
Empirically, PF consistently achieves stronger intervention effects across all three ViT backbones (Table 1). In our additional component ablation on ViT-L/32, replacing PF with PC reduces RAD by 1.5 percentage points under otherwise matched settings.
We therefore adopt PF as the default based on its empirical performance, while retaining PC as an alternative for studying prefix-conditioned group selection. We will clarify this motivation and the distinction between the two strategies in the revised manuscript.
W5: "RAD/EAD measure accuracy changes after zeroing or doubling selected units. ...it is unclear how much the enhancement results support the claimed faithfulness."
We thank the reviewer for this important observation regarding the interpretation of intervention-based faithfulness.
Following NeuronPath (Wang et al., 2025) [1], we adopt RAD and EAD as intervention-based metrics for assessing the influence of selected neurons, maintaining consistency with prior work.
We agree that these metrics do not establish the complete or unique computational path of the model. Our objective is to identify compact sets of causally influential neurons across layers, rather than recover the model's full computational circuits.
Additional removal-based evidence. To strengthen the evaluation, we examine both RAD and target-logit margin changes under neuron removal. Our component ablations show that replacing GG-JAS with random group selection reduces RAD by 7.9 / 4.8 percentage points on ViT-L/32 / ViT-B/16, while random neuron refinement reduces it by 6.5 / 4.0 points.
We also evaluate the selected groups before neuron refinement. Removing these groups at every layer decreases classification accuracy by 11.7 / 6.4 percentage points more than removing the same number of randomly selected neurons on ViT-L/32 / ViT-B/16, respectively. This provides additional evidence that the coarse selection identifies influential neuron groups.
We acknowledge that doubling activations may induce unnatural internal states. Therefore, we treat removal-based evidence (RAD and target-logit margin drop) as the primary support for causal influence, with EAD serving as supplementary evidence.
These intervention results support the causal importance of the selected neurons under the evaluated perturbations, without establishing a complete computational circuit or the unique information-flow path. We will clarify this distinction and refine the corresponding claims in the revised manuscript.
Reference
[1] Wang, Y., et al. Discovering Influential Neuron Path in Vision Transformers. ICLR, 2025. https://arxiv.org/abs/2503.09046
We thank the reviewer again for the constructive feedback, which has helped us improve the presentation, component validation, and interpretation of our results.
Reply to Reviewer uAqz
We thank Reviewer uAqz for the constructive comments and suggestions. We have revised the manuscript to improve its clarity and conducted additional experiments to address the concerns.
W1: "The paper is hard to follow..."
We thank the reviewer for this feedback. We agree that the presentation could be clearer. We have redrawn several figures, particularly Figure 2 (method overview), and revised the manuscript to better explain the methodology and experimental results.
The revised overview more clearly separates the three stages of our framework: (1) offline concept-group construction, (2) coarse group scoring using GG-JAS, and (3) fine-grained neuron refinement. We have also clarified the corresponding mathematical definitions and implementation details.
W2: "...However, the paper does not clearly explain why each part is needed...."
We thank the reviewer for highlighting the need to validate the contribution of each component.
Additional component ablation. We conducted a systematic ablation study on ViT-L/32 and ViT-B/16, using 1,000 ImageNet validation images and three seeds per random control. We replaced each component with an alternative at its respective stage and recomputed all downstream selections.
The table reports the change in RAD (percentage points) relative to the full method, which achieves 7.8% on ViT-L/32 and 4.9% on ViT-B/16.
The results clarify the contributions of the components:
1. Group scoring and neuron refinement drive intervention-based faithfulness. Replacing GG-JAS with random group selection reduces RAD by 7.9 / 4.8 percentage points (ViT-L/32 / ViT-B/16), while random neuron refinement reduces it by 6.5 / 4.0 points. These substantial drops support the importance of both stages. Replacing the probability objective with logits or the most-active neuron criterion with JAS-based refinement also reduces performance.
2. Activation-based grouping improves semantic interpretability. Replacing activation clustering with size-matched random groups changes RAD by only −0.2 points (not significant), but substantially reduces semantic quality. Specifically:
3. Other design choices improve performance or efficiency. PF outperforms PC in intervention-based evaluation, supporting its selection as the default. Uniform patch selection has a limited effect on RAD (−0.3 points), indicating that attention-based patch selection primarily serves as a construction heuristic. Additional sensitivity analyses across the concept-bank size, patch budget, and selected layers are provided in Appendix I.4.
Together, these ablations show that coarse-to-fine attribution is the primary contributor to intervention-based faithfulness, while activation-based grouping and concept alignment support semantic interpretability. We will incorporate these results and clarify the role of each component in the revised manuscript.