- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper introduces Complete Path Contribution, a framework for analyzing neural network computations at the level of individual input-to-output pathways. The motivation is neuron-level attribution methods may obscure important excitatory and inhibitory interactions when strong positive and negative pathway contributions cancel each other. The proposed method defines the contribution of an individual path by integrating the product of its local Jacobians along a trajectory from a baseline input to the actual input. By aggregating path contributions, the method recovers attribution measures, including Integrated Gradients and neuron conductance, while providing hierarchical completeness across different levels of network granularity. To address the combinatorial complexity of enumerating all possible paths, the authors develop a best-first branch-and-bound algorithm for identifying the top-k paths with the largest absolute contributions. The framework is evaluated on MNIST-trained MLPs, ViT-B, and a mouse retinal response prediction model. The experimental results show the effectiveness of the proposed method.
Critical strengths
-
A small contribution from a neuron does not necessarily imply that the neuron participates in weak computations. Instead, positive and negative pathway contributions may cancel each other. The distinction between weak contributions and strong but mutually canceling pathways is valuable.
-
The formulation of pathway, neuron, edge, and subnetwork contributions looks meaningful. The hierarchical completeness property guarantees that contributions from disjoint partitions of paths aggregate to the contribution of their parent subnetwork. This connects fine-grained pathway explanations to coarser component-level analyses.
-
The proposed best-first branch-and-bound algorithm addresses a computational challenge related to pathway-level analysis. By combining upper-bound estimates, selective expansion, and adaptive refinement, the method avoids exhaustive path enumeration.
-
The experiments cover multiple model architectures and demonstrate different applications of the method. The retinal model analysis provides an interesting connection between pathway contributions, excitatory-inhibitory interactions, and dynamic receptive fields.
Critical weaknesses
-
The novelty appears relatively limited. The authors acknowledge that the definition of path contribution is mathematically equivalent to the influence paths formulation. Hierarchical completeness follows directly from defining a subnetwork's contribution as the sum of its constituent path contributions and applying a partition of the corresponding path set. It would be great to compare with prior approaches in terms of computational efficiency and interpretability.
-
The conclusions rely on selected examples. For the MNIST experiment, the analysis focuses on a selected neuron. It was unclear how frequently similar cancellation phenomena occur across other neurons and model architectures. The interpretation of identified motifs as biological computational mechanisms should be distinguished from their role within the fitted predictive model. The ViT experiment analyzed 50 images from the pill bottle class and a particular pair of transformer blocks. The generality of these observations is unclear. After rebuttal, the generality concerns have been addressed.
-
It was unclear whether similar computational efficiency can be maintained for larger and more complex architectures.
Questions for authors
- What is the main theoretical novelty relative to existing influence-path methods?
- What is the meaning of causal contribution in the proposed method?
- How general are the observed excitatory-inhibitory cancellation patterns?
- What are the practical limitations of the proposed top-k search algorithm?
- How sensitive are the conclusions to baseline selection, numerical integration, and aggregation across inputs?
6 comments
Response to Reviewer b41H [2/2]
[W4, Q4] Regarding the top k algorithm, we acknowledge the limited settings we have tested it on. We address the limitations of the algorithm in a dedicated paragraph in Appendix D. In short, for large, modern networks, memory becomes the bottleneck (one layer-wise Jacobian for GPT 2 small costs 2.5 TB at float32) and thus scaling up to those networks requires modifications that trade off time for space, such as matrix-free Jacobians evaluated on the fly. We explain some initial efforts we have taken to mitigate this issue (sibling entries, adaptive refinement, and interval bound) and note that specialized efficient algorithms for specific architectures might be possible even though to our knowledge there is no guaranteed polynomial time algorithm for the general problem of finding top k paths in a DAG with sum-of-products path lengths (like Lu et al. 2021 acknowledged).
Reference: Lu, K., Wang, Z., Mardziel, P. & Datta, A. Influence Patterns for Explaining Information Flow in BERT. in (2021).
[Q2] The meaning of causal contribution comes from the fact that path contribution is a causal measure, just not motivated by intervention. As explained in Appendix G, the chained gradient is inherently the causal effect of a perturbation of the input propagating along that chain. Path contribution accumulates that infinitesimal effect along a straight line to obtain a cumulative causal effect that has the completeness property. The argument that integrated gradients is causal is not new. Geiger et al. 2021 argues that “IGi(x) is measuring the average “limiting” causal effect of increasing neuron i along the straight line from the baseline vector to the input vector x, weighted by the difference at i between input and baseline.”
Reference: Geiger, A., Lu, H., Icard, T. & Potts, C. Causal Abstractions of Neural Networks.
[Q5] We have updated the manuscript with new Figures 6, 7 and 8 to show that top-k path rankings and cancellation patterns remain highly consistent across alternative baselines and reasonable integration steps. Figure 6 shows that both the amount and pattern of cancellation remain stable on gray, white, and MNIST-mean baselines. Figure 7 shows that the overlap of the top k paths under an alternative baseline with those of the default, black baseline remains high (>0.8) for k=10, 100, and 1000 in the class mean contributions (Figure 3’s setting) and over individual images (the box plots). The reason we used top-k overlap for different k’s instead of a rank distance measure over all paths is the many small or exactly zero paths confound such measure. Figure 7 also shows that across different baselines, no path flips sign on the class mean, and almost no path flips sign on the vast majority of images. On the few images where paths flip, h1[13] is so weakly active that its activation lies between its values at the two baselines, so the two integrations run in opposite directions. Figure 8 shows when individual path contributions converge and when completeness error becomes negligible. We find that in the values we used for the main figures (64 for the MLPs, 32 for the retina, 512 for the ViT), all converge well.
I have read other reviews and the authors response. Thank the authors for addressing most of the concerns. I just have one more point regarding causal contribution. In this paper, does causal refer to casual propagation within the model's computational graph based on its structural equations? Or is it more related with intervention-based causal effect in conventional causal inference? What assumptions are required to interpret integrated path contribution as casual effect?
Response to Reviewer b41H [1/2]
Thank you for your insightful review and for acknowledging the novelty of our work.
[W1, Q1] We compare path contribution to influence paths and path patching in appendix G. Influence paths analyzes paths, but does not state or derive hierarchical completeness, which is the key property that enables a fair comparison between E and I values across granularity levels, as they have to sum to the same output difference. Path patching measures the intervention effect, which is in general a distribution over whether other paths are already patched, and path contribution tracks it closely and has guaranteed completeness that path patching does not have. The Shapley value is the mean of that distribution, and it coincides with path contribution whenever the treefied model (in path patching) is multilinear on the input copies, which the toy model satisfies even though it is nonlinear. Therefore, path contribution can be viewed as a representative statistic of that distribution with a completeness guarantee without having to compute the exponential number of contexts to get the Shapley value. Regarding circuit-attribution methods, we briefly explain in the introduction that they involve first finding the important units or edges in each layer, and combining them post hoc into a circuit. To our knowledge, the resulting circuits have no completeness guarantee, and a unit with a small net effect may have strongly excitatory and inhibitory paths through it, which are important components of a circuit but will be missed by a unit-first search.
In terms of computational efficiency, evaluating a single path’s contribution takes a backward pass for each integration step, which can be parallelized. This is on the same order of cost as path patching, while the Shapley value is exponentially more expensive to compute. Even though integration steps necessitate a constant multiplier that can be expensive, we note that the structure of the definition (sum of products) enables the design of the top k algorithm that path patching does not. We have shown in Figure 13 and Figure 3D,G that the top paths capture key network properties much more sparsely than neurons and edges, which enables path-level analyses for the top paths at scales unreachable for path patching.
[W2, W3, Q3] We show in Figure 14 and Figures 10 and 11 that the cancellation phenomena we observe holds across neurons, random seeds, and architectures. For ViT, we additionally analyzed three classes and three block pairs (early-middle, early-late, middle-late), and we actually found the E-I head-head motifs are quite similar at the same block pair for different classes, but quite different at different block pairs for the same class, suggesting shared computations across classes and serial progression across layers. We also find that paths through the heads of the middle block carry a large share of the output, while at early and late blocks most of the output bypasses the heads. We leave the thorough investigation of these findings to future work.
For the retina, across the full population of 36 recorded units, the motifs remain highly consistent: the primary principal component accounts for 44% of the overall variance, with a median cancellation factor of 6.2 (Figure 18). Units displaying the same polarity consistently share common circuit structures (0.35 vs. 0.18 across functional types, p < 1/4000; Figure 19). We clarify that the identified computational mechanisms within the model serve as candidate hypotheses for biological retinal processing. This framing is supported by prior work demonstrating that such predictive models capture ganglion-cell responses and accurately recover underlying interneuron properties without direct supervision (Maheswaranathan et al., 2023).
Reference: Maheswaranathan, N. et al. Interpreting the retinal neural code for natural scenes: From computations to neurons. Neuron 111, 2742-2755.e4 (2023).