acceptodds
Under review as a conference paper at ICLR 2027

Revealing Pathway Interactions in Network Computations with Complete Path Contribution

Abstract

Understanding how neural networks compute is central to the study of artificial and biological intelligence. Networks pass signals through many layers of interconnected units, so explaining what they compute means tracing how information travels from input to output and how pathways interact. Most existing approaches analyze only pairwise interactions between components such as units, layers or attention heads. Although some uses of these methods combine these measurements ad hoc to form a circuit from input to output, these approaches inherently do not capture the higher-order interactions along full causal chains (paths) from input to output, and therefore cannot address how the exponential number of causal paths interact to construct network computation. This leaves a large set of mechanistic questions unanswerable, for example whether inhibitory pathways veto excitatory ones, and how the two combine to yield network decisions. To address this gap, we introduce Complete Path Contribution, a method to decompose the network output into mathematically complete contributions of individual causal chains to study path-level interactions directly and at scale. This decomposition induces definitions of contributions for any circuit at any granularity level via partial summation, with completeness guaranteed at every level, enabling observations that are not feasible with methods that analyze neurons or synapses in isolation. We further present an efficient algorithm that identifies the strongest causal paths without examining all of them, which would be exponential in the number of layers. Instead, the runtime is empirically linear in the number of chosen paths. By applying Path Contribution to benchmark image classification networks, we quantify excitatory-inhibitory interactions and show how they collectively construct network decisions at a level of detail previously unmeasurable. Furthermore, applying Path Contribution to an interpretable model of the mouse retina reveals how dynamic receptive fields emerge from excitatory-inhibitory interaction motifs among pathways between different cell types in response to time-varying natural stimuli. Overall, Path Contribution reveals how a network’s causal pathways interact at scale to transform input to output.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, neuroscience

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_b41HOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces Complete Path Contribution, a framework for analyzing neural network computations at the level of individual input-to-output pathways. The motivation is neuron-level attribution methods may obscure important excitatory and inhibitory interactions when strong positive and negative pathway contributions cancel each other. The proposed method defines the contribution of an individual path by integrating the product of its local Jacobians along a trajectory from a baseline input to the actual input. By aggregating path contributions, the method recovers attribution measures, including Integrated Gradients and neuron conductance, while providing hierarchical completeness across different levels of network granularity. To address the combinatorial complexity of enumerating all possible paths, the authors develop a best-first branch-and-bound algorithm for identifying the top-k paths with the largest absolute contributions. The framework is evaluated on MNIST-trained MLPs, ViT-B, and a mouse retinal response prediction model. The experimental results show the effectiveness of the proposed method.

Critical strengths
  • A small contribution from a neuron does not necessarily imply that the neuron participates in weak computations. Instead, positive and negative pathway contributions may cancel each other. The distinction between weak contributions and strong but mutually canceling pathways is valuable.

  • The formulation of pathway, neuron, edge, and subnetwork contributions looks meaningful. The hierarchical completeness property guarantees that contributions from disjoint partitions of paths aggregate to the contribution of their parent subnetwork. This connects fine-grained pathway explanations to coarser component-level analyses.

  • The proposed best-first branch-and-bound algorithm addresses a computational challenge related to pathway-level analysis. By combining upper-bound estimates, selective expansion, and adaptive refinement, the method avoids exhaustive path enumeration.

  • The experiments cover multiple model architectures and demonstrate different applications of the method. The retinal model analysis provides an interesting connection between pathway contributions, excitatory-inhibitory interactions, and dynamic receptive fields.

Critical weaknesses
  • The novelty appears relatively limited. The authors acknowledge that the definition of path contribution is mathematically equivalent to the influence paths formulation. Hierarchical completeness follows directly from defining a subnetwork's contribution as the sum of its constituent path contributions and applying a partition of the corresponding path set. It would be great to compare with prior approaches in terms of computational efficiency and interpretability.

  • The conclusions rely on selected examples. For the MNIST experiment, the analysis focuses on a selected neuron. It was unclear how frequently similar cancellation phenomena occur across other neurons and model architectures. The interpretation of identified motifs as biological computational mechanisms should be distinguished from their role within the fitted predictive model. The ViT experiment analyzed 50 images from the pill bottle class and a particular pair of transformer blocks. The generality of these observations is unclear. After rebuttal, the generality concerns have been addressed.

  • It was unclear whether similar computational efficiency can be maintained for larger and more complex architectures.

Questions for authors
  • What is the main theoretical novelty relative to existing influence-path methods?
  • What is the meaning of causal contribution in the proposed method?
  • How general are the observed excitatory-inhibitory cancellation patterns?
  • What are the practical limitations of the proposed top-k search algorithm?
  • How sensitive are the conclusions to baseline selection, numerical integration, and aggregation across inputs?
6 comments
AuthorsOpenReview ↗

Response to Reviewer b41H [1/2]

Thank you for your insightful review and for acknowledging the novelty of our work.


[W1, Q1] We compare path contribution to influence paths and path patching in appendix G. Influence paths analyzes paths, but does not state or derive hierarchical completeness, which is the key property that enables a fair comparison between E and I values across granularity levels, as they have to sum to the same output difference. Path patching measures the intervention effect, which is in general a distribution over whether other paths are already patched, and path contribution tracks it closely and has guaranteed completeness that path patching does not have. The Shapley value is the mean of that distribution, and it coincides with path contribution whenever the treefied model (in path patching) is multilinear on the input copies, which the toy model satisfies even though it is nonlinear. Therefore, path contribution can be viewed as a representative statistic of that distribution with a completeness guarantee without having to compute the exponential number of contexts to get the Shapley value. Regarding circuit-attribution methods, we briefly explain in the introduction that they involve first finding the important units or edges in each layer, and combining them post hoc into a circuit. To our knowledge, the resulting circuits have no completeness guarantee, and a unit with a small net effect may have strongly excitatory and inhibitory paths through it, which are important components of a circuit but will be missed by a unit-first search.

In terms of computational efficiency, evaluating a single path’s contribution takes a backward pass for each integration step, which can be parallelized. This is on the same order of cost as path patching, while the Shapley value is exponentially more expensive to compute. Even though integration steps necessitate a constant multiplier that can be expensive, we note that the structure of the definition (sum of products) enables the design of the top k algorithm that path patching does not. We have shown in Figure 13 and Figure 3D,G that the top paths capture key network properties much more sparsely than neurons and edges, which enables path-level analyses for the top paths at scales unreachable for path patching.


[W2, W3, Q3] We show in Figure 14 and Figures 10 and 11 that the cancellation phenomena we observe holds across neurons, random seeds, and architectures. For ViT, we additionally analyzed three classes and three block pairs (early-middle, early-late, middle-late), and we actually found the E-I head-head motifs are quite similar at the same block pair for different classes, but quite different at different block pairs for the same class, suggesting shared computations across classes and serial progression across layers. We also find that paths through the heads of the middle block carry a large share of the output, while at early and late blocks most of the output bypasses the heads. We leave the thorough investigation of these findings to future work.

For the retina, across the full population of 36 recorded units, the motifs remain highly consistent: the primary principal component accounts for 44% of the overall variance, with a median cancellation factor of 6.2 (Figure 18). Units displaying the same polarity consistently share common circuit structures (0.35 vs. 0.18 across functional types, p < 1/4000; Figure 19). We clarify that the identified computational mechanisms within the model serve as candidate hypotheses for biological retinal processing. This framing is supported by prior work demonstrating that such predictive models capture ganglion-cell responses and accurately recover underlying interneuron properties without direct supervision (Maheswaranathan et al., 2023).

Reference: Maheswaranathan, N. et al. Interpreting the retinal neural code for natural scenes: From computations to neurons. Neuron 111, 2742-2755.e4 (2023).

AuthorsOpenReview ↗

Response to Reviewer b41H [2/2]

[W4, Q4] Regarding the top k algorithm, we acknowledge the limited settings we have tested it on. We address the limitations of the algorithm in a dedicated paragraph in Appendix D. In short, for large, modern networks, memory becomes the bottleneck (one layer-wise Jacobian for GPT 2 small costs 2.5 TB at float32) and thus scaling up to those networks requires modifications that trade off time for space, such as matrix-free Jacobians evaluated on the fly. We explain some initial efforts we have taken to mitigate this issue (sibling entries, adaptive refinement, and interval bound) and note that specialized efficient algorithms for specific architectures might be possible even though to our knowledge there is no guaranteed polynomial time algorithm for the general problem of finding top k paths in a DAG with sum-of-products path lengths (like Lu et al. 2021 acknowledged).

Reference: Lu, K., Wang, Z., Mardziel, P. & Datta, A. Influence Patterns for Explaining Information Flow in BERT. in (2021).


[Q2] The meaning of causal contribution comes from the fact that path contribution is a causal measure, just not motivated by intervention. As explained in Appendix G, the chained gradient is inherently the causal effect of a perturbation of the input propagating along that chain. Path contribution accumulates that infinitesimal effect along a straight line to obtain a cumulative causal effect that has the completeness property. The argument that integrated gradients is causal is not new. Geiger et al. 2021 argues that “IGi(x) is measuring the average “limiting” causal effect of increasing neuron i along the straight line from the baseline vector to the input vector x, weighted by the difference at i between input and baseline.”

Reference: Geiger, A., Lu, H., Icard, T. & Potts, C. Causal Abstractions of Neural Networks.


[Q5] We have updated the manuscript with new Figures 6, 7 and 8 to show that top-k path rankings and cancellation patterns remain highly consistent across alternative baselines and reasonable integration steps. Figure 6 shows that both the amount and pattern of cancellation remain stable on gray, white, and MNIST-mean baselines. Figure 7 shows that the overlap of the top k paths under an alternative baseline with those of the default, black baseline remains high (>0.8) for k=10, 100, and 1000 in the class mean contributions (Figure 3’s setting) and over individual images (the box plots). The reason we used top-k overlap for different k’s instead of a rank distance measure over all paths is the many small or exactly zero paths confound such measure. Figure 7 also shows that across different baselines, no path flips sign on the class mean, and almost no path flips sign on the vast majority of images. On the few images where paths flip, h1[13] is so weakly active that its activation lies between its values at the two baselines, so the two integrations run in opposite directions. Figure 8 shows when individual path contributions converge and when completeness error becomes negligible. We find that in the values we used for the main figures (64 for the MLPs, 32 for the retina, 512 for the ViT), all converge well.

I have read other reviews and the authors response. Thank the authors for addressing most of the concerns. I just have one more point regarding causal contribution. In this paper, does causal refer to casual propagation within the model's computational graph based on its structural equations? Or is it more related with intervention-based causal effect in conventional causal inference? What assumptions are required to interpret integrated path contribution as casual effect?

Reviewer_myjwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Complete Path Contribution (CPC), which decomposes an output difference relative to a reference input into contributions from computational paths. Aggregating these recovers contributions associated with neurons, edges, and subnetworks. A branch-and-bound algorithm identifies paths with the largest absolute contributions. Applications to MNIST-trained MLPs, a vision transformer, and a retinal model illustrate cancellation between positive and negative contributions.

Critical strengths
  1. The path contribution definition has a clear mathematical basis in the chain rule and Integrated Gradients. Summing over all paths recovers the output difference from the baseline, and grouping paths provides consistent contributions for neurons and larger components.
  2. The method addresses a useful limitation of neuron-level analysis. A small net contribution can arise from either weak pathways or strong opposing pathways. The proposed decomposition distinguishes these cases and can reveal computational structure hidden by cancellation.
  3. The top-k search provides a practical approach to analyzing networks with many possible paths. The reported runtime benchmarks are encouraging and support its feasibility in the tested MLPs.
Critical weaknesses
  1. The description of the top-k algorithm contains an inconsistency that affects its correctness guarantee. Section 2.2 describes replacing a completed path’s bound with its actual contribution before returning it. Appendix C instead states that the bound automatically becomes exact. This is not generally true when contributions change sign across integration steps or inputs. The implementation should be clarified.
  2. The completeness result holds analytically, but the implementation approximates the integral using a finite number of steps. The paper does not clearly establish how numerical error affects output reconstruction, individual path contributions, or the ranking of the strongest paths.
  3. The dependence on the baseline needs further examination. A valid decomposition can still yield different contributions under different reference inputs. It is unclear how stable the reported pathway signs, rankings, and cancellation patterns are under reasonable alternative baselines.
  4. The method provides an attribution of the model’s computation, but its relationship to selective pathway interventions needs clearer explanation. Completeness does not by itself establish that removing a pathway will change the output by its assigned contribution.
Questions for authors
  1. Test the proposed cancellation mechanisms through targeted perturbations within the model. For example, perturb pathways with opposing contributions separately and together, and assess whether the response changes support the proposed explanation.
  2. Repeat representative analyses using alternative reasonable baselines and report which conclusions remain consistent. For the retinal application, clarify whether contributions reconstruct the firing rate or its difference from the baseline response.
  3. Clarify the meaning of “causal contribution.” Controlled model perturbations could test whether the identified opposing pathways behave as proposed, while recognizing that attribution scores and intervention effects are different quantities.
1 comment
AuthorsOpenReview ↗

Response to Reviewer myjw

We thank you for your careful review.


[W1] Thank you for pointing out this oversight. You are correct in that the bound for a complete prefix may not be tight if paths flip sign across integration steps or inputs. We double checked the code and the code is correct: when a prefix is expanded into complete paths, each path’s actual absolute contribution is computed and pushed into the queue in place of a bound. Then, when it is popped, it is guaranteed to be larger than or equal to any other paths not yet popped. We have made this change to the manuscript in Appendix D.


[W4, Q1, Q3] We address this comparison in appendix G. We show that both for individual paths and many paths at once, path contribution tracks the patching effect closely and has guaranteed completeness that path patching does not have. We want to note that path patching measures the intervention effect, which is in general a distribution over whether other paths are already patched. The Shapley value is the mean of that distribution, and it coincides with path contribution whenever the treefied model (in path patching) is multilinear on the input copies, which the toy model satisfies even though it is nonlinear. Therefore, path contribution can be viewed as a representative statistic of that distribution with a completeness guarantee without having to compute the exponential number of contexts to get the Shapley value.

The meaning of causal contribution comes from the fact that path contribution is a causal measure, just not motivated by intervention. As explained in Appendix G, the chained gradient is inherently the causal effect of a perturbation of the input propagating along that chain. Path contribution accumulates that infinitesimal effect along a straight line to obtain a cumulative causal effect that has the completeness property. The argument that Integrated Gradients is causal is not new. Geiger et al. 2021 argues that “IGi(x) is measuring the average “limiting” causal effect of increasing neuron i along the straight line from the baseline vector to the input vector x, weighted by the difference at i between input and baseline.”

Reference: Geiger, A., Lu, H., Icard, T. & Potts, C. Causal Abstractions of Neural Networks.


[W2, W3, Q2] We have updated the manuscript with new Figures 6, 7 and 8 to show that top-k path rankings and cancellation patterns remain highly consistent across alternative baselines and reasonable integration steps. Figure 6 shows that both the amount and pattern of cancellation remain stable on gray, white, and MNIST-mean baselines. Figure 7 shows that the overlap of the top k paths under an alternative baseline with those of the default, black baseline remains high (>0.8) for k=10, 100, and 1000 in the class mean contributions (Figure 3’s setting) and over individual images (the box plots). The reason we used top-k overlap for different k’s instead of a rank distance measure over all paths is the many small or exactly zero paths confound such measure. Figure 7 also shows that across different baselines, no path flips sign on the class mean, and almost no path flips sign on the vast majority of images. On the few images where paths flip, h1[13] is so weakly active that its activation lies between its values at the two baselines, so the two integrations run in opposite directions. Figure 8 shows when individual path contributions converge and when completeness error becomes negligible. We find that in the values we used for the main figures (64 for the MLPs, 32 for the retina, 512 for the ViT), all converge well. We also show that the cancellation patterns generalize across input samples and random seeds. In Figure 14, we show the net contribution against the total excitatory and inhibitory effects for all h1 neurons in all 100 input-output class scenarios in the same model as in Figure 3 trained from three random seeds, revealing the same cancellation phenomenon across this distribution.

For the retina, path contributions reconstruct output differences from baseline down to a median relative error of 1.6×10−41.6\times10^{-4}. For the cell in the main text, the response to the baseline is 0.05 Hz, so the paths account for essentially the whole firing rate, and we have stated this in the caption of Figure 5. Additionally, these results remain invariant under a uniform gray reference, with robust cancellation observed across both gray and dark baselines (Figure 20).

Reviewer_ZaBUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces Complete Path Contribution, a method that decomposes the difference between a network’s output on an input and on a baseline into contributions from full input-to-output paths. A subnetwork’s contribution is defined as the sum of the contributions of its constituent paths, giving a hierarchy of additive contributions at the path, edge, neuron, and subnetwork levels. The authors also propose a branch-and-bound algorithm to find the k paths with the largest absolute contributions without enumerating all paths. They evaluate the method on a toy example, an MNIST MLP, ViT-B/16, and a retinal CNN. Their examples show that positive and negative path contributions can cancel, a pattern that component-level net contributions may conceal. They also examine how path contributions vary across inputs and over time.

Critical strengths

The method is clearly defined, and hierarchical completeness provides a unified additive framework for attributing contributions to paths, edges, neurons, and subnetworks. Path-level analysis can also distinguish cases that component-level net contributions cannot: downstream paths may be weak individually, or strong positive and negative paths may cancel each other out. The toy model and MNIST case studies illustrate this analytical value. The proposed top-k branch-and-bound method avoids brute-force enumeration of exponentially many paths, and shows reasonable runtime for tested MLP sizes. Applications on image models, transformer and retinal model show broad applicability and point to promising future directions.

Critical weaknesses

The manuscript does not clearly specify the baselines used in the main experiments, nor does it evaluate how finite-step integration or changes in the baseline, input samples, or random seeds affect path rankings and cancellation patterns. The current case studies show that these phenomena can occur, but do not establish whether they are robust to key experimental choices. Comparison against prior related work is insufficient. Influence Paths analyzes individual paths, and Path Patching tests the effects of path sets on model behavior. CODEC is also closely related, but the manuscript does not compare the Path Contribution with these methods under matched settings. Moreover, there are no ablation experiments for the top-k algorithm, making it hard to tell which design choices drive practical performance gains. As a result, the concrete advantages of Path Contribution over existing work remain undersupported.

Questions for authors
  1. The manuscript does not clearly explain how the baselines were obtained. Please specify how the baselines for the MNIST, ViT, and retinal experiments were selected or constructed. The manuscript also does not report the number of integration steps or the numerical completeness error for each experiment. Please provide these details and examine whether the top-k path rankings and major positive/negative cancellation patterns remain stable when the baseline is changed or the number of integration steps is increased. Can these findings also be reproduced across different input samples and random seeds?
  2. The manuscript claims that “previous methods cannot analyze path interactions,” which seems overly broad. Influence Paths has analyzed network paths, and Path Patching has tested the behavioral effects of sets of paths. What is Path Contribution’s main capability beyond Influence Paths, Path Patching, and CODEC? Can the authors provide direct comparisons using comparable models, tasks, and metrics?
  3. Do individual path contributions remain invariant across functionally equivalent models with different computational graphs or parameterizations? Please clarify which modeling choices path scores depend on and how their uniqueness should be interpreted.
  4. The work lacks ablation studies. Could the authors ablate the key design choices in the algorithm and explain the benefit of each?
  5. The retinal experiment reports the firing-rate prediction correlation coefficient (r=0.57) for only one retinal ganglion cell. Could the authors report the fit quality for all 36 cells on held-out test data and discuss how model fit quality affects the biological interpretation of the path results?
  6. The path count reported on line 230 appears to be incorrect. Please check the total number of paths and the corresponding formula. If the count includes additional dimensions, such as the input and output layers, please state the complete counting procedure.
2 comments
AuthorsOpenReview ↗

Response to Reviewer ZaBU [1/2]

Thank you for your thorough and constructive feedback.


[W1, Q1] We have updated the manuscript to state the baselines and integration steps used for each experiment. In addition, we show in new Figures 6, 7 and 8 that top-k path rankings and cancellation patterns remain highly consistent across alternative baselines and reasonable integration steps. Figure 6 shows that both the amount and pattern of cancellation remain stable on gray, white, and MNIST-mean baselines. Figure 7 shows that the overlap of the top k paths under an alternative baseline with those of the default, black baseline remains high (>0.8) for k=10, 100, and 1000 in the class mean contributions (Figure 3’s setting) and over individual images (the box plots). The reason we used top-k overlap for different k’s instead of a rank distance measure over all paths is that many small or exactly zero paths confound such measure. Figure 8 shows when individual path contributions converge and when completeness error becomes negligible. We find that in the values we used for the main figures (64 for the MLPs, 32 for the retina, 512 for the ViT), all converge well. We also show that the cancellation patterns generalize across input samples and random seeds. In Figure 14, we show the net contribution against the total excitatory and inhibitory effects for all h1 neurons in all 100 input-output class scenarios in the same model as in Figure 3 trained from three random seeds, revealing the same cancellation phenomenon across this distribution.

In the retina, the baseline is the clip's mean image, and the contributions are 32-step integrated gradients. Completeness holds to a median relative error of 1.6×10−41.6\times10^{-4}, and the top-10 paths of every cell are identical from 4 to 128 steps (Figure 21). Under a uniform gray baseline, path rankings correlate at 0.96 and the dominant circuits at 0.98. The median cancellation factor is 6.2 under the mean image, 8.2 under gray and 6.4 under black (Figure 20). The findings hold for all 36 recorded cells (Figure 18) and for a model trained from a different seed (CKA 0.91 against a 0.03 null; Figure 22).


[W1, Q2] We address this comparison in appendix G. In short, CODEC only operates at the neuron level and thus cannot measure path-level effects. Influence paths analyzes paths, but does not state or derive hierarchical completeness, which is the key property that enables a fair comparison between E and I values across granularity levels, as they have to sum to the same output difference. Path patching measures the intervention effect, which is in general a distribution over whether other paths are already patched, and path contribution tracks it closely and has guaranteed completeness that path patching does not have. The Shapley value is the mean of that distribution, and it coincides with path contribution whenever the treefied model (in path patching) is multilinear on the input copies, which the toy model satisfies even though it is nonlinear. Therefore, path contribution can be viewed as a representative statistic of that distribution with a completeness guarantee without having to compute the exponential number of contexts to get the Shapley value.

Furthermore, none of these methods study patterns of cancellation at scale, and none of the three has a guaranteed top-k algorithm; finding the top paths is important because we have shown in Figure 13 and Figure 3D,G that they capture key network properties much more sparsely than neurons and edges.

AuthorsOpenReview ↗

Response to Reviewer ZaBU [2/2]

[W1, Q3] Models with the same overall function may solve the task differently, so we expect individual path contributions to differ for different computational graphs or parameterizations. For example, in the retina, models trained on the same ganglion cell starting from two random seeds can learn permuted internal channels (cell types). Therefore, cross-model evaluations require permutation-invariant metrics (such as CKA). In two retina models trained with distinct random seeds, key functional patterns remain highly consistent: the primary circuit closely mirrors the IRF (cosine similarities of 0.96 in both cases), cancellation factors are virtually identical (3.0 vs 3.1), and temporal path dynamics exhibit strong alignment (CKA 0.91; Figure 22).


[W1, Q4] Thank you for pointing this out. The somewhat elaborate designs of interval bound, the sibling entries, and adaptive refinement were originally meant to tackle different challenges that were not present in the test case presented. We ablated these choices and the resulting barebone algorithm actually ran faster on all cases we tested. We have updated the manuscript to now only describe the base algorithm for clarity, with these design choices explained in a Limitations paragraph and what challenges they were intended to solve.


[Q5] Evaluating across the entire population of 36 recorded units using unobserved evaluation data yields a median correlation coefficient of 0.42 (peaking at 0.85), which represents 0.60 of the noise ceiling once it is accounted for (Figure 16). Tracking path dynamics throughout model training reveals that pathway configurations stabilize alongside overall predictive quality, achieving stability well before the classical spatial receptive fields fully mature. Specifically, once network performance reaches 98% of its asymptotic fit, the resulting path contribution structures already share a 0.97 cosine similarity relative to their terminal states (Figure 23).


[Q6] Thank you for pointing this out. We have corrected this typo in the manuscript. Here, we are referring to the network making a specific classification, so the total number of paths is 784784 (input pixels) ×12812×1\times 128^{12} \times 1 (logit).

Comments on the paper

1 comment
AuthorsOpenReview ↗

All reviewer response

We thank all the reviewers for their thoughtful reading, questions, and suggestions for improvement. We apologize for this late reply, because we believed that the feedback was valuable and deserved careful additional analysis and clear presentation. We have made substantial improvements to the manuscript, and we proactively thank the reviewers again for their time in the remaining window to consider the improvements we have made since submission.

We appreciate that all reviewers noted the value of Path Contribution as a unified interpretability framework for paths, neurons, and subnetworks, the important limitations of neuron-level analyses that Path Contribution addresses, and the practical advantages of the top k algorithm. We acknowledge the main concern about the generality of cancellation phenomena and motifs and the lack of analyses on how the discovered phenomena depend on different baselines and integration steps. We believe that these are essential points to make thoroughly, so we have updated the manuscript with additional supplementary figures to address these issues. In summary, we find that cancellation phenomena remain stable under alternative reasonable baselines, are prevalent across neurons, architectures, and random seeds, and integration error becomes negligibly small at the step counts we use (64 for the MLPs, 32 for the retina, 512 for the ViT). We hope the new evidence we present addresses the reviewers’ concerns, and we are willing to make additional improvements for the final version of the paper.

Summary of new figures:

  • Figures 6 and 7: How cancellation phenomena, top-k ranking, and path signs depend on baselines, in the class mean and across individual images.
  • Figure 8: How individual path contributions converge and overall completeness error diminish over integration steps.
  • Figure 14: The prevalence of cancellation across neurons and input-output scenarios and random seeds.
  • Figures 10 and 11: For ViT, the generality of cancellation phenomena and head-head interaction motifs for different image classes and block pairs.
  • Figure 15: Comparison between path patching and Path Contribution.
  • Figure 16: Fit quality of all 36 retinal ganglion cells.
  • Figure 17: Consistent retinal RF structure whether using integration or not.
  • Figure 18 and 19: Path motifs for all ganglion cells and how similar they are, depending on ganglion cell type.
  • Figure 20: Consistent retinal RF structure under alternative baselines.
  • Figure 21: Convergence of retina path contributions over integration steps.
  • Figure 22: Consistent path-level interactions and summary statistics between functionally equivalent models after matching permutation-symmetric channels.
  • Figure 23: The effect of model fit quality on the interpretation of retinal computation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.