- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors study to what extent task-relevant information is encoded in neural networks’ hidden states without using it to produce outputs, while components can causally influence outputs without linearly encoding the answer. The study distinguishes transport, which determines where information can be recovered, from selection, which determines which components affect the output. Results suggest that increased transport broadens information availability with little change in task accuracy. Experiments across multiple architectures show that representational availability and causal responsibility are distinct, such that information may be recoverable by a new readout yet unused by the original model, while a component can control the answer without making it linearly decodable.
Critical strengths
-
The paper studies representational availability, causal responsibility and recoverable sufficiency in neural networks, which I found to be a relevant view on the foundations of representation learning. It connects it to interpretability through asking: What determines whether a component actually uses the task-relevant information available to it, and how can we identify which component is causally responsible for a behavior?
-
The authors acknowledge important limitations, including the dependence of causal responsibility on the intervention and the restricted decoder class used to assess availability.
-
The empirical setup appears reproducible and detailed descriptions are provided, alongside the code repository. Figure 1 gives a useful overview of the key concepts and aims of the study.
Critical weaknesses
-
The four quantities D, X, B, S are individually defined in Section 3, but their relationships are not made sufficiently clear and the paper introduces several new terms and abbreviations in quick succession, e.g. “Transplant responsibility, denoted B and called break below, is a difference of two follow rates under matched transplants.” (l. 222-223).
-
The paper lacks clarity in its empirical description. It is challenging to track quantities that are observational, which use interventions, how the carrier is identified, and which experimental settings have exceptions, which overall limit readability.
-
While the number of tested hypotheses is appreciated, the large number of settings makes the overarching evidence harder to synthesize and difficult to draw clear and practical conclusions. The paper would benefit from a more focused perspective and more clear selection of experiments, e.g. the reader has to work out which combination of measures is used in each experiment and how to interpret the combination and why it was chosen.
-
The motivation of Section 5.2 is appreciated, but the description of results lacks clarity, e.g. “These capacity sweeps establish generality but not causality because deeper stacks also add computation. The inference-time cuts below isolate transport without retraining.” (l 353-252). Where is this shown? What is the setup to measure this?
-
The connection to (mechanistic) interpretability is not worked out sufficiently, with connections drawn loosely to several directions, e.g. causal mediation analysis, intervention studies, circuit discovery, neuron analyses, tracing, routing, etc. Complementary parallels are drawn to modularity, identifiability, Oversmoothing, and load balancing. All these connections span a rather large network of ideas that hinders a clear positioning of the presented work. The paper would benefit from a more focused angle and more clearly aligned experiments.
Questions for authors
- How is the carrier determined across different architecture settings?
- What is the intervention strength and how is it determined?
Thank you for your feedback, and invaluable suggestions.
We thank the three reviewers for reading the submission carefully. Their reviews identified problems that made the paper admittedly difficult to assess. We introduced terms before defining them, gave little detail about the networks we trained, and placed key evidence in the appendix. In checking the results for this response, we also found an error in how we evaluated some of the probes. So we have decided to withdraw the submission and rewrite/refine it. We summarise here our answers to reviewers’ concerns and questions which we hope can help:
Central claim: A frozen network can contain enough information outside the original source nodes for a new readout to solve the task, even when removing those nodes reduces the original readout’s accuracy to chance. In the graph task, for example, the input might provide A = 2 and B = 3, then ask for the value of A. We call the node initially given A = 2 the carrier. We define transport then how information moves between components, through message passing in this case and selection as how the readout draws on the resulting states to produce an answer.
With four rounds of message passing, zeroing both input value nodes after message passing reduced accuracy from 0.969 to 0.240. Keeping those nodes masked and the rest of the network frozen, we retrained the readout and recovered 0.865 on held-out data. Without message passing, the same procedure reached 0.248 (Table 4). The probe measures what can be decoded. The ablation measures the trained readout’s dependence on the removed nodes under that intervention. Retraining establishes what a new readout can do with the remaining states.
Probe evaluation: In the constructed binding experiments, we scored linear probes on the same examples we used to fit them. We reran the graph probes on held-out data. When the probe reads the mean state of the non-carrier nodes, accuracy is 0.77 after four rounds of message passing and 0.25 without message passing. The submission reported 0.88 and 0.38, respectively. The gap between carrier and non-carrier probe accuracy still narrows, from +0.72 to +0.20. We have yet to rerun the other affected settings. The pretrained-model and pointer probes already used held-out data.
Readout impact: Earlier runs led us to expect a larger effect at intermediate transport, where other components contain usable information and the trained readout still depends on the carrier. We fixed the statistic and decision rule before testing on 15 fresh seeds. The difference between the average effect at the two intermediate settings and at the two endpoints was +0.371 for the CNN (95% CI +0.322 to +0.421) and +0.267 for the transformer (95% CI +0.226 to +0.309). All 15 seeds in each model family agreed in direction. The effect also held when we froze the transformer body and retrained only the readout (+0.155). The corresponding frozen CNN comparison was inconclusive because one setting fell below the prespecified accuracy threshold (Appendix B).
Concerning other reviewer questions:
How do we identify the carrier and choose the intervention strength? The task identifies the carrier in the constructed graph, CNN and transformer experiments. In the recurrent-slot and expert settings, we select the component with the highest probe accuracy on a separate discovery split. For the transplant, we replace the entire state with a state from an example with the same query and a different answer. We make the same replacement at a control component and compare how often the prediction switches to the donor’s answer.
How should we interpret the citation-graph results? Across the propagation depths that met the accuracy threshold, the transplant contrast fell from +0.097 to +0.015 on Cora and from +0.075 to +0.006 on CiteSeer. These are modest declines in the measured intervention effect. The probe gap stayed near zero because the neighbourhood predicted the label about as well as the target node did.
Does the pointer result generalise to pretrained models? We demonstrated the effect in a graph task designed to separate the address from the answer. Whether the same dissociation occurs in standard pretrained models remains open.
In the rewrite, we will use the spread of decodable information as motivation and centre the paper on the frozen-network comparisons. We will define the terms where they first appear, give the architecture, data, training and intervention rules for each experiment, move the main evidence into the body of the paper, and use held-out probes throughout. We will focus the related work on probing and causal interventions, self-repair, and binding in pretrained language models.