acceptodds
Withdrawn from ICLR 2027

Separating representational redundancy from causal responsibility

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Neural network interpretations often treat decodability as evidence of functional use. If a linear classifier predicts a task variable from a component's hidden state, that component may be taken both to contain and contribute the information. This work separates those properties. A component may encode task information the model does not use, while another may control the output without linearly encoding the answer. Two architectural mechanisms organise the distinction. Transport spreads task information between components and determines where it can be recovered. Selection governs which available components affect the output. In two transport sweeps, selection has its largest effect after transport creates usable alternatives but before the original component loses measured responsibility. A follow-up statistic and decision rule for this non-monotonic pattern were fixed after the initial result and confirmed on fresh seeds. Across recurrent, graph, convolutional, transformer, and mixture-of-experts networks, greater transport capacity is accompanied by broader recoverability with little change in task accuracy. Fixed-weight route cuts isolate transport where such interventions are available. In a graph network with its body frozen, the task variable is decoded from components other than its source with 0.88 accuracy, yet source removal reduces the trained readout to chance. Retraining only the readout recovers 0.87 accuracy, showing that the information was available to a new readout but unused by the original one. Conversely, transplanting a pointer node redirects the answer even though the answer remains linearly undecodable there. Representational availability and causal responsibility are distinct and should be measured separately when attributing functions to network components.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, causal attribution, representation probing, causal interventions, activation patching, representational redundancy, AI auditing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_cJXUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors study to what extent task-relevant information is encoded in neural networks’ hidden states without using it to produce outputs, while components can causally influence outputs without linearly encoding the answer. The study distinguishes transport, which determines where information can be recovered, from selection, which determines which components affect the output. Results suggest that increased transport broadens information availability with little change in task accuracy. Experiments across multiple architectures show that representational availability and causal responsibility are distinct, such that information may be recoverable by a new readout yet unused by the original model, while a component can control the answer without making it linearly decodable.

Critical strengths
  • The paper studies representational availability, causal responsibility and recoverable sufficiency in neural networks, which I found to be a relevant view on the foundations of representation learning. It connects it to interpretability through asking: What determines whether a component actually uses the task-relevant information available to it, and how can we identify which component is causally responsible for a behavior?

  • The authors acknowledge important limitations, including the dependence of causal responsibility on the intervention and the restricted decoder class used to assess availability.

  • The empirical setup appears reproducible and detailed descriptions are provided, alongside the code repository. Figure 1 gives a useful overview of the key concepts and aims of the study.

Critical weaknesses
  • The four quantities D, X, B, S are individually defined in Section 3, but their relationships are not made sufficiently clear and the paper introduces several new terms and abbreviations in quick succession, e.g. “Transplant responsibility, denoted B and called break below, is a difference of two follow rates under matched transplants.” (l. 222-223).

  • The paper lacks clarity in its empirical description. It is challenging to track quantities that are observational, which use interventions, how the carrier is identified, and which experimental settings have exceptions, which overall limit readability.

  • While the number of tested hypotheses is appreciated, the large number of settings makes the overarching evidence harder to synthesize and difficult to draw clear and practical conclusions. The paper would benefit from a more focused perspective and more clear selection of experiments, e.g. the reader has to work out which combination of measures is used in each experiment and how to interpret the combination and why it was chosen.

  • The motivation of Section 5.2 is appreciated, but the description of results lacks clarity, e.g. “These capacity sweeps establish generality but not causality because deeper stacks also add computation. The inference-time cuts below isolate transport without retraining.” (l 353-252). Where is this shown? What is the setup to measure this?

  • The connection to (mechanistic) interpretability is not worked out sufficiently, with connections drawn loosely to several directions, e.g. causal mediation analysis, intervention studies, circuit discovery, neuron analyses, tracing, routing, etc. Complementary parallels are drawn to modularity, identifiability, Oversmoothing, and load balancing. All these connections span a rather large network of ideas that hinders a clear positioning of the presented work. The paper would benefit from a more focused angle and more clearly aligned experiments.

Questions for authors
  • How is the carrier determined across different architecture settings?
  • What is the intervention strength and how is it determined?
Reviewer_UPC5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

None

Critical strengths

None

Critical weaknesses

After careful reading, I found the paper difficult to follow. The central argument, including the notions of transport, selection, and carrier, and how these quantities relate to one another, was not conveyed clearly enough for me to understand it or assess the claims. Given this, I believe the paper currently falls considerably short of the standard expected for this venue, and I recommend rejection.

The following are several specific examples illustrating the presentation issues that substantially hindered my understanding of the paper. These examples are not exhaustive, but they highlight recurring difficulties with the introduction of terminology, the clarity of the exposition, and the motivation of the proposed framework.

  • Introduction: The introduction begins with a general discussion contrasting probing with causal intervention, but does not clearly articulate the specific research question that the paper seeks to address. Beginning in the third paragraph, it introduces numerous technical terms, including architecture variables, lateral messages, architecture sweeps, route cuts, available components, and realised computation, without sufficiently establishing their meanings or explaining how they relate to the central research question.

  • Paragraphs 3, 4, and 6: I found these paragraphs particularly difficult to follow. The arguments are not presented clearly enough to identify their intended conclusions or understand how they contribute to the paper’s overall motivation and line of reasoning.

  • Figure and caption: The figure is difficult to interpret independently. Its caption begins with the phrase “Randomised factor order” without explaining what this refers to or why it is relevant. It also uses terminology such as binding by content, and readout without providing sufficient explanation for readers to understand the concepts or their roles in the figure.

  • Section 3, lines 191–192: The text states: “The factor f is this value, 2 here, and has four possible values, giving a chance level of 0.25.” However, it does not adequately explain what the factor (f) represents, why it takes four possible values, or how these choices relate to the experimental setup and the stated chance level.

  • Section 3, lines 193–194: The text states: “It is marked by the task when components have fixed roles and nominated on separate data when components are exchangeable.” The distinction between components with “fixed roles” and “exchangeable” components is not sufficiently explained. In particular, it is unclear what constitutes a component’s role, what exchangeability means in this context, and why these conditions require different procedures.

  • Section 3, line 202: The term component states is introduced without enough contextual explanation to make clear what these states represent, how they are obtained, or how they are used in the subsequent analysis.

  • Section 3, line 203: The phrase downstream selection stage is used without adequately explaining what stage of the computation it refers to or how selection operates within the proposed framework.

  • Section 3, line 206: The phrase downstream access mechanism is similarly unclear. The text does not sufficiently specify what this mechanism consists of, how it relates to the previously introduced concepts, or how it is distinguished from the downstream selection stage.

Questions for authors

None

Reviewer_GRTXOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper analyzes the gap between decodability (whether a component encodes task information) and causal responsibility (whether a component is used). The authors propose that two mechanisms, transport and selection, govern the two properties respectively. Experiments are done on recurrent, graph, convolutional, transformer, and mixture-of-experts networks, both trained from scratch and pretrained. By varying transport and selection, the authors discover regimes where task-relevant information is localized and used, widespread but only used in specific locations, and widespread and used widely.

Critical strengths
  • Even though the decodability-responsibility gap is well-known, the redundancy of task-relevant information remains understudied. This paper makes a good effort in quantifying the spread and causal importance of redundant information using four metrics, D, X, B, and S, which tease apart how much redundant information is encoded, how much the network is currently using, and whether the network can use more if the readout is retrained.
  • The paper contains extensive experiments on a wide range of architectures and datasets.
Critical weaknesses
  • The presentation is very hard to follow. The prose is unnecessarily convoluted and introduces a lot of jargon without defining them precisely, such as “donor”, “factor”, “query”, “carrier”, “episode”, “component”, “battery”, to name a few. More importantly, from what I gathered, the paper never clearly defines each experimental setting, which is especially detrimental to the trained graph and recurrent settings, since they are not standard architectures one could look up elsewhere. Also, figures are hard to parse and most figures making important claims are in the appendix.
  • The main finding does not seem surprising or novel and it is unclear what the main claim is. From what I understood, transport is defined as an injected piece of information at a component (which I think means an input location such as a token) spreading to other locations as the computation progresses (e.g. later layers, later message passing rounds). It is hardly surprising that this occurs, and that one can linearly decode information from more tokens at a deep layer of a transformer, for example. It also doesn’t seem surprising that a readout may be pooling from many locations or not, and that a retrained readout may pick up on redundant information elsewhere. While the four metrics seem novel to me, I am not sure what insights they may be able to reveal.
Questions for authors
  • Please make the paper easier to read—reduce jargon, define necessary terms, specify experimental details, and move important figures into main. For example, what do you mean by “component” precisely? Why is transport operationalized as depth and receptive field, for example, in Table 1? What is the architecture, task, and data of each network you train from scratch?
  • What is the central claim? Other than the observation that redundancy and responsibility can vary independently, there seem to be many smaller claims scattered throughout the narrative. What is it that your metrics reveal that previous methods don’t?
  • In Section 5.3, for citation networks, B and X seem quite small overall. Can you give an interpretation of these values and whether it’s still reasonable to claim a trend over the quantity varied on the x axes?
  • In Section 5.4, you give an example of a pointer being causally important but not containing the task-relevant value. Does this phenomenon generalize to the standard pretrained networks?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you for your feedback, and invaluable suggestions.

We thank the three reviewers for reading the submission carefully. Their reviews identified problems that made the paper admittedly difficult to assess. We introduced terms before defining them, gave little detail about the networks we trained, and placed key evidence in the appendix. In checking the results for this response, we also found an error in how we evaluated some of the probes. So we have decided to withdraw the submission and rewrite/refine it. We summarise here our answers to reviewers’ concerns and questions which we hope can help:

Central claim: A frozen network can contain enough information outside the original source nodes for a new readout to solve the task, even when removing those nodes reduces the original readout’s accuracy to chance. In the graph task, for example, the input might provide A = 2 and B = 3, then ask for the value of A. We call the node initially given A = 2 the carrier. We define transport then how information moves between components, through message passing in this case and selection as how the readout draws on the resulting states to produce an answer.

With four rounds of message passing, zeroing both input value nodes after message passing reduced accuracy from 0.969 to 0.240. Keeping those nodes masked and the rest of the network frozen, we retrained the readout and recovered 0.865 on held-out data. Without message passing, the same procedure reached 0.248 (Table 4). The probe measures what can be decoded. The ablation measures the trained readout’s dependence on the removed nodes under that intervention. Retraining establishes what a new readout can do with the remaining states.

Probe evaluation: In the constructed binding experiments, we scored linear probes on the same examples we used to fit them. We reran the graph probes on held-out data. When the probe reads the mean state of the non-carrier nodes, accuracy is 0.77 after four rounds of message passing and 0.25 without message passing. The submission reported 0.88 and 0.38, respectively. The gap between carrier and non-carrier probe accuracy still narrows, from +0.72 to +0.20. We have yet to rerun the other affected settings. The pretrained-model and pointer probes already used held-out data.

Readout impact: Earlier runs led us to expect a larger effect at intermediate transport, where other components contain usable information and the trained readout still depends on the carrier. We fixed the statistic and decision rule before testing on 15 fresh seeds. The difference between the average effect at the two intermediate settings and at the two endpoints was +0.371 for the CNN (95% CI +0.322 to +0.421) and +0.267 for the transformer (95% CI +0.226 to +0.309). All 15 seeds in each model family agreed in direction. The effect also held when we froze the transformer body and retrained only the readout (+0.155). The corresponding frozen CNN comparison was inconclusive because one setting fell below the prespecified accuracy threshold (Appendix B).

Concerning other reviewer questions:

How do we identify the carrier and choose the intervention strength? The task identifies the carrier in the constructed graph, CNN and transformer experiments. In the recurrent-slot and expert settings, we select the component with the highest probe accuracy on a separate discovery split. For the transplant, we replace the entire state with a state from an example with the same query and a different answer. We make the same replacement at a control component and compare how often the prediction switches to the donor’s answer.

How should we interpret the citation-graph results? Across the propagation depths that met the accuracy threshold, the transplant contrast fell from +0.097 to +0.015 on Cora and from +0.075 to +0.006 on CiteSeer. These are modest declines in the measured intervention effect. The probe gap stayed near zero because the neighbourhood predicted the label about as well as the target node did.

Does the pointer result generalise to pretrained models? We demonstrated the effect in a graph task designed to separate the address from the answer. Whether the same dissociation occurs in standard pretrained models remains open.

In the rewrite, we will use the spread of decodable information as motivation and centre the paper on the frozen-network comparisons. We will define the terms where they first appear, give the architecture, data, training and intervention rules for each experiment, move the main evidence into the body of the paper, and use held-out probes throughout. We will focus the related work on probing and causal interventions, self-repair, and binding in pretrained language models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.