acceptodds
Withdrawn from ICLR 2027

Finding Algorithms in Neural Networks: Convergent Predictions Between Distributed Alignment Search and Speed of Transfer Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

How can we identify the algorithm that a neural network learned on a task? One behavioral method, transfer learning analysis (TLA), is to read it off how quickly it learns related tasks. One mechanistic method, distributed alignment search (DAS), is to read it off from interventions on the network's internal structure. Whether these two methods converge is as of yet untested. We find that, on toy arithmetic and vision tasks, the algorithms identified by DAS and TLA mostly agree: the internal structure of neural networks predicts how quickly they will transfer to a related task (and conversely, speed of transfer predicts internal structure). Crucially, only DAS with non-linear localization agrees with TLA in our vision task. This means that TLA isn't merely a description of the network's behavior. DAS and TLA are two convergent sources of evidence for identifying the algorithm implemented by a neural network.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, causal abstraction, transfer learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ThhJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows that different methods of identifying algorithms in neural networks, behavioral analysis and mechanistic interpretations, lead to the same result. Previous investigations in behavioral methods could not identify such a link. In line with previous work, validation is limited to 2 small toy examples.

Critical strengths

Paper tells a clear story, after reading the paper and the related work their combination of TLA and DAS seems like an obvious combination.

They clearly build on and improve the previous work of Morrison et al, and find relevant patterns the original authors could not identify.

Critical weaknesses

The authors build heavily on the TLA by Morrison et al, but in their experiments significantly deviate from the therein described procedures without providing a motivation for this deviation. These arbitrary deviations without indication make it difficult to see the results in the context of previous work and might indicate that they varied their approach until they found some correlation.

“in the vision experiment, we originally looked for the distinctions found by transfer learning in linear subspaces of the network, and couldn’t find them. But because transfer learning gave us independent reason to think the distinction was real, we then searched non-linear subspaces of varying dimensions using a range of techniques and strategies until we found a partial alignment” this seems like a circular argument: the authors assume that there is a connection between TLA and DAS, so they search and vary parameters until they are able to identify a correlation, and even then the effect is very small and only visible if outliers are removed (for which the authors provided no rationale).

Questions for authors

Why did you deviate from the experimental setup described by Morrison et al? Arithmetic: Instead of analyzing a one variable function, you investigate a two variable function, and for comparison you use checkpoints over 75 epochs instead of the results after each batch in the first epoch; 65 are labeled as ambiguous. Vision: Instead of 60 networks they train 362, from which 160 are labeled as ambiguous, and instead of training on all 6 possible separations of 4 shapes into 2 groups, you only train on one and do not provide details for training of the vision tasks.

Morrison et al. have a much lower fraction of ambiguous results, and it is not obvious whether this is due to the difference in tasks or evaluation, do you have a hypothesis for this?

The discussion of the results in Fig. 3 would benefit from tests for statistical significance (In line 371 there is a leftover note (amir: REPORT T-TEST)), while the difference in layer 1 is very pronounced, the differences in later layers are probably less significant. Do you have a hypothesis why?

In the vision task: why are you not discussing the possibility that the increase in IIA in the non-linear DAS is simply due to the increased complexity of the alignment map, as this could be a manifestation of the non-linear representation dilemma by Sauter et al? How significant is the difference when outliers are not included?

Did you also apply DAS to some of the networks that were labeled as ambiguous by TLA? If so, did DAS also find them ambiguous?

1 comment

Thank you for your time and for the detailed review. We agree that additional analysis of the networks labeled "ambiguous" would help strengthen the paper, as well as accounting for the complexity of the alignment map in DAS. Due to limited time, we choose to withdraw our submission, and we will continue on refining the paper in line with these comments.

Reviewer_EvHwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper takes two methods that attempt to identify algorithms learned by DNNs, i.e. Transfer Learning Analysis (TLA) and Distributed Aligment Search (DAS), and attempts to determine, on toy tasks, whether these two methods agree with each other.

While an interesting research question in principle, I found the paper very weak on several aspects, starting from motivating the precise choice of methods to not even attempting to discuss how this could generalize to non-toy tasks and larger neural networks. I am going to expand on my criticism below, but at present I find that this paper clearly does not meet the bar for acceptance at ICLR.

Critical strengths
  1. The broad, high-level question of trying to determine whether two interpretability methods agree with each other is timely, important and overall well-posed.
  2. The proposal that a more "behavioral" method like TLA could constrain how expressive DAS’s alignment map is allowed to be is interesting and a potentially nice re-framing of this work in a future iteration.
Critical weaknesses
  1. The paper's central analysis has elements of circularity that bleed into its interpretation. Either TLA and DAS converge independently, or one can help guide the other, but the two cannot be true at the same time. The title mentions convergence, which hints at the methods independently getting to the same result, but then reading the paper's methods and results sections it is clear that the TLA and DAS analyses clearly influenced each other. For example, in the vision experiment, when linear DAS did not reproduce TLA, the authors turned to non-linear maps and searched for effects “searched non-linear subspaces of varying dimensions using a range of techniques and strategies until we found a partial alignment.” (lines 437-438). In the arithmetic case, the opposite thing happened, where linear DAS has been declared the right stopping point when non-linear erased the agreement between the methods. Given that the DAS search space is quite large, the null where the methods do not converge under any configuration becomes quite slim if the search space (and the null hypothesis) is not specified prior to the analysis itself. This is very close to the "double-dipping" problem described by Kriegeskorte et al. (2009) in Systems Neuroscience.
  2. The paper is concerned with a toy setting, but does not even make use of the strengths of toy settings. I think it is quite problematic to present a paper at ICLR 2027 concerned only with toy examples and relatively small neural networks, but that would not be a problem if the traditional strengths of toy settings were used to their full extent, e.g. by having some sort of ground truth in what algorithms the networks are implementing (for example through some auxilliary training losses etc.). On top of this, there is no discussion (or, more importantly, a limitations section) of how this toy setting would translate to realistic models or tasks.
  3. The TLA method is relatively novel, without much follow-up research on it, yet its reliability is assumed by the authors here. There is also not much discussion of why this method was chosen over others. It also seems to give very low labelling rates, and the authors do not attempt any analysis of the unlabelled networks.

A few more minor points:

  • There is a potential small double-blind leak / todo marker at line 317
  • Confidence intervals seem to not be given for AUCs
Questions for authors
  1. How was the reported vision configuration chosen? Was it fixed before TLA agreement? How many configurations were tried in total and how were they determined?
  2. Would TLA and DAS recover the algorithms in networks that are trained or constrained to implement a known ground-truth one?
  3. How would the approach scale to beyond toy networks, e.g. within a single large model across many variables?
1 comment

Thank you for your time and for the detailed review. We respect the point that independent convergence and the notion of using the algorithms to guide each other deserves a longer discussion. We also appreciate the suggestion to utilize the toy setting to a greater extent, such as by establishing a ground truth for which algorithm a neural network is implementing. Due to limited time, we choose to withdraw our submission, and we will continue on refining the paper in line with these comments.

Reviewer_seKUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares two approaches for understanding the algorithms implemented by neural networks. The first approach, transfer learning analysis (TLA), recently proposed, looks at how quickly a model learns related tasks. The second approach, Distributed Alignment Search (DAS), a popular method in mechanistic interpretability literature, probes how high-level interpretable causal variables are represented in the distributed representations of deep learning models. On toy arithmetic and vision tasks, the authors show that the two approaches largely agree with each other.

Critical strengths
  • The paper compares two very different approaches (one based on learning and the other based on causal interventions on network internals) for understanding the algorithm implemented by a neural network
  • It provides a nice robustness check showing that the two approaches agree (albeit their differences).
  • The authors use different toy setups (vision and arithmetic) to validate their conclusions.
  • The paper is clearly written and the experiment and results are easy to follow.
Critical weaknesses
  1. I found the distinction between the two approaches slightly confusing. The paper describes TLA as grounded in “learning and representations,” but TLA does not directly examine internal representations here as it infers algorithms from transfer speed. DAS is the approach that directly examines and intervenes on representations. Describing the distinction as learning-based versus intervention-based would be clearer.

  2. The agreement between TLA and DAS is a useful consistency check, but the broad finding is not entirely surprising. If both methods identify the algorithms a network implements, we would hope they agree. The more interesting result is that this agreement depends on nonlinear DAS in the vision task. I would appreciate more discussion of when and why the methods agree or disagree, and what this teaches us beyond providing reassurance about the methods. More specifically, the consistency in the vision case does not seem very significant (the AUC is quite low even with the non-linear DAS approach).

  3. The experiments are limited to toy tasks. I appreciate the experimental control this provides, especially because the candidate algorithms and relevant variables can be clearly specified. However, it remains unclear whether the agreement would hold for larger networks on more realistic tasks, where these choices are less straightforward. The paper would benefit from either a more complex experiment or a clearer discussion of what might limit generalization. Could the authors also discuss how TLA could be used for more realistic task-trained models?

  4. The paper could be strengthened significantly if you were to show a case where TLA discovers an algorithm that DAS misses or vice versa. Until then, it is an overstatement to say that the two approaches could complement each other in understanding neural network algorithms.

  5. In the current version, the related work section appears the end after discussion. Would be better to move it earlier before the Methods.

  6. Some of the text is redundant. eg TLA and DAS are explain multiple times in the first couple of pages.

Questions for authors

See weaknesses above

1 comment

Thank you for your time and for the detailed review. We appreciate the suggestions for future experiments that would strengthen the paper, especially the idea for analyzing cases where DAS and TLA identify different algorithms. Due to limited time, we choose to withdraw our submission, and we will continue on refining the paper in line with these comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.