- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper shows that different methods of identifying algorithms in neural networks, behavioral analysis and mechanistic interpretations, lead to the same result. Previous investigations in behavioral methods could not identify such a link. In line with previous work, validation is limited to 2 small toy examples.
Critical strengths
Paper tells a clear story, after reading the paper and the related work their combination of TLA and DAS seems like an obvious combination.
They clearly build on and improve the previous work of Morrison et al, and find relevant patterns the original authors could not identify.
Critical weaknesses
The authors build heavily on the TLA by Morrison et al, but in their experiments significantly deviate from the therein described procedures without providing a motivation for this deviation. These arbitrary deviations without indication make it difficult to see the results in the context of previous work and might indicate that they varied their approach until they found some correlation.
“in the vision experiment, we originally looked for the distinctions found by transfer learning in linear subspaces of the network, and couldn’t find them. But because transfer learning gave us independent reason to think the distinction was real, we then searched non-linear subspaces of varying dimensions using a range of techniques and strategies until we found a partial alignment” this seems like a circular argument: the authors assume that there is a connection between TLA and DAS, so they search and vary parameters until they are able to identify a correlation, and even then the effect is very small and only visible if outliers are removed (for which the authors provided no rationale).
Questions for authors
Why did you deviate from the experimental setup described by Morrison et al? Arithmetic: Instead of analyzing a one variable function, you investigate a two variable function, and for comparison you use checkpoints over 75 epochs instead of the results after each batch in the first epoch; 65 are labeled as ambiguous. Vision: Instead of 60 networks they train 362, from which 160 are labeled as ambiguous, and instead of training on all 6 possible separations of 4 shapes into 2 groups, you only train on one and do not provide details for training of the vision tasks.
Morrison et al. have a much lower fraction of ambiguous results, and it is not obvious whether this is due to the difference in tasks or evaluation, do you have a hypothesis for this?
The discussion of the results in Fig. 3 would benefit from tests for statistical significance (In line 371 there is a leftover note (amir: REPORT T-TEST)), while the difference in layer 1 is very pronounced, the differences in later layers are probably less significant. Do you have a hypothesis why?
In the vision task: why are you not discussing the possibility that the increase in IIA in the non-linear DAS is simply due to the increased complexity of the alignment map, as this could be a manifestation of the non-linear representation dilemma by Sauter et al? How significant is the difference when outliers are not included?
Did you also apply DAS to some of the networks that were labeled as ambiguous by TLA? If so, did DAS also find them ambiguous?
Thank you for your time and for the detailed review. We agree that additional analysis of the networks labeled "ambiguous" would help strengthen the paper, as well as accounting for the complexity of the alignment map in DAS. Due to limited time, we choose to withdraw our submission, and we will continue on refining the paper in line with these comments.