- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a communication map of a transformer that charts every possible communication path from weights alone. They extend the work of [1] and propose a generalized composition score that includes neurons, which they call “coupling coefficient”.
The main contributions are: i) generalizing the composition score of [1] to include neurons; ii) mapping the full GPT-2 and Pythia communication paths using the coupling coefficient; iii) finding induction critical subspaces (deleting these subspaces destroy the induction capability).
The validation is performed in two main experiments: i) finding head-to-head communities by using a community detection algorithm on their map; ii) finding induction critical subspaces.
While the work has interesting points, the theory and validation are limited. The text also lacks clarity in some points, and most of the required methodological explanations were in the Appendix. Therefore, I find it insufficient for acceptance.
[1] Elhage, N., et al. (2021). A mathematical framework for transformer circuits. Transformer Circuits Thread. https://transformer-circuits.pub/2021/framework/index.html
Critical strengths
The idea of extending the work from [1] to build the entire communication map of the transformer using only the weights (feed-forward and attention blocks) is potentially interesting. Particularly, Application 1 usage of Louvain partition to compute different “communities” of heads over the communication map was an interesting way of finding communities in an unsupervised way. However, I believe that even this part was underexplored in the results.
Critical weaknesses
- The theoretical novelty was very limited: the paper's main methodological contributions were to i) extend the coupling coefficient to include feed-forward layer neurons; and ii) construct the full communication map of a transformer using the coefficient. However:
- For i): there is very limited explanation on the main paper of why this extension works. Also, both the geometric interpretation and properties of the coefficient are directly related to the Frobenius norm. These are interesting properties of the metric, but I don’t see how this is a significant improvement to the state-of-the-art.
- For ii): the construction itself could be interesting (although not sufficient), however, both applications focus on the head-to-head paths, ignoring the neurons paths for most of it. Therefore, I don’t see how extending the coefficient was necessary for this downstream analysis.
- The experimental results were very limited: i) Application 1 results were not unexpected or new considering related work; ii) Application 2 results were potentially more novel (although not sufficient by themselves), but I don’t think the baselines were justified.
- For i): As previously mentioned, the community detection algorithm was an interesting way of finding subgraphs that communicate with each other. However, the choice of application was not very novel in my opinion: choosing to ablate the green community should destroy the performance of the model in induction tasks, because this group contained known name-movers and induction heads. I think that this result would be much more interesting if instead of focusing on a community that is well described in previous work (like the IOI circuit paper), there was, for example, an exploration of what each community is responsible for.
- For ii): I think this was a more novel result, however, I don’t think the baselines were justified. For example, why removing rogue dimensions would be considered a good baseline for subspaces that carry positional information?
Questions for authors
Clarification requests
- Following Weakness 2b: why were that set of baselines chosen for Application 2? How are they comparable to a method that was specifically designed for finding the induction-critical bands?
- How were the neuron paths used in the experiments? Both applications appeared to have ignored these paths completely and used primarily the head’s weights.
Suggestions for future work
- Application 1 was underexplored in my opinion. Instead of ablating a community that contains known induction heads and showing that it indeed destroys its induction capabilities, a more interesting question would be to characterize the other communities found: what are their functions? Do they activate on a particular set of prompts?