- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
As far as I understand, the paper asks whether it matters which input weight vector of an MLP hidden unit is paired with which output weight vector, by comparing the trained pairing against random shuffles of it. On small GELU language models trained from scratch (29M to 405M parameters, three seeds) and on released models, the authors report that the trained pairing differs clearly from the shuffles, that in the most strongly paired units the output vector mostly points against the input vector, and that this sign appears early in training and only when the text is in its natural order.
Critical strengths
The measurement is careful: a clean exchangeability null, initialisation controls, multiple seeds, and a token-order ablation (Figure 21).
Critical weaknesses
I struggle to see the significance of the result. Stated plainly, the paper shows that an MLP neuron's input and output weight vectors tend to point in opposite directions, more than random re-pairings would, which is partly expected because each output vector is trained only where its input vector fires (Section 5.3). The effect is not specific to language models (the MNIST control in Appendix D shows it too), similar observations were made by Gurnee et al. (2024, https://arxiv.org/abs/2401.12181) and Voita et al. (2024, https://arxiv.org/abs/2309.04827), and breaking the pairing costs only 4 to 9 millinats (Table 13). The paper does not show a consequence for interpretability, training or practice.
Furthermore, I find the paper is difficult to read. The reader/writer terminology is misleading: each neuron computes f(u . x) v, and u passing through a nonlinearity does not make it a "reader" and v a "writer"; the labels add an interpretation that the measurement does not support. Together with coined terms such as "couple", "core" and "stream couple", it hides a simple measurement, the sign of u . v and the symmetry of the MLP Jacobian, and almost every number is deferred to the appendix.
Questions for authors
What would change in practice, for interpretability or for training, if this pairing were absent?