acceptodds
Under review as a conference paper at ICLR 2027

Reader-Writer Coupling in Transformer MLPs

Abstract

A transformer MLP gives each hidden channel two roles: a reader that decides when the channel responds, and a writer that decides what it adds back to the residual stream. Read as a key–value memory, the layer can be measured one factor set at a time: the pairing of readers to writers is carried only by their shared index, invisible to any quantity computed from one factor set alone. Yet training couples the two factors through their joint update, so we ask what reading them together shows. The block supplies the joint instrument: the MLP writes back into the very stream it reads, so the exact Jacobian of its update is square, and its agreement with its own transpose resolves into reader–writer terms whose inner products give each channel, read as a _couple_, a signed _pairing coordinate_. Ranked by this coordinate, each block we measure has a small _core_ of strongly coupled channels; where initialization gives either sign equally, the core is nearly all opposed: each writer points against the direction its reader responds to. This opposed core recurs from the smallest models trained from scratch to released checkpoints; its sign is in place within a few percent of training and holds while the coupling grows. Read as couples, most channels return their contribution against the reader that fired it, and the readers of later blocks take it up; few show in the logits. Training holds the couples in place: flipping the opposed writers moves the output toward common tokens, and a writer permuted onto another reader regrows to fit that reader while the reader stays.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, transformer MLPs, reader-writer coupling, key-value memories, training dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_PhbgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

As far as I understand, the paper asks whether it matters which input weight vector of an MLP hidden unit is paired with which output weight vector, by comparing the trained pairing against random shuffles of it. On small GELU language models trained from scratch (29M to 405M parameters, three seeds) and on released models, the authors report that the trained pairing differs clearly from the shuffles, that in the most strongly paired units the output vector mostly points against the input vector, and that this sign appears early in training and only when the text is in its natural order.

Critical strengths

The measurement is careful: a clean exchangeability null, initialisation controls, multiple seeds, and a token-order ablation (Figure 21).

Critical weaknesses

I struggle to see the significance of the result. Stated plainly, the paper shows that an MLP neuron's input and output weight vectors tend to point in opposite directions, more than random re-pairings would, which is partly expected because each output vector is trained only where its input vector fires (Section 5.3). The effect is not specific to language models (the MNIST control in Appendix D shows it too), similar observations were made by Gurnee et al. (2024, https://arxiv.org/abs/2401.12181) and Voita et al. (2024, https://arxiv.org/abs/2309.04827), and breaking the pairing costs only 4 to 9 millinats (Table 13). The paper does not show a consequence for interpretability, training or practice.

Furthermore, I find the paper is difficult to read. The reader/writer terminology is misleading: each neuron computes f(u . x) v, and u passing through a nonlinearity does not make it a "reader" and v a "writer"; the labels add an interpretation that the measurement does not support. Together with coined terms such as "couple", "core" and "stream couple", it hides a simple measurement, the sign of u . v and the symmetry of the MLP Jacobian, and almost every number is deferred to the appendix.

Questions for authors

What would change in practice, for interpretability or for training, if this pairing were absent?

Reviewer_eyujOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper investigates the relationship between the two sides of an MLP neuron: what directions activate it (reader) and what directions it contributes to the residual stream (writer). The main finding is that many neurons write in a derection opposite to the one that increases their activation. The authors train small models and observe that this phenomenon appears early in the training, across several different model sizes. Additionally, they compare the trained neurons with a control where the output weights are randomly shuffled between neurons, and examine how the weights adapt when training is continued after this intervention. They also test how changes in the output weights affect the predictions of the model, and study how individual neurons influence later layers, documenting a recurring relationship between what neurons respond to and what they write.

Critical strengths
  • The paper identifies a consistent and potentially interesting geometric pattern, with reasonable controls to support its robustness, making the observations reasonably convincing.Overall, the experiments are very well thought out and executed!

  • The figures are actually very helpful, as they, suprisingly, clearly communicate the ideas the text is struggling to do so.

Critical weaknesses
  • Writing and Presentation: The paper is unnecessarily difficult to read and to be honest it reads like Claude. I had trouble identifying even the main ideas and spend significant amount of time rereading the paper. The problem strarts from the abstract and carries over every section. I strongly suggest a significant revision of the manuscript. The authros have included that they used AI for "writing and editing (grammar, drafting, and restructuring passages)", but this is not an excuse for the writing of the paper to border on incomprehensible at times.

  • Novelty and Framing: As you acknowledge in Section 2 (L114–L121), previous works also examine the relationship between a neuron's input and output vectors, including negative alignment and weakening behavior [1,2] (I know [2] is very recent, I am just listing it as an additional work you might want to consider, not as a complaint that you did not discuss it). Your effective reader formulation, permutation tsts and trianing trajectories provide more specific additions, but they should be the focus of the motivation and comparison to prior work. Showing that seperate norms or spectra cannot capture the assigment does not establish a limitation of existing works that already consider both vectors jointly. In the same vain, it is far more interesting to study what particular geometry the reader-writer pairs develop and what it explains, than just to document that there is a non-random relationship.

[1] Gurnee, Wes, et al. "Universal neurons in gpt2 language models." arXiv preprint arXiv:2401.12181 (2024).

[2] Gerstner, Sebastian, et al. "Weakening Neurons: An Input-Output Functionality in Transformers with Outsize Influence." arXiv preprint arXiv:2609.18612 (2026).

Questions for authors

See weaknesses.

In my opinion, you have spend a large amount of time working on this project and it is a shame it is so difficult to go over. Unfortunatelly, I am not sure the writing can be improved enough during the rebuttal phase to warrant chaning my assesment of the paper.

Nevertheless, I am more than happy to engage with you during the rebuttal period to see what we can do :-).

Reviewer_ad7LOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors study reader-writer coupling in Transformer MLPs using Jacobian symmetry and permutation tests. Opposed input–output directions are reported, together with early emergence during training, and dependence on sequential structure, supported by experiments across multiple language models.

Critical strengths

The experiments include meaningful controls across multiple models. The early emergence of opposed coupling and its dependence on sequential structure are interesting findings.

Critical weaknesses

1. Limited novelty and scientific significance.

Negative input–output alignment in Transformer MLPs was already studied by Elhage et al. (2021), Gurnee et al. (2024), and Gerstner et al. (2026). The Jacobian analysis relies mainly on standard matrix identities, and the additional experiments do not establish a sufficiently significant conceptual advance over these works.

2. Insufficient mechanistic understanding.

Non-random coupling between jointly trained weight matrices is not particularly surprising, and the assignment gap persists even on shuffled data and in a simple autoencoder (Appendix D). Moreover, Jacobian symmetry is insensitive to coupling polarity, and the interventions do not convincingly establish the functional importance of negative coupling.

3. Poor presentation.

The paper is extremely tedious to read and reads as heavily AI-written. The convoluted writing, excessive metaphors, and repetitive explanations unnecessarily obscure relatively straightforward ideas, making the scientific contribution difficult to identify.

Questions for authors
  • What fundamentally new insight does this work provide beyond the input–output neuron analyses of Gurnee et al. (2024) and Gerstner et al. (2026)?

  • Can the authors establish a more direct functional role for negative coupling, independently of generic learned organization between weight matrices?

  • Given that the assignment gap also appears on shuffled data and in a simple autoencoder, what makes this phenomenon specifically informative about Transformer computations?


I am open to increasing my score if the authors convincingly address the concerns above and substantially improve the clarity and readability of the manuscript.

Reviewer_SQeMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether a transformer MLP neuron writes back along the direction that makes it more active or pushes back against it. Across the models studied, many neurons push back, especially those with the largest reader–writer dot products. The authors track when this pattern emerges and what happens when they disrupt it.

This work investigates reader-writer pairs in a transformer MLP. The authors find the input and output vectors are consistently in opposite directions (negative cosine sim), supported with experiments.

Critical strengths

The authors find an interesting phenomenon, in particular the opposite-aligned input output pairs being present early and consistently. The experiments largely support the finding and provide some understanding.

Critical weaknesses
  1. Writing is very difficult to parse and understand. Unfortunately, while I think the core idea may be interesting and potentially useful, I can't adequately understand this paper well enough to defend my conclusion.
  2. The novelty of this paper is not fully clear. Prior work has studied alignment between input and output weights. The most novel contribution (in my opinion) is the authors' investigating of this pattern during training, which isn't as highlighted. Likewise, the "recovery" experiment after weight swapping is interesting and deserves more attention. In particular this experiment might give some additional clarity on the development of these neurons during training.

Overall, I recommend the authors should focus their efforts on rewriting this paper rather than conducting additional experiments.

Questions for authors

See weaknesses. If this paper is AI written, I would recommend going over it with a fine-toothed comb and rewriting the expository sections by hand.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.