acceptodds
Withdrawn from ICLR 2027

The Communication Map of a Transformer

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The components of a transformer communicate by writing to and reading from a shared residual stream, and the mechanistic interpretability literature has mapped these connections by hand, one circuit at a time. We present the communication map, which charts every potential communication channel from the geometry of the model's weights alone, generalizing the composition score of Elhage et al. (2021) into a single coupling coefficient covering all 18 connection classes, from head-to-head to neuron-to-neuron and everything in between. We provide an account of the properties of the coupling coefficient, including its geometric interpretation and its exact chance level. The census finds that 70-89% of head pairs are oriented far from chance, some coupled strongly and others actively avoiding each other. We demonstrate the communication map in two novel applications. In Application 1, we recover the known induction circuits blind from the strongest head-to-head couplings and group the heads into communities, and ablating one such community destroys the model's in-context copying. In Application 2, we pool the coupling coefficients of every head to identify a distinct two-dimensional residual stream subspace, whose deletion abolishes the induction capability in six models up to Pythia-6.9B. We show that this subspace is different from those identified by either activation PCA or outlier dimensions. We release the map, the statistical machinery, and the intervention suite.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, transformers, residual stream, attention heads, circuit analysis, language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_LnqmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a communication map of a transformer that charts every possible communication path from weights alone. They extend the work of [1] and propose a generalized composition score that includes neurons, which they call “coupling coefficient”.

The main contributions are: i) generalizing the composition score of [1] to include neurons; ii) mapping the full GPT-2 and Pythia communication paths using the coupling coefficient; iii) finding induction critical subspaces (deleting these subspaces destroy the induction capability).

The validation is performed in two main experiments: i) finding head-to-head communities by using a community detection algorithm on their map; ii) finding induction critical subspaces.

While the work has interesting points, the theory and validation are limited. The text also lacks clarity in some points, and most of the required methodological explanations were in the Appendix. Therefore, I find it insufficient for acceptance.

[1] Elhage, N., et al. (2021). A mathematical framework for transformer circuits. Transformer Circuits Thread. https://transformer-circuits.pub/2021/framework/index.html

Critical strengths

The idea of extending the work from [1] to build the entire communication map of the transformer using only the weights (feed-forward and attention blocks) is potentially interesting. Particularly, Application 1 usage of Louvain partition to compute different “communities” of heads over the communication map was an interesting way of finding communities in an unsupervised way. However, I believe that even this part was underexplored in the results.

Critical weaknesses
  1. The theoretical novelty was very limited: the paper's main methodological contributions were to i) extend the coupling coefficient to include feed-forward layer neurons; and ii) construct the full communication map of a transformer using the coefficient. However:
  • For i): there is very limited explanation on the main paper of why this extension works. Also, both the geometric interpretation and properties of the coefficient are directly related to the Frobenius norm. These are interesting properties of the metric, but I don’t see how this is a significant improvement to the state-of-the-art.
  • For ii): the construction itself could be interesting (although not sufficient), however, both applications focus on the head-to-head paths, ignoring the neurons paths for most of it. Therefore, I don’t see how extending the coefficient was necessary for this downstream analysis.
  1. The experimental results were very limited: i) Application 1 results were not unexpected or new considering related work; ii) Application 2 results were potentially more novel (although not sufficient by themselves), but I don’t think the baselines were justified.
  • For i): As previously mentioned, the community detection algorithm was an interesting way of finding subgraphs that communicate with each other. However, the choice of application was not very novel in my opinion: choosing to ablate the green community should destroy the performance of the model in induction tasks, because this group contained known name-movers and induction heads. I think that this result would be much more interesting if instead of focusing on a community that is well described in previous work (like the IOI circuit paper), there was, for example, an exploration of what each community is responsible for.
  • For ii): I think this was a more novel result, however, I don’t think the baselines were justified. For example, why removing rogue dimensions would be considered a good baseline for subspaces that carry positional information?
Questions for authors

Clarification requests

  1. Following Weakness 2b: why were that set of baselines chosen for Application 2? How are they comparable to a method that was specifically designed for finding the induction-critical bands?
  2. How were the neuron paths used in the experiments? Both applications appeared to have ignored these paths completely and used primarily the head’s weights.

Suggestions for future work

  1. Application 1 was underexplored in my opinion. Instead of ablating a community that contains known induction heads and showing that it indeed destroys its induction capabilities, a more interesting question would be to characterize the other communities found: what are their functions? Do they activate on a particular set of prompts?
Reviewer_dZcwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper suggests using a quantity termed "coupling coefficient" to characterize and study the connectivity in LLMs. The coupling coefficient is simply the cosine similarity between pairs of input-output matrices. The manuscript then discusses a baseline from a rotationally invariant ensemble (the Haar measure on orthogonal matrices) and finds incredibly large deviations from it. Then, within-sample measures are used to identify important couplings, which are subsequently tested.

Critical strengths

The manuscript is easy to follow and uses very simple concepts.

Critical weaknesses

The manuscript barely goes beyond the work of Elhage et al (2021), though the field of mechanistic interpretability has advanced significantly in these 5 years. To reinforce this impression, the manuscript compares to baselines from 2018 and 2019 (before LLMs appeared).

Additionally, the claim baseline is clearly inappropriate, showing absurd median z-scores of 10 and above.

Questions for authors

What are the values corresponding to Table 2 at initialization (with randomly drawn weights)?

How do the values in Table 2 evolve over training? You can use OLMo or Pythia checkpoints

Reviewer_bFodOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to identify potential information flow between Transformer components directly from model weights. It extends the existing composition score to attention heads, MLP neurons, and other components, and provides reference distributions based on random rotations, along with efficient methods for computing the map. In its applications, the paper groups strongly coupled attention heads into communities and tests their role by fixing selected heads' outputs at their mean values. It also uses RW-PCA to select two directions in the residual representation and examines how projecting out these directions across layers affects the model's ability to exploit repeated context.

Critical strengths
  1. The method analyzes multiple component types within a unified framework and scales to the full model. The paper places attention heads, MLP neurons, and other components in a common reader–writer framework and provides an efficient implementation. This enables researchers to systematically identify components that may exchange information and select targets for further analysis and intervention.

  2. The experiments show that weight analysis can identify structures relevant to model behavior. In GPT-2, ablating the selected community reduces induction gain substantially more than ablating matched random head sets. In Pythia-6.9B, removing the RW-PCA and outlier directions causes similar degradation in natural-text prediction but different reductions in induction gain. These results show that the method can help locate components and directions that affect particular behaviors, beyond describing relationships between weights.

Critical weaknesses
  1. The community experiments do not always guarantee matched random controls, and the illustrated and tested head sets differ. Table 9 states that the random controls are matched to the target community in size and per-layer head counts. However, the released implementation (rng.choice(pool, min(cnt, len(pool)), ...)) silently shrinks a control set whenever a layer has too few non-community heads. For GPT-2 small this cannot happen (the community has at most five heads in any layer), but it may for large communities such as GPT-2-large (n=184), where the larger effect of community ablation could then partly reflect the ablation of more heads. In addition, the community shown in Figure 2 is not identical to the one ablated in Table 10: according to the released result files, they differ by one head (L9H6 appears only in the figure, L4H3 only in the ablation). The authors should clarify which head set each experiment uses and confirm that the controls are exactly size-matched for every model.

  2. The Pythia maps include direct within-layer paths that are absent from the model's computation. Table 8 allows connections from attention-head outputs to MLP-neuron inputs in the same layer. In Pythia, however, attention and the MLP operate in parallel: the MLP does not read the outputs produced by attention heads in that layer. These connections therefore cannot be interpreted as direct information-flow paths as described in the paper. The corresponding maps and statistics need to be corrected.

Questions for authors
  1. Regarding Table 9, please clarify two points about the controls and the head groups.

    • Do the random controls ablate the same number of attention heads in each layer as the target community? When this condition is strictly enforced, does ablating the target community still cause a larger reduction in induction gain?
    • Some of the identified induction heads lie outside the target community. If the comparison is restricted to induction heads within the community and the remaining community heads, do the results still support the interpretation that some heads compensate for others? What evidence supports this interpretation?
  2. Why do the Pythia maps include connections from attention heads to MLP neurons in the same layer? The MLP does not read these heads' outputs. Please explain the basis for including these connections and whether removing them from the maps changes the relevant statistics and conclusions.

  3. How do prediction losses on the first and repeated blocks change after removing the two directions selected by RW-PCA? The paper measures induction gain as the difference between these two losses. This difference can shrink because the repeated-block loss increases, the first-block loss decreases, or both. Please report each loss before and after the intervention to clarify what drives the reduction in induction gain.

Overall: the paper provides a useful analysis method and reports findings related to model behavior, but the community experiments do not yet establish a fully matched comparison, and the Pythia maps contain connections inconsistent with the model's computation. If the authors provide properly matched controls, correct the maps, and show that the revised results still support the main conclusions, I would consider raising my recommendation to weak acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.