- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce a mean-field analysis of attention to understand how the geometry of each token's representation evolves. A kernel defined by the average attention a head pays from one token to another describes how token geometry is transformed. The kernel predicts the trajectory of each token's mean representation from input embeddings under fixed weights. The paper shows that early in training, replacing every attention head’s output with its context-conditional mean field leaves the model’s loss unchanged. The observed findings also generalize to large-scale models. As training progresses, the model behavior and mean-field behavior diverge, with representations becoming contextualized.
Critical strengths
-
The paper is clearly written and easy to follow. The research question of how token representation geometry evolves during initialization and training provides a better understanding of the inner workings of transformer models.
-
The authors first introduce the contribution $ z =\sum\limits_{j=1}^{q} p_q(x_i) v(x_i) $ (Eq. (1)) that a single head computes for a single context. They then use the mean-field approach to aggregate contributions over a set of contexts that share the same structure.
-
The generalization of the proposed average representation dynamics across both Pythia and OLMo model families significantly strengthened the paper.
-
A key strength of the work is its parameter-free mean-field framework, which predicts the evolution of token representations across transformer layers without requiring additional training or fitted parameters.
Critical weaknesses
-
The mean-field approximation in Section 3 relies on attention–value independence and negligible curvature assumptions. How these assumptions relate to standard transformer models is not clear.
-
The experiments in Section 5 replace attention only for the most frequent token types, leaving other queries unchanged. Although Appendix L examines larger vocabularies, the results do not establish whether the entire model can be replaced by its mean field.
-
The authors associate the divergence between model and mean-field behavior with the onset of induction in Section 5. However, the results mainly establish temporal correspondence, without isolating whether induction is responsible for the observed transition.
-
The paper lacks the anonymous code repo or supplementary zip for reproducing the results provided in the paper.
Questions for authors
Questions:
-
Section 3.1 defines a token type as an entry in the model's vocabulary. However, the authors could explain what a token type represents from a natural language perspective. In particular, it would be helpful to clarify whether token types correspond to distinct lexical units, how they differ from token positions and grammatical categories (e.g., nouns, pronouns, and adjectives), and how they are identified and grouped across a large corpus. A concrete example illustrating how multiple occurrences of the same token type are handled, despite differences in their positions and linguistic contexts, would help understand the interpretation of token types and the dynamics of their representation across layers.
-
The authors associate the divergence between model and mean-field behavior with the onset of induction in Section 5. However, the results mainly establish temporal correspondence, without isolating whether induction is responsible for the observed transition. Did the authors observe the induction head behavior?
3 comments
Responses to Questions for Authors
-
Provide better illustration of how token types relate to lexical units and token positions, and how multiple occurrences of the same token type are handled despite differences in position and context. A token type is an entry in the model’s tokenizer vocabulary. Types therefore do not correspond to lexical units or grammatical categories. A single type can span several senses and parts of speech, and one word can map to several types. For example, in GPT-2, “ May” (with a leading space) is one type, used both for the month (“delayed until May”) and for the surname (“Theresa May”); also “ may” (the modal), and “May” (no leading space, e.g. at the start of a line) are distinct types. An occurrence is a type at a particular position in a particular context. Both occurrences of “ May” above share the type, but each has its own representation at every layer. At the input these representations differ only by position embedding; as attention mixes in context, the month and surname uses diverge. A type’s centroid at layer ℓ is the average of its representations over all its occurrences in the corpus sample (2M tokens), regardless of position or sense, so the centroid of “ May” is a frequency-weighted blend of its uses. The mean field predicts how this blend evolves across layers. Separating the senses within a particular context is contextual processing, which is exactly what the mean field does not capture and what the departure measures of §5 (ΔL, D_MF) quantify. We will add an illustration of this sort to the revision.
-
Relationship between onset of induction and mean-field deviation. We address this in response 3 to critical weaknesses (above).
Response to Rebuttal
I thank the authors for their careful response to the review. The authors have responded to all my concerns.
Responses to Critical Weaknesses
Relationship of attention-value independence and negligible curvature assumptions to real models. In the paper, we test both of these assumptions directly in standard pretrained models (GPT-2, Gemma-2, Qwen-3-14B, Llama-3). Negligible curvature holds because occurrences of a type cluster tightly around their centroid (Appendix C): the value map at the centroid matches the average value map in direction to within 1 − cos < 3×10⁻⁴ in every layer of both GPT-2 and Gemma-2. A per-layer error decomposition (Appendix G, Fig 15) shows that this approximation costs at most ~0.005 in centered cosine per layer. Attention–value independence holds at the corpus level in trained models: dropping the covariance term likewise costs at most ~0.005 per layer (Fig 15). This is why the mean field predicts average representations well. On the other hand, at the level of individual contexts, independence is exactly what contextual processing violates. A head that attends according to what a particular context’s values carry, rather than according to type, departs from its mean field. The mean field therefore serves as a null model for contextual processing, and §5 uses it in this way.
Replacing the entire model by its mean field. We agree that our experiments do not replace the entire model, and we will say so explicitly. However, the substitution is more complete than "frequent types only" suggests. At every covered position, every head in every layer is replaced, and attention to all key types, including the rarest, is accounted for: untracked keys are pooled into a shared "other" bucket. Only queries whose own type is untracked keep the model's computation (35% of positions at N = 1000). The results also do not depend on where that boundary falls, which we demonstrate in Appendix L. As shown there, raising coverage from 65% to 80% (N = 1000 to 4000, at matched estimation quality) leaves ΔL within 0.001 nats through step 64, and every landmark of the trajectory falls at the same checkpoint. We note that the trends shown in Appendix L across varying N don’t show signs that the early-phase agreement would fail at full coverage. Only the size of ΔL after the departure grows with coverage, as expected if the remaining positions depart too. Full coverage would require kernel rows for rare query types, whose pair statistics cannot be estimated reliably, because support per pair falls quadratically with N (Appendix M).
Is induction responsible for mean-field deviation during training? We do not claim a causal link; suppressing induction-head formation would require retraining the models, which is beyond our scope. We can, however, show that the mean-field departure is specific to induction heads. For each head of Pythia-160m and each frequent query type, we average the head's actual output over all corpus contexts with that query, and likewise its mean-field prediction. The head's mean-field deviation is one minus the cosine between the two averages, averaged over query types. It is near zero when the mean field reproduces the head's type-level behavior. We also score each head two ways. First, we use a standard test of induction: random token sequences, repeated twice. The induction score is the attention a second-copy token pays to the position after its first occurrence. Second, we use a test that detects previous-token behavior: the previous-token score is the attention a token pays to the immediately preceding position.
We have added this experiment as the new Figure 5 in the updated paper. The figure shows the Spearman correlation across all 144 heads between mean field deviation and each score over training. Through step 256, deviation is unrelated to induction score (all correlations are not significant, ie p ≥ 0.16). The correlation rises to 0.71 at step 1000 and stays between 0.47 and 0.63 thereafter. On the other hand, previous-token heads form in the same window but show the opposite relation (−0.60 at step 1000, −0.39 to −0.55 after). Their deviations rank in the 14th–26th percentile, versus 83rd–94th for induction heads. Deviation thus localizes to induction heads, not to attention reorganization in general. The earliest rise in ΔL (steps 128–256) precedes both head types; we attribute it to the mean field's insensitivity to token order.
Code Repository. Thank you for alerting us to this oversight. An anonymous code repository that reproduces all results of our paper is at https://anonymous.4open.science/r/attention-meanfield-repro-818B/