acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Contextualization by Reinterpreting Attention Head Channels

Abstract

Contextualization, the core operation of language modeling, transmits information across words to build sentence-specific word representations. Prior works mainly study contextualization, focusing on individual words and attention heads as a growing discrete dictionary, lacking a global view of their general behavior. Therefore, we propose a general principle: Globally, we find and estimate that different words carry different amounts of information, and less-informative words tend to absorb more contextual information. Specifically, these low-information words do not absorb contextual words uniformly, and finer-grained selectivity enables more precise routing to promote information transmission between matched words. Moreover, to find what mechanism causes such processing, we reinterpret attention heads as channels gated by their singular vectors and find that: (1) these singular vectors point to the hidden states of more informative words, allowing such words to write their information to others more strongly to act as information sources, and vice versa; and (2) these singular vectors can be viewed equally as hidden state features, enabling automated interpretation of attention heads beyond prior heuristic head discovery, also embedding heads into a continuous space rather than treating them as discrete, independent dictionary entries.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GSy2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the general mechanism of contextualization in Transformer language models. Instead of analyzing individual attention heads or token interactions through heuristic discovery, the authors propose a global principle: Information tends to flow from high-information words to low-information words, where low-information words absorb contextual information through selective routing. The paper decomposes this principle into three hypotheses:

  1. Static Information Hypothesis Different words contain different amounts of intrinsic information before contextualization.
  2. Information Gradient Hypothesis High-information words tend to remain stable and act as information sources, while low-information words are more strongly rewritten by context.
  3. Selective Contextualization Hypothesis Information transmission is not uniform; semantic compatibility determines which tokens exchange information. The authors validate these hypotheses empirically by measuring token information gain, hidden-state variation, and activation patching effects. They further provide a mechanistic interpretation by decomposing attention matrices through singular vectors and viewing attention heads as gated information channels. The paper argues that singular vectors correspond to interpretable latent features and provide a continuous representation of attention-head functionality. The central motivation is that previous interpretability works often construct discrete dictionaries of attention-head functions through heuristic observation, while this work attempts to provide a more unified scientific principle. The paper explicitly positions itself as moving from individual token/head enumeration toward a macroscopic description of contextualization.
Critical strengths

Strength 1: A compelling global perspective on contextualization The biggest strength is the attempt to move beyond token-level or head-level observations toward a unified principle. The paper identifies a meaningful pattern: informative tokens tend to preserve representations, while less informative tokens absorb contextual information. The experiments across multiple token categories provide evidence that different words indeed have different levels of influence on model outputs. The authors measure this using output divergence against a calibrated background distribution.


Strength 2: Connecting phenomenology with mechanisms The paper does more than observe correlations. The singular-vector decomposition of attention parameters attempts to explain why the observed behavior happens. The authors argue that: • Q/K singular directions determine routing; • V/O directions determine writing magnitude; • singular vectors correspond to interpretable operational modes. This connection between observed contextualization behavior and attention parameter geometry is the most technically interesting part of the paper.

Critical weaknesses

Weakness 1: The definition of “information” is not sufficiently justified This is the most important concern. The paper repeatedly discusses: • information amount; • information-rich words; • information sources. However, the actual measurement models output sensitivity, not necessarily information content. A token can strongly influence output probability because of: • positional effects; • frequency bias; • tokenizer artifacts; • prompt-template compatibility; • memorization. The paper partially addresses tokenization effects, but the conceptual gap remains. The authors should avoid claiming that they measure “information” unless they provide a stronger information-theoretic justification.


Weakness 2: Correlation is sometimes interpreted as causation The paper argues: high-information words become information sources, low-information words absorb information. But many experiments demonstrate correlation. For example: • high-information words have more stable hidden states; • low-information words vary more across contexts. This does not necessarily prove a causal direction. Alternative explanations include:

  1. frequent tokens naturally have more contextual variance;
  2. rare proper nouns naturally activate specialized circuits;
  3. training distributions cause asymmetric representation stability. The paper needs stronger causal intervention experiments. The activation patching experiments are a step in this direction, but the causal interpretation remains incomplete.

Weakness 3: Limited scale of mechanistic validation The paper evaluates many models, but the detailed mechanistic analysis appears concentrated on smaller models such as Llama 3.2-1B. This raises concerns: • Does the singular-vector interpretation scale? • Do large models exhibit the same channel structure? • Are these mechanisms preserved after instruction tuning? A major claim of the paper is generality, but the strongest mechanistic evidence is relatively limited.


Weakness 4: AutoInterp evaluation is insufficient The paper uses AutoInterp to interpret singular vectors. However, the authors themselves acknowledge a problem: automatic interpretation systems can produce plausible explanations even for random features. Therefore, F1 score alone is not enough. A stronger evaluation would include: • human expert evaluation; • comparison against random directions; • comparison against SAE features; • causal intervention verifying the discovered feature.

Questions for authors

Q1: The paper defines word information through output divergence. Why should this quantity correspond to semantic information rather than merely output sensitivity? Could the authors compare their metric with established measures such as: • surprisal; • entropy reduction; • mutual information; • conditional entropy?


Q2: The proposed principle suggests a directional flow: high-information → low-information. Can the authors provide stronger causal evidence showing that changing the high-information token representation changes the low-information token more strongly than the reverse?


Q3: How stable are singular-vector interpretations across: • model scales; • training objectives; • instruction tuning? The paper should include larger-scale mechanistic validation.


Q4: How different is the proposed framework from existing circuit discovery methods? The authors criticize heuristic head naming, but many circuit methods already analyze distributed mechanisms rather than isolated heads. A more direct comparison would strengthen the positioning.

4 comments

Thank you for your review. We address your concerns as follows:


  1. Definition of Information. In this paper, information is simply a term we adopt for our explicitly defined quantity, measuring how far a word shifts the output distribution from the average distribution. This quantity can also be related to LM loss, as discussed in Appendix A. One could equally call it semantic specificity, saliency, or importance. We happen to use the term information, but our argument does not rely on information-theoretic definitions or derivations, nor do we claim to measure information in the information-theoretic sense. Therefore, we see no necessity for comparisons with information-theoretic quantities. If your primary concern is merely this terminology, we respectfully disagree that it constitutes a major weakness, since information does not exclusively refer to concepts from information theory.

  2. Causality. We made two improvements:

    • As described in Revision Note 2, we replaced our previous similarity-based metric with a more causally grounded metric based on attention-head outputs.
    • As described in Revision Note 1, we added a PatchScope-based experiment demonstrating that ablating the attention-head singular vectors identified and interpreted by AutoInterp produces semantic changes in hidden states consistent with their interpretations.
    • Furthermore, we believe our experiments already control for your proposed confounders. In particular, we use fixed input templates, eliminating differences in contextual variation across words. Note that rigorous variable control is itself a valid method of causal inference, without necessarily requiring explicit interventions. Your other two proposed explanations, namely that rare proper nouns activate specialized circuits and that training distributions produce asymmetric representation stability, describe possible implementations of our principle rather than alternatives that contradict it.
  3. Mechanistic validation across models. We extended our mechanistic analysis to additional models, as described in Revision Note 4.

  4. AutoInterp evaluation. As explicitly stated in our Discussion, developing alternatives to or improving AutoInterp is not the objective of this paper and is far beyond its scope. We fully acknowledge the limitations of AutoInterp, but these do not undermine our central contribution: demonstrating that attention-head operations can be interpreted through AutoInterp applied to their singular-vector decomposition. The quality of such interpretations naturally depends on the underlying AutoInterp method.

    • Human expert evaluation: Replacing AutoInterp with human experts would be impractical at this scale. Even highly automated AutoInterp evaluations cost several dollars per feature using inexpensive APIs, making equivalent human evaluation prohibitively demanding. Nevertheless, Figures 8 and 11 provide concrete case studies demonstrating how our AutoInterp interpretations causally affect attention scores and hidden-state semantics, allowing readers to evaluate these examples themselves.
    • Comparisons with random directions or SAE features: We consider these comparisons largely orthogonal to our contribution. Both would still rely on AutoInterp, while SAE features are defined in hidden-state space rather than directly through the attention parameter operators that we investigate.
  5. Ethics review flag. We also noticed that you selected “Flag For Ethics Review: Yes.” Could you please clarify the specific ethical concern motivating this flag?


We hope these explanations and revisions address your concerns, and we respectfully ask you to reconsider your rating.

Dear Reviewer GSy2,

As the end of the discussion period is just two days away, we would like to kindly ask if our response addresses your main concerns. If it does, we would greatly appreciate it if you could reconsider the score.

Thank you very much for dedicating your valuable time to our paper!

Authors.

Dear Reviewer GSy2,

As the end of the discussion period is just one day away, we would like to kindly ask if our response addresses your main concerns. If it does, we would greatly appreciate it if you could reconsider the score.

Thank you very much for dedicating your valuable time to our paper!

Authors.

Dear Reviewer GSy2,

As the discussion period ends today, we would like to make one final request for you to check whether our responses have addressed your concerns. If they have, we would greatly appreciate it if you could reconsider the score.

If we do not receive a response, we may consider asking the AC to mediate or assess whether these concerns have been adequately addressed.

Thank you for your time and consideration!

Authors.

Reviewer_GLZsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes that more informative words tend to serve as information sources, while less informative words absorb contextual information selectively. It investigates this hypothesis through representation analysis and activation patching, and uses singular directions of attention-head QK and OV operators for mechanistic analysis and automated interpretation.

Critical strengths
  1. A useful framework for analyzing multifunctional attention heads. Decomposing QK and OV operators into multiple directions offers a finer-grained description than assigning a single functional label to each head. Connecting these directions to automated feature interpretation is a promising application.
  2. An effort to connect behavioral observations with parameter structure. The combination of representation analysis, output measurements, and interventions provides several complementary perspectives on contextualization.
  3. Broad phenomenological evaluation. The experiments cover multiple model families and include tokenization-length and grammaticality controls. The paper also appropriately acknowledges that plausible automated explanations alone do not establish mechanistic validity.
Critical weaknesses
  1. The novelty requires clearer positioning. Prior work has already reported highly context-specific representations for stopwords using cross-context self-similarity [1], attention patterns corresponding to syntax and coreference [2], and quantitative relationships between word informativeness and embedding norms [3]. These findings do not establish the paper’s exact hypothesis, but the additional explanatory or predictive value should be clarified.

  2. The generality claim exceeds the cross-model evidence. Figures 50 and 54 show weak or negative correlations for Granite and Qwen3-8B, respectively. Granite also exhibits substantial negative correlations under the two-token control in Figure 76. These exceptions require explanation and a more qualified statement of the principle’s scope.

  3. The causal mechanism remains insufficiently established. Larger singular-direction projections do not necessarily imply greater realized information transfer, which also depends on attention weights and competing interactions. The importance of considering attention-weighted transformed vectors was established [4]. The presented ablations change attention patterns but do not directly demonstrate the transfer of identifiable semantic information or explain the aggregate contextualization relationship.

Key implementation details need clarification. The paper should specify how positional transformations and normalization are handled in the spectral analysis, and how candidate-answer losses are computed, including length handling and scoring for encoder-only models. These choices affect both reproducibility and the interpretation of the proposed measurements.

[1] How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. [2] What Does BERT Look at? An Analysis of BERT’s Attention. [3] Norm of Word Embedding Encodes Information Gain. [4] Attention is Not Only a Weight: Analyzing Transformers with Vector Norms.

Questions for authors

Seeing Weaknesses

3 comments

Thank you for your review. We address your concerns as follows:


  1. Novelty.

    • [1] only investigates how stopwords collect contextual information, without offering a more general conclusion about which words, beyond stopwords, tend to absorb contextual information and which tend to emit it (and its main claim is the anisotropy of hidden states instead of contextualization). Our work makes a substantial and novel advance in this direction.

    • [2] investigates attention patterns associated with syntax and coreference. The closest connection to our work is perhaps one case in Figure 6, but such attention patterns are not our focus. Moreover, Figure 6 employs causal activation patching rather than merely observing attention patterns, whose insufficiency you also acknowledge.

    • [3] is even less relevant, as our focus is not how informativeness is encoded in static embeddings, but how it influences dynamic contextualization.

  2. Counterexamples. The counterexamples observed in Granite and Qwen3-8B resulted from an implementation bug. After fixing it, we obtained substantially stronger correlations consistent with our hypothesis, eliminating these counterexamples. Please refer to Revision Note 3.

  3. Causality. We made two improvements:

    • As described in Revision Note 2, we replaced our previous similarity-based metric with a more causally grounded metric based on attention-head outputs.

    • As described in Revision Note 1, we added a PatchScope-based experiment demonstrating that ablating the attention-head singular vectors identified and interpreted by AutoInterp causes semantic changes in hidden states consistent with their interpretations.

  4. Experimental details. We have added the requested implementation details to the appendix. Please refer to Revision Note 6.


Thank you again for your review, if these additions address your concerns, we would appreciate it if you could reconsider your rating.

Thank you for your response! The rebuttal solved most of my questions, so I will raise my score to 3.

Thank you very much for your thoughtful comments and recognition of the improvements in our revisions! We sincerely appreciate your constructive suggestions, which have greatly enhanced this paper.

Reviewer_J7rkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is an empirical paper that studies how a language model's vector representation of a word changes with the surrounding text. The authors first insert each word into a collection of questions and compare the model's answer probabilities with those averaged over all tested words. This gives a word information score measuring how strongly the particular word affects the answers. They then place each word in different sentences and compare its vector representations using cosine similarity. In many tested settings, words with higher scores have more similar representations across sentences. The authors interpret this as evidence that words with more specific meanings, such as names, depend less on surrounding text than words such as pronouns.

To investigate the underlying mechanism, the authors perform SVD for both WQ⊤WKW_Q^\top W_K and WOWVW_OW_V. They examine correlations between word information scores and weighted projections onto the leading singular directions, interpreting these directions as features controlling the output of an LLM.

Critical strengths

I appreciate the authors' efforts to identify general patterns in how transformers process information across tokens. Their extensive experiments could help formulate concrete hypotheses to guide further theoretical study.

Critical weaknesses

From a theoretical perspective, I would appreciate further discussion of how the empirical findings might be explained mathematically. In particular, could the authors suggest a simplified model or explicit assumptions under which the observed relationships between their information score, hidden-state variation, and singular-vector projections would be expected to hold?

Questions for authors

The SVD analysis appears closely related to current research on steering language models. I would appreciate the authors' comments on this connection and on promising directions for future research, particularly from a theoretical perspective.

6 comments

Thank you for your review.

We have added a theoretical framework, as detailed in Revision Note 5. The framework can be summarized as follows:

  1. Collecting information from context allows an LM to reduce its language modeling loss by moving beyond 2-gram modeling toward higher-order n-gram modeling.
  2. Under an idealized Bayesian setting, we show that words with weaker semantic representations or less information are more likely to acquire information from context, and that this process is favored by the pre-training objective.
  3. Extending this idealized setting to our empirical observations, we explain why informative source words tend to produce longer attention writing vectors to context tokens, while less-informative destination words tend to receive longer attention writing vectors from context tokens.
  4. Finally, we explain why the directions carrying this information align with the singular vectors of the attention matrices.

Thank you again for your review, if these additions address your concerns, we would appreciate it if you could reconsider your rating.

Dear Reviewer J7rk,

As the end of the discussion period is just two days away, we would like to kindly ask if our response addresses your main concerns. If it does, we would greatly appreciate it if you could reconsider the score.

Thank you very much for dedicating your valuable time to our paper!

Authors.

Dear Reviewer J7rk,

As the end of the discussion period is just one day away, we would like to kindly ask if our response addresses your main concerns. If it does, we would greatly appreciate it if you could reconsider the score.

Thank you very much for dedicating your valuable time to our paper!

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Note

Dear PCs, SACs, ACs, and Reviewers,

We appreciate the reviewer’s feedback. To address the valuable comments raised by the reviewer, we have made the following adjustments to the paper.

  1. (Reviewer GLZs W3, GSy2 W2) Additional causal experiment. We conducted an additional causal intervention experiment to verify semantic information transfer for Fig. 10 in Fig. 11. Please refer to Line 462-Line 479 for details. In principle, remove the “Halloween” singular vector of Layer 11 Head 25 indeed reduce the semantic magnitude of “Halloween” in the later text.
  2. (Reviewer GLZs W3) Revised information transmission metric. We replaced the previous similarity-based metric, which primarily captured correlational relationships, with a more operational and causally grounded metric based on attention write vectors. This new metric allows us to characterize both information absorption and emission in a more causal style. Specifically, we measure absorption by the magnitude of attention write vectors from template tokens to the inserted token, and emission by the magnitude of write vectors from the inserted token to template tokens.
  3. (Reviewer GLZs W2) Bug fix and improved results. We corrected a bug in our experimental implementation (i.e., some of the inputs in the first “information content” experiment was not correctly calculated), yielding stronger results and eliminating the counterexamples identified by the reviewers.
  4. (Reviewer GSy2 W3) Expanded mechanistic analysis on larger models. In response to the reviewers’ requests, we extended our mechanistic analysis and AutoInterp evaluation to four additional, larger models. The results are consistent with our conclusions.
  5. (Reviewer J7rk) Theory. We proposed a theoretical framework in Appendix A to explain why language model training trigger the information transmission patterns observed in our experiments. A summary of the theory is provided in our responses to the reviewers.
  6. (Reviewer GLZs W4) Clarification of experimental details. In response to the reviewers’ requests, we clarified several implementation details and updated Appendix C accordingly.
  7. Corresponding writing revisions.
  8. (11/17 21:00 AOE) Revised a figure import error.

Again, thank you for the time and thoughtful reviews you devoted to this paper.

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.