- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the general mechanism of contextualization in Transformer language models. Instead of analyzing individual attention heads or token interactions through heuristic discovery, the authors propose a global principle: Information tends to flow from high-information words to low-information words, where low-information words absorb contextual information through selective routing. The paper decomposes this principle into three hypotheses:
- Static Information Hypothesis Different words contain different amounts of intrinsic information before contextualization.
- Information Gradient Hypothesis High-information words tend to remain stable and act as information sources, while low-information words are more strongly rewritten by context.
- Selective Contextualization Hypothesis Information transmission is not uniform; semantic compatibility determines which tokens exchange information. The authors validate these hypotheses empirically by measuring token information gain, hidden-state variation, and activation patching effects. They further provide a mechanistic interpretation by decomposing attention matrices through singular vectors and viewing attention heads as gated information channels. The paper argues that singular vectors correspond to interpretable latent features and provide a continuous representation of attention-head functionality. The central motivation is that previous interpretability works often construct discrete dictionaries of attention-head functions through heuristic observation, while this work attempts to provide a more unified scientific principle. The paper explicitly positions itself as moving from individual token/head enumeration toward a macroscopic description of contextualization.
Critical strengths
Strength 1: A compelling global perspective on contextualization The biggest strength is the attempt to move beyond token-level or head-level observations toward a unified principle. The paper identifies a meaningful pattern: informative tokens tend to preserve representations, while less informative tokens absorb contextual information. The experiments across multiple token categories provide evidence that different words indeed have different levels of influence on model outputs. The authors measure this using output divergence against a calibrated background distribution.
Strength 2: Connecting phenomenology with mechanisms The paper does more than observe correlations. The singular-vector decomposition of attention parameters attempts to explain why the observed behavior happens. The authors argue that: • Q/K singular directions determine routing; • V/O directions determine writing magnitude; • singular vectors correspond to interpretable operational modes. This connection between observed contextualization behavior and attention parameter geometry is the most technically interesting part of the paper.
Critical weaknesses
Weakness 1: The definition of “information” is not sufficiently justified This is the most important concern. The paper repeatedly discusses: • information amount; • information-rich words; • information sources. However, the actual measurement models output sensitivity, not necessarily information content. A token can strongly influence output probability because of: • positional effects; • frequency bias; • tokenizer artifacts; • prompt-template compatibility; • memorization. The paper partially addresses tokenization effects, but the conceptual gap remains. The authors should avoid claiming that they measure “information” unless they provide a stronger information-theoretic justification.
Weakness 2: Correlation is sometimes interpreted as causation The paper argues: high-information words become information sources, low-information words absorb information. But many experiments demonstrate correlation. For example: • high-information words have more stable hidden states; • low-information words vary more across contexts. This does not necessarily prove a causal direction. Alternative explanations include:
- frequent tokens naturally have more contextual variance;
- rare proper nouns naturally activate specialized circuits;
- training distributions cause asymmetric representation stability. The paper needs stronger causal intervention experiments. The activation patching experiments are a step in this direction, but the causal interpretation remains incomplete.
Weakness 3: Limited scale of mechanistic validation The paper evaluates many models, but the detailed mechanistic analysis appears concentrated on smaller models such as Llama 3.2-1B. This raises concerns: • Does the singular-vector interpretation scale? • Do large models exhibit the same channel structure? • Are these mechanisms preserved after instruction tuning? A major claim of the paper is generality, but the strongest mechanistic evidence is relatively limited.
Weakness 4: AutoInterp evaluation is insufficient The paper uses AutoInterp to interpret singular vectors. However, the authors themselves acknowledge a problem: automatic interpretation systems can produce plausible explanations even for random features. Therefore, F1 score alone is not enough. A stronger evaluation would include: • human expert evaluation; • comparison against random directions; • comparison against SAE features; • causal intervention verifying the discovered feature.
Questions for authors
Q1: The paper defines word information through output divergence. Why should this quantity correspond to semantic information rather than merely output sensitivity? Could the authors compare their metric with established measures such as: • surprisal; • entropy reduction; • mutual information; • conditional entropy?
Q2: The proposed principle suggests a directional flow: high-information → low-information. Can the authors provide stronger causal evidence showing that changing the high-information token representation changes the low-information token more strongly than the reverse?
Q3: How stable are singular-vector interpretations across: • model scales; • training objectives; • instruction tuning? The paper should include larger-scale mechanistic validation.
Q4: How different is the proposed framework from existing circuit discovery methods? The authors criticize heuristic head naming, but many circuit methods already analyze distributed mechanisms rather than isolated heads. A more direct comparison would strengthen the positioning.
4 comments
Dear Reviewer GSy2,
As the end of the discussion period is just two days away, we would like to kindly ask if our response addresses your main concerns. If it does, we would greatly appreciate it if you could reconsider the score.
Thank you very much for dedicating your valuable time to our paper!
Authors.
Dear Reviewer GSy2,
As the end of the discussion period is just one day away, we would like to kindly ask if our response addresses your main concerns. If it does, we would greatly appreciate it if you could reconsider the score.
Thank you very much for dedicating your valuable time to our paper!
Authors.
Dear Reviewer GSy2,
As the discussion period ends today, we would like to make one final request for you to check whether our responses have addressed your concerns. If they have, we would greatly appreciate it if you could reconsider the score.
If we do not receive a response, we may consider asking the AC to mediate or assess whether these concerns have been adequately addressed.
Thank you for your time and consideration!
Authors.
Thank you for your review. We address your concerns as follows:
Definition of Information. In this paper, information is simply a term we adopt for our explicitly defined quantity, measuring how far a word shifts the output distribution from the average distribution. This quantity can also be related to LM loss, as discussed in Appendix A. One could equally call it semantic specificity, saliency, or importance. We happen to use the term information, but our argument does not rely on information-theoretic definitions or derivations, nor do we claim to measure information in the information-theoretic sense. Therefore, we see no necessity for comparisons with information-theoretic quantities. If your primary concern is merely this terminology, we respectfully disagree that it constitutes a major weakness, since information does not exclusively refer to concepts from information theory.
Causality. We made two improvements:
Mechanistic validation across models. We extended our mechanistic analysis to additional models, as described in Revision Note 4.
AutoInterp evaluation. As explicitly stated in our Discussion, developing alternatives to or improving AutoInterp is not the objective of this paper and is far beyond its scope. We fully acknowledge the limitations of AutoInterp, but these do not undermine our central contribution: demonstrating that attention-head operations can be interpreted through AutoInterp applied to their singular-vector decomposition. The quality of such interpretations naturally depends on the underlying AutoInterp method.
Ethics review flag. We also noticed that you selected “Flag For Ethics Review: Yes.” Could you please clarify the specific ethical concern motivating this flag?
We hope these explanations and revisions address your concerns, and we respectfully ask you to reconsider your rating.