acceptodds
Withdrawn from ICLR 2027

Why Language Models Disambiguate Contexts Despite Similar Embeddings

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language models present an apparent paradox where the representations of a token across distinct contexts exhibit very high cosine similarity, and yet they cleanly disambiguate contexts. Here, we analyze the tangent space dynamics of the embedding and unembedding spaces, to understand the influence of context and layer depth on the output. We introduce a two-fold decomposition that separates representations into (i) a shared component common across contexts and its context-specific orthogonal complement, and (ii) readout-sensitive and readout-insensitive subspaces determined by their direct linear projection through the unembedding matrix. From large scale empirical studies on 16 models of 160M to 9B parameters, we observe that the shared context-invariant subspace is high-dimensional, occupying a large fraction of the ambient embedding space. The readout is dominated by the component shared across contexts, which is why representations across contexts look nearly identical. Separation survives because the shared components cancel and the contrast component is enriched tenfold in the subspace the readout acts on.

Primary area:
interpretability and explainable AI
Keywords:
Transformer Interpretability, Contextual Disambiguation, Representation Geometry, Tangent Space Dynamics, Readout-Sensitive Subspaces

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_v3M6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks why language models can distinguish different meanings of a word even when their hidden vectors are similar. The authors split the average vectors for two meanings into a shared part and a difference vector. They then study how these parts relate to the model’s output weights. The experiments cover 16 models and examine changes across layers and context lengths. The paper reports that the difference vector is more aligned with output-sensitive directions than a random vector, and that the hidden states become more aligned with these directions at the final layer.

Critical strengths
  1. The paper gives a useful way to study why similar vectors can still carry different meanings. Looking at both the shared part and the difference between the vectors makes this question easier to understand. The analysis also connects these differences to the model’s own output weights.

  2. The experiments include 16 models with different sizes and architectures. The authors compare different layers and context lengths and use both linear probes and ablations. This gives a broader view than results from a single model or measurement.

Critical weaknesses
  1. Equation (1) holds when the two sense centroids have equal norms, but the paper only gives numerical evidence that this is a reasonable assumption. Section 5.3 correctly calls the equation an identity. However, Section 6.1 describes it as predicting how sense separation changes across layers. Both sides are calculated from the same pair of vectors, so their agreement mainly shows that the equal norm approximation works. It does not explain why the vectors change across layers. The authors should show how close the two norms are, or gives explanation of when this approximation is valid and how unequal norms affect it.

  2. The result in Figure 2 is largely expected. With more context, the target token receives more clues about its meaning, so the two groups of representations may become easier to separate. It is unclear what this result tells us beyond that. The experiment also changes input length, topic information, and word sense clues together. A comparison with equally long but irrelevant context would help show which factor matters.

  3. The introduction says that the final-layer sense centroids have cosine similarity above 0.99 across all 16 models. Table 6 shows otherwise. Only GPT-2 and Pythia-160M exceed 0.99. Figure 2 also shows values of r above 1, which would not give cosine similarity close to 1 under the equal norm assumption.

Questions for authors

See the weakness.

Reviewer_uSnBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why representations of the same polysemous word in contexts corresponding to different senses exhibit high cosine similarity while still allowing the senses to be distinguished. The authors decompose the sense centroids into a shared component and a sense-distinguishing component, and analyze how these components align with a readout-sensitive subspace. Through empirical analyses of 16 models, the paper evaluates how sense separation varies with context length and model depth, and how these components relate to the readout. The results show that sense contrasts are more concentrated in the readout-sensitive subspace than random directions are on average. High cosine similarity can be explained by the relative magnitude of the shared component and does not necessarily imply a loss of sense-distinguishing information.

Critical strengths
  1. The paper provides a useful analysis of how highly similar representations can retain sense-distinguishing information. By separating shared and sense-distinguishing components, it clarifies how a large shared component relative to the contrast can obscure differences under cosine similarity. The authors also relate these components to the readout geometry to further connect representation similarity to output sensitivity.

  2. The layerwise analysis distinguishes sense separation from alignment with the final readout. Sense distinctions emerge before the final layer, while the largest increase in hidden-state alignment with the readout-sensitive subspace occurs at the final layer across all 16 models. This consistent observation suggests that forming a sense distinction and aligning the representation with the final readout are distinct aspects of processing.

Critical weaknesses
  1. The evidence for terminal sense-readout alignment remains indirect. Section 5 shows that the sense-distinguishing component is enriched in the readout-sensitive subspace at the final layer. Appendix Q shows that sense distinctions emerge earlier. However, Section 6 measures the alignment for full hidden states across all token positions, rather than the sense-distinguishing component itself. These results do not establish that sense-readout alignment specifically emerges in a single terminal step. Tracking the sense-distinguishing component across layers would clarify whether the terminal increase reflects sense-specific alignment.

  2. The empirical coverage is broad across models but limited across linguistic contexts. The analysis evaluates 16 models using the same controlled set of 226 sentences covering 11 polysemous words. While the limitation discusses the low statistical power of across-word correlations, the robustness of the main findings to different sets of words and more varied contexts remains unclear.

Questions for authors
  1. Appendix L acknowledges that general logit movement is not sense-specific. Is there evidence that removing the sense-distinguishing component changes the relative probabilities of continuation that are consistent versus inconsistent with the intended sense? This would help clarify how the geometric findings relate to the model's actual disambiguation behavior.

  2. The introduction states that the final-layer sense centroids have cosine similarity above 0.99 in all sixteen models. However, Table 6 reports final-layer values between 0.78 and 0.94 for most models. Could the authors clarify which measurement the result refers to?

Reviewer_yinpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates how language models distinguish word senses when their contextual representations have high cosine similarity. It decomposes sense centroids into a shared component and a sense-specific contrast, then examines their alignment with a subspace defined by the model’s readout operator. Using sixteen decoder-only models and 226 sentences covering 11 polysemous words, the authors study how these geometric properties vary with context length and network depth. They report that additional context increases sense separation, that sense contrasts are enriched in the selected readout subspace, and that alignment emerges abruptly at the final layer. The paper further relates centroid similarity to the relative magnitudes of the shared and contrast components, proposing a geometric explanation for the coexistence of high similarity and distinguishable sense information.

Critical strengths

I appreciate the paper’s focus on the limitations of representation similarity, particularly its examination of how representations with high cosine similarity can still encode distinguishable word senses.

Critical weaknesses

There are three main problems: (1) the claimed final-layer effect is not robustly established; (2) the experimental setup does not cleanly isolate contextual disambiguation; and (3) the geometric analysis does not establish the proposed causal mechanism.

First, the paper argues that readout alignment emerges abruptly at the final layer across all models (Section 6), but its normalization checks do not consistently support this claim. Table 11 places GPT-2’s largest increase at layer 10 rather than 12 under an alternative convention, while the Gemma models shift to the penultimate layer. This contradicts Appendix K’s claim that the location is consistent across conventions. The implementation in the supplementary material provided also handles normalization inconsistently across layers, leaving uncertainty about how much of the terminal effect reflects learned computation versus the measurement procedure.

Second, the evaluation does not consistently satisfy its own requirements for contextual disambiguation. Appendix P.2 states that distinguishing information precedes the target, yet the stimulus bank includes different sense labels attached to identical visible prefixes—for example, bird and machine examples both present "The crane" at the evaluated position ("The crane waded through the shallow marsh." versus "The crane lifted the steel beam into place."). The intended distinction is therefore unavailable to the model. Longer-context conditions also exclude sentences with insufficient preceding text, changing the evaluated sample. Consequently, the reported trends combine additional context with changes in the examples being measured, weakening their interpretation as evidence of improved semantic disambiguation.

Third, the geometric results provide limited independent support for the proposed mechanism. Section 5.3 acknowledges that the central relationship is an algebraic identity, while the ablations in Table 2 remove directions derived from sense-centroid differences and then measure reductions in that same separation. The outcome is therefore partly expected from the experiment’s construction. Appendix L further acknowledges that general logit movement is not sense-specific. These findings describe representation geometry, but stronger causal claims require evidence that the identified components affect sense-appropriate model predictions on held-out examples.

Questions for authors
  1. What evidence shows that the final-layer effect comes from the last layer itself rather than the final normalization step?

  2. How could the model distinguish different meanings when the text it sees is identical up to the target word, such as “The crane”?

  3. After removing the selected component, did you recalculate the model’s output with normalization, as described in Appendix P.4?

  4. How do you explain the claim of final-layer cosine similarity above 0.99 for every model given the lower values in Table 6?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.