acceptodds
Under review as a conference paper at ICLR 2027

The Geometry of Inference in Transformer Residual Streams

Abstract

Transformer language models build predictions through successive residual updates, but how their representations become specific to an eventual outcome remains unclear. We study this process by comparing intermediate residual states with their own final states and an empirical bank of final states from other contexts. Across six pretrained language models, the own endpoint becomes preferable to the average alternative early, while many individual endpoints remain closer. These competing sets generally shrink with depth, but their membership changes and their surviving endpoints need not become more similar to one another. Directional alignment and endpoint rank can therefore improve while Euclidean distance to the final state changes little. We develop a simple high-dimensional model that separates the roles of norm, alignment, and endpoint geometry, showing how gradual directional changes can produce sharp reductions in competition. We also prove that a straight path toward the own endpoint cannot introduce new competitors under either Euclidean or cosine distance; observed entries thus establish departures from straight-line convergence. Finally, endpoints associated with lower-ranked output tokens tend to lie farther away in cosine distance across all studied models, connecting residual geometry to output organization. Together, these findings characterize increasing geometric specificity during transformer inference and explain why distance, competitor count, and concentration of the surviving endpoints provide distinct views of that process.

Primary area:
foundation or frontier models, including LLMs
Keywords:
geometry, LLM, explainability, probability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KMS9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper measures and characterizes the cosine distance between a context position's intermediate activation vectors and various competing final activation vectors (endpoints). The central quantity is, for each intermediate vector, the fraction of other contexts’ endpoints that lie closer to it than the context’s own endpoint. Many metrics are defined and reported based on this core idea. The authors do not provide a specific way or path that this kind of investigation could lead to a better understanding of frontier models, but they do show that distances between endpoints being small is (weakly) correlated with them making similar predictions. They show that the population of which endpoints are closer to the intermediate activations vectors over layers is not consistent with the residual trajectories being a simple linear interpolation towards the final vector. Additionally, they provide evidence that the distribution of endpoints is not well described by a uniform distribution on a a sphere by comparing this hypothesis against several candidate distribution models fit on FineWeb data.

Critical strengths

I believe this paper to be largely sound in its methods. The writing is reasonably clear, and the left panel of figure 1 gives a nice intuitive sense of their central object of study. The authors avoid overclaiming significance that is not earned as well.

Critical weaknesses

I feel that the central "hypothesis" is largely vacuous, even without paraphrasing-- it is stated as: "intermediate residual states express partial distinctions among possible final states and that residual updates refine those distinctions through changes in relative geometry". To paraphrase, it simply seems to state that "intermediate residual states get closer to their final values through the layers". This seems to be largely a tautology, and while there might be interesting phenomenology to explore from this idea, I think the authors fall short of this. The interesting contributions are in two quite short paragraphs: "A straight-path baseline for turnover" and potentially "Fitting endpoint distributions." as well. I imagine that a significant condensing of the rest of the manuscript and an extension of the mathematical and phenomenological analysis of the trajectories through the layers would provide a much more compelling set of results. In general, I think the authors would be well served by considering how this kind of analysis relates to general ideas about how transformers do computation throughout the layers-- and perhaps provide a mechanistic model or explanation for why the competitor count drops exponentially only near the final layer. Perhaps, even, this could be tied in with behavioral tests on LLMs or toy models. I believe there would be plenty of room for this, if the manuscript was condensed so as to spend less time delving into the

The organization of the paper seems to unnecessarily extend it's length, as well as to make simple ideas and arguments appear more complicated and confusing than they are. Figures 2,3,4 contain redundant panels that show similar behavior qualitatively; Each could be reduced to 1X3 figures, including figure -- which currently has 12 subplots. Additionally, much of the paper is spent defining and reporting metrics that appear to me to be very similar in nature and could likely be condensed into one or two definitions. For example, I imagine that defining the fraction in equation 8 much earlier, and then having the counts introduced as its expected value would provide a lot of help in both condensing and making the manuscript more understandable. Euclidean and Cosine distances are defined throughout, though the paper relies only on cosine in all the main figures. This is explained around equaition 7, which is more than halfway trough the paper. Instead, the authors could only define cosine distances at the beginning-- briefly mentioning the arguments in the "why directional progress can coexist with flat distance" section right there and relying on an appendix to talk in detail about why euclidean are not as informative here (this would simplify their notation as well, which to me was a bit difficult to follow. currently nearly all symbols carry an index for the distance metric which is rarely used).

Questions for authors

Why define both the Euclidean and cosine distances? I imagine this paper would flow much better with the Euclidean in the appendix.

Whats do the background bands mean in figure 2? I did not see it defined or explained.

Speaking of figure 2: the trend looks incredibly weak to me-- but it is presented as the only evidence that this is something worth studying. I'd say I was almost shocked at how insensitive the token predictions are to the distance between endpoints. I read this plot as mostly showing that, at the very least, the cosine distance isn't decoupled or inversely coupled to next token predictions. Perhaps this is because I am reading the plot wrong though. Is there a control value, reference band, or axis scale that would reveal this trend to be significant compared to other dependencies?

Figure 3's caption refers to "cosine counts". I find that term confusing. What does it mean?

Section 3 is where the main content of the manuscript seems to lie, but I find myself a bit confused about the unifying thread that ties it together. What is the core narrative of the series of investigations there? Can the manuscript be organized to make this more clear?

Where did the uniform sphere as a baseline come from? It seems obvious that this wont model things well, especially when compared to distributions that are fit to data. but perhaps I am missing a reason that this is a reasonable baseline to start with.

While the hypothesis being tested seems to me to be largely tautological-- I am aware that this may simply be a difference of research taste or background. Can you offer an alternative phrasing that helps me understand wether the central claim is more than "residual vectors eventually get closer to their own final values throughout the layers"?

Reviewer_7oT9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a "geometric inference hypothesis": intermediate residual states express partial distinctions among possible final states, and residual updates refine these through changes in relative geometry. It operationalizes this with a fixed endpoint bank — the final residual states (before final norm) of 1,024 FineWeb contexts — per model, across six pretrained LMs (Gemma-2B/7B, Qwen2.5-1.5B/7B, Mistral-7B, Llama-3-8B). For each context, an alternative endpoint is a competitor at layer ℓ if it is closer (Euclidean or cosine) to the current state than the own endpoint; the competitor fraction measures geometric specificity.

Empirical findings: (1) cosine endpoint distance increases with the final-rank index of the top token (Spearman ρ = 0.73–0.98, one-sided p small; Fig. 2), linking endpoint geometry to output organization; (2) the document-averaged margin favors the own endpoint from the earliest layers, yet many individual competitors remain (Fig. 3); (3) cosine competitor sets shrink with depth but with genuine membership turnover (entries as well as exits; Fig. 4), and cosine competition can fall while Euclidean distance plateaus. Theory: margin-update equations (Eq. 6) explain simultaneous gain/loss of competitors; a proposition shows straight-line convergence yields nested competitor sets under both metrics, so observed entries establish departures from straight-line convergence; a high-dimensional shared-axis model (Eqs. 9–11) shows how small alignment gains can sharply cut competitor count and how counts can fall at constant Euclidean distance. Finally, endpoint distributions (sphere/cap/vMF/vMF-mixture/projected-normal) fitted only to final states predict cosine competitor fractions along held-out trajectories, with projected-normal best in 5/6 models and both anisotropic models beating the uniform-sphere baseline (Fig. 5).

Critical strengths
  1. A rigorous, falsifiable framework linking final-state geometry to intermediate refinement. The combination of a fixed endpoint bank, the competitor-set definition, and the straight-line-convergence nestedness proposition gives the paper real analytical teeth: the observed cosine entries (Fig. 4) are not just a qualitative curiosity but a proven departure from the simplest convergence baseline. This is a meaningful advance over purely descriptive depth-wise analyses and is consistent across six architectures.
  2. Predictive validation from population geometry. Showing that endpoint distributions fit only to final states predict competitor fractions along held-out trajectories (projected-normal best in 5/6; anisotropic >> uniform sphere, Fig. 5) is a strong test: it converts the geometric story into a quantitative, generalizing model rather than a post-hoc narrative, and it connects to known residual-stream anisotropy (Ethayarajh 2019). The rank–distance association (Fig. 2, ρ up to 0.98) independently grounds the premise that endpoint geometry tracks output organization.
Critical weaknesses
  1. Descriptive, not functional — no causal/interventional evidence. The paper characterizes geometry but never shows it matters for model behavior. The competitor fraction is explicitly not a calibrated probability, the bank is a modeling choice, and the set reaches zero at the own endpoint by construction. The authors themselves note that "interventions near competitor-margin crossings could test whether changes in geometric preference affect token predictions" — but no such intervention is run. The predictive test (Fig. 5) does not close this gap: it predicts the competitor fraction — itself a geometric quantity — from geometry, so it validates geometry→geometry, not geometry→behavior. On close reading of Fig. 5 the parametric families (cap/vMF/vMF-mixture/projected-normal) also nearly coincide, so "projected-normal best in 5/6" is a marginal ranking; the robust, interpretable signal there is only the anisotropic-vs-uniform-sphere gap. Without a behavioral test, the practical significance of "competitor turnover" for interpretability or control remains conjectural, which is the main barrier to impact.
  2. Scope and construct dependence. All results use FineWeb, 1,024 contexts of length 256, with the endpoint taken before final normalization; competitor sets and entries/exits depend on the chosen bank, metric, and these preprocessing decisions. The paper shows robustness to subsampling (Appendix J) and some controls, but not to corpus/domain, context length, sequence position (only the final position is used), or the before-vs-after-final-norm choice. It is therefore hard to know how much of the geometry reflects the model's inference process versus properties of this particular measurement setup.
Questions for authors
  1. Can you provide even a small causal test — e.g. nudging a state across a competitor-margin crossing (Eq. 6 direction) and measuring the effect on the output distribution — to show that geometric competition has behavioral consequences rather than being an epiphenomenon?
  2. How sensitive are the central phenomena (early average preference, cosine turnover, Euclidean plateau with directional progress) to: (a) corpus/domain beyond FineWeb; (b) context length; (c) non-final token positions; (d) measuring the endpoint after final normalization?
  3. The rank–distance trend (Fig. 2) uses one-sided tests with n = 20 rank bins. How robust is it to the number of bins and to using per-token (not per-bin-mean) distances, and does it hold for lower-frequency tokens?
  4. Competitor membership turnover counts entries without distinguishing first-time entries from returns. Can you quantify how much turnover is genuine re-ordering vs oscillation near the margin?
  5. How does the endpoint-bank competitor fraction relate quantitatively to tuned-lens / logit-lens top-1 agreement at the same layers — does geometric specificity track decodability?
  6. A concurrent arXiv paper, Predictive Geometry of Hidden Trajectories in Transformers (arXiv:2609.37717), appeared the same day on a closely related theme. Can you contrast your endpoint-bank / competitor-set framework with its trajectory-predictability framing, so the distinct contribution is clear?
Reviewer_bgJSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes investigating the dynamics of getting to the final residual layer across the transformer layers through the residual layer states

The paper calculates the competitor set of an input i -- the set of all inputs whose residual layer states are closer to the correct endpoint (the final residual layer state) than the the residual layer state of input i itself.

As is mathematically expected, at the final layer, the competitor set shrinks to the input i itself. The paper explores the dynamics of the competitor set through the layers.

The paper shows that the final representation is favored surprisingly early; the number of competitors generally (but not always) decreases with depth; directional refinement can occur without much reduction in Euclidean distance; final-state geometry correlates with token predictions.

Theoretical results show that those empirical observations can arise

Critical strengths

As far as I know, the observations as framed is novel

Critical weaknesses

It is difficult to see that the observation is framed is not in many ways trivial.

Mathematically, the competitor set will start out large and end up of size 1. One might imagine proving some kinds of regularities about in the abstract, but those weren't really observed.

Questions for authors

What are surprising findings in this paper/findings that could be used to advance the understnading of Transformers even if they are not surprising?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.