- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper measures and characterizes the cosine distance between a context position's intermediate activation vectors and various competing final activation vectors (endpoints). The central quantity is, for each intermediate vector, the fraction of other contexts’ endpoints that lie closer to it than the context’s own endpoint. Many metrics are defined and reported based on this core idea. The authors do not provide a specific way or path that this kind of investigation could lead to a better understanding of frontier models, but they do show that distances between endpoints being small is (weakly) correlated with them making similar predictions. They show that the population of which endpoints are closer to the intermediate activations vectors over layers is not consistent with the residual trajectories being a simple linear interpolation towards the final vector. Additionally, they provide evidence that the distribution of endpoints is not well described by a uniform distribution on a a sphere by comparing this hypothesis against several candidate distribution models fit on FineWeb data.
Critical strengths
I believe this paper to be largely sound in its methods. The writing is reasonably clear, and the left panel of figure 1 gives a nice intuitive sense of their central object of study. The authors avoid overclaiming significance that is not earned as well.
Critical weaknesses
I feel that the central "hypothesis" is largely vacuous, even without paraphrasing-- it is stated as: "intermediate residual states express partial distinctions among possible final states and that residual updates refine those distinctions through changes in relative geometry". To paraphrase, it simply seems to state that "intermediate residual states get closer to their final values through the layers". This seems to be largely a tautology, and while there might be interesting phenomenology to explore from this idea, I think the authors fall short of this. The interesting contributions are in two quite short paragraphs: "A straight-path baseline for turnover" and potentially "Fitting endpoint distributions." as well. I imagine that a significant condensing of the rest of the manuscript and an extension of the mathematical and phenomenological analysis of the trajectories through the layers would provide a much more compelling set of results. In general, I think the authors would be well served by considering how this kind of analysis relates to general ideas about how transformers do computation throughout the layers-- and perhaps provide a mechanistic model or explanation for why the competitor count drops exponentially only near the final layer. Perhaps, even, this could be tied in with behavioral tests on LLMs or toy models. I believe there would be plenty of room for this, if the manuscript was condensed so as to spend less time delving into the
The organization of the paper seems to unnecessarily extend it's length, as well as to make simple ideas and arguments appear more complicated and confusing than they are. Figures 2,3,4 contain redundant panels that show similar behavior qualitatively; Each could be reduced to 1X3 figures, including figure -- which currently has 12 subplots. Additionally, much of the paper is spent defining and reporting metrics that appear to me to be very similar in nature and could likely be condensed into one or two definitions. For example, I imagine that defining the fraction in equation 8 much earlier, and then having the counts introduced as its expected value would provide a lot of help in both condensing and making the manuscript more understandable. Euclidean and Cosine distances are defined throughout, though the paper relies only on cosine in all the main figures. This is explained around equaition 7, which is more than halfway trough the paper. Instead, the authors could only define cosine distances at the beginning-- briefly mentioning the arguments in the "why directional progress can coexist with flat distance" section right there and relying on an appendix to talk in detail about why euclidean are not as informative here (this would simplify their notation as well, which to me was a bit difficult to follow. currently nearly all symbols carry an index for the distance metric which is rarely used).
Questions for authors
Why define both the Euclidean and cosine distances? I imagine this paper would flow much better with the Euclidean in the appendix.
Whats do the background bands mean in figure 2? I did not see it defined or explained.
Speaking of figure 2: the trend looks incredibly weak to me-- but it is presented as the only evidence that this is something worth studying. I'd say I was almost shocked at how insensitive the token predictions are to the distance between endpoints. I read this plot as mostly showing that, at the very least, the cosine distance isn't decoupled or inversely coupled to next token predictions. Perhaps this is because I am reading the plot wrong though. Is there a control value, reference band, or axis scale that would reveal this trend to be significant compared to other dependencies?
Figure 3's caption refers to "cosine counts". I find that term confusing. What does it mean?
Section 3 is where the main content of the manuscript seems to lie, but I find myself a bit confused about the unifying thread that ties it together. What is the core narrative of the series of investigations there? Can the manuscript be organized to make this more clear?
Where did the uniform sphere as a baseline come from? It seems obvious that this wont model things well, especially when compared to distributions that are fit to data. but perhaps I am missing a reason that this is a reasonable baseline to start with.
While the hypothesis being tested seems to me to be largely tautological-- I am aware that this may simply be a difference of research taste or background. Can you offer an alternative phrasing that helps me understand wether the central claim is more than "residual vectors eventually get closer to their own final values throughout the layers"?