- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes that representation geometry seen across models is driven by input and output geometry. The paper demonstrates that model embeddings and unembeddings share structure across models based on dot-product kernels. They also show that a 1-layer model can recover much of this structure suggesting that embedding and unembedding geometry is driven by simple statistics. They then attempt to attribute intermediate representation geometry to either the input or output or both.
Critical strengths
-
The use of kernel geometry and global metrics like CKA in addition to local metrics like mean neighborhood overlap help provide support for the findings about the shared geometry and ensure that the results are not simply due to high-variance directions.
-
The comparison with shallow models supports the idea that the embedding and unembedding geometry are primarily low-order statistics.
Critical weaknesses
- The main concern is the fact that it is unclear whether the key claim that representation geometry can be mapped back to the embedding or unembedding advances our understanding of model representations and is novel. The fact that representations towards the beginning have similarities with the initial embedding follows from the residual connections and the fact that the model must transform this embedding by adding on new components at each layer. Similarly, the fact that representations towards the end have similarities with the final unembedding follows from the fact that the model aims to eventually decode its representation into a token. As a result, it is unclear whether explaining intermediate representations in this manner provides new insights, especially when Figure 2 demonstrates that agreement with representations at either end is quite low in the middle of the network, and such relationships between low-order statistics and representations have been explored in works such as [1]. Additionally, the analysis is primarily focused on settings where the concepts/structures are primarily based on single words/tokens and may have limited applications beyond this scope. Furthermore, analysis such as the one in Section 5.1 seem to simply be a verification that the output representation more strongly appears after layer 18 which was shown to contain key addition neurons in previous work, and it does not seem to add any additional explanation for how these representations transformed beyond the fact that the model eventually needs to decode the answer.
[1] Symmetry in language statistics shapes the geometry of model representations, Dhruva Karkada, Daniel J Korchinski, Andres Nava, Matthieu Wyart, Yasaman Bahri
Questions for authors
- Could you provide clarification on how this work adds to the existing literature such as works using LogitLens or works that analyze the relationship between low-order statistics and representations?
5 comments
Author response (part 2)
Middle layers. While low-order statistics do not explain all representation geometry, we show that they explain a surprisingly large fraction of the geometry shared by different language models, without trying to explain the geometry that is not even shared by different LLMs. We have now made this clearer and added a new line to Figure 2 showing the agreement between the hidden representations of two different LLMs as a reference. Even taking this baseline into account, it is true that a larger fraction of geometry is unexplained in the middle layers. This unexplained geometry in middle layers that is shared across models is an interesting place to look for structure that appears with scale and reflects more sophisticated computation. We think this is another way in which our work advances our understanding of model representations by discarding structure that has simple explanations and allowing interpretability researchers to focus on structure that is not explained by the shape of the input or the shape of the answer.
Section 5.1. The goal of Section 5.1 is not to relocate the addition computation, which Feucht et al. (2026) identify at the layer-18 MLP. It is to explain an observation their account leaves open. Feucht et al. find that the period-12 month circle is absent where the computation happens: “we cannot reliably probe for circular structure at the final token position until layers 22-25 … This suggests that the causal mechanism at layer 18 does not rely on task-specific circular geometry.” If the circle is not used by the computation, why does it appear at all? We answer this by showing that the late circle is the shape of the answer: since the residual stream must produce month logits through the unembedding, it inherits the cyclic arrangement of the month unembedding rows. Consistent with this, answer geometry overtakes offset geometry around layer 19, and prompts are organised by answer month at layers 22 and 29 (Figure 3), matching the layers where Feucht et al. recover the circle. The location of the circle is therefore predicted by where the answer forms, not by where the computation happens.
We hope our response has alleviated the reviewer's concern with novelty and would allow them to increase their score. We remain available for the remainder of the rebuttal period to address any further questions or concerns.
References
- Engels, J., Michaud, E. J., Liao, I., Gurnee, W., & Tegmark, M. (2025). Not All Language Model Features Are One-Dimensionally Linear. International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=d63a4AM4hb
- Feucht, S., Haklay, T., Bhalla, U., Wurgaft, D., Rager, C., Sarfati, R., Merullo, J., McGrath, T., Lewis, O., Lubana, E. S., Fel, T., & Geiger, A. (2026). Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts. arXiv preprint arXiv:2605.01148. https://arxiv.org/abs/2605.01148
- Kantamneni, S., & Tegmark, M. (2025). Language Models Use Trigonometry to Do Addition. arXiv preprint arXiv:2502.00873. https://arxiv.org/abs/2502.00873
- Karkada, D., Korchinski, D. J., Nava, A., Wyart, M., & Bahri, Y. (2026). Symmetries in Language Statistics Shape the Geometry of Model Representations. International Conference on Machine Learning (ICML). https://arxiv.org/abs/2602.15029
- Modell, A., Rubin-Delanchy, P., & Whiteley, N. (2025). The Origins of Representation Manifolds in Large Language Models. arXiv preprint arXiv:2505.18235. https://arxiv.org/abs/2505.18235
- Park, C. F., Lee, A., Lubana, E. S., Yang, Y., Okawa, M., Nishi, K., Wattenberg, M., & Tanaka, H. (2025a). ICLR: In-Context Learning of Representations. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2501.00070
- Prieto, L., Stevinson, E., Barsbey, M., Birdal, T., & Mediano, P. A. M. (2026). From Data Statistics to Feature Geometry: How Correlations Shape Superposition. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2603.09972
- Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., et al. (2025). Open Problems in Mechanistic Interpretability. Transactions on Machine Learning Research (TMLR). https://arxiv.org/abs/2501.16496
- Wurgaft, D., Rager, C., Kowal, M., Shyam, V., Feucht, S., Bhalla, U., Haklay, T., Bigelow, E., Sarfati, R., McGrath, T., Lewis, O., Merullo, J., Goodman, N. D., Fel, T., Geiger, A., & Lubana, E. S. (2026). Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior. arXiv preprint arXiv:2605.05115. https://arxiv.org/abs/2605.05115
Thank you for the in-depth response and the clarifications! I appreciate the clarifications around the related works and section 5.1.
While I also appreciate the shared baseline to demonstrate the shared structure ceiling, I believe this remains a limitation of the input or output shape explanation itself.
The main concern that remains is that it is unclear whether the proposed idea goes beyond identifying correlations between the embedding/unembedding structure and structure from previous findings. While this provides a quantitative measure for identifying how similar the input/output structure is to intermediate embeddings, it remains unclear whether interventions on the shared parts of the representations result in the input or output-level representation shifting similarly and more generally, how it should be used as a baseline explanation. Furthermore, in Figure 3, it seems that the circular structure actually has more noise and explains less of the variance at layer 29 than at layer 25, which makes it uncertain how strongly the answer structure corresponds to the structure being studied such as the circular months.
I do appreciate all of the additional results and explanations and do believe the informative and soundness have been strengthened, and will raise my score from 1 to 2.
Author response
We thank the reviewer for their time and feedback. Below we make the case that our work is novel in providing a simple explanation of observed representation geometry that has predictive power over the geometry of early and late layers of real LLMs and is validated with quantitative metrics beyond specific examples like months or integers. We also argue against the notion that these results 'follow' from residual connections and note that connection to embedding and unembedding geometry was not considered as a hypothesis despite extensive work on this topic over the past 2 years.
We position our work with respect to two existing literatures:
1) Works that identify geometry/manifolds in LLM representations: Engels et al. (2025), Kantamneni & Tegmark (2025), Wurgaft et al. (2026), Feucht et al. (2026), Park et al. (2025a), Modell et al. (2025).
2) Works that link these structures to low-order statistics: Karkada et al. (2026), Prieto et al. (2026).
Relation to (1). None of the works in (1) consider the embedding or unembedding geometry as an explanation for the structures they find, even those that focus on early and late layers. They focus instead on what the geometry might do computationally, or on what it implies for the linear representation hypothesis and for steering. Wurgaft et al. explicitly leave the origin of the representation–behaviour correspondence outside their scope. Understanding where such geometry comes from is listed among the main open problems of mechanistic interpretability (Sharkey et al., 2025).
Two of our novel empirical findings from this work make the connection to embedding and unembedding geometry worth testing:
a) Most of the previously studied cases of representation geometry/manifolds in the hidden representations of LLMs are already present in the embedding and unembedding.
b) The same structures are also present in the embedding and unembedding of 1-layer models.
Why this does not simply follow from residual connections and LogitLens.
With respect to the works in (2), we believe our work makes several important contributions. While Karkada et al. (2026) and Prieto et al. (2026) establish a connection between low-order statistics and representations, their main objects of study are classical word embeddings like word2vec (for Karkada et al., 2026) and shallow autoencoders (for Prieto et al., 2026). Their connection to language models is limited to noting that a few of the same structures in representations of specific concepts like months or years look qualitatively similar between LLMs and word embeddings. We build on this by showing that even the embeddings and unembeddings of LLMs with billions of parameters are driven by similar statistics and measure this agreement using quantitative metrics like CKA across the full vocabulary, beyond specific word groups like months or days. While works in (2) show the appearance of similar structures to (1) in simpler settings, we replicate the results from (1) and show that they can be mapped to the shape of the input or the shape of the answer in real LLMs.