- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper investigates whether correspondences between hidden representations, plausibly of semantically equivalent inputs, also bring to similar representation trajectories across layers. The authors study affine transformations between hidden states at a given layer and between successive representations . They apply this analysis to musical notation, and arithmetic tasks, they explore the extent to which these correspondences can predict the effects of interventions.
Critical strengths
The paper may contain interesting results, and addresses an important problem, but its conceptual contribution is not communicated with sufficient clarity to allow a meaningful assessment of its significance. A substantial rewriting, starting from the abstract and introduction, is in my opinion necessary.
Critical weaknesses
I found the abstract and introduction exceptionally difficult to follow. The authors introduce the notion of a "correspondence between computations" without providing a clear qualitative explanation of what this means, or how it differs from the correspondence between hidden-state trajectories. Since the transformations of hidden states are fully specified by the architecture and trained weights, the meaning of a correspondence between computations needs to be explained much more carefully. The reader should be provided with a qualitative idea of the goal of the research before jumping into formal derivations.
Moreover, the introduction introduces mathematical objects such as "domains", hidden states, and affine correspondence maps without adequately defining them or illustrating their meaning with a concrete example. A simple example involving two alternative representations would make the problem considerably more accessible.
Questions for authors
I think that the papers unclear at a level that cannot be addressed by revising the paper. I suggest a careful and in depth revision aimed at explaining the problem in qualitative and intuitive terms. `
4 comments
Dear Reviewer ZrEV, thank you for your candid review. Following your suggestion, we rewrote the abstract and introduction from scratch, starting with a qualitative explanation and a concrete example before any formalism (pages 1-3, marked in red in the revised PDF). Even a quick look at these pages would help us greatly. We would appreciate your view on whether the paper is now clear enough to assess.
Thanks very much for your reply. Indeed I will be happy to read the paper in its revised version, but I think that such a revision cannot be performed in such a short time, by adding a few paragraphs. This, in my opinion, a clear example of a potentially interesting paper which requires a complete redesign to make it understandable to human readers, avoiding as much as possible the usage of AI.
We take the point about readability seriously, and we partly share it. The submitted introduction moved to formalism too quickly. AI-assisted writeup also tires the readers.
That is why the revision did not add paragraphs to the old introduction but replaced it. Pages 1-3 were rewritten by the author from a blank page, starting from the "7" vs. "seven" example and the question of whether a map that aligns two states also aligns what each layer does to them; AI tools were used afterwards only for grammar. Sec. 3.1 and the table in Appendix A define every symbol. Three further sections report the controls requested by the other reviewers. The revised PDF has been available since Nov 9 (AoE), and the final version since Nov 13. We understand that the discussion window is short. If you are able to read only pages 1-3, that would already tell us whether the rewrite succeeds. And if the thread is still lost, it would help us most to know at which step: the running example, the definition of a correspondence, or the step from state correspondence to update correspondence. That is the kind of feedback we could act on.
Thank you for your frank assessment. We agree that the introduction moves to formalism too quickly, and we are rewriting it. Below is the qualitative explanation that will open the revised introduction.
The idea. Take two ways of writing the same concept, e.g. the number written as "7" and as "seven". At every layer, the hidden states of the two inputs can be related by a map, which we call a correspondence. Such maps are often found to be accurate, and accurate maps are commonly read as evidence that the model "treats the two inputs the same way". We ask whether that inference is justified. As the model processes the two inputs layer by layer, are the changes it makes to each input also related by the correspondence? If so, the model performs corresponding computations on the two inputs, not merely holding similar representations of them at some layer.
On "the transformations are fully specified by the weights": yes. Each transformer block computes the same function for both inputs. What differs is where in state space the two inputs lie, and the same block can act very differently in different regions. A correspondence between computations therefore means that the block commutes with the correspondence (Eq. 2): mapping a state and then running the block gives the same result as running the block and then mapping.
Why it matters. This gives two practical answers. (1) When can we trust that a model computes the same thing on equivalent inputs? Neither similar representations nor identical answers are sufficient. A correspondence that aligns states at 0.954 transports their updates at only 0.096. Three arithmetic phrasings are all answered correctly, but only two of them share a computational trajectory. (2) Where in the network does the information that carries a computation live? Not in any single layer's geometry, nor in one map shared across depth. A correspondence that transports the updates of one layer segment (0.60) fails one layer away (0.11 / 0.27), and one map for all middle layers reaches only 0.23.
Concretely, the revision will include: