- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper presents a gauge theory framing of attention and applies it to understand what happens to the inner products in attention if the vectors are first tranformed into different coordinate systems by two different maps. I.e., each "entity" (vector) is in its own "frame" (coordinate system). Entity representations are defined only up to independent rotations , with standard attention being the case where all entities have the same coordinate system. Classical invariant theory gives results that are stated in Thm1 and Thm2. Some theoretical results are presented that study orthogonal recurrences of the form . Small-scale experiments are presented covering small group word problems and parsed versions of CLUTRR and StepGame.
Critical strengths
The paper has some significant writing issues which made it too hard to assess the paper's contributions with confidence. I think section 3 might contain some interesting analysis, but I couldn't judge its significance or novelty from the presentation.
Critical weaknesses
- The writing is difficult to follow in my opinion. It tends to use overly complex sentences structure, with many nested clauses, and awkward phrasing. Em dashes are overused. Simple things are said in convoluted ways. Overall, the paper is very hard to understand.
- The paper is poorly motivated. It takes its premise of different frames per entity for granted without explaining when or why real models would have entities in different cooridnate systems.
- The experiments do not test a transport-attention model, which is the main framing and motivation of the paper.
Questions for authors
- What are concrete applications or instantiations of this work? How does it relate to real models? In what settings would different entities have different coordinate systems?
Thank you for the review and for pointing to specific sentences. Taking your examples in order:
“Each entity owns its frame.” We meant that each entity’s vector is written in its own coordinate system. In our story experiments, an entity is a person such as Alice. The parsed input represents each person as a graph node and each stated relationship as an edge. Different mentions, such as “Alice” and “she,” can refer to the same person.
By a “fresh entity,” we mean a person or object newly introduced in the current story or task, even if its name is already familiar to the model. Giving that entity an independent coordinate frame is a modeling choice we make, not a consequence of it being newly introduced. The revision makes this distinction explicit in the introduction.
“Two coordinate languages.” This meant the two coordinate systems illustrated in Figure 1(a). The metaphor did not help, and we have removed it.
“The pair (width, readout).” Width is the number of coordinates in the model’s state vector. The readout is the classifier that turns the final state into an answer. We used both terms before defining them. Section 2.3 now defines them alongside Eq. (3), which shows how the model updates its state as it reads a sequence of operations.
“What are C3 and A4?” C3 describes three cyclic shifts, and A4 describes the twelve rotations that carry a regular tetrahedron onto itself. Combining these choices gives 36 possible answers. We use this example to show that the required state dimension depends on the classifier. With orthogonal updates, meaning rotations or reflections of the state vector, exact predictions on every sequence require at least five dimensions when the classifier has convex decision regions, while a quadratic classifier works in four. Section 3 now introduces this concrete task before stating the theorems.
For a route through the revised paper, please start with Eq. (3) in Section 2.3: each input symbol tells the model which matrix to apply to its state. Section 3.1 then asks how many coordinates the state needs to predict every sequence’s combined operation correctly. Figure 1(c) and Section 3.2 explain how changing the classifier saves one dimension. Finally, Section 5.2 tests the readout’s role by keeping the states fixed and changing only the classifier. The general response provides a fuller reading guide.