acceptodds
Under review as a conference paper at ICLR 2027

Relational Composition as Parallel Transport: What Per-Entity Frames Force in Attention

Abstract

A dot product means something only when both vectors share a coordinate system. Content words have one – the corpus fixes what every direction means – but a fresh entity, a name in a story or a variable in a program, does not: rotating its coordinates alone changes nothing. We take this per-entity frame freedom as a symmetry of the representation and ask what it forces in attention. Every invariant score depends on two entities only through their norms and the transported inner product , and every equivariant message lies in the span of and . A transport computed from the endpoints' own channels has rank at most the number of channels, never a group element from one vector each, so relations must be supplied; the standard transformer is the flat sector of one shared frame. Representation theory then attaches an integer to each axis of a mechanism, and the readout decides which integer. A readout with convex decision cells composes a finite group exactly only at widths of at least its smallest faithful real representation, and at that width every exact solution is a faithful representation with a finite latent group – as a conclusion, not a hypothesis. An even readout can decode a cover of the group instead, at a smaller width, and the gap is unbounded. Below its integer a mechanism is capped by the part of the group it cannot see – the commutator subgroup for a commutative mechanism, the derived series for a stack of commutative layers, the lower central series for a signature readout – all computable from the group before training. Empirically, every fitting seed at the minimal width generalizes exactly to length ( of , against of at twice the width), with certified horizons read from the weights. On , a -element group whose faithful width is , gradient descent at width learned the -state double cover under a quadratic readout and under sign-blind attention ( of seeds, half declared in advance; monotone attention of ), and on a frozen cover the cross-entropy-optimal affine head is provably uniform while a quadratic head is exact. On parsed CLUTRR, transported composition beats a matched transformer by points over four to ten hops (trained on two and three), and forcing isometries reverses that by – no isometry represents an idempotent rule such as sibling of sibling, as Krohn–Rhodes predicts – while on StepGame's commutative algebra isometries help instead. Static embeddings gave each word one vector; contextual models gave each occurrence its own; per-entity frames make each relation an operator, the unit in which relations compose. This points toward attention that produces its operators from context rather than retrieving them from a library, and the carrier theorem writes its specification in advance: not from the two entities, but from a carrier the context supplies.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
geometric deep learning, state tracking, finite groups, relational composition, parallel transport, gauge equivariance, representation theory, neuro-symbolic reasoning, transformers, attention mechanisms

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_PBPbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a gauge theory framing of attention and applies it to understand what happens to the inner products in attention if the vectors are first tranformed into different coordinate systems by two different maps. I.e., each "entity" (vector) is in its own "frame" (coordinate system). Entity representations are defined only up to independent rotations gi∈O(d)g_i \in O(d), with standard attention being the case where all entities have the same coordinate system. Classical invariant theory gives results that are stated in Thm1 and Thm2. Some theoretical results are presented that study orthogonal recurrences of the form xw=RaL⋯Ra1x0x_w = R_{a_L} \cdots R_{a_1} x_0. Small-scale experiments are presented covering small group word problems and parsed versions of CLUTRR and StepGame.

Critical strengths

The paper has some significant writing issues which made it too hard to assess the paper's contributions with confidence. I think section 3 might contain some interesting analysis, but I couldn't judge its significance or novelty from the presentation.

Critical weaknesses
  • The writing is difficult to follow in my opinion. It tends to use overly complex sentences structure, with many nested clauses, and awkward phrasing. Em dashes are overused. Simple things are said in convoluted ways. Overall, the paper is very hard to understand.
  • The paper is poorly motivated. It takes its premise of different frames per entity for granted without explaining when or why real models would have entities in different cooridnate systems.
  • The experiments do not test a transport-attention model, which is the main framing and motivation of the paper.
Questions for authors
  • What are concrete applications or instantiations of this work? How does it relate to real models? In what settings would different entities have different coordinate systems?
1 comment

Thank you for the review and for pointing to specific sentences. Taking your examples in order:

  • “Each entity owns its frame.” We meant that each entity’s vector is written in its own coordinate system. In our story experiments, an entity is a person such as Alice. The parsed input represents each person as a graph node and each stated relationship as an edge. Different mentions, such as “Alice” and “she,” can refer to the same person.

    By a “fresh entity,” we mean a person or object newly introduced in the current story or task, even if its name is already familiar to the model. Giving that entity an independent coordinate frame is a modeling choice we make, not a consequence of it being newly introduced. The revision makes this distinction explicit in the introduction.

  • “Two coordinate languages.” This meant the two coordinate systems illustrated in Figure 1(a). The metaphor did not help, and we have removed it.

  • “The pair (width, readout).” Width is the number of coordinates in the model’s state vector. The readout is the classifier that turns the final state into an answer. We used both terms before defining them. Section 2.3 now defines them alongside Eq. (3), which shows how the model updates its state as it reads a sequence of operations.

  • “What are C3C_3 and A4A_4?” C3C_3 describes three cyclic shifts, and A4A_4 describes the twelve rotations that carry a regular tetrahedron onto itself. Combining these choices gives 36 possible answers. We use this example to show that the required state dimension depends on the classifier. With orthogonal updates, meaning rotations or reflections of the state vector, exact predictions on every sequence require at least five dimensions when the classifier has convex decision regions, while a quadratic classifier works in four. Section 3 now introduces this concrete task before stating the theorems.

For a route through the revised paper, please start with Eq. (3) in Section 2.3: each input symbol tells the model which matrix to apply to its state. Section 3.1 then asks how many coordinates the state needs to predict every sequence’s combined operation correctly. Figure 1(c) and Section 3.2 explain how changing the classifier saves one dimension. Finally, Section 5.2 tests the readout’s role by keeping the states fixed and changing only the classifier. The general response provides a fuller reading guide.

Reviewer_kqYzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors see standard attention as a comparison between two vectors that assumes the vectors are already represented in some shared coordinate system. They consider what the implications are for the attention mechanism when this assumption doesn't hold. In particular, if two entities don't share a coordinate system, what would be required mathematically and practically of a model to afford their meaningful comparison? This paper constructs settings where the comparison between entities without a shared frame can occur exactly and develops guarantees for when comparison can happen in learned models. The authors then test their theoretical predictions in models trained on relational tasks.

Critical strengths

I think the paper brings up an interesting point about how standard attention implicitly assumes a shared coordinate system between the vectors it is comparing. If this assumption breaks, what does this imply about what attention is computing? I think this is an interesting question and it makes me want to better understand when in the real world we would expect a fresh entity to actually show up / be a practical concern.

Critical weaknesses

It is pretty difficult for me to get something substantial out of this paper given its presentational difficulties. It seems like much of the paper is premised on this idea that standard attention is not set up to handle fresh entities. While I suspect there is something interesting here, the paper never explicitly says what a fresh entity is or how it relates to any real world object. This makes it quite hard to understand the value of this paper and is just one example of ambiguity out of many that plague this paper. Words and concepts are repeatedly introduced without clear definitions of how we should understand them nor how important it is to understand them and this makes following the paper's claims, evidence for them, and why they are important, quite taxing. What is an entity? In particular, what is a fresh entity? What is a relation? How should I distinguish this from a content word? What are decision cells? What is a mechanism, message, or frame? These words and others are introduced without much care in orienting the reader to the domain in which they take on precise meaning despite their use and context suggesting importance. Reading this work feels like trying to fill in a crossword puzzle where to unambiguously reveal the meaning of a phrase, word, or result requires first filling in the meaning of other equally ambiguous phrases, words, and results. Even in sections which are dedicated to theorems and proofs, which should have some inherent precision, the authors provide little context for why we should care about them nor much connective tissue between them. This makes the paper on the whole read at times like a series of statements (potentially even true ones) whose relationships are left to the reader to infer.

In the end, there very well may be interesting and important results here but the cognitive effort required to interpret the presentation such that I may adequately evaluate its soundness and informativeness feels unreasonable, and frankly, inconsiderate.

Questions for authors

What is a fresh entity and what does this object actually correspond to in the context of neural network training and inference?

1 comment

Thank you for the review, and for saying what you found interesting. We accept the criticism of the presentation; the revision adds explicit definitions of the terms you identified and introduces the question before the theorems.

To your question. A training story and a test story can both use the name Alice. Each story introduces its own Alice, and later mentions in that story refer back to her. That newly introduced person is what we call a fresh entity: a distinct node in the parsed relation graph, with later mentions linked to that node and the stated facts represented by edges. In a trained network this is the situation the binding literature studies: Feng and Steinhardt (2024) identify binding-ID vectors associated with corresponding entities and attributes in the language models they study. Our frames are a modeling choice for the same situation. We give each such node its own coordinate system, so comparing feature directions across nodes requires a transport; vector norms and other invariant attributes can still be compared without one. This is an assumption we adopt for relational reasoning, not something that follows from a name being new; a standard transformer places names in one shared space, and we do not claim otherwise. The revision says this in the introduction.

In the benchmark experiments (Section 5.3), the models receive relations already extracted from the text. "Alice is Bob's sister" gives two people and one fact connecting them; the name identifies a node in this story and "sister of" is a relation type. In the matrix-composition model, a matrix is trained for each relation type and reused for that type in new stories. The new Section 5.4 places each person in an independent random coordinate system and asks whether a ten-layer transport-attention stack and standard attention can still compose the supplied relations. The experiments of Sections 5.1 and 5.2 test synthetic finite-group state tracking, rather than entity identification from raw text: they examine how matrices compose and what a classifier can recover, as described in the general response.

For the other terms you list:

  • Frame: the coordinate system in which one entity's vector is written.
  • Relation: a fact linking two entities, such as "Alice is Bob's sister", represented as a matrix that carries vectors from one entity's coordinates to the other's.
  • Entity versus content word: in the parsed task, "Alice" identifies a node, while "sister of" specifies a relation between two nodes.
  • Message: the vector one node sends to another in an attention step, as in graph neural networks. In Section 5.4, if one person's axes are changed and the matrices on the edges touching that person are adjusted, the transport model gives the same answer.
  • Decision cell: the set of states the classifier maps to one label.
  • Mechanism: one of the restricted model classes in Section 3.3; for example, if the update matrices commute, swapping two adjacent symbols leaves the state unchanged.
Reviewer_BUmiOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper starts with an interesting premise: suppose transformer has to consume new token (unseen before, presumably after training, though that's not specified). It makes an observation that one may as well rotate arbitrarily the tokens (and for two such tokens we can put them in any position). After that I'm losing the thread: what exactly is suggested and what's the goal here.

Example of (inadequate) writing in the introduction already:

  • "Each entity owns its frame". What does this mean? Isn't there a more standard language for what you are saying here? (What is an entity, a token?)
  • "two coordinate languages". Is language used a metaphor here? I'd suggest a more mathematically consistent language.
  • "the pair (width, readout)". What is width and readout? I didn't see the definitions there.
  • what is C_3, A_4 that suddenly appear?
Critical strengths

n/a

Critical weaknesses

Impossible to read for humans.

Questions for authors

Not sure what to start, but I would appreciate a roadmap of how to read and understand this paper (to be read by humans).

1 comment

Thank you for the review and for pointing to specific sentences. Taking your examples in order:

  • "Each entity owns its frame" means that each entity's vector is written in its own coordinate system. An entity is a node in the parsed relation graph, such as a person in a story. It is not a token, since one entity can be mentioned by several tokens ("Alice", "she"). The paper is not about tokens unseen during training: the same name "Alice" can introduce a different person in a training story and in a test story, and that newly introduced person is what we call fresh. Giving that person its own coordinate system is a modeling choice we make, not a consequence of the name being new; the revision says both things in the introduction.
  • "Two coordinate languages" just meant the two coordinate systems of Figure 1(a). The metaphor did not help and is gone.
  • Width is the number of coordinates in the state vector; the readout is the classifier that turns the final vector into a label. We used both before defining them. They are now defined in Section 2.3, at Eq. (3), where composition becomes the question.
  • C3C_3 is the cyclic group of order three, and A4A_4 is the group of the twelve rotations of a tetrahedron. Their product has 36 elements. We use it because, with orthogonal state updates, a classifier with convex decision regions needs five dimensions for it while a quadratic classifier works in four; Section 3 now gives this concrete 36-label example before the formal statements.

As a route through the paper: please start with Eq. (3) in Section 2.3, where each symbol tells the model which matrix to apply to its state; then Section 3.1, which asks how wide the state must be to get every sequence right; then Figure 1(c) and Section 3.2, where changing the classifier saves one dimension; then Section 5.2, which tests this by keeping the states fixed and changing only the classifier. The general response provides a longer order.

Comments on the paper

1 comment

Thank you all for reading the paper closely. All three reviews found it hard to follow, and we agree: we introduced terms before defining them, and we took too long to get to the question the paper answers. We have uploaded a revised PDF, and we state the question here.

Each input symbol stands for an element of a finite group, and the model must predict the product of the symbols, in order, for inputs of any length. It multiplies its state vector by one orthogonal matrix per symbol, and a classifier reads the answer from the final state. The question is how many dimensions the state needs for exact predictions, and whether that depends on the classifier. We find that it does.

Our main example is a group with 36 elements, C3×A4C_3 \times A_4: a cyclic shift of order three combined with one of the twelve rotations of a tetrahedron. With a classifier whose decision regions are convex, such as an affine one, five dimensions are needed, and an affine classifier attains them (Theorem 3). With a quadratic classifier, four are enough (Theorem 4). Figure 1(c) and Section 3.2 show the four-dimensional construction: each label is stored as a pair of opposite states, zz and −z-z.

To classify all 72 states correctly, each label’s convex decision region would need to contain both zz and −z-z, and therefore their midpoint, the origin. But decision regions for different labels must be disjoint, so they cannot all contain the origin. A quadratic classifier reads every label, because zz⊤zz^\top is the same for both states in a pair and differs between pairs. Section 5.2 isolates the readout by freezing the state sets and refitting only the classifier. Appendix B.3 distinguishes the raw approximately closed states from the snapped cover. On an exact balanced antipodal cover, the cross-entropy-optimal affine classifier assigns probability 1/361/36 to each label. In the reported fits, the quadratic classifier labels all 72 states correctly, and the fitted affine loss is approximately log⁡36\log 36.

Section 2 motivates the relation matrices by asking how to compare features written in different local coordinate systems. Separate coordinate systems are our modeling assumption, not a claim about standard transformers; the width results of Section 3 do not depend on it. Reviewer PBPb also noted that no transport-attention model was tested. The revision adds one (Section 5.4, Appendix B.7), together with an experiment on the carrier theorem's implications (Appendix B.8).

What changed in the revision: the introduction now defines entity, frame, transport, score and message, with Figure 1 illustrating the setup; Section 2.3 adds a worked two-edge frame-change example; width and readout are defined in Section 2.3, where composition becomes the question, and decision cells in Section 3; "fresh entity" is defined in the introduction; each theorem's section heading says in words what it establishes. The central mathematical results and previously reported empirical findings are retained; the revision clarifies their assumptions and exposition and adds the experiments identified above. Appendix A gives the proofs, and Appendix B gives detailed protocols and results. We would welcome specific technical questions about either.

To help navigate the revised paper, we suggest the following reading order:

(1) Section 1 and Figure 1: the five basic terms (entity, frame, transport, score and message), and the distinction between our modeling assumption and standard attention. Section 2.3 gives a concrete two-edge example showing how coordinate changes cancel during composition.

(2) Section 2.1 and Theorem 1: how to compare vectors expressed in different frames, and what invariance requires of the score. The six-candidate matching experiment illustrates the transported comparison.

(3) Section 2.2 and Theorem 2: why two endpoint vectors alone cannot produce a full-rank transport in dimensions greater than one, and what additional information can supply it.

(4) Eq. (3), Sections 3.1–3.2 and Figure 1(c): the central width/readout question. The 36-label example explains why convex decision regions require five dimensions, while a quadratic readout works in four.

(5) Section 3.3 and Table 1: how restrictions on composition produce conditional accuracy ceilings.

(6) Section 4, Section 5.1 and Figure 2: why passing long tests is not an all-length guarantee, and how finite horizons are certified. Section 5.2 isolates the readout’s role by holding the states fixed.

(7) Section 5.4 and Appendix B.7: the new attention-stack experiment under independent frames, including its controls and limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.