acceptodds
Desk rejected at ICLR 2027

Self-Attention as Connection Propagation: An Exact Geometric Operator Representation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-attention is a ubiquitous primitive in modern sequence models, yet its operator-level geometry is only partially understood. We view a token sequence as a *vector field over the token-position graph* and identify attention as a *connection walk*: messages are aggregated by a nonnegative walk matrix while being *transported* along each edge by a learned linear map. Within this framework, we derive an exact representation of *single-head attention (SHA)* through a connection propagation step with *constant* transport, and of *multi-head attention (MHA)* through a *single edge-dependent connection walk* whose effective transport is an attention-gated mixture of headwise transports. We further clarify the conditions under which the corresponding operator reduces to a *random-walk connection Laplacian*, highlighting the roles of stochasticity, reversibility, and metric-compatible transports. Empirically, we examine trained Transformers across scales and structures (encoder/decoder). Layerwise diagnostics reveal low-drift middle-layer routing regimes and pronounced diagonal structure in averaged transport Gram matrices, while a separate selected-edge spectral analysis reveals concentrated transport energy with unequal directional gains. Overall, the paper provides a precise connection-walk formalism that links self-attention to classical geometric operators, along with a set of operator-level tools for analyzing transformer models from a geometric perspective.

Primary area:
learning theory
Keywords:
LLM, Geometry, Connection Laplacian, Connection Walk

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_krx1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose an exact representation of single- and multi-head attention by interpreting W_{V,h}W_{O,h} as feature transports, thus showing that attention can be represented as a connection walk consisting of aggregating messages that are transported along edges. They give bounds to analyze routing gains, the rank of the transports, and their joint action. They also give the conditions under which the resulting connection walk defines a metric connection Laplacian. The authors then experimentally analyze how much source selection changes between layers, how messages are transported, and the sensitivity to context length. They find that attention weights often remain relatively similar between consecutive middle layers, that averaged transport Gram matrices show a pronounced diagonal structure, and that the transports concentrate most of their energy in a relatively small number of feature directions, with unequal gains across directions.

Critical strengths

This paper's main contribution is an original point of view on the (self) attention mechanism, therefore its main strengths are related to this change of perspective:

  • The paper establishes an interesting link between walks on graphs and attention, for vector-valued signals that are transported along edges
  • Thorough analysis of aggregation, transport, and their joint action
  • Finds the conditions under which a metric connection Laplacian is obtained
  • Separates the actions of aggregation and transport, also in the experiments
  • uncovers an interesting link to geometric parallel transport, while showing experimentally that they are usually not orthogonal
Critical weaknesses

Weaknesses or avenues for improvement mostly concern clarifications, limits of the analysis and broadly speaking impact on the Transformer architecture.

  • Maybe briefly restate the standard formulation of attention so that references to it become clearer (e.g., W_{V,h})
  • Explain more explicitly the limitations of analyzing F_X(V) instead of the actual F_X(X). The analysis does not take into account the nonlinear dependence of attention on its input, and it is not entirely clear to me how much the properties found analyzing F_X(V) tell us about the behavior of the actual Transformer.
  • As a consequence, the paper it analyzes a single operator A_X, not the dynamical system generated by layer composition X_{l+1}=A_{X_l}V(X_ℓ) while this feedback loop between attention and representation is typically important.
  • Likewise, beyond looking at Transformer from a different (original) point of view, what could be modifications of the architecture this new point of view might bring ? Is the proposed study purely diagnostic ?
  • In the discussion of prior art, in particular using differential geometric ideas, I haven't seen this recent paper: RiemannInfer: improving transformer inference through Riemannian geometry, Mao et al
Questions for authors

Questions are naturally related to the above mentioned weaknesses:

  • Can the authors provide a concrete example of a novel or even non-trivial phenomenon that becomes explainable in the connection-walk formalism but is difficult or impossible to derive from standard formulations of (self) attention?
  • can you briefly restate the standard formulation of attention so that references to it become clearer (e.g., W_{V,h}) ?
  • briefly restate the standard formulation of attention so that references to it become clearer (e.g., W_{V,h}) ?
  • Right now the paper seems mostly an interesting point of view. Do any of the proposed (geometric) diagnostics relate to usual metrics (e.g., perplexity, long-context performance, scaling trends, ...)? That would certainly update the paper from an original exposition to an impactful reformulation.
  • As a follow-up, I haven't seen interventions that use the proposed diagnostics. For instance, can you modify layers with high/low drift, or perturbing transport operators ? Again that would add to the purely diagnostics aspects of the paper.
  • Can you expand on the analysis of F_X(V) vs F_X(X) and the limit of linear regime ?
  • You clearly give conditions under which an actual connection laplacian appears, but these conditions do not necessarily materialise in a Transformer. what is therefore the practical role of the metric connection Laplacian analysis?
Reviewer_6iVZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper reformulates multi-head attention as a geometric connection-walk operator, separating token routing from feature transformations. The authors derive mathematical properties of this represenation and analyze pretrained Transformers, reporting stable middle-layer attention patterns, approximately diagonal transport Gram matrices, and concentration of operator energy in relatively few dimensions.

Critical strengths
  1. Mathematical rigor: The authors derive an exact representation of multi-head attention as a connection-walk operator, rather than relying on an approximation.
  2. Unified theoretical framework: The formulatin connects attention to geometric graph operators and establishes mathematical properties concerning amplification, rank, and conditions for diffusion.
  3. Broad empirical analysis: The authors examine multiple Transormer families, including GPT-2, Qwen, and Lama, across a substantial range of model sizes.
  4. Useful operator-level diagnostics: Separating routing, feature transport, and their joint action provides tools for investigating Transformer representations beyond attention weights alone.
Critical weaknesses

The contribution appears largely incremental. the decomposition of attention into routing and feature transformations is already well established, and similar diagonal Gram structures have reportedly been observed before. the remaining empirical findings are primarily descriptive: stable routing lacks a clear functional interpretation, while spectral concentration is difficult to assess without appropriate random or untrained baselines. Most imporantly, the theoretical framework neither predicts these observations nor demonstrates their relevance to Transformer performance or efficiency. Overall, the paper offers a mahtematically coherent reinterpretation of existing attention mechanisms, but limited conceptual novelty and few substantial new insights.

Questions for authors

no questions

Reviewer_8Ht4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The goal of this paper is to understand self-attention as a connection-walk operator where tokens are represented as vector fields over a graph. The main theoretical contribution is an exact representation of multi-head attention as a single connection walk, where routing is given by the mean attention matrix and edge-dependent feature transports are conditional mixtures of the headwise value-output maps.

The authors derive several theoretical results, including bounds on routing amplification, feature-transport rank, and the operator norm of the combined routing and transport action. They also identify conditions under which attention can be viewed as a metric connection Laplacian. Empirical results across GPT-2, Qwen, Llama, and BERT analyze routing similarity, transport Gram-matrix structure, and singular value distributions.

Critical strengths
  1. The paper provides a unified framework that separates token routing from feature transformation, offering an interesting perspective on how information propagates through attention layers.
  2. The authors derive an exact connection-walk representation of multi-head attention and establish several theoretical properties, including bounds on routing amplification, transport rank, and operator norms.
  3. The experiments cover multiple transformer architectures and model sizes, including GPT-2, Qwen, Llama, and BERT. The analysis of routing drift, transport Gram matrices, and singular-value distributions provides useful insights into the structure of learned attention operators.
Critical weaknesses
  1. The main result, Theorem 1, appears to be a relatively straightforward algebraic reformulation of the standard multi-head attention decomposition using Kronecker products. Previous work, such as Elhage et al., A Mathematical Framework for Transformer Circuits, 2021, already separates attention routing from value-output transformations. Similarly, several subsequent propositions follow from standard linear algebra results. It is unclear what fundamentally new theoretical insights the connection-walk representation provides beyond existing formulations.
  2. Although the authors motivate the connection-walk formulation as a new geometric perspective on self-attention, it is unclear why this new representation is necessary or what it enables beyond existing approaches. The introduction should better explain what limitations of existing attention analyses are addressed by the proposed representation and what new insights can be obtained.
  3. Proposition 5 establishes a connection to metric diffusion under strong assumptions, including reversibility and orthogonal feature transformations. As the authors acknowledge, these conditions generally fail for standard causal Transformers. While the general connection-walk representation remains valid, the paper does not clearly establish what additional geometric insights can be obtained in the non-reversible and non-orthogonal setting encountered in practice.
  4. Although Section 5.5 discusses operator approximation and edge removal, the paper does not demonstrate that the proposed representation yields better approximations, interpretability, or performance predictions compared with existing analyses. A concrete application, such as attention sparsification based on the joint routing-transport contribution, would substantially strengthen the contribution.
  5. The experiments report diagonal structure in transport Gram matrices and concentrated singular-value energy, but do not compare these observations against appropriate baselines, such as randomly initialized or randomized operators. Similar patterns can arise naturally in high-dimensional random matrices, making it unclear whether the observed properties reflect meaningful learned structure or generic mathematical properties.
Questions for authors
  1. What is the central theoretical result that cannot be obtained directly from the existing Kronecker-product representation of multi-head attention? Can the authors identify a nontrivial theorem or consequence that specifically requires the connection-walk perspective?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.