- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose an exact representation of single- and multi-head attention by interpreting W_{V,h}W_{O,h} as feature transports, thus showing that attention can be represented as a connection walk consisting of aggregating messages that are transported along edges. They give bounds to analyze routing gains, the rank of the transports, and their joint action. They also give the conditions under which the resulting connection walk defines a metric connection Laplacian. The authors then experimentally analyze how much source selection changes between layers, how messages are transported, and the sensitivity to context length. They find that attention weights often remain relatively similar between consecutive middle layers, that averaged transport Gram matrices show a pronounced diagonal structure, and that the transports concentrate most of their energy in a relatively small number of feature directions, with unequal gains across directions.
Critical strengths
This paper's main contribution is an original point of view on the (self) attention mechanism, therefore its main strengths are related to this change of perspective:
- The paper establishes an interesting link between walks on graphs and attention, for vector-valued signals that are transported along edges
- Thorough analysis of aggregation, transport, and their joint action
- Finds the conditions under which a metric connection Laplacian is obtained
- Separates the actions of aggregation and transport, also in the experiments
- uncovers an interesting link to geometric parallel transport, while showing experimentally that they are usually not orthogonal
Critical weaknesses
Weaknesses or avenues for improvement mostly concern clarifications, limits of the analysis and broadly speaking impact on the Transformer architecture.
- Maybe briefly restate the standard formulation of attention so that references to it become clearer (e.g., W_{V,h})
- Explain more explicitly the limitations of analyzing F_X(V) instead of the actual F_X(X). The analysis does not take into account the nonlinear dependence of attention on its input, and it is not entirely clear to me how much the properties found analyzing F_X(V) tell us about the behavior of the actual Transformer.
- As a consequence, the paper it analyzes a single operator A_X, not the dynamical system generated by layer composition X_{l+1}=A_{X_l}V(X_ℓ) while this feedback loop between attention and representation is typically important.
- Likewise, beyond looking at Transformer from a different (original) point of view, what could be modifications of the architecture this new point of view might bring ? Is the proposed study purely diagnostic ?
- In the discussion of prior art, in particular using differential geometric ideas, I haven't seen this recent paper: RiemannInfer: improving transformer inference through Riemannian geometry, Mao et al
Questions for authors
Questions are naturally related to the above mentioned weaknesses:
- Can the authors provide a concrete example of a novel or even non-trivial phenomenon that becomes explainable in the connection-walk formalism but is difficult or impossible to derive from standard formulations of (self) attention?
- can you briefly restate the standard formulation of attention so that references to it become clearer (e.g., W_{V,h}) ?
- briefly restate the standard formulation of attention so that references to it become clearer (e.g., W_{V,h}) ?
- Right now the paper seems mostly an interesting point of view. Do any of the proposed (geometric) diagnostics relate to usual metrics (e.g., perplexity, long-context performance, scaling trends, ...)? That would certainly update the paper from an original exposition to an impactful reformulation.
- As a follow-up, I haven't seen interventions that use the proposed diagnostics. For instance, can you modify layers with high/low drift, or perturbing transport operators ? Again that would add to the purely diagnostics aspects of the paper.
- Can you expand on the analysis of F_X(V) vs F_X(X) and the limit of linear regime ?
- You clearly give conditions under which an actual connection laplacian appears, but these conditions do not necessarily materialise in a Transformer. what is therefore the practical role of the metric connection Laplacian analysis?