- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper aims to give a normative, axiomatic derivation of Transformer self-attention. Its basic strategy is to view a Transformer as an approximation of a continuous matrix-valued flow defined by the Wirtinger differential of a complex-valued potential, and then to show that the flows generated by potentials with a token-wise invariance resemble a constrained form of linear attention via a low-order Taylor approximation. Given that result, it aims to show how depth and width affect how well a Transformer can approximate the idealized flow.
Critical strengths
The general idea of giving a normative proof of how attention arises from basic symmetry considerations is interesting, and the central mathematical argument (in particular the proof of Theorem 5) seems sound, though more or less a consequence of classical invariant theory.
Critical weaknesses
My primary concern is that there is a mismatch between the axioms the authors impose and the symmetries of actual softmax attention. The central argument in Section 3.2, leading up to Theorem 5, assumes that the potential is invariant under the action of on the token dimension. To fourth order, this leads to a field of the form (from eq. 17), meaning that each row of the output is obtained by applying the same transformation to the corresponding row of the input. In contrast, ordinary softmax attention has in general only permutation symmetry along the token dimension. Schematically, following the authors' eq. (1), the corresponding field would be for trainable and , which mixes rows. Thus, the results concern an architecture with enhanced symmetry relative to the one the authors aim to explain.
A similar issue arises in the authors' argument for the emergence of the MLP branch: in their simplified model it is just the right-multiplication , which is substantially simpler than the full MLP branch used in standard Transformers (and as stated in eq. 1 of this paper).
As I touch upon under Questions, the gap between the central results and realistic architectures is also not adequately bridged by the other theorems and experiments. For these reasons, I think the paper's results fall short of its stated goals.
Questions for authors
-
Why is symmetry the right starting point? Can you establish a similar Taylor expansion result for permutation symmetry?
-
Related to this question, the general expansion in (19) does not single out attention-type computations. Indeed, thanks to associativity one is free to group the matrices in this expression in any way; this underlies the equivalence of linear attention with other architectures (see, for instance, Irie et al, "Going Beyond Linear Transformers with Recurrent Fast Weight Programmers").
-
The paper's terminology substantially exceeds the scope of its results. From the title on, it refers to "effective field theory", but nowhere is there a nontrivial notion of scale or the other trappings of a true EFT. Moreover, the authors refer to "symmetry breaking", but this is just the decomposition of the matrix field into integrable (i.e., potential-gradient) and non-integrable components. There is no physical mechanism by which the symmetry is broken, it is an architectural design choice.
-
Theorem 7 presents itself as a strikingly uniform approximation result, but it already assumes uniform approximation of the branch fields rather than starting from a realizable architecture. In particular, having skimmed the proof, the assumptions used in passing from eq. (40) to eq. (41) regarding the possibility of selecting "near-minimizers" seems very strong. Can the author elaborate on how they view the practical implication of this theorem? Also, the Gr"onwall-type exponential factor seems to have disappeared between (40) and (41).
-
Several of the experiments seem to test generic phenomena rather than specific aspects of the theory. For instance, looking through the figures, Figure 3 is just a demonstration that Euler's method achieves first-order accuracy; this would be true of any smooth dynamical system. Moreover, the experiments showing that the ViT Jacobian includes skew components does not specifically test whether those components lie within the particular sector identified by the authors in their discussion of the operator . Now, they themselves acknowledge this, writing "The full-Jacobian audit in Appendix G measures the aggregate departure from potential compatibility and does not attribute all observed skew exclusively to missing exchange tying" in lines 348-350, but this raises the question of what this experiment is intended to test. Is a more direct test of the proposed mechanism possible?