acceptodds
Under review as a conference paper at ICLR 2027

Transformers as effective fields: from symmetry breaking to architecture

Abstract

Can Transformer-like structures be derived from structural assumptions on a learnable field? We formulate an axiomatic effective-field theory that does not postulate attention in advance. For analytic matrix potentials with token-isometry invariance and feature-basis covariance, we classify the Taylor expansion through fourth order. The quadratic term generates token-wise linear feature mixing, while the quartic Hessian generates a cubic interaction with a finite paired linear-attention decomposition. This pairing reflects the trace-self-adjoint structure imposed by a scalar potential; relaxing the exchange tying opens a trace-anti-self-adjoint, locally non-potential sector that accommodates generic untied attention. A separate low-complexity assumption yields a small number of bottleneck heads. Row-softmax weights remain token-permutation equivariant and admit a row-wise entropy-regularized variational characterization, but softmax normalization does not in general restore potential compatibility for an untied update. We further quantify how a finite Transformer-like model tracks a fixed effective flow. Under uniform regularity, its trajectory error separates into an \(O(T^-1)\) depth-discretization term and width-dependent best field-approximation errors for attention and MLP. Controlled experiments diagnose these terms through frozen-field depth refinement and learned field surrogates. The field perspective therefore provides a low-order route from symmetry to architecture, with explicit limits on potential compatibility and approximation.

Primary area:
learning theory
Keywords:
transformer architectures, effective field theory, symmetry breaking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YPFEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to give a normative, axiomatic derivation of Transformer self-attention. Its basic strategy is to view a Transformer as an approximation of a continuous matrix-valued flow defined by the Wirtinger differential of a complex-valued potential, and then to show that the flows generated by potentials with a token-wise U(N)U(N) invariance resemble a constrained form of linear attention via a low-order Taylor approximation. Given that result, it aims to show how depth and width affect how well a Transformer can approximate the idealized flow.

Critical strengths

The general idea of giving a normative proof of how attention arises from basic symmetry considerations is interesting, and the central mathematical argument (in particular the proof of Theorem 5) seems sound, though more or less a consequence of classical invariant theory.

Critical weaknesses

My primary concern is that there is a mismatch between the axioms the authors impose and the symmetries of actual softmax attention. The central argument in Section 3.2, leading up to Theorem 5, assumes that the potential is invariant under the action of U(N)U(N) on the token dimension. To fourth order, this leads to a field of the form F(X)=X[A+L(X∗X)]F(X) = X[A + \mathcal{L}(X^{\ast}X)] (from eq. 17), meaning that each row of the output is obtained by applying the same transformation to the corresponding row of the input. In contrast, ordinary softmax attention has in general only permutation symmetry along the token dimension. Schematically, following the authors' eq. (1), the corresponding field would be F(X)=softmax⁡(XBX∗)XCF(X) = \operatorname{softmax}(XBX^{\ast}) X C for trainable BB and CC, which mixes rows. Thus, the results concern an architecture with enhanced symmetry relative to the one the authors aim to explain.

A similar issue arises in the authors' argument for the emergence of the MLP branch: in their simplified model it is just the right-multiplication XAXA, which is substantially simpler than the full MLP branch used in standard Transformers (and as stated in eq. 1 of this paper).

As I touch upon under Questions, the gap between the central results and realistic architectures is also not adequately bridged by the other theorems and experiments. For these reasons, I think the paper's results fall short of its stated goals.

Questions for authors
  1. Why is U(N)U(N) symmetry the right starting point? Can you establish a similar Taylor expansion result for permutation symmetry?

  2. Related to this question, the general expansion in (19) does not single out attention-type computations. Indeed, thanks to associativity one is free to group the matrices in this expression in any way; this underlies the equivalence of linear attention with other architectures (see, for instance, Irie et al, "Going Beyond Linear Transformers with Recurrent Fast Weight Programmers").

  3. The paper's terminology substantially exceeds the scope of its results. From the title on, it refers to "effective field theory", but nowhere is there a nontrivial notion of scale or the other trappings of a true EFT. Moreover, the authors refer to "symmetry breaking", but this is just the decomposition of the matrix field into integrable (i.e., potential-gradient) and non-integrable components. There is no physical mechanism by which the symmetry is broken, it is an architectural design choice.

  4. Theorem 7 presents itself as a strikingly uniform approximation result, but it already assumes uniform approximation of the branch fields rather than starting from a realizable architecture. In particular, having skimmed the proof, the assumptions used in passing from eq. (40) to eq. (41) regarding the possibility of selecting "near-minimizers" seems very strong. Can the author elaborate on how they view the practical implication of this theorem? Also, the Gr"onwall-type exponential factor seems to have disappeared between (40) and (41).

  5. Several of the experiments seem to test generic phenomena rather than specific aspects of the theory. For instance, looking through the figures, Figure 3 is just a demonstration that Euler's method achieves first-order accuracy; this would be true of any smooth dynamical system. Moreover, the experiments showing that the ViT Jacobian includes skew components does not specifically test whether those components lie within the particular sector identified by the authors in their discussion of the operator T\mathsf{T}. Now, they themselves acknowledge this, writing "The full-Jacobian audit in Appendix G measures the aggregate departure from potential compatibility and does not attribute all observed skew exclusively to missing exchange tying" in lines 348-350, but this raises the question of what this experiment is intended to test. Is a more direct test of the proposed mechanism possible?

Reviewer_MygvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper provide an analysis for analytic token-isometry-invariant potentials through fourth order, showing that their Wirtinger gradients produce linear feature mixing and paired linear attention.

Critical strengths
  • Using an energy function to explain or design a Transformer is already an established idea. The earlier constructions are as follows: Choose an energy function and derive attention-like updates from it. This work starts with symmetry assumptions, classify the possible quadratic and quartic potential terms, and show that the gradients produce linear feature mixing and paired linear attention. This theoretical part of the paper is sound. Symmetry does not imply few heads, low head rank, softmax, nonlinear MLPs, or positional structure. The softmax derivative also identifies why a natural log-partition potential does not generate standard attention.
Critical weaknesses
  • The classification does not have softmax, normalization, positional structure, and nonlinear MLPs.

  • There is no code submission. No explicit wall-clock runtime is reported, while paper provides training epochs, optimizer updates, solver steps, and some software details.

Questions for authors
  • The work base on a workshop paper at NeurIPS 2025, Transformers Are Optimal Effective Fields. Which results are new or strengthened?

  • Please provide training/inference timings, type of GPUs, total GPU-hours, peak memory for all compared models?

Reviewer_614fOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper classifies the fourth-order Taylor expansion of analytic token-isometry-invariant matrix potentials, where the quadratic term generates token-wise linear feature mixing, while the trace-self-adjoint quartic Hessian generates paired linear attention. It also characterizes how the width and depth of the induced idealized Transformer affect the approximation errors of effective fields.

Critical strengths
  1. The paper is generally well written and structured.
  2. The surrounding literature is reviewed properly.
Critical weaknesses
  1. The main results are largely theoretical under ideal assumptions (e.g., no normalization, nonlinear activation, and softmax) with little to no empirical evidence showing the practical advantages. Viewing Transformers as ODEs is not totally new; this perspective has been largely unified into an architectural design framework as the hyperspherical energy minimization in Hyper-SET [1], which already shows some practical applications. Given this work, how do the authors justify the contributions when restricted merely to idealized theoretical analysis?

  2. The motivation is not clear and compelling. The paper lists its contributions as proofs and mathematical characterizations. It is unclear why these theoretical results are needed at all, or whether they bring any new understanding or inform any technical design, especially given the highly simplified toy setup. The proofs appear to be for their own sake rather than in the context of practical Transformers.

[1] Hyper-SET: Designing Transformers via Hyperspherical Energy Minimization. ICLR, 2026.

Questions for authors
  1. What technical claims is Section 3 attempting to establish, beyond simply deriving descriptive rather than constructive theoretical properties? What actionable insights does it provide?

  2. What, precisely, are "effective fields"? The authors argue for Transformers as effective fields, yet I don’t find any formal definition or description of what effective fields are. How is it fundamentally different from ODE formulation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.