- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper rewrites the Transformer block into a combination of Advection and Diffusion term, calling it ADR representation.
They measure the magnitudes of each component in well known models, and reports the tendency of these magnitudes with respect to depth.
Critical strengths
The paper attempts to tackle the recurring idea of Symmetry-Assymetry decomposition of the Transofmer operator and endeavors to make use of it in model analysis.
Critical weaknesses
The research itself is poorly motivated and it is hard to follow what it is trying to communicate to the community.
It was a difficult read not because of the content itself, but because of its motivation.
If I am not reading wrongly, the paper's central mathematical theorem is essentially the composition of two elementary algebraic substitutions, as stated itself in its own proof --- and I am finding it difficult to identify the notable novelty of these theorems in the literature of mathemtical theory of transformers.
On the other hand, the main contribution is primarily an empirical application of this decomposition, which seems not well motivated in practical sense as well. I believe they are inspired from layer reuse and local alignments, but the paper does not present an algorithm that utilizes ADR or so-derived measure of geometric quantities.
The paper seems to merely choosing to call the symmetric and skew parts of attention as diffusion and advection, followed by the analysis of their magnitudes.
Questions for authors
What is the practical use of this study's measured quantities of the magnitudes of each component? Please provide explicit example of how they can be used for compression, architectural modification, decision in model selection, etc and support its practical utility with experiment, even possibly providing a useful hindsight whose merit can be evaluated quantitatively.
Did you consider a connection to Helmholtz–Hodge decomposition in the gradient flow formulation of Geshkovski et al, which is mentioned in the introduction of this paper?
6 comments
Part 2/3
Question 1: Practical Uses of Component Measurements.
What is the practical use of this study's measured quantities of the magnitudes of each component? Please provide explicit example of how they can be used for compression, architectural modification, decision in model selection, etc and support its practical utility with experiment, even possibly providing a useful hindsight whose merit can be evaluated quantitatively.
Response to Question 1:
The quantities distinguish mechanisms within the complete update and provide a basis for controlled modification and evaluation. Equation (17) separates skew action, discrepancy dissipation, and signed potential gain; Eq. (18) distinguishes action magnitude from its directional contribution to the update. Norms alone do not establish causal importance. Indeed, §4.2 shows that some component magnitudes can change under a normalization rescaling that leaves the block function unchanged.
The revised paper provides three concrete uses and their supporting evidence.
1. Constraining architectural modifications. Proposition 3 (§4.1, Eq. (14)) shows that, for a fixed causal connection with nonzero cross-token transport,
\quad\text{has causal support on arbitrary fields} \quad\Longleftrightarrow\quad \alpha=\delta.$$ For the input-dependent native map, the potential’s coefficients must also be considered. Coupling all three gains gives $$I+\gamma(\mathcal A_\pi-\mathcal L_\pi+\mathcal V_\pi) =(1-\gamma)I+\gamma T,$$ which preserves native input causality for a fixed layerwise gain. This supplies an explicit family of admissible modifications and explains why independently changing the geometric components can violate causality. Learning these gains remains a proposed architectural experiment (§4.1; Appendix E). **2. Evaluating approximation after the FFN responds.** An attention perturbation changes both the attention output and the FFN input. Equation (19) therefore evaluates the complete error through $$\delta F=\delta t+\delta r,\qquad |\delta F|^2-|\delta t|^2 =2\langle\delta t,\delta r\rangle+|\delta r|^2,$$ where $\delta r$ is the native FFN response. Across six checkpoints and 128 held-out windows per checkpoint, the FFN response opposes the attention error **yet amplifies the block-output error in 36.3–95.8% of evaluated cases** (§5.3; Appendix C.6, Table 9). This gives a quantitatively testable lesson for approximation: opposition alone does not establish compensation, and local fidelity must be measured after sequential composition. Prediction loss remains a separate endpoint; adding this response measure to the tested loss predictor did not meet the prespecified improvement threshold. **3. Testing an executable compression candidate.** Motivated by the transport spectra, Appendix D.2 approximates one middle-layer attention output projection using a calibration-fitted low-rank basis while preserving its affine term. On GPT-2 Small/Large and Qwen3-1.7B/4B, the approximation meets the declared **0.02-nat/token upper loss-change bound on 64 held-out windows**, including at approximately **25% of the selected projection’s original matrix multiply–accumulate budget** (Figure 7). At that budget, projection working memory falls by 57–71%. The stronger advantage over matched magnitude/random controls is established in the two GPT-2 cases; measured whole-model latency shows no meaningful speedup (Figure 8). These experiments connect operator analysis to concrete modifications and measurable consequences. The compression example is motivated by transport spectra and fitted to calibration outputs; we do not infer compressibility directly from ADR component magnitudes. Appendix D.1 additionally uses geometrically constrained interventions to distinguish native strength, task sensitivity, and the FFN’s response in an exploratory task study.Part 3/3
Question 2: Connection to Helmholtz–Hodge Decomposition.
Did you consider a connection to Helmholtz–Hodge decomposition in the gradient flow formulation of Geshkovski et al, which is mentioned in the introduction of this paper?
Response to Question 2:
The connection is relevant, with a distinction between the objects being decomposed. Helmholtz–Hodge decomposes a vector field into gradient, divergence-free, and harmonic components under appropriate geometry and boundary conditions [2]. Our construction decomposes the realized attention operator acting on token-feature fields, then retains its sequential composition with the nonlinear FFN.
For fixed routing, transports, and reference, Proposition 1 gives the quadratic energy
=\frac12\langle x,\mathcal L_ \pi x\rangle_ \pi, \qquad -\mathcal L_ \pi x=-\nabla_ \pi E_ {\mathcal L}(x), \qquad \langle x,\mathcal A_ \pi x\rangle_ \pi=0.$$ Thus the frozen diffusion component is a gradient flow of a nonnegative discrepancy energy, while the frozen skew component preserves the weighted norm in continuous time. This is a consequence of our operator identities; it does not establish a Helmholtz–Hodge decomposition of the full nonlinear Transformer dynamics or contraction of the native finite residual step (Appendices A.2 and B). Geshkovski et al. [3] distinguish normalized attention dynamics from an unnormalized surrogate admitting a Wasserstein gradient-flow formulation. Their formulation concerns evolution of token distributions, whereas ours describes a finite native block on token-feature fields. For input-dependent attention, differentiating a candidate energy or computing the divergence also differentiates the operator coefficients. The realized operator is therefore distinct from the full Jacobian, as stated in §2 after Eq. (3). Establishing a nonlinear Helmholtz–Hodge correspondence requires further analysis. **The exact finite-block representation in Theorem 2 does not require such a correspondence or a gradient-flow formulation of the Transformer.** **References:** > [2] Comments on the “Meshless Helmholtz-Hodge Decomposition” > > [3] A mathematical perspective on Transformers
Part 1/3Weakness: Research Motivation, Theoretical Novelty and Geometric Interpretation
Response to Weaknesses:
Our motivation is to understand how attention, normalization, residual connections, and nonlinear FFNs jointly organize computation within a complete Transformer block, and how this organization changes through depth. Extensive research studies these components individually; their sequential interaction remains challenging to characterize. Building on the connection-walk representation and block-level ADR perspective of Lin et al. (2026) [1], Theorem 2 (§3.2, Eq. (11)) gives an exact representation of the native sequential pre-norm block, preserving attention’s normalization, head-sum scale, affine terms, and the full nonlinear FFN evaluated at the attention-updated state.
The revision makes the mathematical content beyond the symmetric/skew-adjoint split more explicit. Proposition 1 (§3.1, Eqs. (7)–(8)) separates the realized connection as
\qquad \langle x,\mathcal L_ \pi x\rangle_ \pi =\frac12\sum_ {i,j}\pi_ iP_ {ij}|x_i-O_{ij}x_ j|^2\geq0.$$ Here, diffusion is defined by transported disagreement, while the token-diagonal potential accounts for gain/loss. Dissipativity holds for every positive token reference, without assuming stationary routing or orthogonal transports. The split is unique once the routing–transport pair, reference, and discrepancy form are fixed. Theorem 2 then embeds these coordinates into the complete native computation. **The decomposition is the starting point for the analysis.** It enables us to establish causal modification constraints (§4.1), identify which action vectors survive a function-preserving normalization rescaling (§4.2), compare component geometry and complete updates through depth (§§5.1–5.2), and quantify how the FFN changes attention-approximation errors (§§4.3 and 5.3). These are the substantive questions addressed by the paper. Convergence remains a further question; Appendix B specifies the additional assumptions needed for a continuous-depth limit. The operator viewpoint also supports analyses beyond the ADR split. Our transport-spectrum studies (Appendices D.3–D.4) motivate the executable approximation experiment in Appendix D.2, discussed below. **References:** > [1] From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers