- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces a dynamical-systems perspective on Transformers, using pattern-formation theory to explain how different architectural components influence the emergence of structured token representations. The authors derive a matrix-valued dispersion relation predicting which spatial and feature patterns are amplified and investigate how nonlinearities affect their stability. They further demonstrate that task-aligned initialization can improve learning efficiency on synthetic tasks and CIFAR-10.
Critical strengths
The paper presents a mathematically sophisticated and reasonably well-structured theoretical framework that extends existing analyses of rank collapse and clustering. The derivations provide a systematic characterization of how different architectural components influence representation dynamics. The authors also make an efort to connect their theoretical predictions to empirical results, including controlled experiments and an image-classification benchmark.
Critical weaknesses
The main weakness is the unclear conceptual and practical significance of the proposed framework. While the mathematical analysis is interesting, it remains difficult to understand what fundamentally new insights it provides into Transformer representations or learning. The observation that Transformers can amplify different spatial patterns is largely descriptive, and the functional relevance of these patterns remains unclear. While the proposed framework provides a mathematical characterization of architectural inductive biases, its practical significance remains unclear. The improvements obtained through task-aligned initialization largely confirm the established principle that appropriate inductive biases facilitate learning. The experiments do not convincingly demonstrate that the proposed theoretical framework offers substantial advantages over existing approaches to initialization or architectural design, particularly for realistic tasks where the relevant representational structures are not known in advance.
Questions for authors
no questions
5 comments
Response to Critical Weaknesses - Part 1
1. “While the mathematical analysis is interesting, it remains difficult to understand what fundamentally new insights it provides into Transformer representations or learning.”
We thank the reviewer for this remark, which gives us an opportunity to clarify the new insights brought by our analysis. We believe there is a lot of food for thought in our results, both for theorists and practitioners, and your remark suggests that we could make more of an effort to bring these insights to the forefront. We have therefore added takeaway boxes throughout the revised manuscript to highlight the main messages.
To keep things self-contained, we briefly summarize them here, together with their potential practical relevance.
(i) Understanding where emerging structures come from, and how architectural components select them.
Previous theoretical efforts have shed light on transformer token-representation through the lens of rank collapse, dynamical systems and mean-field analysis, often under simplifying assumptions. Pattern-formation theory allows us to go further and characterize the emergence of structures while retaining the main Transformer components.
Our central result is the Transformer Matrix Dispersion Relation (Theorem 4.5):
where is the Fourier transform of head ’s positional attention profile, its OV matrix, and the FFN Jacobian.
This reveals that positional encoding doesn’t only provide positional information but dictates the emerging structure via the positional part of attention, the commonly used relative positional encoding inducing quasi-Fourier modes. At linear order, token-space modes evolve independently, with amplification determined by . It also exhibits a division of labor: positional attention provides the available token structures (Fourier modes for translation-invariant PE), while its interplay with OV geometry and the FFN selects which modes are amplified.
We believe this has practical significance. It suggests that positional encodings could be designed to promote desired token structure (pattern formation theory generalizes different positional encoding schemes) and their spectral responses, and, importantly, that positional encoding and OV initialization should be considered jointly rather than independently. This sheds light on the empirical success of Mimetic initialization (Trockman & Kolter, 2023) and suggests principled design levers beyond heuristic choices.
(ii) A richer family of structures naturally supported by Transformers.
Our analysis accounts for previously identified clusters, but also characterizes standing waves, traveling waves, feature-rotating waves, and their combinationns as shown in Sec.4.3.
The existence of traveling waves is particularly interesting to us. Such structures have motivated dedicated architectures, including Neural Wave Machines, for modeling spatiotemporal data. We show that Transformers naturally support these structures as soft inductive biases, without requiring them to be hard-coded.
(iii) Understanding what happens beyond linear amplification: stabilization and competition.
The linear analysis determines which structures initially grow, but not what happens afterward. Our nonlinear analysis addresses this through amplitude equations, of the form
This exposes another division of labor: the dispersion relation determines which modes grow (), while content-dependent attention and FFN nonlinearities determine whether they stabilize or break apart ().
In particular, we show how activation function choices influence the stablization of token structure, an effect that, to our knowledge, has received little attention in Transformer theory. Indeed activation functions are mostly analyzed with forward and backward signal propagation considerations for trainability. We also characterize competition between simultaneously amplified modes possibly coming from different heads, deriving conditions for coexistence () versus winner-takes-all selection ().
We find this particularly interesting as it raises questions about representational redundancy and whether such competition could help explain phenomena like attention-head pruning. Although we do not establish this connection, our framework suggests directions to investigate it.
Taken together, we believe these insights go well beyond a descriptive account: they explain where structures come from, how architectural components select them, and what determines whether they stabilize, propagate, or compete.
We thank the reviewer again for encouraging us to make these insights more apparent in the revised manuscript.
Response to Critical Weaknesses - Part 2
2. “The improvements obtained through task-aligned initialization largely confirm the established principle that appropriate inductive biases facilitate learning.”
We agree with the reviewer, but would like to emphasize that we do not claim to rediscover the well-established fact that appropriate inductive biases accelerate learning. Rather, we take this principle as motivation for our work.
Our contribution is to characterize which soft inductive biases, which we call dynamical priors, are naturally present in Transformers, where they come from, and how they can be controlled.
Knowing that inductive biases facilitate learning does not tell us which structures a Transformer favors or how to promote one over another. Our framework provides precisely this understanding through .
The controlled experiments then establish that these theoretically identified priors have consequences for learning. Importantly, we compare identical architectures with matched maximal amplification gains , varying only the selected token–feature structures. The task–initialization alignment observed in Fig. 4 therefore suggests that it is not simply amplification strength that matters, but which structure is amplified.
In fact, motivated by the question of their functional relevance, we’ve conducted an additional experiment on Rotating MNIST, inspired by the Neural Wave Machines (Keller & Welling, 2023) setting.
In this experiment, a weight-tied Transformer sees the first 6 frames of a digit rotating by 20° per frame and must then continue the rotation autonomously, including a second full turn never seen in training. We compare the accuracy of Transformers that differ only in their initialization: standard, standing-wave, and traveling-wave.
Table. Rotating MNIST, weight-tied Transformer, test MSE (summed over pixels), 3 seeds, mean ± std. All models share the same architecture, parameter count and gain; only the initialization differs. Early learning: validation forecast MSE after 250 steps. Forecast: frames 7–18 (within the training horizon). Extrapolation: frames 19–36 (a second turn, never trained).
| init | early learning ↓ | forecast ↓ | extrapolation ↓ |
|---|---|---|---|
| traveling wave, 4 coexisting modes | 22.7 ± 0.5 | 1.83 ± 0.08 | 9.6 ± 3.8 |
| traveling wave, single mode | 27.3 ± 1.4 | 2.39 ± 0.05 | 87.4 ± 15.9 |
| standing wave | 33.1 ± 0.2 | 2.89 ± 0.06 | 78.5 ± 4.1 |
| standard | 32.9 ± 1.7 | 2.48 ± 0.10 | 62.1 ± 10.4 |
The 4-mode traveling-wave initialization learns faster, forecasts better, and extrapolates 6.5× better than the standard initialization: it is the only one that keeps rotating the digit beyond the training horizon. More details and visualizations (rollouts for every digit class, learning curves) are in the revised PDF (Sec. 6.2, Fig. 5, App. K).
This Rotating MNIST experiment complements the task-alignment one of Sec.6.2: while the latter focuses on mode selection within different classes of representations, the former highlights the importance of selecting a class of representations appropriate to the task.
We hope this provides additional evidence of the practical relevance of these newly exhibited structures, and illustrates how our theoretical characterization could help practitioners exploit them within Transformers rather than necessarily resorting to dedicated architectures. We believe this is an important validation of the practical relevance of the dynamical priors identified by our theory.
Answer to gPKt's strengths
Response to Critical Strengths
We thank the reviewer for their careful reading and constructive feedback. We appreciate that you recognize the mathematical sophistication of our work.
Characterizing structures emerging from Transformer dynamics is a difficult task that has motivated substantial theoretical efforts, often requiring simplified architectures or strong assumptions. We have found pattern-formation theory to be a productive framework, allowing us to retain positional encoding, multi-head attention, and FFN nonlinearities, understand their interplay, and characterize previously unreported structures.
We believe this understanding is important both theoretically, in identifying the soft inductive biases of Transformers, and practically, in shedding light on architectural choices and suggesting new directions for research. Although our work is primarily theoretical, we value practical relevance and appreciate that you recognized our experimental efforts.