acceptodds
Under review as a conference paper at ICLR 2027

Pattern Formation in Transformers

Abstract

What are the inductive biases of a Transformer architecture? Existing theory on how the forward pass shapes representations either considers whether Transformers escape from rank collapse or demonstrates that self-attention drives tokens toward cluster patterns. The latter view arises from an elegant dynamical systems perspective, but relies on simplified architectural assumptions, and does not explain the rich structures observed in practice. This leaves a major open question: when a full Transformer escapes rank collapse, how does it structure token representations? Using pattern-formation theory, we show that the dynamical view of Transformers can account for Positional Encoding, Multi-Head Attention, and Output-Value geometry. We demonstrate that a full Transformer architecture imposes an inductive prior by selectively amplifying a rich set of previously unreported patterns, including traveling or rotating waves among others. We characterize the role of each architectural component in controlling which pattern is amplified, which ones stabilize, compete, or coexist. Finally, we show that these structures can act as a controllable dynamical prior that facilitates learning. By choosing both task-aligned positional encoding and weight initialization, we demonstrate improved data efficiency and accelerated optimization on controlled sequence tasks and with ConViT on CIFAR-10.

Primary area:
learning theory
Keywords:
dynamical systems, representation learning, inductive bias, pattern formation, dispersion relation, complex systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gPKtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a dynamical-systems perspective on Transformers, using pattern-formation theory to explain how different architectural components influence the emergence of structured token representations. The authors derive a matrix-valued dispersion relation predicting which spatial and feature patterns are amplified and investigate how nonlinearities affect their stability. They further demonstrate that task-aligned initialization can improve learning efficiency on synthetic tasks and CIFAR-10.

Critical strengths

The paper presents a mathematically sophisticated and reasonably well-structured theoretical framework that extends existing analyses of rank collapse and clustering. The derivations provide a systematic characterization of how different architectural components influence representation dynamics. The authors also make an efort to connect their theoretical predictions to empirical results, including controlled experiments and an image-classification benchmark.

Critical weaknesses

The main weakness is the unclear conceptual and practical significance of the proposed framework. While the mathematical analysis is interesting, it remains difficult to understand what fundamentally new insights it provides into Transformer representations or learning. The observation that Transformers can amplify different spatial patterns is largely descriptive, and the functional relevance of these patterns remains unclear. While the proposed framework provides a mathematical characterization of architectural inductive biases, its practical significance remains unclear. The improvements obtained through task-aligned initialization largely confirm the established principle that appropriate inductive biases facilitate learning. The experiments do not convincingly demonstrate that the proposed theoretical framework offers substantial advantages over existing approaches to initialization or architectural design, particularly for realistic tasks where the relevant representational structures are not known in advance.

Questions for authors

no questions

5 comments
AuthorsOpenReview ↗

Answer to gPKt's strengths

Response to Critical Strengths

We thank the reviewer for their careful reading and constructive feedback. We appreciate that you recognize the mathematical sophistication of our work.

Characterizing structures emerging from Transformer dynamics is a difficult task that has motivated substantial theoretical efforts, often requiring simplified architectures or strong assumptions. We have found pattern-formation theory to be a productive framework, allowing us to retain positional encoding, multi-head attention, and FFN nonlinearities, understand their interplay, and characterize previously unreported structures.

We believe this understanding is important both theoretically, in identifying the soft inductive biases of Transformers, and practically, in shedding light on architectural choices and suggesting new directions for research. Although our work is primarily theoretical, we value practical relevance and appreciate that you recognized our experimental efforts.

AuthorsOpenReview ↗

Response to Critical Weaknesses - Part 1

1. “While the mathematical analysis is interesting, it remains difficult to understand what fundamentally new insights it provides into Transformer representations or learning.”

We thank the reviewer for this remark, which gives us an opportunity to clarify the new insights brought by our analysis. We believe there is a lot of food for thought in our results, both for theorists and practitioners, and your remark suggests that we could make more of an effort to bring these insights to the forefront. We have therefore added takeaway boxes throughout the revised manuscript to highlight the main messages.

To keep things self-contained, we briefly summarize them here, together with their potential practical relevance.

(i) Understanding where emerging structures come from, and how architectural components select them.

Previous theoretical efforts have shed light on transformer token-representation through the lens of rank collapse, dynamical systems and mean-field analysis, often under simplifying assumptions. Pattern-formation theory allows us to go further and characterize the emergence of structures while retaining the main Transformer components.

Our central result is the Transformer Matrix Dispersion Relation (Theorem 4.5):

J(q)=C⋆[I+∑h=1Hλh(q)Mh],J(q)=C_\star\left[I+\sum_{h=1}^H\lambda_h(q)M_h\right],

where λh(q)\lambda_h(q) is the Fourier transform of head hh’s positional attention profile, MhM_h its OV matrix, and C⋆C_\star the FFN Jacobian.

This reveals that positional encoding doesn’t only provide positional information but dictates the emerging structure via the positional part of attention, the commonly used relative positional encoding inducing quasi-Fourier modes. At linear order, token-space modes evolve independently, with amplification determined by ρ(J(q))\rho(J(q)). It also exhibits a division of labor: positional attention provides the available token structures (Fourier modes for translation-invariant PE), while its interplay with OV geometry and the FFN selects which modes are amplified.

We believe this has practical significance. It suggests that positional encodings could be designed to promote desired token structure (pattern formation theory generalizes different positional encoding schemes) and their spectral responses, and, importantly, that positional encoding and OV initialization should be considered jointly rather than independently. This sheds light on the empirical success of Mimetic initialization (Trockman & Kolter, 2023) and suggests principled design levers beyond heuristic choices.

(ii) A richer family of structures naturally supported by Transformers.

Our analysis accounts for previously identified clusters, but also characterizes standing waves, traveling waves, feature-rotating waves, and their combinationns as shown in Sec.4.3.

The existence of traveling waves is particularly interesting to us. Such structures have motivated dedicated architectures, including Neural Wave Machines, for modeling spatiotemporal data. We show that Transformers naturally support these structures as soft inductive biases, without requiring them to be hard-coded.

(iii) Understanding what happens beyond linear amplification: stabilization and competition.

The linear analysis determines which structures initially grow, but not what happens afterward. Our nonlinear analysis addresses this through amplitude equations, of the form

B+−B=μB−βB3,μ=ρ(J(qc))−1B^+-B=\mu B-\beta B^3,\qquad \mu=\rho(J(q_c))-1

This exposes another division of labor: the dispersion relation determines which modes grow (μ\mu), while content-dependent attention and FFN nonlinearities determine whether they stabilize or break apart (β\beta).

In particular, we show how activation function choices influence the stablization of token structure, an effect that, to our knowledge, has received little attention in Transformer theory. Indeed activation functions are mostly analyzed with forward and backward signal propagation considerations for trainability. We also characterize competition between simultaneously amplified modes possibly coming from different heads, deriving conditions for coexistence (βcross<βself\beta_{\mathrm{cross}}<\beta_{\mathrm{self}}) versus winner-takes-all selection (βcross>βself\beta_{\mathrm{cross}}>\beta_{\mathrm{self}}).

We find this particularly interesting as it raises questions about representational redundancy and whether such competition could help explain phenomena like attention-head pruning. Although we do not establish this connection, our framework suggests directions to investigate it.

Taken together, we believe these insights go well beyond a descriptive account: they explain where structures come from, how architectural components select them, and what determines whether they stabilize, propagate, or compete.

We thank the reviewer again for encouraging us to make these insights more apparent in the revised manuscript.

AuthorsOpenReview ↗

Response to Critical Weaknesses - Part 2

2. “The improvements obtained through task-aligned initialization largely confirm the established principle that appropriate inductive biases facilitate learning.”

We agree with the reviewer, but would like to emphasize that we do not claim to rediscover the well-established fact that appropriate inductive biases accelerate learning. Rather, we take this principle as motivation for our work.

Our contribution is to characterize which soft inductive biases, which we call dynamical priors, are naturally present in Transformers, where they come from, and how they can be controlled.

Knowing that inductive biases facilitate learning does not tell us which structures a Transformer favors or how to promote one over another. Our framework provides precisely this understanding through J(q)J(q).

The controlled experiments then establish that these theoretically identified priors have consequences for learning. Importantly, we compare identical architectures with matched maximal amplification gains max⁡qρ(J(q))\max_q\rho(J(q)), varying only the selected token–feature structures. The task–initialization alignment observed in Fig. 4 therefore suggests that it is not simply amplification strength that matters, but which structure is amplified.

In fact, motivated by the question of their functional relevance, we’ve conducted an additional experiment on Rotating MNIST, inspired by the Neural Wave Machines (Keller & Welling, 2023) setting.

In this experiment, a weight-tied Transformer sees the first 6 frames of a digit rotating by 20° per frame and must then continue the rotation autonomously, including a second full turn never seen in training. We compare the accuracy of Transformers that differ only in their initialization: standard, standing-wave, and traveling-wave.

Table. Rotating MNIST, weight-tied Transformer, test MSE (summed over pixels), 3 seeds, mean ± std. All models share the same architecture, parameter count and gain; only the initialization differs. Early learning: validation forecast MSE after 250 steps. Forecast: frames 7–18 (within the training horizon). Extrapolation: frames 19–36 (a second turn, never trained).

initearly learning ↓forecast ↓extrapolation ↓
traveling wave, 4 coexisting modes22.7 ± 0.51.83 ± 0.089.6 ± 3.8
traveling wave, single mode27.3 ± 1.42.39 ± 0.0587.4 ± 15.9
standing wave33.1 ± 0.22.89 ± 0.0678.5 ± 4.1
standard32.9 ± 1.72.48 ± 0.1062.1 ± 10.4

The 4-mode traveling-wave initialization learns faster, forecasts better, and extrapolates 6.5× better than the standard initialization: it is the only one that keeps rotating the digit beyond the training horizon. More details and visualizations (rollouts for every digit class, learning curves) are in the revised PDF (Sec. 6.2, Fig. 5, App. K).

This Rotating MNIST experiment complements the task-alignment one of Sec.6.2: while the latter focuses on mode selection within different classes of representations, the former highlights the importance of selecting a class of representations appropriate to the task.

We hope this provides additional evidence of the practical relevance of these newly exhibited structures, and illustrates how our theoretical characterization could help practitioners exploit them within Transformers rather than necessarily resorting to dedicated architectures. We believe this is an important validation of the practical relevance of the dynamical priors identified by our theory.

Reviewer_CjwrOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how a Transformer's architecture favors particular patterns in its internal representations before training. It introduces a mathematical framework explaining how positional information, attention, and feed-forward layers amplify patterns such as waves and feature rotations, and when these patterns stabilize or compete. Simulations support these predictions, while controlled tasks show benefits from matching the initial pattern preference to the task. Experiments with a weight-tied ConViT on CIFAR-10 also demonstrate faster learning and improved performance with limited training data, reaching 91.5% accuracy in the strongest configuration. The main contribution is an interpretable connection between architectural choices, representation structure, and initialization.

Critical strengths
  1. The paper is well-written and the story is sound.

  2. The paper offers a useful explanation of why Transformer representations can develop particular structures even before learning begins.

  3. The experiments are helpful and help connect the mathematical predictions to observable patterns.

Critical weaknesses
  1. It seems unclear to me how well the findings generalize to standard Transformers, as the learning experiments reuse block across layers. The extra benefit of positional sharpening also disappears with pre-LN, which might suggest that the initialization advice depends on the architecture.

  2. The CIFAR-10 results show better performance, but do not establish which predicted patterns help classification. It remains unclear how much the proposed explanation adds beyond the known benefits of local attention and negative output-value initialization.

Questions for authors
  1. Is there any example of how a traveling wave or feature rotation helps the model solve a task? It would be much useful to understand what do the patterns represent.

  2. In the synthetic tasks, the relevant shift, frequency, or rotation is known in advance. How can we choose an initialization when the useful structure is unknown? Or does it still require empirical tuning?

  3. Does the analysis of pattern competition and coexistence suggest that encouraging several patterns may help tasks require information at different spatial scales?

7 comments
AuthorsOpenReview ↗

Answer to Cjwr's strengths

Response to Critical Strengths

We thank the reviewer for their thoughtful and constructive review. We appreciate that you recognize the introduction of pattern-formation theory and its effectiveness in shedding light on the role of different Transformer components in the emergence of structured token representations.

Previous works on Transformer theory have often had to resort to simplified architectures (omitting positional encoding, FFNs, etc.) or restrictive assumptions on the input data. We’re glad you appreciated our pattern formation theory framework, which allows us to retain these components, understand their interplay, and predict new structures and their origins.

Although our work is mainly theoretical, we value practical relevance, so we’re thankful that you recognized our experimental efforts. We also appreciate your remarks on the presentation. We’ve dedicated considerable effort to making the work as pedagogical and digestible as possible, hoping that our insights can benefit the community and encourage future work, both theoretical and practical.

AuthorsOpenReview ↗

Response to Critical Weaknesses

1. Generalization to untied Transformers and LayerNorm.

Thank you for raising this concern. We considered weight-tied Transformers mainly for convenience, as this facilitates the theoretical exposition, particularly when studying nonlinear stabilization through repeated application of the same map.

However, your concern is valid, and we’ve added experiments on the untied ConViT variant to our benchmark (Fig. 6 in the revised article). We observe that the ordering of the initialization strategies remains the same, suggesting that our predictions also transfer to the untied setting.

Table. CIFAR-10 test accuracy (%), ConViT, 3 seeds. Untied: 12 independent blocks, each initialized i.i.d. as in Sec. 6.3 (lr 10−310^{-3}).

inittied, full data (Fig. 6)untied, full datauntied, 25% datauntied, 10% data
default86.9 ± 0.386.3 ± 0.5––
default + pos89.6 ± 0.490.7 ± 0.180.5 ± 0.569.6 ± 0.3
−I-I + pos91.1 ± 0.391.2 ± 0.181.8 ± 0.570.7 ± 0.9
−I-I + sharp pos91.5 ± 0.291.6 ± 0.182.5 ± 0.672.2 ± 0.2

Regarding your concerns on the generalization to pre-LN architectures, we omitted it mainly to streamline the analysis, also motivated by previous works suggesting that it is not strictly necessary for successful Transformer training (see Transformers without normalization. (Zhu et al, 2025)), at least in certain settings. Theoretically, the exposition in A.9 supports the use of normalization, but also shows a flattening of the dispersion relation with depth. The loss of sharpening with pre-LN may therefore be due to that effect, combined with an overall lower accuracy of pre-LN architectures with respect to no normalization for this task. Alternative methods may be developed to have a sharpening resistant to normalization, although we believe this goes out of the scope of our work.

2. What do the CIFAR-10 results establish beyond the known benefits of localized attention and negative OV initialization?

This is a fair point: our CIFAR-10 experiments do not establish which particular patterns are useful for classification. Rather, our goal was to show that the theoretical understanding provided by our framework could help explain and guide initialization choices in a more realistic setting.

Indeed, while the benefits of localized attention and negative OV initialization were already known, our analysis sheds light on why these two components work together. The amplification of a mode qq is determined by the matrix dispersion relation

J(q)=C⋆[I+∑hλh(q)Mh],J(q)=C_\star\left[I+\sum_h\lambda_h(q)M_h\right],

which shows that positional attention and OV geometry should not be considered in isolation. In particular, their interplay can shift amplification away from the homogeneous mode q=0q=0 and toward heterogeneous structures. This also provides an explanation of what is really at play when we use localized attention and negative OV weights: relevant quasi-fourier modes emerge upon initialization, which consitutes a useful prior, at least for the image modality.

We believe this is an interesting insight, as it provides a theoretical explanation for previously heuristic design choices, and suggests ways to go beyond them. For example, rather than simply favoring localized attention or negative OV weights, one could aim to design their joint spectral response. Additionally, since relative positional encoding induces quasi-fourier modes, maybe designing positional encoding schemes can be performed with these modal considerations in mind.

We agree that identifying which particular structures benefit classification would be interesting. Our controlled experiments address this more directly, while CIFAR-10 provides evidence that the design principles suggested by the theory can also be useful beyond synthetic tasks.

AuthorsOpenReview ↗

Response to Question 1

1. Is there any example of how a traveling wave or feature rotation helps the model solve a task?

We thank the reviewer for this question on the interpretability of the obtained patterns. Firstly, we would like to emphasize that, to the best of our knowledge, the emergence of these structures from Transformer dynamics had not previously been characterized.

Traveling waves have actually been considered useful inductive biases for spatiotemporal data (eg brain activity, or physical systems), motivating dedicated architectures such as Neural Wave Machines (Keller & Welling, 2023), which hard-code such dynamics. We found it interesting that these structures are already naturally supported by Transformers, albeit as soft dynamical priors.

To answer your question more directly, our controlled experiments on transport (T2) and feature rotation (T3) provide a first illustration: initializing the Transformer to favor the corresponding propagation phase $ \omega_c=\arg\Lambda_c $ or feature-space geometry vcv_c facilitates learning.

However, we agree that this is only a partial confirmation. To address your concern, we have conducted additional experiments on Rotating MNIST, inspired by the setting of the Neural Wave Machines paper.

In this experiment, a weight-tied Transformer sees the first 6 frames of a digit rotating by 20° per frame and must then continue the rotation autonomously, including a second full turn never seen in training. We compare Transformers that differ only in their initialization: standard, standing-wave, and traveling-wave.

Table. Rotating MNIST, weight-tied Transformer, test MSE (summed over pixels), 3 seeds, mean ± std. All models share the same architecture, parameter count and gain; only the initialization differs. Early learning: validation forecast MSE after 250 steps. Forecast: frames 7–18 (within the training horizon). Extrapolation: frames 19–36 (a second turn, never trained).

initearly learning ↓forecast ↓extrapolation ↓
traveling wave, 4 coexisting modes22.7 ± 0.51.83 ± 0.089.6 ± 3.8
traveling wave, single mode27.3 ± 1.42.39 ± 0.0587.4 ± 15.9
standing wave33.1 ± 0.22.89 ± 0.0678.5 ± 4.1
standard32.9 ± 1.72.48 ± 0.1062.1 ± 10.4

The 4-mode traveling-wave initialization learns faster, forecasts better, and extrapolates 6.5× better than the standard initialization: it is the only one that keeps rotating the digit beyond the training horizon. The benefit requires several coexisting traveling modes: with a single mode, extrapolation is no better than with the standard initialization (see also Q3). More details and visualizations (rollouts for every digit class, learning curves) are in the revised PDF (Sec. 6.2, Fig. 5, App. K).

We hope this provides further evidence of the practical relevance of these structures. More generally, we believe that exhibiting traveling waves and characterizing the conditions under which they emerge gives practitioners useful knowledge: a properly initialized Transformer could serve as an alternative to dedicated architectures for certain spatiotemporal tasks.

Reviewer_RHp4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how small perturbations around homogeneous token states evolve through Transformer blocks. It derives a matrix-valued dispersion relation linking positional encoding, multi-head attention, OV geometry, and the FFN to the amplification of joint token–feature modes. The nonlinear analysis describes saturation, phase drift, and competition near the instability threshold. The experiments validate these predictions numerically, test task-aligned initialization on controlled sequence tasks, and evaluate a weight-tied ConViT on CIFAR-10, where the authors report improvements in optimization and accuracy.

Critical strengths

I find the analysis of how Transformer components shape token representation patterns particularly interesting. The analysis brings positional encoding, multi-head attention, and feature transformations into a common framework and goes beyond initial growth to characterize nonlinear saturation and competition. This gives a useful account of representation dynamics within the stated assumptions.

The controlled sequence experiments make the proposed dynamical prior concrete. Task-aligned and mismatched initializations are compared with the same architecture, parameter count, and maximal linear gain. The results support the importance of which structure is amplified, beyond the amount of amplification. I find this connection between the theory and learning behavior valuable.

Critical weaknesses

My main concern is the indirect connection between the predicted patterns and representations produced by real data. The finite-depth validation starts from a homogeneous state with a small prescribed perturbation (Appendix K.3.1). Figure 12 measures token heterogeneity, while Figure 13 tracks the dispersion relation computed from model checkpoints. These results support the local analysis and show that the representations remain non-collapsed. But they do not directly establish whether actual image inputs undergo the mode-selective amplification predicted by the theory. This limits the interpretation of the CIFAR-10 gains as evidence for the proposed pattern-formation mechanism, without undermining the theoretical results under their stated assumptions.

Second, the implications for model design remain unclear when the useful target pattern is unknown. The paper explains why positional locality and negative OV geometry should be considered jointly. In the synthetic tasks, the target pattern is specified, but it is less clear how a practitioner should choose a spectral preference for a new task. The pre-LN results illustrate this issue: the benefit of negative OV with positional locality persists, whereas the additional benefit of sharpening does not (Table 1). I recognize that Appendix A.9 offers an explanation. What remains unclear is how this explanation should guide the design of a pre-LN, untied ViT. A clearer account of which design choices the results support, and which remain hypotheses, would strengthen this part of the paper.

Questions for authors

For additional validation, could you quantify whether the modes predicted to be favored at initialization are preferentially amplified across depth for actual CIFAR-10 inputs, and whether the differences between initialization variants follow the predictions? Accounting for the input spectrum would help separate selective amplification from structure already present in the images. A limited quantitative correspondence would be useful; this need not cover every pattern type.

For clarification, what design guidance follows for a pre-LN, untied ViT when the useful target spectrum is unknown, beyond considering positional locality and OV geometry jointly? How should the pre-LN analysis and the loss of the sharpening benefit inform these choices? Please distinguish guidance supported by the present results from directions that remain hypotheses.

4 comments
AuthorsOpenReview ↗

Response to critical strengths

Thank you for your constructive review. We are glad that you found our approach interesting, and for acknowledging the effort we put in considering all the components of the Transformer. This allowed us to explore the whole set of emerging representations these architectures can support.

We appreciate that you found the experiments concrete, and for your recognition of the rigour of the experimentation. This allowed us to draw solid links between our theoretical findings and practical implications.

AuthorsOpenReview ↗

Response to Critical Weakness - Part 1

1. Connection between the predicted patterns and representations produced by real data

Thank you for this valuable question. We’d like to start by clarifying why our analysis focuses on the homogeneous state.

Previous works have established that Transformer token representations can be driven toward homogeneity as they propagate through the network, even for different inputs. This phenomenon is problematic not only because the resulting representations become uninformative, but also because it has been linked to vanishing gradients, hindering learning.

This motivated us to focus our efforts on understanding which architectural components, initializations, and parameterizations can make this state unstable, and what structures emerge in its place, independently of any particular input. This motivation has been made clearer in the revised paper.

However, your concern is valid: do these predictions also describe the evolution of representations produced by real images? To address this more concretely, we’ve conducted two additional experiments.

(i) Mode-selective amplification on real CIFAR-10 inputs.

Rather than starting from a prescribed perturbation around the homogeneous point, we pass 2,000 CIFAR-10 test images through the network and measure, mode by mode, how their Fourier components are amplified after 12 layers. Importantly, we account for the input spectrum, so as to distinguish the amplification induced by the architecture from the structure already present in the images.

We find remarkably strong agreement between the measured amplification and the theoretical predictions obtained using the measured input covariance, with no fitted parameters.

The different initializations also behave as predicted.

Table. Spectral transfer on 2,000 CIFAR-10 test images (tied ConViT). Pearson: measured vs predicted per-mode transfer after 12 layers at initialization (no fitted parameters). Energy share: fraction of heterogeneous energy in the 10% of modes predicted to be most amplified, input → layer 12 (prediction in brackets).

initPearson (init)energy share, input → layer 12
default1.000.8% → 1.0% (0.9%)
default + pos0.990.8% → 0.9% (0.9%)
+I+I + pos1.0067.3% → 90.4% (89.8%)
−I-I + pos0.983.8% → 8.9% (9.5%)
−I-I + sharp pos0.984.1% → 15.3% (17.2%)

(ii) Linearization around actual image representations.

We also went a step further, computing the Jacobian of each block at the actual token representations produced by images, rather than at the homogeneous state.

Interestingly, we find that the per-mode gains remain very close to those predicted by the homogeneous dispersion relation J(q)J(q).

Table. Block Jacobian at actual CIFAR-10 token states (64 images, initialization): Spearman correlation over all 256 modes between the measured per-mode gain and the homogeneous ρ(J(q))\rho(J(q)), and leakage into other modes (max over layers, relative to the in-mode gain).

initblock 1block 4block 7block 12leakage
default0.950.980.970.92≤ 2.4%
default + pos1.001.001.000.96≤ 2.4%
+I+I + pos1.000.990.980.98≤ 7.0%
−I-I + pos1.001.001.001.00≤ 1.8%
−I-I + sharp pos1.001.001.001.00≤ 1.9%

These results are particularly interesting, as they show that our analysis around the homogeneous state transfer to the spectral preferences of Transformer dynamics on real images. It provides additional evidence that the mode-selective amplification mechanism predicted by our theory is indeed at play on real data. We have included these experiments in App.L.3.2 of our revised paper.

Overall, we are thankful that you raised this point, as we believe these results considerably strengthen the connection between our theory and practical settings.

AuthorsOpenReview ↗

Response to Critical Weakness - Part 2

2. Design guidance when the useful target structure is unknown

This is a very good point. We agree that in realistic settings, the relevant target structure is rarely known in advance, and choosing an appropriate initialization would probably still require some empirical tuning.

However, we believe the insights provided by our theory can make this process more targeted and informed.

For instance, Mimetic initialization (Trockman & Kolter, 2023) was partly motivated by analyzing the weights of pretrained Transformers. Our analysis suggests that OV weights alone are insufficient to characterize the resulting dynamical prior: their interplay with positional attention must be considered jointly, as captured by the matrix dispersion relation

J(q)=C⋆[I+∑hλh(q)Mh].J(q)=C_\star\left[I+\sum_h\lambda_h(q)M_h\right].

This already provides a useful design principle: positional encoding and OV initialization should be considered together, rather than as independent choices.

Additionally, the token representation at initialization is strongly tied to the modes of the positional part of attention. For commonly used relative positional encoding, this induces quasi-Fourier modes. We believe this insight provides actionable levers, such as opting for encodings that induce alternative modal structure.

An interesting possibility would therefore be to inspect the dispersion relation of pretrained Transformers, rather than only their weights, and use this information to jointly design positional encodings and OV matrices. This is not something we’ve investigated, but we believe it provides solid ground for dedicated practical follow-up work.

We do not claim to provide a universal recipe for optimal initialization. Rather, we believe that characterizing these dynamical priors, understanding the role of each architectural component, and identifying the mechanisms through which they can be controlled already provides useful knowledge for practitioners and suggests new design directions.

As for untied Transformers, our linear theory does not require weight tying (Appendix A.10), we’ve added the untied Transformer to our CIFAR-10 benchmark and show that the performance ordering is maintained, suggesting that our theoretical insights on tied-transformers transfer to the more realistic untied case.

Table. CIFAR-10 test accuracy (%), ConViT, 3 seeds. Untied: 12 independent blocks, each initialized i.i.d. as in Sec. 6.3 (lr 10−310^{-3}).

inittieduntied
default86.9 ± 0.386.3 ± 0.5
default + pos89.6 ± 0.490.7 ± 0.1
−I-I + pos91.1 ± 0.391.2 ± 0.1
−I-I + sharp pos91.5 ± 0.291.6 ± 0.1

We thank the reviewer for addressing this issue that encouraged us to provide more robust practical implications. We have added a selection of the experiment above (in Fig.6 of the revised paper).

AuthorsOpenReview ↗

Response to Questions

1. Can the predicted mode amplification be measured on real CIFAR-10 inputs?

Thank you for this suggestion. This is precisely what motivated the two additional experiments described in our response to the first weakness.

By measuring the spectral amplification of actual CIFAR-10 representations, accounting for the input spectrum, and comparing the results with theoretical predictions, we find strong quantitative agreement across all initialization variants.

We hope these new results address your concern. We believe they provide particularly valuable evidence that the theoretical mechanisms we characterize are also present in the representations of real data.

2. What design guidance follows for pre-LN, untied ViTs, and what remains hypothetical?

We thank the reviewer for encouraging us to clarify this distinction.

The Table in "Weakness Part 2" above shows that the designs provided by our theory is adaptable to untied ViTs, which supports the adaptability of our approach.

As you have pointed out in A.9, the theory supports the use of normalization, but also shows a layer-wise flattening of the dispersion relation. The loss of sharpening with pre-LN may therefore be due to that effect, combined with pre-LN architectures being less accurate than no normalization for this task. This choice of removing normalization is actually supported by Transformers without normalization. (Zhu et al, 2025) on ViTs, which is why we chose to simplify the architecture. Nevertheless, we admit that within this experimental setup, the practical design guidance of ViTs is not adapted to the use of pre-LN Transformers. Alternative architectural designs may be developed to have a sharpening resistant to normalization, and we believe this could be an interesting practical perspective to our work.

More generally, we believe that using the dispersion relation to guide initialization, possibly by inspecting the spectral preferences of pretrained models, is an interesting direction. However, automatically identifying which structures should be favored for a new task remains an open question.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Revisions

Dear Reviewers and AC,

We thank all reviewers for their constructive feedback. We have uploaded a revised version of the paper, with all changes marked in blue, and have responded to each reviewer individually. If our responses address your concerns, we would be grateful if you would consider updating your score

For the AC, we would like to briefly summarize our main contributions.

We study the non-trivial structures of Transformer token representations through the novel lens of pattern-formation theory. Contrary to previous dynamical approaches, which typically rely on simplified architectures, our framework allows us to account for the main Transformer components (with an extension to LayerNorm in the appendix). It provides us with tools such as dispersion relations and amplitude equations, allowing us to characterize the structures Transformers naturally promote, their stability, and the role of each architectural component.

Our contributions are the following:

On the theoretical side:

  • We derive the Transformer Matrix Dispersion Relation J(q), showing that emerging token representations are strongly tied to the modal structure of positional attention. To first order, these modes evolve independently, with their amplification or decay governed by J(q).
  • This allows us to characterize the role of each component: positional encoding determines the modal structure, its specific form shapes the spectral response, and multi-head attention, through its interplay with OV geometry, allows us to engineer band-pass filters. We also extend the previous taxonomy beyond cluster formation, revealing standing waves, traveling waves, and feature rotations, which we exhibit through simulations.
  • Through a nonlinear analysis, we derive amplitude equations that provide stability conditions for these structures, allowing us to establish phase diagrams and scaling laws that we verify numerically. This also reveals previously unappreciated roles of activation functions and characterizes coexistence and winner-takes-all regimes between modes from different heads.

On the practical side:

  • Our analysis shows that what matters for the emergence of these dynamical priors is the interplay between OV geometry and positional encoding, refining our understanding of previous heuristics such as Mimetic and ConViT initialization.
  • We show that these dynamical priors can facilitate learning, both on controlled tasks and on the more realistic Rotating MNIST experiment added in the revision, where traveling-wave priors and mode coexistence prove useful.
  • Through new ablations, we show through experiments that our theory transfers to untied transformers.
  • We demonstrate how the dispersion relation can serve as a principled tool for Transformer initialization through experiments with ConViT on CIFAR-10. We have also added experiments with untied Transformers and measurements on actual CIFAR-10 representations, showing strong agreement between theoretical predictions and observed mode amplification.

We have tried to address each reviewer’s comments to the best of our ability, given our resources and time constraints. We have uploaded a revised paper with additional experiments and a clearer exposition of our insights, notably through added takeaway boxes and remarks.

We believe these additions make the paper clearer and more convincing, and we are thankful to the reviewers for that. We believe the insights provided here can serve both theorists and practitioners, and we will release our code to ensure full reproducibility of our results and facilitate follow-up work.

We thank the reviewers and AC again for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.