- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors consider the clustering evaluation of token representations for transformers whose parameter are randomly initialized. They find that the long-time behavior is controlled by and which are explicit quantities. They also connect token synchronization and Lyapunov exponents based on the classical theory of random dynamical system.
Critical strengths
The strength of this paper is to characterize clustering behavior in the long time regime. Some quantities appeared in several sufficient conditions are explicit, which directly provides the practical benefits. The experiments provide good empirical support for the theoretical results.
Critical weaknesses
The weakness of this paper is that, although the sufficient conditions are explicit, the analysis depends on Gaussian initialization, as the authors already acknowledged. Also, this paper studies the long-time clustering behavior of tokens in deep random transformers at initialization, while it does not study the evolution of model parameters during training.
Questions for authors
-
Is there any intrinsic difficulty to extend the result to non-Gaussian settings ? It might be difficult to discuss general distributions, but what about some concrete distribution, or the setting close to Gaussian, for instance, sub-Gaussian distributions ?
-
To be closer to practical settings, could authors discuss the evolution of model parameters during training ? I do not ask the authors to add details, but just a brief discussion on this possibility.
We thank the reviewer for their positive feedback and their comments, which we address below.