- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies high-dimensional embeddings of natural language under the lens of dynamical systems. Using various embedding and (transformer-based) language models, text in several languages is mapped into a trajectory in vector space, with the token-axis acting as the traditional "time" axis. The major finding appears to be that pooled representations (one embedding for an entire piece of text) evolve slowly along the token axis, while in contrast last-token representations in language models show much more variation.
While the approach is generally interesting, it is not particularly new, and the findings are limited in scope and lack proper contextualization and implications. The main flaw of the paper is that the presentation is really poor: writing is dense and jargony, and there are no figure to illustrate empirical results. Given these shortcomings, in my opinion the paper -- in its current form -- does not bring enough scientific and didactic value to the ICLR community, and should not be accepted.
Critical strengths
None.
Critical weaknesses
Several substantial issues with the paper in its current form.
-
It's very difficult to understand the paper. The paper appears to rely heavily on AI-generated text, which itself is not a problem (as far as I am concerned) as long as the clarity and rigor are not compromised (and, ideally, improved). Here that's not the case. The abstract itself is overly dense and obscure. Consider for example sentences like: "Accepted causal coordinates are moderately predictable and usually contract rapidly under a fitted output-space surrogate; pooled coordinates are far more predictable but evolve near the unit timescale"
-
There are no figures to illustrate the empirical findings! All results are presented as textual summaries and long tables of results. The lack of figures, itself unusual, also makes reading and understanding the findings particularly difficult.
-
Findings are limited. While the results presented are interesting, there is almost no contextualization, description of implications for the field.
Questions for authors
What are the implications of your findings to our general understanding of language models, language itself, or dynamical systems?
Thank you for your careful reading and straightforward criticisms. We appreciate that you found the approach and results interesting, and address the weaknesses and questions one by one in the following.
W0: Novelty
With the table below, we explain the novelty and contributions of this work compared to previous papers.
W1: Poor presentation
Thank you for pointing this out. In the revised manuscript, we have manually polished the writing.
W2: No figures
Thank you for the comments. In the revised manuscript, we have added several figures to demonstrate our experimental results:
W3: Limited findings & Q1: Implications for the field
We list our findings as follows: