acceptodds
Under review as a conference paper at ICLR 2027

Neural Language Dynamics: Short-Horizon Stability and Dissipation of Perturbations

Abstract

We study neural language dynamics: treating the text rollout as a dynamic system, quantized by a pretrained embedding model one token at a time. We formulate this inference-time process as a discrete input–state–output system in which tokens drive an internal Markov state, read out by the embedding model. Our benchmark spans 15 model configurations, 7 languages, parallel sentence-scale FLORES+, and document-scale FineWeb/FineWeb2 trajectories. Local geometry separates sharply by readout: last-token trajectories retain large turning and approximately constant token-scale speed, whereas pooled trajectories slow with prefix length. A predictive gate accepts a reduced or delay coordinate on 267 of 280 shards. Accepted causal coordinates are moderately predictable and usually contract rapidly under a fitted output-space surrogate; pooled coordinates are far more predictable but evolve near the unit timescale, while Chinese causal trajectories form a slower, more metastable intermediate regime. Paired interventions on English FineWeb across three Qwen3-Base sizes expose the central stability distinction. Under a shared suffix, plausible and random one-token edits contract with half-lives of roughly 1.5–5.5 tokens. Once the branches generate freely, their deviation instead returns to the ordinary sampling-noise scale, so shared-input contraction does not imply synchronization under free generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, Embedding models, Dynamic system

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yEw8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies high-dimensional embeddings of natural language under the lens of dynamical systems. Using various embedding and (transformer-based) language models, text in several languages is mapped into a trajectory in vector space, with the token-axis acting as the traditional "time" axis. The major finding appears to be that pooled representations (one embedding for an entire piece of text) evolve slowly along the token axis, while in contrast last-token representations in language models show much more variation.

While the approach is generally interesting, it is not particularly new, and the findings are limited in scope and lack proper contextualization and implications. The main flaw of the paper is that the presentation is really poor: writing is dense and jargony, and there are no figure to illustrate empirical results. Given these shortcomings, in my opinion the paper -- in its current form -- does not bring enough scientific and didactic value to the ICLR community, and should not be accepted.

Critical strengths

None.

Critical weaknesses

Several substantial issues with the paper in its current form.

  1. It's very difficult to understand the paper. The paper appears to rely heavily on AI-generated text, which itself is not a problem (as far as I am concerned) as long as the clarity and rigor are not compromised (and, ideally, improved). Here that's not the case. The abstract itself is overly dense and obscure. Consider for example sentences like: "Accepted causal coordinates are moderately predictable and usually contract rapidly under a fitted output-space surrogate; pooled coordinates are far more predictable but evolve near the unit timescale"

  2. There are no figures to illustrate the empirical findings! All results are presented as textual summaries and long tables of results. The lack of figures, itself unusual, also makes reading and understanding the findings particularly difficult.

  3. Findings are limited. While the results presented are interesting, there is almost no contextualization, description of implications for the field.

Questions for authors

What are the implications of your findings to our general understanding of language models, language itself, or dynamical systems?

1 comment

Thank you for your careful reading and straightforward criticisms. We appreciate that you found the approach and results interesting, and address the weaknesses and questions one by one in the following.

W0: Novelty

With the table below, we explain the novelty and contributions of this work compared to previous papers.

WorkModelFormulationMethodCross-lingual comparisonIntervention
Maheswaranathan et al. (2019)RNNRecurrent state dynamicsFixed-point analysis; local linearization✗Zero-input rollouts
Hosseini & Fedorenko (2023)TransformerToken prefix dynamicsTrajectory geometry✗—
Zhang et al. (2025)Pretrained LLMToken prefix dynamicsTrajectory geometry; clustering; persistent homology✗Sampling temperature adjustment
Zhou et al. (2025)Pretrained LLMReasoning-step prefix dynamicsTrajectory geometry✓Step-order shuffling
Wilson & Akrout (2026)Pretrained embedding modelsToken embedding dynamicsKoopman / EDMD✗—
Barenholtz (2026)Pretrained LLMWord prefix dynamicsLinear extrapolation✗Syntactic disambiguation
Quan (2026)Pretrained embedding modelsWord prefix dynamicsTrajectory geometry✗Context edits
OursPretrained embedding modelsToken prefix dynamicsTrajectory geometry; affine DMD; fixed-point analysis✓Paired token edits

W1: Poor presentation

Thank you for pointing this out. In the revised manuscript, we have manually polished the writing.

W2: No figures

Thank you for the comments. In the revised manuscript, we have added several figures to demonstrate our experimental results:

  • In Section 4.2, a figure to explain the geometric measurements we study.
  • In Section 4.6, a figure to demonstrate the intervention operation.
  • In Appendix C (for now), a figure to analyse how the capability of embedding models affects the dynamics.

W3: Limited findings & Q1: Implications for the field

We list our findings as follows:

  • Based on the analysis and comparison of different languages, we find that Chinese representations exhibit more predictable, slower-decaying, and more regionally persistent dynamics than the other languages in our study.
  • Based on the stability analysis, we discover that language embeddings are robust to interventions. This explains why LLM chatbots are robust to typos, even without specific training (according to the information disclosed in the technical reports of mainstream LLMs).
  • We discovered that more capable embedding models provide better alignment among different languages, and that their trajectories turn more sharply at each step but the step lengths stay more stationary.
  • We provide new probing methods and metrics for analysing embedding models, which can be applied to tasks such as analysing the technical specifics of closed-source models through API access.
Reviewer_EXxoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors study how the representations of a language evolve when you add new tokens. They study three types of evolutions: 1) as you consistently add new tokens to the text, how distance, angle or trajectory length change, 2) can these embedding evolutions be predicted, and 3) what is the effect when one changes a single token. They find that embedding have clear temporal structure and a single single change has a decaying effect on the last token embedding under fixed subsequent input.

Critical strengths

One main strength that I appreciate in this work is that the breadth of the experimental validation is vast. The authors test 15 model configurations across 7 languages. This includes sharing the code and material necessary to reproduce the work.

One impressive part is that the paper differentiates several different behaviors that previous work might have conflated eg.: 1) the difference in effect/behavior between last-token and mean-pooled representations. 2) language dependent behavior e.g. some languages exhibit persistent dynamics, 3) even though shared inputs might show contraction, it does not mean that one would observe convergence during generation.

Critical weaknesses

My main weakness is that several of the techniques and some of the insights in this work have been observed in the literature before, so there is limited novelty of both new insight and methodology. For example, some of the work that the paper cites directly identify some of these findings. e.g.

  1. Maheswaranathan et. al. identify low-dimensional attractor dynamics,
  2. Hossieni et. al. (Neurips 2023, LLMs implicitly learn to straigten neural sentence trajectories)

While these papers above are not as a comprehensive, empirically, as the one here, I think there are enough similarities.

Questions for authors

I am not an expert on this literature. However, I found it a surprising omission that there is no link between the analysis and the performance/capabilities of the models. Has this type of work been done before or what can your findings imply about models that perform well on a task and the ones that don't?

2 comments

Thank you for your inspiring comments and your appreciation of the breadth of the experiments and the analysis on readout, language-specific behavior, etc. In our response below, we address the weaknesses and questions that you have brought up.

W1: Limited novelty against prior attractor and trajectory results

We recognise that there are previous works studying the dynamics of neural representations and curvature of token-time trajectories, including both papers you mentioned, which are already cited in our manuscript. Below, we briefly summarize the contributions of these two papers, and explain the difference and novelty of our work.

Maheswaranathan et al. (2019, 2020) study the dynamics of RNNs (LSTM, GRU, etc.) in sentiment analysis tasks, and reveal how RNNs leverage low-dimensional dynamical structures to accumulate sentiment information and dynamically modulate its processing based on context. In our study, we instead use pretrained transformer-backboned embedding models in an unlabeled natural language context.

Hosseini et al. (NeurIPS 2023) show that the curvature of sentence representations decreases with model layers, model sizes, and training. In their formulation, each point on the trajectory is the hidden state at each token position in a given layer, extracted from a single forward pass on the whole sentence. In our study, we consider only the final output embeddings of pretrained models, and focus on language-driven dynamics.

Our approach to this work was to start with the formulation of language representations as a dynamical system, and study its evolution and properties. Inevitably, there is overlap with existing work, which we acknowledge, but our work differs in terms of problem formulation, research focus, model architecture, and learning task. Moreover, we also introduce several controlled comparison analyses along axes that are not covered in previous works, such as readout, language, intervention, etc.

Q1: Link between these dynamics and model capability

Thank you for this suggestion; it is indeed interesting to analyse how the dynamics of the embedding models could be correlated with their performance. To rank model capabilities, we use the MMTEB benchmark. As of November 14, 2026, the rankings of the models used in our study are (we omit Voyage-4 models because their rankings are unreasonably low):

RankModel
4Qwen/Qwen3-Embedding-8B
6Qwen/Qwen3-Embedding-4B
20Qwen/Qwen3-Embedding-0.6B
45openai/text-embedding-3-large
52BAAI/bge-m3
70openai/text-embedding-3-small
245perplexity-ai/pplx-embed-v1-4b
249perplexity-ai/pplx-embed-v1-0.6b

The figure and analysis are provided in Appendix C of the modified manuscript. We plot the metrics of these models on FLORES+ in order of their rankings as shown above. The small markers represent one dataset (one language or shard of one language) each, the large markers indicate the median, and the colors indicate the readout type.

From the plots, we observe that apart from the differences caused by readout, some metrics do show a significant trend with the model ranking. The turning-angle deviation increases with the model rank, while the early/late speed ratio rises significantly. These phenomena imply that for more capable models, the turning angle is larger at each step, but the step length stays mostly constant. Parallel-endpoint cosine falls slightly with the model rank, indicating that stronger models may provide better alignment among different languages. For predictive NMSE, spectral radius, and metastability, there are no significant trends across models other than that from the differing readout types.

Reviewer_EXxoOpenReview ↗

Understood

I have read the response to my review. I think this work still has some merit, and can be seen has modernizing some of the previous results. Thanks to the authors for clarifying the new insight and empirical differences between your work and the previous work. From the analysis, it seems that there is no a big correlation between some of these findings and performance. It would be interesting to directly develop a way to penalize or control the model's dependence during training as a way to see the effect on other downstream behavior. For example, if we penalize the spectral radius metric during training, what happens? Overall, I think this work is compelling, and I keep my rating.

Reviewer_8iPAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the evolution of prefix representations as text unfolds one token at a time. It formulates the process as a token-driven input–state–output system and analyzes trajectory geometry, predictive adequacy of reduced coordinates, affine DMD surrogates, and regional persistence across 15 model configurations and seven languages. Paired token interventions on three Qwen3-Base models distinguish rapid contraction under a shared suffix from sustained branch separation under free generation. The main empirical finding is that representation dynamics differ substantially across readouts, while shared-input contraction does not imply synchronization under free generation.

Critical strengths

A useful distinction between shared-input contraction and free-generation behavior. The paired interventions directly test whether a one-token edit is attenuated under an identical suffix and whether the branches subsequently synchronize under free generation. This is more informative than inferring stability from observational trajectories alone, and provides a concrete empirical contribution. Careful separation of observations, surrogates, and internal states. The paper explicitly limits its spectral and fixed-point conclusions to fitted output-space models. Its document-level train–test splits, multilingual coverage, and detailed shard-level reporting also provide a useful basis for inspecting the reported regularities.

Critical weaknesses

The main geometric findings lack sufficiently matched null models and controlled readout comparisons. Independent isotropic state vectors already produce approximately 120-degree angles between successive differences, corresponding to an excess of approximately 0.524 radians over the independent-increment baseline. This is close to the reported Qwen range of 0.32–0.52 radians. Likewise, averaging fixed token features naturally produces shrinking increments as the prefix grows. These observations do not invalidate the measurements, but make it unclear how much language-specific structure they establish. The cross-model comparison also changes architecture, objective, and readout simultaneously. In addition, Appendix A.2 uses mean pooling for BGE-M3, whereas its standard dense representation uses normalized CLS. Controlled comparisons within the same checkpoint are needed to substantiate the attribution to readout. The predictive and spectral analyses do not yet establish sufficiently informative structure beyond ordinary mean reversion, and their connection to the intervention results remains weak. A stationary AR(1) process can beat persistence, attain horizon-eight NMSE below one, and yield a stable surrogate fixed point. Thus, passing the proposed gate and observing autonomous surrogate contraction are not by themselves strong evidence of distinctive structure. Several accepted entries in Table 9 have horizon-eight NMSE rounded to 1.000, making effect sizes and uncertainty important. Furthermore, the English FineWeb spectral radii for the three intervened models imply decay times of approximately 17–24 tokens, rather than the roughly five-token comparison invoked in Section 4.6. The intervention results remain interesting, but their quantitative relationship to the fitted dynamics needs stronger support.

Questions for authors

Geometry and readout controls: How do the reported turning-angle excess and slowing trajectories compare with temporally permuted state sequences and simple averaging baselines? On a representative subset, do the main differences persist when comparing multiple readouts within the same checkpoint, including BGE-M3’s native CLS readout? Predictive-gate strength: What are the horizon-eight improvements over a training-mean predictor, with document-level uncertainty estimates, particularly for accepted shards whose NMSE rounds to 1.000? Please also clarify whether rank/lag selection and final performance evaluation use separate held-out data. Connection to interventions: For the same English FineWeb checkpoints, how should the DMD decay times be reconciled with the much shorter perturbation half-lives? Does contraction remain substantial when deviations are normalized at the first shared suffix token, and is it also reflected in next-token distribution differences?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8iPA (Part 1)

We thank the reviewer for a careful and constructive review, and also appreciate the reviewer's recognition of our experimental designs. We have since revised and extended our work in response to the issues raised, as listed below:

W/Q 1: matched null models

We now report differences against the independent isotropic state vector baseline and temporally permuted state sequences, which the reviewer pointed out is the stronger and more relevant null. For the default (last-token) trajectories, the theoretical isotropic state baseline is 2.09 rad (120 deg), while the temporally permuted states sit essentially on this value, with empirical values of 2.07-2.12 rad. The observed trajectories have turning angles of 1.93–2.06 rad, i.e., real trajectories are straighter. We also compare with the simple averaging baseline (mean-pooled trajectories), which have turning angles of 1.24–1.35 rad, and for which the corresponding temporally permuted baseline is 1.55–1.59 rad. The averaged trajectories have deviations (−0.26 rad) approximately twice as large as the last-token ones.

modellast-token: obs / nulldeviationmean-pooled: obs / nulldeviation
Qwen3-0.6B-Base1.96 / 2.07−0.111.31 / 1.57−0.25
Qwen3-4B-Base1.96 / 2.08−0.121.31 / 1.57−0.26
Qwen3-8B-Base1.96 / 2.07−0.111.29 / 1.55−0.25
Qwen3-14B-Base2.06 / 2.12−0.071.35 / 1.57−0.23
Qwen3-Embedding-0.6B1.93 / 2.08−0.151.24 / 1.59−0.35
Qwen3-Embedding-4B1.96 / 2.08−0.121.29 / 1.59−0.30
Qwen3-Embedding-8B1.97 / 2.08−0.111.28 / 1.56−0.28

You are also right to point out that averaging fixed features shrinks the steps as 1/t, which accounts for much of the reported slowing of pooled trajectories. Below, we show the relative speed in the corresponding sigma bins, averaged within model families. We also derive an analytic 1/t result, using token length distributions from the Qwen tokeniser.

model (readout)12345
Qwen3-Base (last-token)1.010.970.990.991.05
Qwen3-Embedding (last-token)1.020.991.001.010.98
analytic 1/t4.361.590.960.680.53
Qwen3-Base (mean-pooled)4.241.540.930.660.54
Qwen3-Embedding (mean-pooled)4.071.560.950.690.53
bge-m3 (cls)2.981.320.870.650.60
pplx-embed3.341.400.930.680.53
text-embedding-31.961.220.960.820.69
voyage3.131.400.880.670.56

We have updated the text in the manuscript to reflect the new baselines, and to clarify the natural 1/t trend for pooled trajectories.

W/Q 2: within-checkpoint readout comparison

We agree that the cross-model comparison changes the checkpoint, objective, and readout together, so it cannot isolate readout alone as the cause. We also note your point on CLS being the native readout scheme for bge-m3, and revise our results in our manuscript accordingly.

We take this opportunity to compare trajectories of bge-m3 under both mean-pooling and native CLS readout schemes. On the same checkpoint and dataset shard, the recorded deviation is -0.25 (mean-pooling) vs -0.29 (CLS) against the matched null models. For the Qwen3 Base and Embeding models, which share architecture but differ in objective, the difference is much smaller between the model families (~0.04 on average) than between different readout schemes (~0.16 on average). Hence, putting this and other results together, we believe the readout, and not the architecture, dominates the local geometry.

modellast-token: obs / nulldeviationmean-pooled: obs / nulldeviation
Qwen3-0.6B-Base1.96 / 2.07−0.111.31 / 1.57−0.25
Qwen3-4B-Base1.96 / 2.08−0.121.31 / 1.57−0.26
Qwen3-8B-Base1.96 / 2.07−0.111.29 / 1.55−0.25
Qwen3-14B-Base2.06 / 2.12−0.071.35 / 1.57−0.23
Qwen3-Embedding-0.6B1.93 / 2.08−0.151.24 / 1.59−0.35
Qwen3-Embedding-4B1.96 / 2.08−0.121.29 / 1.59−0.30
Qwen3-Embedding-8B1.97 / 2.08−0.111.28 / 1.56−0.28
modelcls: obs / nulldeviationmean-pooled: obs / nulldeviation
bge-m31.72 / 2.01−0.291.75 / 2.00−0.25
AuthorsOpenReview ↗

Response to Reviewer 8iPA (Part 2)

W/Q 3: predictive baselines

The horizon-8 NMSE of the paper's predictor and a training-mean predictor are shown below. The analysis was performed with separate held-out data for rank/lag selection and final evaluation.

familyreadoutdatahorizon-8 NMSEtraining-meandifference
Qwen-Baselast_tokenFLORES0.5280.533−0.005 [−0.011, −0.001]
Qwen-Baselast_tokenFineWeb0.5410.549−0.010 [−0.015, −0.007]
Qwen-Basemean_poolFLORES0.4842.038−1.52 [−2.71, −0.78]
Qwen-Embmean_poolFLORES0.5240.544−0.022 [−0.041, −0.006]
Qwen-Embmean_poolFineWeb0.5970.629−0.020 [−0.031, −0.013]
bge-m3mean_poolFLORES0.5031.653−1.10 [−1.95, −0.54]
bge-m3CLSFLORES0.6921.037−0.24 [−0.58, −0.08]
pplx-0.6bmean_poolFLORES0.3171.433−1.08 [−1.89, −0.52]
pplx-4bmean_poolFLORES0.5562.108−1.51 [−2.39, −0.90]

For last-token trajectories, our predictor shows only a slight improvement over the training-mean baseline at horizon-8, at about 0.5–1%. However, for mean-pooled trajectories, the effect is much stronger. We interpret this as a real but mostly mechanical effect, reflecting the inertia of the mean-pooling readout rather than dynamics beyond mean reversion. The Qwen3-Embedding models with mean-pooling readout are a notable exception; their trajectories cluster near a common mean, which we believe to be an artifact of their contrastive training objective.

To assess the significance of our predictor against this baseline, we look at the document-level passing rates across different datasets:

family (readout)datamedian fractionrange
Qwen-Base (last_token)FLORES0.800.58–0.98
Qwen-Base (last_token)FineWeb0.9990.92–1.00
Qwen-Emb (mean_pool)FLORES0.870.67–0.94
Qwen-Emb (mean_pool)FineWeb1.0000.99–1.00
Qwen-Base (mean_pool)FLORES0.980.91–1.00
Qwen-Base (mean_pool)FineWeb1.0001.00–1.00
bge-m3 (cls)FLORES0.890.48–0.97
pplx-embed (mean_pool)FLORES0.990.98–1.00

An interesting pattern we notice is that FLORES tends to have lower document-level passing rates in general. We hypothesize that FLORES documents tend to be shorter and thus lack required context for accurate prediction. If we consider only FineWeb datasets, overall, we find that 99.8% out of the 70,000+ documents show an improvement over the training-mean baseline.

We acknowledge that the gate is weak, and that several accepted entries are numerically at the boundary you flag. We also acknowledge that a stationary AR(1) model can beat persistence, and indeed, our one-step affine DMD surrogate is itself a multivariate AR(1) model in the selected coordinate. Passing the gate and having a locally stable surrogate fixed point are therefore not evidence of structure beyond ordinary mean reversion.

However, we believe the document-level passing rates do indicate that our predictor has meaningful horizon-8 improvements over the training-mean predictor, though the effects are admittedly small in the case of last-token trajectories. Based on this, we retain the current definition, but make the limitations clear in our manuscript. The work in sections 4.4 and 4.5 remains conditional on passing the predictive gate.

W/Q 4: reconciliation of DMD decay times and perturbation half-lives

Thank you for catching this. Here, we inadvertently compared multi-language average spectral radii with the half-lives from the perturbation results, which are English-only. The spectral radii (from the appendix) are 0.942, 0.957, and 0.959 for the Qwen3-Base 0.6B, 4B, and 8B models respectively, which agree with the 17- 24 tokens you state. We have removed the statement regarding comparable timescales from our manuscript.

The "mismatch" between these two lifetimes reflects two fundamentally different modes in the system: a fast mode that damps external perturbations, and a slow mode that reflects the underlying, unperturbed dynamics, which is what the spectral analysis isolates, by construction. hence, we believe that these two modes can coexist, and are not in conflict.

AuthorsOpenReview ↗

Response to Reviewer 8iPA (Part 3)

W/Q 5: contraction cross-checks

As suggested, we normalise the deviations at the first shared suffix token, and calculate the contraction at various steps. The contraction effect survives the normalisation, and has a consistent trend across models.

model/scenario2481632>384
0.6B plausible0.520.270.140.0810.0480.014
0.6B random0.630.380.250.170.120.043
4B plausible0.510.270.150.0820.0520.019
4B random0.630.400.290.220.170.078
8B plausible0.520.270.150.0910.0570.024
8B random0.660.440.320.260.210.118

We calculate the KL divergence of the next-token distribution at various steps after the perturbation:

model/scenario012415
0.6B plausible6.170.850.1170.0350.004
0.6B random10.110.820.2440.0590.009
4B plausible7.100.970.1250.0300.004
4B random11.080.990.2110.0550.008
8B plausible6.660.980.1250.0260.003
8B random11.490.990.1650.0480.008

The results above show that contraction remains substantial and is reflected strongly in the next-token distributions. The large immediate divergence at step 0 (directly after the perturbation) is reduced by an order of magnitude after the first shared suffix token, and a further two orders of magnitude 3 tokens out. The difference between plausible and random perturbations is also visible clearly at step 0, and interestingly, suppressed to the same level at the first shared suffix token, but regains strength at the next shared suffix token and is persistent out to longer horizons. We interpret this as an effect of the tokenisation side-effect, where immediate next-token distributions are more strongly correlated with the immediate preceding token, but the perturbation retains a slight residual semantic influence at longer horizons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.