acceptodds
Under review as a conference paper at ICLR 2027

Linear Dynamical Modeling of Language Models

19 views

Abstract

How can we characterize the internal dynamics of a language model during generation? Modern LLMs produce text through high-dimensional, nonlinear hidden-state trajectories, making exact mechanistic understanding difficult. In this paper, we study these trajectories through linear dynamical systems (LDS), a classical model of state evolution whose eigenvalues provide interpretable measures of model memory and stability. Because recovering the high-dimensional full transition matrix or true latent state of an LLM is infeasible, we focus on a practical signature that approximates the dominant eigenmode of latent state evolution: the time-lagged covariance of observable states. We show that LDS-inspired summaries capture useful low-dimensional structure in both synthetic settings and real LLM generations. We further demonstrate that such summaries quantify generation stability across three downstream settings: evaluating the difficulty of safety-relevant jailbreaks, predicting mathematical reasoning correctness, and detecting hallucinations in retrieval-augmented generation. Most notably, classifiers built on these summaries predict answer correctness more accurately than frontier LLM-as-a-judge baselines (such as GPT-5.6 Terra), providing an interpretable and computationally efficient alternative requiring negligible additional inference cost.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Linear Dynamical Systems, Language Model Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jUUTOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a novel approach to model the evolution of the latent states of LLMs using linear dynamical systems (LDSs) and analyze their properties using statistics of time-lagged covariance. Given a fixed prompt, the authors consider centered trajectories by subtracting the conditional expectation given the prompt and focus on the time-lagged covariance of these deviations. Under strong assumptions, the mapping from the time lag to the log trace of the covariance provides spectral information of the unobservable, high-dimensional transition in the latent space, both in terms of eigenvalues and the degrees to which they are preserved under observations. Two statistics are proposed to recover such spectral information from observations: the first one is a prompt-level estimator constructed with repeated continuations, while the second estimator is defined using a single continuation averaged across different time lags. The authors conduct experiments on toy settings (fully observable 1D/2D LDS and a small Markov process) to illustrate their theoretical findings. The framework is further applied to predictions for jailbreaking, mathematical reasoning correctness and hallunication. Compared to LLM-as-judges, this approach achieves similar or better performance at negligible cost.

Critical strengths
  1. This work introduces a novel theoretical framework to analyze the dynamics of transformer-based representations. This perspective may have great future potential in developing understanding of the internal states of LLMs.
  2. The effectiveness of the proposed framework is validated by the superior and cost-effective performance on three downstream predictions tasks: the ease of GCG attacks, correctness of mathematical reasoning and the presence of hallucination. This shows that the time-lagged covariance capture relevant information which is widely applicable.
  3. The authors provide detailed information for both theoretical derivations and experimental setup.
Critical weaknesses

The major weaknesses lie in the disconnect between theoretical framework and experimental setup.

  1. Section 3 introduces the main estimators as the traces of (7) and (8), whereas results in Section 4 are all reported in Frobenius norm. As such, the experiments do not serve to back up the proposed theory without further justification.
  2. Section 4.1 "Controlled linear systems" seem to consider LDS in dimensions one and two, instead of dynamics in higher dimension with only one/two active modes. This extremely limited setting (with xt=ztx_t=z_t) does not really provide insight and appears to be side-stepping the assumption behind Eq (5) that AA needs to be positive definite with all modes active.
  3. Results from a toy transformer in Section 4.1 serve the main empirical evidence for the proposed theoretical framework. However, Fig 2c) is unclear and incomplete, as there should be 10 per-prompt and 10 per-step-centered curves according to Line 323. To substantiate their claims, the authors should instead provide the log trace curves for a wide class of open-weight models for both estimators (e.g. as in Fig 11).

I would be willing to increase my score if the authors provide convincing explanations.

Questions for authors
  1. How should the matrix AA be conceptualized? The authors explain in Line 205 that "At transformer scale it has millions of parameters" whereas the last paragraph of section 2.1 implies that the transformer only gives us the (uncentered) observables hth_t. Should the matrix AA be though of as a lift for the nonlinear dynamic xtx_t (in which case wouldn't the operator AA be infinite dimensional)? Furthermore, how does it relate to the transformer from a language modeling perspective (e.g. can AA be defined via some sort of conditioning on a given prompt as in Line 165)?
  2. Eq (5) holds provided that pk(τ)p_k(\tau) sum to one. This is only true when all eigenvalues of AA are inside (0,1)(0,1) and all coefficients ckc_k are positive. These assumptions seem very restrictive. In comparison, Fact A.3 is general while Fact A.4 completely disregard all the other eigenvalues but the top two (though a straightfoward extension in the style of Fact A.3 seems likely to hold). Does the interpretation of mean/variance in Eq (5) have fundamental importance (i.e. why not focus on Fact A.3 and some generalization of Fact A.4 instead)? Are there good theoretical reasons to believe that the assumptions behind Eq (5) should hold? What happens when the assumptions fail?
  3. Have the authors tried deriving similar theoretical framework for the Frobenius norm instead of the trace? Shifting from eigenvalues to singular values might yield statements with less restrictive assumptions.
  4. In the jailbreak experiment, the choice of intervals for fitting the slops (lags 1–2 and lags 751–800) appears arbitrary. How are these intervals selected and how sensitive are the results to such selections?
  5. In Eq (25), the term CsumC_\text{sum} is of order ∑t=1∞∥WAtΣzW⊤∥F\sum_{t=1}^\infty \Vert W A^t \Sigma_z W^\top\Vert_F. Even when CsumC_\text{sum} is finite, the bound appears too loose to be informative. In addition, what can we conclude from (25) in terms of the (log) trace statistics like Fact A.8, which serve as the actual theoretical justification?
Reviewer_BPdiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper models the generation trajectory of an LLM as an effective linear dynamical system. At each generation step, it takes the final-layer hidden state and subtracts the mean across sampled continuations of the same prompt. It then treats these fluctuations as linear observations of a latent state that evolves under a fixed transition matrix with additive innovations. Motivated by Koopman theory, the authors show that the lagged covariance of hidden states then decomposes into a sum of decaying modes. The slope of its log-curve gives an effective decay rate, which they split into fast short-lag and slow long-lag components. They validate the estimator on synthetic linear systems and on a small Transformer trained on a Markov chain. They then relate the decay signatures to three behaviours. For jailbreak robustness, requests that are harder to break with GCG show faster short-lag decay, and adding a successful adversarial suffix slows it. For math correctness and hallucination detection on RAGTruth, classifiers built on covariance features are reported to be competitive with frontier LLM judges at a small fraction of the cost.

Critical strengths

Studying how LLM hidden states evolve along the generation trajectory, rather than at single positions, is an interesting direction. The Koopman / LDS direction is theoretically appealing. It offers a principled vocabulary for timescales and memory in autoregressive generation. The paper also checks the framework on controlled systems where the true decay rates are known, and the estimator recovers them. The evaluation targets topical problems: jailbreak robustness, hallucination detection on RAGTruth, and math correctness. Within each setting, the covariance features carry some signal. However, the connection between the theory and these downstream behaviours remains tenuous, as elaborated in the weaknesses.

Critical weaknesses

The phenomenology of this paper feels largely "manufactured." On the theory side, the central object, Eq. (3), is just the lagged covariance between lagged terminal states. It can be simply motivated without Koopman theory, which I think is largely redundant. None of the experiments estimate A, W, or the eigenvalues. They either fit slopes of log-norm curves or feed raw curves into CNN ensembles, so the LDS structure does no work in any result. It is also unclear what zyz_y is designed to capture: just hth_t, or the entire context. If it is hth_t alone, the missing context leaks into the innovation and violates the orthogonality that Eq. (3) relies on. If it is the entire context, a fixed-dimensional with time-invariant A is a strong, untested assumption. Relatedly, I am not sure the innovation captures only the effect of sampling the next token. In the RAGTruth experiment, nothing is sampled at all, since the recorded answers are teacher-forced.

Secondly, the paper relates the log decay of covariance to safety and hallucination, which are topical. However, it gives no justification, heuristic or theoretical, for why a simple covariance measure meaningfully tracks these higher-level behaviours, and the confounds are obvious.

Attack difficulty. Easy and Hard requests (by how many GCG optimization steps it takes to jailbreak them) differ only modestly. The slow-mode tail goes the wrong direction for the stability story. Safe vs Jailbroken. This contrast is confounded by refusal vs compliance. Refusals stop early (48 of 50 vs 2 of 50 runs hit EOS within 100 tokens), and compliant text plausibly has more divergent dynamics. Math correctness. Generations are forced 2,000 tokens past their natural stop, so EOS timing and rambling could drive both the features and the labels. As a result, the investigation is to be interpreted as correlational at best. No causal or counterfactual intervention is provided, such as steering along slow directions, comparing comply vs refuse within the same prompt, or length-matched controls.

Finally, the practical contribution seems oversold. The headline numbers come from heavily engineered supervised classifiers, not from the decay rate. On RAGTruth, the covariance-only model (67.44% BACC) falls below an open-weight judge (72.06%) and within the confidence interval of a last-token probe (65.81%). The judge comparison is also not apples-to-apples. The judges receive no reference answer and are zero-shot, while the classifier is trained on about 180k in-distribution labels. And if one can afford the larger LLM as a judge, why not use it in the first place? More importantly, predicting correctness and hallucination from hidden states is an established line of work, yet none of it is benchmarked: SAPLMA, Zhang et al. (2025), EigenScore, LLM-Check, ReDeEP, and simple length or entropy baselines. Only accuracy and balanced accuracy are reported, so it is unclear whether the covariance dynamics add anything beyond a single hidden-state probe.

Questions for authors

What is the heuristic or theoretical link between covariance decay and safety or hallucination? Is there any causal or counterfactual evidence for it, such as steering hidden states and measuring the effect on behaviour?

What does the Koopman framing add beyond the plain lagged autocovariance, given that A, W and the eigenvalues are never estimated?

Reviewer_QsUuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The core idea of this paper is to extract temporal covariance features from LLM hidden states, motivated by linear dynamical systems (LDS), and train classifiers to predict hallucinations and answer correctness.

I am not convinced by the motivation or practical significance of this work. At its core, the paper applies supervised classifiers to statistical features extracted from hidden states, which is not fundamentally different from existing probing approaches. More importantly, hallucination is fundamentally about factual correctness, and I find it difficult to see why simple temporal statistics of hidden states should provide a reliable or generalizable signal for detecting it. The empirical results show some correlations, but do not convincingly establish the proposed connection between dynamical stability and hallucination.

Critical strengths

See Summary

Critical weaknesses

See Summary

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.