- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a novel approach to model the evolution of the latent states of LLMs using linear dynamical systems (LDSs) and analyze their properties using statistics of time-lagged covariance. Given a fixed prompt, the authors consider centered trajectories by subtracting the conditional expectation given the prompt and focus on the time-lagged covariance of these deviations. Under strong assumptions, the mapping from the time lag to the log trace of the covariance provides spectral information of the unobservable, high-dimensional transition in the latent space, both in terms of eigenvalues and the degrees to which they are preserved under observations. Two statistics are proposed to recover such spectral information from observations: the first one is a prompt-level estimator constructed with repeated continuations, while the second estimator is defined using a single continuation averaged across different time lags. The authors conduct experiments on toy settings (fully observable 1D/2D LDS and a small Markov process) to illustrate their theoretical findings. The framework is further applied to predictions for jailbreaking, mathematical reasoning correctness and hallunication. Compared to LLM-as-judges, this approach achieves similar or better performance at negligible cost.
Critical strengths
- This work introduces a novel theoretical framework to analyze the dynamics of transformer-based representations. This perspective may have great future potential in developing understanding of the internal states of LLMs.
- The effectiveness of the proposed framework is validated by the superior and cost-effective performance on three downstream predictions tasks: the ease of GCG attacks, correctness of mathematical reasoning and the presence of hallucination. This shows that the time-lagged covariance capture relevant information which is widely applicable.
- The authors provide detailed information for both theoretical derivations and experimental setup.
Critical weaknesses
The major weaknesses lie in the disconnect between theoretical framework and experimental setup.
- Section 3 introduces the main estimators as the traces of (7) and (8), whereas results in Section 4 are all reported in Frobenius norm. As such, the experiments do not serve to back up the proposed theory without further justification.
- Section 4.1 "Controlled linear systems" seem to consider LDS in dimensions one and two, instead of dynamics in higher dimension with only one/two active modes. This extremely limited setting (with ) does not really provide insight and appears to be side-stepping the assumption behind Eq (5) that needs to be positive definite with all modes active.
- Results from a toy transformer in Section 4.1 serve the main empirical evidence for the proposed theoretical framework. However, Fig 2c) is unclear and incomplete, as there should be 10 per-prompt and 10 per-step-centered curves according to Line 323. To substantiate their claims, the authors should instead provide the log trace curves for a wide class of open-weight models for both estimators (e.g. as in Fig 11).
I would be willing to increase my score if the authors provide convincing explanations.
Questions for authors
- How should the matrix be conceptualized? The authors explain in Line 205 that "At transformer scale it has millions of parameters" whereas the last paragraph of section 2.1 implies that the transformer only gives us the (uncentered) observables . Should the matrix be though of as a lift for the nonlinear dynamic (in which case wouldn't the operator be infinite dimensional)? Furthermore, how does it relate to the transformer from a language modeling perspective (e.g. can be defined via some sort of conditioning on a given prompt as in Line 165)?
- Eq (5) holds provided that sum to one. This is only true when all eigenvalues of are inside and all coefficients are positive. These assumptions seem very restrictive. In comparison, Fact A.3 is general while Fact A.4 completely disregard all the other eigenvalues but the top two (though a straightfoward extension in the style of Fact A.3 seems likely to hold). Does the interpretation of mean/variance in Eq (5) have fundamental importance (i.e. why not focus on Fact A.3 and some generalization of Fact A.4 instead)? Are there good theoretical reasons to believe that the assumptions behind Eq (5) should hold? What happens when the assumptions fail?
- Have the authors tried deriving similar theoretical framework for the Frobenius norm instead of the trace? Shifting from eigenvalues to singular values might yield statements with less restrictive assumptions.
- In the jailbreak experiment, the choice of intervals for fitting the slops (lags 1–2 and lags 751–800) appears arbitrary. How are these intervals selected and how sensitive are the results to such selections?
- In Eq (25), the term is of order . Even when is finite, the bound appears too loose to be informative. In addition, what can we conclude from (25) in terms of the (log) trace statistics like Fact A.8, which serve as the actual theoretical justification?