acceptodds
Under review as a conference paper at ICLR 2027

Deep latents, long memory: Latent persistence explains temporal correlations in transformer activations

Abstract

The correlations between elements of a time series stem from the structure of the process that generated it. Here, we apply this lens to the internal representations of Large Language Models (LLMs): We define the autocorrelation function of the neural activity as the average dot product between the activations of a fixed layer at two context positions separated by a time lag. First, we find that the autocorrelations of deeper layers develop a slowly-decaying tail over training. We conjecture that this decay reflects the abstraction of the encoded features: Deeper layers encode more abstract variables, an abstract variable governs a longer stretch of the input, and its code persists over that stretch. We then test this hypothesis in two settings. In LLMs trained on synthetic data with known hierarchical latent structure, the persistence of latent codes predicts the qualitative structure of autocorrelations, including their development during training, and a large fraction of the empirical measures. In open LLMs trained on real text, we take sparse-autoencoder features as proxies for latent variables, and define their persistence time as the number of contiguous tokens where the feature is active. We find that the persistence time linearly correlates with the magnitude of autocorrelations across datasets and LLMs. In particular, the persistence time increases with depth as autocorrelations do, implying that deeper latent features persist for longer context spans. We complement this analysis by explicitly ablating the persistent sparse-autoencoder features and observing a consequent decrease in the autocorrelations.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Large Language Models, Physics of Learning, Activation Geometry, Abstraction, Long-range Correlations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3XfHOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies whether transformer trained on time series of LLM on real text follow the pattern where earlier layers model shorter autocorrelations, and later layers model longer lags.

Critical strengths

Understanding what LLMs encode in their internal representations is important. The paper's central methodology is the autocorrelation function of equation (1), where dot product for a layer's embedding on token at time T and token at time T - tau is computed. Since such dot product on normalized activations is also used as a similarity measure in other fields such as RAG, it is to be a reasonable measurement. The main results such as Figure 1 supports the finding regarding earlier vs later layer in what time lapse tau they encode.

Critical weaknesses

A central weakness is takeaway or contribution from the paper. Intellectually: it is understood, prior to this paper, that generally speaking later layers tend to encode more abstract concepts, and earlier layers encode more local and lexical aspects. The paper does point out related works that share this finding. Therefore, the paper's contribution can be viewed as adding more evidence towards this known aspect. Application-wise: the paper examines transformer trained on time series and LLM applied to general text data like WikiText. However, it is not discussed whether this interpretability angle sheds new light on how LLMs encode concepts (except earlier vs later layer distinction in autocorrelation lag), how we should view or use LLMs, or how deep neural nets can be used to model time series more effectively.

A second weakness is organization of the paper. The paper is has 3 sections: Intro, Results, Discussion and Perspectives. Its methodology is written embedded within the Results, making the paper sometimes difficult to follow. A roadmap and motivation for what experiments and why these experiments would be helpful.

Questions for authors

There're no additional questions that would materially affect my assessments. Main thoughts are in Strengths and Weaknesses.

3 comments
AuthorsOpenReview ↗

The takeaway/contribution is not "later layers encode more abstract concepts"

We thank the reviewer for the detailed critique. We agree that the submitted version did not state the contribution sharply enough. The revision addresses this as follows. A clarification on scope: the paper does not train models on time series: this term appeared only as an analogy in the old abstract. The synthetic data are sentences generated by a hierarchical grammar (the RHM), and all the other experiments use LLMs pretrained on text.

1. The takeaway/contribution is not "later layers encode more abstract concepts"

We take this statement as a starting point, not our conclusion. We argue that the literature lacks a way to quantify abstraction in a model-native way: current evidence relies on labelled probes, paired data such as translations; or SAE features, whose abstraction is judged by the subjective interpretation of features. The last route is especially fragile: what an SAE feature represents is not well defined, and its meaning is usually assigned by asking another LLM to summarise its activating examples. As a result, "abstract" has no operational definition, and no theory can predict the outcome of its measurement.

Instead of defining abstraction in trained LLMs, we use an observable that quantifies how long the information they encode persists across tokens: the autocorrelation cℓ(τ)c_\ell(\tau).

For the theory, we use the RHM, where all latent variables are known and cℓ(τ)c_\ell(\tau) decomposes into contributions of named latents. The persistence of these latents predicts

  • where the autocorrelation steps down (at the subtree boundaries, Fig. 1a);
  • which latent carries each plateau (Fig. 2);
  • when each plateau forms during training (one stage per level of the hierarchy, Fig. 3).

The test pretrained LLMs use SAE features only to validate the relationship between cℓ(τ)c_\ell(\tau) and persistence (Fig. 4c), through when they are active and not through what they mean. We reframed the abstract of the revised manuscript to clarify the scope.

2. New light on how LLMs encode concepts.

It does, as it suggests that LLMs hold the code of a variable fixed over the span of the input governed by that variable. Therefore, deeper layers show longer-range autocorrelations because the variables they encode govern longer spans, not because of a generic smoothing with depth.

3. How should we view, use or train LLMs?

Since cℓ(τ)c_\ell(\tau) requires only raw activations, it applies where labels, paired data or SAEs are unavailable. We are running two experiments to test possible applications, we will report on the outcomes when completed.

  • Locating language-independent content without paired data. Neighbour-based measures locate the layers that encode language-independent content by comparing the representations of translated sentences (Acevedo et al., 2026). In preliminary experiments, the depth profile of the integrated autocorrelation χℓ\chi_\ell matches that of this representation similarity almost perfectly (r=0.97r = 0.97, averaged over four LLMs). This is promising, as autocorrelations are measured on a single language and are much cheaper to compute.

  • Signal available to latent-prediction objectives. A recent theory (Korchinski, Favero & Wyart, 2026) shows that, on hierarchical data, networks trained to predict their own latent representations (as in data2vec or JEPA) need exponentially fewer samples than networks trained to predict tokens, because correlations decay much faster with distance between tokens than between latents. A modified autocorrelation, cswap(τ)c^{\rm swap}(\tau), measures this signal without knowing the latents: we replace all the tokens after position T−τT-\tau with those of an independent sentence, and subtract the resulting autocorrelation from c(τ)c(\tau). The difference isolates the dependence between the text before and after position T−τT-\tau. In preliminary experiments on the RHM with 2202^{20} sentences, this signal is within the sampling noise at the token level at all lags, whereas layer 2 resolves it up to lag 3 and layer 3 up to lag 6, with a signal-to-noise ratio of 10–30 (below 2 at initialisation). This implies that a next-latent prediction method could leverage a higher learning signal than a next-token prediction method. Whether this is feasible in practice also depends on when the signal becomes available during training, and requires tests on real data, for instance by tracking how c(τ)c(\tau) and cswap(τ)c^{\rm swap}(\tau) develop over the publicly available checkpoints of pretrained LLMs.

4. Organisation of the paper

The revision has a dedicated methods section (Sec. 2) separate from the results. The contribution section now includes a roadmap linking each experiment to the claim it supports. Results on synthetic and real data are split across different sections. We added new schemes to simplify the presentation of our results in Fig. 1d and Fig 2a, and significantly revised Section 3 to improve readability.

I thank the authors for the clarification. It is helpful to clarify that the paper is not framed as further evidence for later layers encoding more abstract concepts, but more about quantifying that aspect with autocorrelation. However, it is still not clear what problem the paper solves or what's surprising about LLMs that the paper revealed. Overall, I raise my score to 2.

Our work does not solve a specific problem: it is aimed at the scientific understanding an empirical property of transformer activations. We appreciate that the reviewer doesn't regard this as a meaningful contribution, but we believe that works like ours, combining insights from toy models with real-world experiments, significantly expand our theoretical understanding of how LLMs process information.

There are many examples of works with a similar goal in the machine-learning literature. For instance, https://arxiv.org/abs/1611.00740 (among others) characterised the advantage of deep over shallow networks from an approximation theory perspective. By then, the community had already realised that 'Deep is better than shallow' for vision models. Yet, this paper improved our theoretical understanding of the phenomenon and spurred further fundamental work.

Scaling laws are another example: they have been driving Industry developments for a few years already, but their origin is still debated (e.g., https://arxiv.org/pdf/2602.07488 ; https://arxiv.org/pdf/2606.25008) and generating valuable theoretical research.

In the same spirit, our work quantifies how feature persistence and autocorrelations increase with transformer depth, and uses the RHM to explain how this layer dependence can arise from a hierarchical structure of data.

Reviewer_Ng1FOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how information persists across token positions in LLMs and how this persistence can explain autocorrelations in their activations. The authors measure autocorrelation using average dot products of activations. They find that deeper-layer representations remain correlated over longer distances. They hypothesize that deeper layers encode more abstract variables whose codes persist over longer context spans. To test this hypothesis, they first use data generated by the RHM to analyze the composition of autocorrelations and their development during training. They then examine this relationship in models trained on natural language through SAE feature persistence measurements and feature-direction ablation.

Critical strengths
  1. The authors use known latent variables in the RHM to decompose autocorrelation into same-node, cross-node, and residual contributions, with detailed mathematical derivations. This lets them quantify the contributions of latent codes and link their persistence to the autocorrelation curves. The authors show that the autocorrelation plateaus develop progressively alongside the learning of different levels of the hierarchy. This provides additional evidence for the proposed explanation.

  2. The natural language experiments cover five models and two datasets, showing that the observations are not limited to a single dataset or model. Experiments such as SAE feature persistence measurements, ablations, and control analyses provide supporting evidence and help assess the robustness of observations.

Critical weaknesses
  1. The natural language experiments show that SAE feature persistence is related to autocorrelation, but they do not examine what the persistent features represent or whether deeper-layer features are more abstract. Features related to formatting or repeated local patterns could also remain active for many tokens. The authors note that their method does not require identifying feature content, which is reasonable for measuring autocorrelation. However, the claim that more abstract variables account for longer-range correlations still needs semantic evidence.

  2. It is unclear how much of the correlation between persistence and autocorrelation comes from both quantities changing with depth. Feature activation frequency (the fraction of tokens on which each feature is active) is another possible factor. A feature that activates more often can have longer consecutive runs even without temporal dependence. Selecting the most frequently active features at each layer does not control for differences in activation frequency across layers.

Questions for authors
  1. Does the relationship between persistence and autocorrelation remain after controlling for depth and feature activation frequency?

  2. The ablation results show that persistent feature directions contribute to autocorrelation. However, the reduction in autocorrelation also depends on how much of the activation is removed. It is unclear whether the non-persistent controls are comparable in this respect. Are the persistent features and non-persistent controls comparable in how much of the activation they remove?

1 comment

W1

We agree that our natural-language experiments characterise the persistence of SAE features without identifying their content.

Our notion of abstraction is grounded in the Random Hierarchy Model (Cagnetta et al., https://journals.aps.org/prx/abstract/10.1103/PhysRevX.14.031001). There, the abstraction of a latent variable is its depth above the leaves in the generative tree. This is known a priori independently of the network, and it sets the stretch of text over which the variable's code persists in the activations, hence the range of the autocorrelations (See Fig. 1c,d and Fig. 2, revised). We use the relationship between depth and persistence to bypass the definition of abstraction in trained models, where we only test the relationship between persistence and depth.

Inspecting the top-activating examples of the most persistent SAE latents of Gemma 2 2B on Neuronpedia confirms the reviewer's intuition. They include formatting variables, such as code blocks (L12:3454, L15:14595), as well as topic variables, such as “clinical medicine” (L20:143), and genre variables, such as “research abstracts” (L12:13227). All of them are deep and persistent, as they describe a span of text rather than an individual token. We also find repeated local patterns (number sequences, L0:11153, notation-heavy spans, L4:16348), but the ones we found are in the early layers.

Interpreting SAE features requires substantial human input, and ranking them by abstraction requires even more. Our measure is instead model-native, as it uses only raw activations, and it is quantitative and falsifiable. The examples above show that it is consistent with an intuitive notion of abstraction, while extending it to variables, such as formatting, that one might not call abstract a priori.

The abstract and introduction of the revised paper clarify this point.

W2

Our motivation for selecting the 1000 most frequently active SAE features at each layer was to ensure sufficient statistics for estimating their mean persistence lengths. Since persistence is estimated by averaging the lengths of individual firing episodes, rarely activated features generally provide fewer observations and consequently noisier estimates. We agree that this selection does not control for frequency, and we added a dedicated control (Appendix G and Fig. 20, summarised in Sec. 4.1). The control randomly shuffles the token positions of each sequence. This preserves how often features are active in every sequence, while destroying the temporal order. For Gemma 2 2B on The Pile, we show how such shuffling affects persistences and autocorrelations, and how to use χ~ℓ\tilde{\chi}_\ell to avoid measuring the contributions induced exclusively by frequency while retaining the correlation between persistence length and integrated autocorrelations.

Q1

See above for frequency. For depth, the comparison across layers has one point per layer, so it cannot be separated from depth. There are two additional results that support correlation beyond what's implied by growth with depth:

  • Same depth profile. Once the sequence-wide plateau is removed, χ~ℓ\tilde\chi_\ell grows through the first half of the network and then saturates, as persistence does (Fig. 21), and their correlation increases in most cases (Table 7).
  • Fixed layer. In the ablation experiment of Sec. 4.2, depth is fixed: removing a persistent feature reduces the autocorrelation, whereas removing a non-persistent one does not (Table 1).

Q2

They are not exactly comparable, as we now quantify in the new Appendix L, Fig. 26, which shows the distributions of ablated norms for Gemma 2 9B on WikiText. On the positions where each feature is active, the persistent feature removes a median 1.2% of the squared norm of the activation and the control 0.29%, a factor of 4 smaller. In the same setting, the mean autocorrelation decreases by about 3.0% after ablating the persistent feature, but instead does not change for the control ($ \Delta \approx 0 $, see Eq. 10 and Table 1). This suggests that the amount of squared norm removed captures only one part of the effect on $ \Delta $, which also depends on how strongly the same feature direction is represented at the reference token. As correctly mentioned by the referee, we remove feature directions, by subtracting the projection of activations along them. Temporal persistence and projection magnitude are, in principle, independent properties: a feature may remain active across many tokens while accounting for only a small fraction of the activation norm, or have a large projection over a short span. Moreover, a large projection in ϕt\phi_t not present at ϕe\phi_e would not make $ \Delta $ increase. Interestingly, our results show that temporal persistence and projection magnitude are indeed coupled in our experimental setting, with persistent features also capturing a larger fraction of the activation norm when active.

Reviewer_4vN2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper explores the temporal axis of language model representations, particularly looking at how more abstract features persist temporally in representation space for longer. The authors look at synthetic data with known structure as well as LM's trained on natural text where SAEs are uses a proxy for latents.

Critical strengths

The study of temporal dynamics of representations in language models is still under explored. Thus the main motivation behind this work is interesting and important.

Critical weaknesses

Many of the results of this paper are known from prior work, namely the TFA and TSAE papers. it is unclear to me wihat this paper contributes beyond what was already shown in those two prior works.

Questions for authors

how does the simplicity or abstractness of a concept contribute to its persistence in representation space? Have the authors considered how non-linear geometry may impact these results?

2 comments

Contribution beyond TFA and T-SAE

Like TFA, we observe that the activations of a layer are correlated along the context. Our aim, however, is different. TFA and T-SAE are dictionary-learning methods that build this temporal structure into the features. We instead propose the autocorrelation profile cℓ(τ)c_\ell(\tau) as a model-native measure that quantifies the relationship between depth and persistence, without relying on prior assumptions about the represented concepts. This relationship is grounded in the Random Hierarchy Model (RHM), where the latent variables are known and cℓ(τ)c_\ell(\tau) decomposes into the contributions of named latents. Accordingly, the following results are in neither paper:

  1. Persistence grows with depth. We measure cℓ(τ)c_\ell(\tau) and the persistence of standard SAE features, trained without any temporal objective, at every layer of five LLMs on two corpora. Both grow with depth and correlate across layers (Fig. 4), also after controlling for the activation frequency of the features (App. G). TFA and T-SAE characterise standard SAEs at fixed layers only.

  2. Ground truth. In the RHM, each latent contributes to cℓ(τ)c_\ell(\tau) only over the span of input it governs, and the predicted codes account for a large fraction of the measurement (Fig. 2).

  3. Training. Each plateau of the autocorrelation forms when the model learns the corresponding level of the hierarchy (Fig. 3).

We stress this point further in our replies to Reviewers Ng1F and 3XfH. If the reviewer has in mind another specific result of ours that appears in TFA or T-SAE, we would be glad to address it.

The revised abstract, introduction and "Our contribution" section clarify the scope of our work, and the related-work section now makes the comparison with TFA explicit.

Simplicity/abstractness and persistence

We avoid ranking the concepts of trained LLMs by simplicity or abstractness, because interpreting SAE features requires substantial human input, and ranking them by abstraction requires even more. Our notion of abstraction is grounded in the RHM. There, the abstraction of a latent variable is its depth in the generative tree, which is known a priori and sets the span of input over which its code persists (Fig. 2). In natural language, we test only the relationship between persistence and depth (Fig. 4).

Nevertheless, we inspected the top-activating examples of the most persistent SAE latents of Gemma 2 2B on Neuronpedia. We found formatting variables, such as code blocks, as well as topic variables, such as "clinical medicine", and genre variables, such as "research abstracts". All of them are deep and persistent, as they describe a span of text rather than an individual token. We also found repeated local patterns, but the ones we found are in the early layers. Our reply to Reviewer Ng1F lists these examples.

Non-linear geometry

The measurement assumes no linearity: cℓ(τ)c_\ell(\tau) is a dot product between raw activations, and a variable that is encoded non-linearly, but in similar ways at both positions, contributes to it. Linearity enters only when we attribute the autocorrelation to individual, linearly encoded, latent variables.

Thank you for your rebuttal! I will maintain my score.

Comments on the paper

1 comment

We thank all the reviewers for their careful reading. Thanks to their comments, we have improved the positioning of the paper and clarified its contribution: we propose the autocorrelation of the activations, cℓ(τ)c_\ell(\tau), as a model-native measure of the relationship between depth and persistence, which does not rely on prior assumptions about the represented concepts. This relationship is grounded in the Random Hierarchy Model, where the latent variables are known.

We invite everyone to read the revised abstract. We reply to each reviewer individually below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.