acceptodds
Under review as a conference paper at ICLR 2027

Overcoming the KV Catch: State-Space Analysis of Transformer Generation Dynamics

Abstract

Transformer-based large language models (LLMs) are feed-forward networks, yet they are deployed autoregressively, giving rise to rich dynamical phenomena such as repetition loops. Analyzing these phenomena with dynamical systems tools, such as fixed points and their stability, faces a basic obstacle: due to their growing memory, these systems lack a fixed-size state. Here, we take a first step towards state-space analysis of growing-memory transformers. We show that an attention head without positional embeddings is exactly described by a non-linear self-interacting Markov chain with a fixed-size state, and use its timescale separation to derive a self-consistency condition for fixed points. Building on this condition, we develop an optimization method that finds both stable and unstable fixed points. Applied to position-ablated heads of a pretrained 1-layer transformer, it reveals stable repetition attractors, asymptotically unstable yet persistent semantic clusters, and saddle states between neighboring attractors. We then introduce a multi-timescale mean-field approximation for general attention layers, which closely matches the full model, and use it to find an unstable repetition whose predicted escape dynamics we confirm with real-model rollouts. Our work opens the door to applying dynamical systems tools to understand LLM generation.

Primary area:
interpretability and explainable AI
Keywords:
LLM, interpretability, dynamical systems, closed loop

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hJW5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a new ansatz to understand autoregressive transformer generation as a state space model with fixed size state. They use the fact that in a one-layer single-head-attention-only transformer without positional encoding, there are no cross-token dependencies prior to the attention, and the output of the attention only depends on the unigram token distribution (which is a state of size \text{vocab_size}). The authors note that in a one-layer multi-head-attention-only transformer (with RoPE/ALiBi-like positional encoding), each attention head has its own dynamical timescale, and propose a multi-timescale mean-field (MTMF) model to handle analysis in this case. They find fixed points of generation (e.g., iterating the same string, "doom loops", etc) using a fixed point algorithm on the induced Markov chain. Experiments (again with one-layer attention-only transformers on simple datasets) show how the theory plays out on real trained networks.

Critical strengths
  • The mathematical derivations appear to be solid and correct.
Critical weaknesses
  • The paper studies one-layer attention-only transformers. The core reduction to a fixed-size state-space Markov chain seems fundamentally limited to exactly this setting and hard/intractable to generalize. I did find something in the appendix discussing this, but it was fairly hand-wavy with no evidence. This limitation severely restricts the applicability of the framework.
  • The central application of the theoretical formulation in this paper is finding fixed points of generation dynamics, which rarely occur in practice (and less frequently with larger and more powerful models). It is not clear how this motivation aligns with the described impact to the AI safety community.
  • The writing and presentation, including the figures, is not that clear. For example, when trying to understand the multi-head setup, it's not clear whether the separation of heads into multiple timescales for context retention is done deliberately or is just a natural phenomenon observed in pretrained transformers. The relevant figure, Figure 5, did not clear this up and it was difficult to understand the rest of the figure. Additionally, several phrases and shorthands are used without being introduced, making it difficult to read; experimental setups are not well specified in all cases, etc.
Questions for authors
  • Is there a tighter characterization of multi layer transformers under this framework?
  • Are there direct applications of the framework besides finding generation fixed points?
6 comments
AuthorsOpenReview ↗

Response to Reviewer hJW5 (Part 1/3)

We thank the reviewer for the candid and detailed review, and for confirming that the mathematical derivations are solid and correct. The concerns about generality, applications and presentation are fair. They led us to run several new experiments and to rewrite parts of the paper, and we believe that the work is considerably stronger as a result. All new results are collected in the supplementary PDF (sections S1–S7), and some of them are already in the revised paper. We answer each point below.

1. Generality beyond one-layer attention-only models (W1 and Q1)

The one-layer case is broader than it appeared. The state representation is exact for any one-layer semantic (position-ablated) model, with any number of heads and with an MLP block. Heads are additive, and MLP and LayerNorm are stateless operations that act on each position separately, so they do not change the state. The submission did not make this clear, and the revised paper now states it explicitly (Section 3.2). To support it with evidence, we test the MTMF approximation error on two further pretrained one-layer models that contain an MLP block, gelu-1l and TinyStories-1L. For both models, the approximation error is as low as for the original model (mean JSD of 0.003–0.015, against 0.5–0.6 for the baseline), with a similar ablation pattern (Appendix J.3; Supp. S4).

A characterization of multi-layer models (Q1). The reviewer is right that the discussion of deep models in the submitted appendix was hand-wavy and came with no evidence. We replaced it with a precise result (Section 3.6 and Appendix M of the revised paper; Supp. S1):

  • The general state. In a deep semantic model, the representation of a past token depends on the context distribution under which it was processed. The state of layer ℓ\ell is therefore a joint probability measure over the states of layer ℓ−1\ell-1 and tokens. The full state is a hierarchy of such measures, which is intractable in general. This makes precise why the general case is hard, as the reviewer anticipated.
  • The collapse at a fixed point. At a fixed point, all past tokens were processed under the same distribution π∗\pi^\ast, so identical tokens again carry identical representations. The hierarchy collapses, and every layer becomes equivalent to the one-layer case we analyzed, but acting on π∗\pi^\ast-dependent token representations (instead of raw token embeddings). The fixed points of a semantic model of any depth, including MLP blocks and LayerNorm, are therefore exactly the solutions of the same self-consistency condition, π∗=π∗Pπ∗\pi^\ast = \pi^\ast P_{\pi^\ast}, differing only in how PπP_\pi is computed (by a layer-by-layer recursion).
  • The algorithm. Our fixed-point finder applies unchanged, with a cost that grows linearly with depth.
  • What it does not give. The description is exact at the fixed points, and not for the transient dynamics around them. The stability of deep fixed points therefore cannot be read from the Jacobian. We state this limitation openly in the revised paper.

Evidence at the scale of a modern model. We applied this construction to Llama-3.2-1B (16 layers, 1.2 billion parameters, 32 query heads per layer), with its rotary positional encoding removed and nothing else changed (Supp. S2). Starting from the unigram distributions of ordinary texts (Wikipedia, web text, code and dialogue), the optimization lowers the loss by about two orders of magnitude. 210 of the 228 runs end below the level that we use in the paper to accept a state as an approximate fixed point. With one exception, these states are not repetitions: they are broad distributions over tens to hundreds of tokens. The same method therefore works, at a moderate cost, at the scale of a modern language model.

What remains open. Deep models with positional information, and the dynamics of deep models away from fixed points, remain open, and we list both under the limitations. We believe that the paper now provides the two components needed to approach them: an approximation for positional one-layer models (MTMF), and an exact fixed-point condition for semantic models of any depth.

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer hJW5 (Part 2/3)

2. Fixed points and applications (W2 and Q2)

Fixed points are not only repetitions. We agree that full repetition collapse is rare in strong models. Repetitions are, however, only one class of fixed points. A fixed point is any token distribution that reproduces itself under generation. This includes the semantic clusters that we find: wide sets of related tokens that sustain each other without repeating. Supp. S7 shows text generated from such a point, which stays in one semantic field for thousands of tokens without becoming a loop. Moreover, a fixed point affects generation without being reached. Unstable points and saddles determine which of several neighbouring behaviours the model commits to (Figure 4), as is well known from the analysis of RNNs. The Llama result points in the same direction: almost all the fixed points that we find there are broad states and not loops.

Applications beyond finding fixed points (Q2). We ran two new experiments that demonstrate direct applications, and describe a third direction.

  • Interpretability: discovering what a head does (Supp. S6). We ran our fixed-point search on a single attention head of a TinyStories model. 17 of its 57 fixed points turn out to be name clusters: states that differ mainly in which character name they contain (Tom, Ben, Anna, …). This suggests that the head keeps track of the character of a story. We then tested this in the full, unmodified model with standard causal tools. Ablating this single head removes about 80% of the model's preference for the correct character name, far more than any other head, and patching in its output from the same story with a different name recovers 83% of the switch to that name. The effect is specific to names. The fixed points thus revealed a function of the head without supervision, since we did not assume a name-tracking role in advance. This example also shows how fixed points give information beyond simple repetitions: the name is never a repetition or a fixed point of the full model, and instead this function is related to copying mechanisms, which were previously linked to in-context learning.
  • Model editing: fixed points as training targets (Supp. S5). Our framework makes the loss of a fixed point a differentiable function of the model weights. Usually we fix the weights and optimize the state. Here we fix the state of a repetition loop and fine-tune the weights so that it stops being a fixed point. In a proof of concept, this removes the loop from the real model (the share of loop tokens drops from 98% to 9%), with only a small change in next-token prediction on ordinary text. The training uses no repeated text at all, only the fixed point. Compared with unlikelihood training, the standard baseline that penalizes the repeated string, it preserves the normal use of the word far better (median probability 0.14 against 0.01, from 0.23 in the original model). This is a proof of concept on one loop of a small model and not a complete method, but it shows that fixed points are actionable and not only descriptive. Future work can lead to more advanced methods targeting more complex behaviours.
  • Control, steering and prediction. A state space is the starting point for the tools of control theory. Knowing the attractors and saddles of a system makes it possible to design small perturbations that steer it towards or away from a basin, and to estimate the probability of transitions between regions of state space, for example from a given state into an undesired behaviour. LLMs are already steered in practice, e.g. by activation addition, but such methods are largely empirical and rarely come with guarantees or predictions about when or why they work. The rigorous application of control theory was not available for LLMs because a state was not defined in the dynamical-systems sense. Future work could use this state space to design new, theory-grounded steering methods for LLMs.

We agree that, since this is a new research direction, its potential applications should be highlighted more clearly, and this was not done in the submission. We will add a discussion of applications to the final paper.

The connection to AI safety. We agree that the paper does not show a direct AI-safety application. Interpretability, editing and control are each relevant to safety, and the two new experiments are first steps in this direction, but the connection is a long-term motivation. We will revise the text of the final version to state this clearly and not to overclaim.

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer hJW5 (Part 3/3)

3. Presentation and clarity (W3)

We agree that the submission was not clear enough in several places, the MTMF part in particular. We rewrote Section 3.5 and Figure 5, and reworked most of the appendices.

The timescales of the heads are a natural phenomenon, not a design choice. We do not train, constrain or modify the model in any way. We measure, on the pretrained model, how each head uses position: the average factor by which position changes the attention weight of a token at a given distance (its positional profile). The heads separate by themselves into two groups, fast heads that attend almost only to the last few tokens, and slowly decaying heads with tails of tens to hundreds of tokens. We find the same separation in the two new models. The MTMF model only uses the measured timescale of each head, and these timescales are not fitted to the benchmark. The revised text states this explicitly.

Figure 5. The figure was redrawn. Panel (a) now shows the measured positional profile of every head together with the exponential tail that MTMF uses in its place. Panel (b) shows the approximation error of MTMF and of its ablations. Panel (c) shows rollouts of the real model that leave an unstable repetition. Each panel is now explained step by step in the text, with full details in Appendices H–K.

Terms and experimental setups. We define terms and shorthands where they first appear, and the appendices now fully specify every experiment: the models, the data, the selection of examples, and all controls (Appendices B, E, G, J and K). We will continue to improve the presentation for the final version. If specific phrases or passages remain unclear, we would be glad to know and will fix them.

Other changes

Following the other reviews, we also corrected the linear stability analysis, which now linearizes the averaged flow itself and not a surrogate. No fixed point changed its stability classification (Appendix F; Supp. S3). We also added a test of whether unstable points persist in finite-length generation (Section 3.4).

We hope that the new results and clarifications address the reviewer's concerns: the framework now covers general one-layer models and the fixed points of deep semantic models, with a demonstration on Llama-3.2-1B, and it has concrete uses beyond locating fixed points. If so, we would be grateful if the reviewer would consider raising the score. We would also be glad to hear which further results or clarifications would be most useful, and are happy to answer any further questions.

Reviewer_4yAAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper considers an one-layer attention head, derives an averaged ODE and a self-consistency equation, and optimizes the equation’s residual to locate fixed points. The paper's core idea is sound, but its main claim about stability and saddles need stronger justification.

Critical strengths
  • Equations (4-6) are sound, reasonable under the stated semantic head assumption. The equivalence between stationary self-consistency and πPπ=π\pi P_\pi=\pi is correct.
  • There are substantial implementation checks. The authors verify agreement with the hooked transformer, re-evaluates final semantic head losses over the full vocabulary, and explains why truncated Jacobian were not good. Table 2 includes paired evaluations on 1000 Wikipedia and 1000 C4 text, with informative ablation.
Critical weaknesses
  • The dynamical system used in stability analysis is different from the derived ODE. The derived ODE is h(π)=πˉ(π)−πh(\pi)=\bar\pi(\pi)-\pi, while the reported eigenvalues come from r(π)=πPπ−πr(\pi)=\pi P_\pi-\pi. Then it correctly derives h(π)=r(π)Zπh(\pi)=r(\pi)Z_\pi. Note that, at an exact fixed point, the linearization includes multiplication by Zπ∗Z_{\pi^*}; it is different from the residual flow Jacobian. However, sharing fixed points does not imply sharing stability, since multiplication by this matrix can change the signs of eigenvalue real parts.

  • Is “Presley” a saddle of the full model? Escaping from an approximate state is insufficient evidence. Some explanations beyond residual drift will strengthen this claim.

  • While this is mentioned in the limitations, I think only consider a simple setting which is one-layer, attention-only transformer is not interesting enough. This does not affect my review.

Questions for authors

Please answer two questions from the weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4yAA

We thank the reviewer for the careful reading, and for acknowledging the soundness of our work. The questions led us to improve the paper, and we answer them below.

W1. The stability analysis used a different dynamical system than the derived ODE. The reviewer is exactly right: sharing fixed points does not imply sharing stability, and the missing factor is the fundamental matrix Zπ∗Z_{\pi^\ast} of the induced Markov chain. We fixed this. The revised analysis linearizes the derived ODE itself, with the Jacobian (J~−I) Zπ∗(\tilde{J} - I)\,Z_{\pi^\ast} at a fixed point.

We recomputed the spectrum of all 355 fixed points of Figure 3 with this Jacobian. No fixed point changes its stability classification: 171 of 178 repetitions are stable and 176 of 177 Wikipedia fixed points are unstable, as before. The number of unstable directions changes for only 5 points, each by a single complex pair of eigenvalues very close to the stability boundary. The agreement has a simple reason: at all of these fixed points the induced chain mixes within about ten tokens, so Zπ∗Z_{\pi^\ast} is close to the identity. The corrected derivation and the updated figure are in Section 3.4, Figure 3 and Appendix F of the revised paper (also Supp. S3).

W2. Is "Presley" a saddle of the full model? We agree that an escape from an approximate fixed point is not sufficient by itself, since any state with a non-zero loss drifts. Three observations distinguish the behaviour of "Presley" from residual drift (Section 3.5 and Appendix K.2 of the revised paper):

  • A control with a similar loss does not escape. The "Preserve" loop was optimized in exactly the same way and has a comparable loss (2.0×10−32.0\times10^{-3}, against 3.1×10−33.1\times10^{-3} for "Presley"). If the escapes were caused by residual drift, a state with a similar residual should also be left. Yet all 32 real-model rollouts launched from "Preserve" stay there, while 28 of the 64 rollouts launched from "Presley" leave it.
  • The escapes are stochastic, not a common drift. Residual drift is a deterministic motion that all rollouts would share. Instead, 36 of the 64 rollouts are still at "Presley" after 450 tokens, and the others leave at different times, spread over the whole rollout. This is the behaviour expected near an unstable point, where sampling noise must first push the state along an unstable direction.
  • The destinations are few and structured. Most escapes end in one of two neighbouring loops, "Preserve" (11) and "Prescription" (8), which share the first token "Pres" and differ in its continuation. This points to a small number of unstable directions, as in a saddle. "Preserve" is also the loop to which the real model switched in the sequence where the example was found.

All of these rollouts are of the real, unmodified model and not of its MTMF approximation. We also re-ran this experiment in the revision with an improved measurement of the MTMF timescales, and the result is unchanged. We agree that this is empirical evidence and not a proof: unlike in the semantic case, we do not compute the Jacobian spectrum of this point. We will phrase the claim accordingly in the final version.

W3. The restriction to a one-layer, attention-only model. Although the reviewer notes that this did not affect the review, we want to point out that the revision goes beyond this setting (details in the general comment):

  • The state is exact for any one-layer semantic model, including multiple heads and an MLP block. The MTMF benchmark on two further models with MLP blocks gives the same accuracy (Appendix J.3; Supp. S4).
  • The fixed-point condition extends to semantic models of any depth (Section 3.6, Appendix M; Supp. S1). On a position-ablated Llama-3.2-1B with 16 layers, the same algorithm finds approximate fixed points in more than 90% of the runs (Supp. S2).

Further new results. Two new experiments show what the framework can be used for: a proof of concept that removes a repetition loop from a model by fine-tuning at its fixed point (Supp. S5), and a family of fixed points organized around character names in a TinyStories model, which identifies the attention head that carries the names in the full model (Supp. S6).

We hope that these answers address the reviewer's concerns. If they do, we would be grateful if the reviewer would consider raising the score. We are happy to answer any further questions or to provide further clarifications.

Area_Chair_TVmpOpenReview ↗

Author-reviewer discussion period is ending soon

Dear Reviewer 4yAA,

If you haven't done so, please read through authors' response and let authors know if you need further clarification or have more questions. Beginning 11/19, authors are not allowed to respond any more, so this is good time to review the response and ask more.

Thank you.

-AC

Reviewer_1fXLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors the dynamics of autoregressive text generation in transformers by developing fixed-dimensional representations of their evolving context. For a single-layer sigle-head model without positional embeddings, the authors show that the generation process can be represented as a self-interacting non-homogenous Markov chain whose state depends on the current token and the empirical unigram distribution of the context. They derive a self-consistency condition for the stationary states of the associated dynamics and develop an optimization procedure to identify these states in pretrained attention heads. The authors also include several experiments that study repetition patterns and semantic clusters under different data distributions. Finally, they try to extend their approach to a full one-layer transformer through a MTMF approximation, which combines a fixed recent token window with exponentially weighted sums of earlier context.

Critical strengths

In my opinion the paper introduces a fascinating perspective on the inference of transformers, by linking it to discrete non-homogeneous Markov processes. Despite a few concerns (see below), I think that this is an interesting theoretical work that is far from complete but that opens interesting research directions that can be further explored in the future. While I believe that the theory is the strongest component of the work, the paper also includes a few interesting experiments to back the theory. The proofs seem correct to me, and everything is well presented.

Critical weaknesses

I have a few concerns on the approach taken by the authors to solve some of the problems arising from their framework, both on the theory and on the experiments. I expounded my concerns in the questions below. More generally, the limit of the theory is that it is by definition exact only for the very simple case of a single-layer single-head attention-only transformer with no positional embeddings. The authors put effort in trying to generalize the framework to multi-head attention with positional embeddings, but there they take a mostly heuristic approach. The limit of the experiments is that they rely on small models and on a few selected examples, limiting possible insights on larger LLMs.

Questions for authors

I would like to ask the following clarification questions to the authors:

  1. The stability classifications in Figure 3 are obtained from the Jacobian of the surrogate flow rather than the original ODE. Although these flows share the same fixed points, their Jacobians differ by a multiplication with Zπ∗Z_{\pi^*}, which can change how these points are classifed in terms of stability. Is there a theoretical or empirical justification for assuming that their stability classifications coincide? In particular, have you compared the eigenvalues of the two Jacobians to rule out the possibility that some supposedly unstable semantic clusters are actually stable under the original dynamics?
  2. You interpret Wikipedia-derived states with branching factor 1 as evidence that unstable semantic clusters can persist over finite generation horizons. While I like the discussion about metastability, the way you define a branching factor of 1 only indicates that the rollouts end in a single cluster. This cluster can in principle be distant from the oiginal fixed point. Did you check this out?
  3. In Appendix J, you report detecting 1,807 transitions between repetition loops, but you only analyse nine of them using MTMF. Could you clarify how these examples were selected and whether they are representative of the full set of detected events?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 1fXL (Part 1/2)

We thank the reviewer for the thoughtful review, and for the encouraging assessment of our work as "a fascinating perspective on the inference of transformers" that "opens interesting research directions". The three questions were precise and led to real improvements in the paper. We answer them first, and then address the concern about the scope of the framework.

Q1. Stability from the Jacobian of the surrogate flow. The reviewer is correct: the two flows share their fixed points but not necessarily their stability. We fixed this. The revised analysis linearizes the averaged ODE itself. Its Jacobian at a fixed point is (J~−I) Zπ∗(\tilde{J} - I)\,Z_{\pi^\ast}, where J~−I\tilde{J} - I is the Jacobian we used before and Zπ∗Z_{\pi^\ast} is the fundamental matrix of the induced Markov chain, which is the factor the reviewer points to.

We recomputed the spectrum of all 355 fixed points of Figure 3 with the correct Jacobian, which is exactly the comparison the reviewer asks for. No fixed point changes its stability classification: 171 of 178 repetitions are stable and 176 of 177 Wikipedia fixed points are unstable, as before. In particular, no unstable semantic cluster turns out to be stable under the original dynamics. The only change is in the number of unstable directions of 5 of the 177 Wikipedia points, each by a single complex pair of eigenvalues very close to the stability boundary. The reason for the agreement is that the induced chains mix within about ten tokens at all of these points, so Zπ∗Z_{\pi^\ast} is close to the identity. See Section 3.4, Figure 3 and Appendix F of the revised paper (also Supp. S3).

Q2. Does a branching factor of 1 imply that the rollouts stay near the fixed point? The reviewer is correct, and it does not. We checked this and revised the text accordingly. A single cluster only shows that the rollouts end together, and they could end together far from the original point. We therefore added a direct test of whether the fixed point π∗\pi^\ast is itself a member of the cluster that the rollouts reach. This holds for 80 of the 98 Wikipedia states with a branching factor of 1 (about 80%), including the semantic-cluster example of Figure 3f. For the remaining 18 states the rollouts move together to a different state, so these are not persistent, as the reviewer suspected. The revised text claims persistence only for the states that pass the test (Section 3.4 and Appendix G).

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer 1fXL (Part 2/2)

Q3. How was the MTMF example selected from the 1,807 transitions? The 1,807 transitions were found by an automatic loop detector, applied to 115,200 generated sequences. The nine events that we analysed (seven candidates and two negative controls) were then selected by hand, as clear cases in which the model holds one loop for a long time and then moves to a different loop and stays there. They are not a random sample, and we do not claim that they are representative of all detected events. Their role is to show that an unstable fixed point of the full model exists and can be found by our method.

In the revision we improved the way the MTMF timescales are measured, and re-ran the full pipeline for the main example, the "Presley" loop of Figure 5c. In its source sequence the model repeats "Presley" about 230 times and then switches abruptly to "Preserve". With the new timescales the result is the same: the optimized state has a low loss (3.1×10−33.1\times10^{-3}), 36 of 64 real-model rollouts launched from it still repeat "Presley" after 450 tokens while the others escape, mostly to "Preserve" (11) or "Prescription" (8), and all 32 control rollouts launched from "Preserve" stay there. The revised paper presents only this example, as a single illustrative case, and states how it was found (Appendix K.2).

The scope of the framework (one-layer models, heuristic extensions, small models). We address this concern in detail in the general comment. In brief:

  • The state is exact for any one-layer semantic model, with multiple heads and an MLP block, and not only for a single-head attention-only model. The revised paper states this explicitly.
  • The MTMF model is indeed an approximation. To test it beyond one model, we repeated its benchmark on two further pretrained one-layer models with MLP blocks (gelu-1l and a TinyStories model). It is as accurate as before, with the same ablation pattern (Appendix J.3; Supp. S4).
  • The fixed-point condition extends to semantic models of any depth (Section 3.6, Appendix M; Supp. S1). We demonstrate this on a position-ablated Llama-3.2-1B with 16 layers, where the same algorithm finds approximate fixed points in more than 90% of the runs (Supp. S2).

Further new results. Two new experiments show what the framework can be used for: a proof of concept that removes a repetition loop from a model by fine-tuning at its fixed point (Supp. S5), and a family of fixed points organized around character names in a TinyStories model, which identifies the attention head that carries the names in the full model (Supp. S6).

We hope that these answers address the reviewer's concerns. If they do, we would be grateful if the reviewer would consider raising the score. We are happy to answer any further questions or to provide further clarifications.

I thank the authors for their convincing answers. I maintain my positive assessment of the paper and advocate for the paper's acceptance.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to reviewers (Part 1/2)

We thank the reviewers for recognizing the conceptual novelty of our work and expressing interest in this direction, as well as noting its soundness. The reviewers’ concerns motivated an analytical extension to multilayer models in the semantic case, new experiments demonstrating the framework’s practical value and broader applicability, and a correction of the stability analysis. We summarize these updates here and provide detailed results and point-by-point responses in the individual replies.

Together with this comment we uploaded a revised paper and a supplementary PDF. The supplementary PDF is self-contained and holds all the new results of the rebuttal, so it can be reviewed by itself. Some of these results are already integrated into the revised paper, as listed at the end of this comment.

The reviews raised three main concerns. We summarize our response to each here, and give the details in the individual replies.

1. "The framework is limited to one-layer, attention-only models." We respond with a clarification and two new results.

  • Clarification. The state representation is exact for any one-layer semantic (position-ablated) model, including multiple heads and an MLP block, and not only for a single head or an attention-only model. Heads are additive, and MLP and LayerNorm are stateless operations that act on each position separately. In the same way, the MTMF approximation applies to any one-layer model. We used an attention-only model in the submission only for simplicity, and the revised paper now states this explicitly.
  • New models. To demonstrate this, we repeated the MTMF approximation error experiment (as in Figure 5b) on two further pretrained one-layer models that contain an MLP block, gelu-1l and a TinyStories model. The approximation is as accurate as in the original model, with the same ablation pattern.
  • Extension to deep models. In a main new result, we show that the self-consistency condition for fixed points extends to semantic models of any depth. At a fixed point every past token was processed under the same context distribution, so every layer reduces to a one-layer semantic layer, and we obtain the same condition π∗=π∗Pπ∗\pi^\ast = \pi^\ast P_{\pi^\ast}. Our fixed-point algorithm then applies to deep models unchanged. The caveat is that for deep models the description is exact at the fixed points themselves, and not for the transient dynamics around them. We demonstrate the extension on a position-ablated Llama-3.2-1B (16 layers, 1.2 billion parameters), where the optimization finds approximate fixed points in more than 90% of the runs.

(Continued in the next comment.)

AuthorsOpenReview ↗

General response to reviewers (Part 2/2)

2. "The applications beyond finding fixed points are not clear." First, we clarify that fixed points are not only repetitions ("doom loops"). The semantic clusters and mixtures that we find are sets of tokens that sustain each other without repeating, and unstable points and saddles shape generation even when the model never converges to them. Second, we give examples of applications, with new experiments to demonstrate them:

  • Interpretability. In a TinyStories model, the fixed points of one attention head are organized around the names of story characters. Ablation and activation patching in the full model confirm that this head carries the character name. The fixed points revealed the function of the head without any prior assumption about it, and even though it is unrelated to repetitions of the full model.
  • Model editing. Our framework makes a fixed point a differentiable function of the weights, so it can serve as a training target. In a proof of concept experiment, we remove a repetition loop from a model by fine-tuning it at the fixed point, with a small effect on the rest of its behaviour.

We will add a discussion of applications to the final paper. We agree that we do not show a direct AI-safety application, and we will state this clearly as a long-term motivation.

3. "The stability analysis used the Jacobian of a surrogate flow." The reviewers are correct, and we fixed this. The stability analysis now linearizes the averaged flow itself. We recomputed the spectrum of all 355 fixed points of Figure 3, and no fixed point changed its stability classification, so the results are unchanged.

Presentation. We also substantially improved the clarity of the paper. The MTMF section and Figure 5 were rewritten, most appendices were reworked, and the experimental setups are now fully specified. We will keep improving the presentation for the final version.

Where to find the new results.

  • Already in the revised paper (and also in the supplementary PDF): the corrected stability analysis (Section 3.4, Figure 3, Appendix F; Supp. S3), the MTMF benchmark on the two new models (Section 3.5, Appendix J.3; Supp. S4), and the fixed-point condition for deep semantic models (Section 3.6, Appendix M; Supp. S1).
  • In the supplementary PDF only (to be integrated into the final paper): fixed points of Llama-3.2-1B (Supp. S2), the model-editing proof of concept (Supp. S5), the name clusters in TinyStories (Supp. S6), and example text generated from the fixed points of Figure 3 (Supp. S7).

Finally, applying dynamical systems tools to autoregressive LLM generation is a new direction. Our guiding principle is to first develop the approach in the simplest setting where an exact, analytic understanding is possible, as was done successfully for RNNs. We believe that the one-layer theory, together with the new extension to deep semantic models, lays the groundwork for analyzing full-scale models.

We thank the reviewers again, and are happy to answer any further questions.

AuthorsOpenReview ↗

Final Revision: all rebuttal results integrated into the main PDF

Dear Area Chair and Reviewers,

Thank you all again for your thoughtful feedback and engagement throughout the discussion. We have uploaded a final revised PDF that brings together all the corrections and new results of the rebuttal:

  • Main text: the corrected stability analysis based on the Jacobian of the averaged flow (Section 3.4, Appendix F) and the new section on deep semantic models (Section 3.6, Appendix M) are updated in the main text.
  • Appendices: all new experiments that were previously only in the supplementary material, namely the deep semantic fixed points of Llama-3.2-1B (Appendix N), the TinyStories name clusters with causal tests in the full model (Appendix O), the fixed-point editing proof of concept (Appendix P), and example text generated from the fixed points (Appendix G.1) are all included and referenced from the main text. The PDF is now self-contained.
  • Polish: typo fixes and improved phrasing throughout.

We will continue to improve the presentation and clarity of the paper for the final version. We are grateful to the reviewers for their constructive feedback, which substantially improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.