- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes a new ansatz to understand autoregressive transformer generation as a state space model with fixed size state. They use the fact that in a one-layer single-head-attention-only transformer without positional encoding, there are no cross-token dependencies prior to the attention, and the output of the attention only depends on the unigram token distribution (which is a state of size \text{vocab_size}). The authors note that in a one-layer multi-head-attention-only transformer (with RoPE/ALiBi-like positional encoding), each attention head has its own dynamical timescale, and propose a multi-timescale mean-field (MTMF) model to handle analysis in this case. They find fixed points of generation (e.g., iterating the same string, "doom loops", etc) using a fixed point algorithm on the induced Markov chain. Experiments (again with one-layer attention-only transformers on simple datasets) show how the theory plays out on real trained networks.
Critical strengths
- The mathematical derivations appear to be solid and correct.
Critical weaknesses
- The paper studies one-layer attention-only transformers. The core reduction to a fixed-size state-space Markov chain seems fundamentally limited to exactly this setting and hard/intractable to generalize. I did find something in the appendix discussing this, but it was fairly hand-wavy with no evidence. This limitation severely restricts the applicability of the framework.
- The central application of the theoretical formulation in this paper is finding fixed points of generation dynamics, which rarely occur in practice (and less frequently with larger and more powerful models). It is not clear how this motivation aligns with the described impact to the AI safety community.
- The writing and presentation, including the figures, is not that clear. For example, when trying to understand the multi-head setup, it's not clear whether the separation of heads into multiple timescales for context retention is done deliberately or is just a natural phenomenon observed in pretrained transformers. The relevant figure, Figure 5, did not clear this up and it was difficult to understand the rest of the figure. Additionally, several phrases and shorthands are used without being introduced, making it difficult to read; experimental setups are not well specified in all cases, etc.
Questions for authors
- Is there a tighter characterization of multi layer transformers under this framework?
- Are there direct applications of the framework besides finding generation fixed points?
6 comments
Response to Reviewer hJW5 (Part 2/3)
2. Fixed points and applications (W2 and Q2)
Fixed points are not only repetitions. We agree that full repetition collapse is rare in strong models. Repetitions are, however, only one class of fixed points. A fixed point is any token distribution that reproduces itself under generation. This includes the semantic clusters that we find: wide sets of related tokens that sustain each other without repeating. Supp. S7 shows text generated from such a point, which stays in one semantic field for thousands of tokens without becoming a loop. Moreover, a fixed point affects generation without being reached. Unstable points and saddles determine which of several neighbouring behaviours the model commits to (Figure 4), as is well known from the analysis of RNNs. The Llama result points in the same direction: almost all the fixed points that we find there are broad states and not loops.
Applications beyond finding fixed points (Q2). We ran two new experiments that demonstrate direct applications, and describe a third direction.
- Interpretability: discovering what a head does (Supp. S6). We ran our fixed-point search on a single attention head of a TinyStories model. 17 of its 57 fixed points turn out to be name clusters: states that differ mainly in which character name they contain (Tom, Ben, Anna, …). This suggests that the head keeps track of the character of a story. We then tested this in the full, unmodified model with standard causal tools. Ablating this single head removes about 80% of the model's preference for the correct character name, far more than any other head, and patching in its output from the same story with a different name recovers 83% of the switch to that name. The effect is specific to names. The fixed points thus revealed a function of the head without supervision, since we did not assume a name-tracking role in advance. This example also shows how fixed points give information beyond simple repetitions: the name is never a repetition or a fixed point of the full model, and instead this function is related to copying mechanisms, which were previously linked to in-context learning.
- Model editing: fixed points as training targets (Supp. S5). Our framework makes the loss of a fixed point a differentiable function of the model weights. Usually we fix the weights and optimize the state. Here we fix the state of a repetition loop and fine-tune the weights so that it stops being a fixed point. In a proof of concept, this removes the loop from the real model (the share of loop tokens drops from 98% to 9%), with only a small change in next-token prediction on ordinary text. The training uses no repeated text at all, only the fixed point. Compared with unlikelihood training, the standard baseline that penalizes the repeated string, it preserves the normal use of the word far better (median probability 0.14 against 0.01, from 0.23 in the original model). This is a proof of concept on one loop of a small model and not a complete method, but it shows that fixed points are actionable and not only descriptive. Future work can lead to more advanced methods targeting more complex behaviours.
- Control, steering and prediction. A state space is the starting point for the tools of control theory. Knowing the attractors and saddles of a system makes it possible to design small perturbations that steer it towards or away from a basin, and to estimate the probability of transitions between regions of state space, for example from a given state into an undesired behaviour. LLMs are already steered in practice, e.g. by activation addition, but such methods are largely empirical and rarely come with guarantees or predictions about when or why they work. The rigorous application of control theory was not available for LLMs because a state was not defined in the dynamical-systems sense. Future work could use this state space to design new, theory-grounded steering methods for LLMs.
We agree that, since this is a new research direction, its potential applications should be highlighted more clearly, and this was not done in the submission. We will add a discussion of applications to the final paper.
The connection to AI safety. We agree that the paper does not show a direct AI-safety application. Interpretability, editing and control are each relevant to safety, and the two new experiments are first steps in this direction, but the connection is a long-term motivation. We will revise the text of the final version to state this clearly and not to overclaim.
(Continued in the next comment.)
Response to Reviewer hJW5 (Part 3/3)
3. Presentation and clarity (W3)
We agree that the submission was not clear enough in several places, the MTMF part in particular. We rewrote Section 3.5 and Figure 5, and reworked most of the appendices.
The timescales of the heads are a natural phenomenon, not a design choice. We do not train, constrain or modify the model in any way. We measure, on the pretrained model, how each head uses position: the average factor by which position changes the attention weight of a token at a given distance (its positional profile). The heads separate by themselves into two groups, fast heads that attend almost only to the last few tokens, and slowly decaying heads with tails of tens to hundreds of tokens. We find the same separation in the two new models. The MTMF model only uses the measured timescale of each head, and these timescales are not fitted to the benchmark. The revised text states this explicitly.
Figure 5. The figure was redrawn. Panel (a) now shows the measured positional profile of every head together with the exponential tail that MTMF uses in its place. Panel (b) shows the approximation error of MTMF and of its ablations. Panel (c) shows rollouts of the real model that leave an unstable repetition. Each panel is now explained step by step in the text, with full details in Appendices H–K.
Terms and experimental setups. We define terms and shorthands where they first appear, and the appendices now fully specify every experiment: the models, the data, the selection of examples, and all controls (Appendices B, E, G, J and K). We will continue to improve the presentation for the final version. If specific phrases or passages remain unclear, we would be glad to know and will fix them.
Other changes
Following the other reviews, we also corrected the linear stability analysis, which now linearizes the averaged flow itself and not a surrogate. No fixed point changed its stability classification (Appendix F; Supp. S3). We also added a test of whether unstable points persist in finite-length generation (Section 3.4).
We hope that the new results and clarifications address the reviewer's concerns: the framework now covers general one-layer models and the fixed points of deep semantic models, with a demonstration on Llama-3.2-1B, and it has concrete uses beyond locating fixed points. If so, we would be grateful if the reviewer would consider raising the score. We would also be glad to hear which further results or clarifications would be most useful, and are happy to answer any further questions.
Response to Reviewer hJW5 (Part 1/3)
We thank the reviewer for the candid and detailed review, and for confirming that the mathematical derivations are solid and correct. The concerns about generality, applications and presentation are fair. They led us to run several new experiments and to rewrite parts of the paper, and we believe that the work is considerably stronger as a result. All new results are collected in the supplementary PDF (sections S1–S7), and some of them are already in the revised paper. We answer each point below.
1. Generality beyond one-layer attention-only models (W1 and Q1)
The one-layer case is broader than it appeared. The state representation is exact for any one-layer semantic (position-ablated) model, with any number of heads and with an MLP block. Heads are additive, and MLP and LayerNorm are stateless operations that act on each position separately, so they do not change the state. The submission did not make this clear, and the revised paper now states it explicitly (Section 3.2). To support it with evidence, we test the MTMF approximation error on two further pretrained one-layer models that contain an MLP block,
gelu-1landTinyStories-1L. For both models, the approximation error is as low as for the original model (mean JSD of 0.003–0.015, against 0.5–0.6 for the baseline), with a similar ablation pattern (Appendix J.3; Supp. S4).A characterization of multi-layer models (Q1). The reviewer is right that the discussion of deep models in the submitted appendix was hand-wavy and came with no evidence. We replaced it with a precise result (Section 3.6 and Appendix M of the revised paper; Supp. S1):
Evidence at the scale of a modern model. We applied this construction to Llama-3.2-1B (16 layers, 1.2 billion parameters, 32 query heads per layer), with its rotary positional encoding removed and nothing else changed (Supp. S2). Starting from the unigram distributions of ordinary texts (Wikipedia, web text, code and dialogue), the optimization lowers the loss by about two orders of magnitude. 210 of the 228 runs end below the level that we use in the paper to accept a state as an approximate fixed point. With one exception, these states are not repetitions: they are broad distributions over tens to hundreds of tokens. The same method therefore works, at a moderate cost, at the scale of a modern language model.
What remains open. Deep models with positional information, and the dynamics of deep models away from fixed points, remain open, and we list both under the limitations. We believe that the paper now provides the two components needed to approach them: an approximation for positional one-layer models (MTMF), and an exact fixed-point condition for semantic models of any depth.
(Continued in the next comment.)