acceptodds
Under review as a conference paper at ICLR 2027

Constructive State Abstraction and Finite Identification in Autoregressive Transformers

Abstract

Understanding which internal information is computationally necessary in a Transformer is a basic question in mechanistic interpretability and state abstraction. We study the state information needed to preserve a fixed-weight, pre-normalized autoregressive Transformer's computation under a declared class of internal interventions. From the weights, we construct explicit updates for residual coordinates in the joint read–write subspace, the squared norm of its omitted complement when normalization requires it, and cache entries grouped by key with their multiplicities and observable value sums. This state reproduces every finite generation distribution without reconstructing the original residual vector. If the centered terminal readout is nonzero, finitely many single-pass interventions identify the state from the probability ratio of one fixed token pair, even for a rank-one readout. Equality of the constructed states is therefore equivalent to equality of observations under all admissible controlled continuations. Controlled cache insertion separates the attention numerator and denominator. We show that residual resets are unnecessary and characterize the information retained when all executed queries lie in specified subspaces. These identification results yield minimum state dimensions for continuous and affine encoders on the stated local domains, and a sharp first-order approximation error under a specified observation metric. Checkpoint analyses examine ranks and sensitivity, while controlled FP64 experiments support fixed-token-pair reconstruction and its dependence on the available interventions, but reveal numerical instability in cache inversion. A separate quantized-checkpoint study finds equal candidate-scalar readings with unequal outputs and intervention effects, distinguishing causal influence from state sufficiency.

Primary area:
interpretability and explainable AI
Keywords:
autoregressive Transformers, mechanistic interpretability, state abstraction, finite identification

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gznfOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper investigates what parts of a transformer’s internal state in the residual stream matter for its outputs. To study this, the authors compress the residual stream, dropping the subspaces that the model does not use for the output, while ensuring that the output remains reproducible. The authors then show that the studied compression function is as strong as it can be, as any changes to the residual stream can be identified from the probability ratios of two tokens. This allows them to determine the minimum state size of the model and lossy compression error below that size.

Critical strengths
  • While I am not able to verify them, the statements seem rigorous and well-backed
  • The high-level aim of the paper, understanding the fundamentals of transformer interpretability, is very relevant and a topic that warrants theoretical treatment
Critical weaknesses
  • The paper is overwhelmingly dense. Admittedly, I am not an expert on the topics covered, but I struggled a lot when trying to understand even the high-level ideas of the contributions. This was my main issue with the paper, as it made it, despite the interesting topic, hard to read and follow.
    • The writing is, as perhaps expected for a theoretical contribution, very dry and to the point, but particularly because of my lack of familiarity with the field, I found the very direct exposition hard to follow. For example, many claims and crucial parts appear without prior justification (e.g., Assumption 2 begins Section 4 with no context given).
    • Similarly, many concepts are used without explanation (again, it might be clear to someone more familiar with the topics), for example (among others), “native execution”, “cache-write suppression”, “probe reachability”, etc.
  • Altogether, this led to me struggling to understand what the different claims and results in the paper are trying to achieve and, most importantly, how they contribute to the main goal of understanding the interpretability of transformers. Experiments seem to aim to apply the derived results to real-world models, but, similarly, it was difficult to discern what they contribute.
  • To the best of my understanding, the strict requirements lead to the conclusion that everything in the model matters (i.e., the compressibility is minimal); experiments confirm this in actual models. Thus, I am not sure about the usefulness of the techniques.
Questions for authors
  • The introduction starts by motivating the work by trying to understand the computation of a transformer. What do the results imply about the steerability of transformer LMs? What would be the implications for, e.g., mechanistic interpretability, if any?
Reviewer_innVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

[I should start by saying, that I do not fully understand what this paper is trying to do. So I would request the authors to correct me, if I have misunderstood what they have written].

The paper is trying to find activations and the cache stored in a Transformer that are actually 'necessary' for its computations, where, the necessity is based on the fact that the overall behavior should change predictably if certain kinds of interventions are applied. Then the authors create what they call an 'executable state', which as I understand is a representation of the said activations and cache, that can be updated in a way that preserves future output distributions. This representation called the executable state can in fact, also be found by finitely many experiments. Across experiments, the same two output tokens are compared, and if the representations of these tokens always agree, then the resulting representation is said to have been found. There are some experiments done to show all of this empirically, but I must admit I don't quite follow what the experiments are exactly testing or showing.

Critical strengths

I feel like there is a strong idea under the heavy jargon sounding like text somewhere. I did not follow the exact method and empirical way of establishing how the 'executable state' is recovered. However if the overall idea is correct, and the experiments do establish this, then this should be a good contribution.

I found the text to be difficult to follow and therefore I would be happy to engage during the discussion period with the authors to change my opinion, and find more strengths.

Critical weaknesses

too much jargon/ esoteric language: Let alone the methodology, and details I was unable to understand what the introduction and the abstract is trying to say clearly. There was a lot of technical jargon whose usage did not seem justified (at least to me). So the most obvious weakness according to me is that the language of the paper is very esoteric, there is use of jargon when simpler language could have been more helpful.

What are the implications?: The discussion in the paper is lacking, I do not quite understand what are the implications of the said results. The experiments and discussion section in the paper just lists the set of results, and does not actually discuss the implications/ takeaways from the experiments.

Difficult to grasp the numbers, lack of figures: The results are listed in tables and sometimes even just in paragraphs. These numbers are a bit hard to interpret and compare in isolation. Some plots / figures might be more useful in visually seeing what's going on, especially to understand statistical significance.

If other reviewer's understand what's going on in the text and it's only me who doesn't follow, then I apologize to the authors for my review. However if it turns out, that most of the reviewers don't follow, then I would suggest to the authors to try and spend some time in trying to reduce the technical jargon where possible, introduce terms before using them, and in general give a bit more thought to the readability.

Questions for authors

I request the authors to kindly explain what they are doing in layman terms. I would really want to try and understand the method of finding the executable state a bit better, also to better understand the experimental setup is, and understand the implications of each experiments are.

Reviewer_oisTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

Please see the weaknesses.

Critical strengths

None.

That is not to say that there is no substantive strength; I cannot comment on that (see weaknesses). It is that I fail to identify any substantive strength.

Critical weaknesses

Based off of my extensive experience supervising projects and interacting with LLMs, I believe that this paper is one of several in my reviewing batch that are entirely LLM written. That does not preclude interaction with a human author, but the guiding force and the main writer of this paper, I claim, is an LLM. That is by itself not a ground for rejection; a valuable contribution is a valuable contribution wherever it comes from, but I have to say that I fail to find the value in this work. It is very difficult to follow, and that is my main ground for rejection. A paper may deal with a difficult topic, making it intrinsically hard to follow without significant background knowledge. However, for example, one may have little knowledge of differential geometry and stochastic calculus, but the core ideas behind Einstein's papers remain more accessible than the ideas presented here.


Before I list a few specific weaknesses, I think you will better understand what I mean through the following.

Consider the abstract. Instead of presenting the idea at a conceptual level, the abstract quickly delves into details which cannot be understood without having read the paper, and are arguably even unimportant for the general summary of the paper.

Consider for example the first sentence that describes the method in any detail: "From the weights, we construct explicit updates for residual coordinates in the joint read–write subspace, the squared norm of its omitted complement when normalization requires it, and cache entries grouped by key with their multiplicities and observable value sums". Why is the concept of the joint read-write subspace significant enough to use it in the first conceptual summary of the method in the text? Why does normalisation constitute such an important part of the theory? Why do you want to group cache entries with their key multiplicities; does this paper deal with computational efficiency? These are very specific details to discuss. If they are important, we should first discus their significance, not immediately delve into the technicalities.

Following that, in the abstract, "This state reproduces every finite generation distribution without reconstructing the original residual vector. If the centered terminal readout is nonzero, finitely many single-forward-pass interventions identify the state from the probability ratio of one fixed token pair, even for a rank-one readout". A very technical sentence that does not help the reader understand anything about the significance of the method. Why are we discussing rank-one readouts? What is a finite generation distribution? Why do we explicitly not want to reconstruct the original readout vector? Everything is correct, I presume, but does not belong in the abstract.

Compare this with the abstract the paper cites, "Causal Abstractions of Neural Networks" by Geiger, Lu, Icard and Potts. This influential technical paper has only a single technical sentence in the abstract that reads: "In this method, neural representations are aligned with variables in interpretable causal models, and then interchange interventions are used to experimentally verify that the neural representations have the causal properties of their aligned variables."

Let us also consider the conclusion section, which states the following: "The constructed state preserves computation and finite generation distributions through explicit updates derived from the weights. With any nonzero centered terminal readout, controlled cache insertion and residual translations identify its retained information from one fixed token-pair probability ratio, yielding local minimum-dimension results and a sharp first-order approximation bound. Resets are unnecessary, and restricting every executed query gives the projected-key state of Corollary 5."

Beyond this part, the section discusses technical limitations. This is the conclusion of the work. Can you understand it?


Now a few specific weaknesses, which can hopefully aid a productive discussion with the authors.

  1. The presentation of the paper is severely underoptimized. Concepts are introduced abruptly, with insufficient explanation.
  2. When introducing the Transformer, you would usually want to describe the full architecture rather than just a generalised version of the normalization function.
  3. Too much generality in the concepts, and too many concepts. We are discussing "token-entry states", "live residuals", "cache-write suppression", "groups" (but not algebraic ones, rather seemingly defined in terms of "cache groups"), "reduced residual-write matrix", even a symbol for the set of "query heads reading group c, after zero centered-write heads are omitted".

If the authors express sincere interest in improving the paper, I will gladly expand on what I consider the weaknesses of the paper in more detail during the rebuttal process. It is in both of our interests to provide a meaningful contribution to science, and the area of mechanistic interpretability, especially considerations of the computational (algorithmic) redundancy in representations that may be extracted by probes or interventions, are interesting to us both. However, at this point, I believe it is not appropriate for a human reviewer to spend a significant amount of effort to decode the signal in a text with this many signs of LLM generation.

Questions for authors

Apart from addressing the weaknesses above, I have several general questions which can hopefully increase the quality of the submission.

  1. How would you position your approach compared to other standard methods of mechanistic interpretability?
  2. Could you, without use of LLMs, summarise your paper for me? I would like to hear about the motivation behind it, the main findings, and what you believe would be interesting extensions.
  3. Can you provide a list of three main results which you think anyone should take away from your work? The list should be written without excessive use of technical concepts, such as e.g. some of those listed in the third weakness.

In general, I would appreciate it if the authors write the responses independently, using LLMs only to translate if needed but not to draft. We all use LLMs to obtain information, please do so too, but consider my claim that you will gain the most if you practice integrating the information provided by an LLM and expressing it yourself.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.