Constructive State Abstraction and Finite Identification in Autoregressive Transformers

Understanding which internal information is computationally necessary in a Transformer is a basic question in mechanistic interpretability and state abstraction. We study the state information needed to preserve a fixed-weight, pre-normalized autoregressive Transformer's computation under a declared class of internal interventions. From the weights, we construct explicit updates for residual coordinates in the joint read–write subspace, the squared norm of its omitted complement when normalization requires it, and cache entries grouped by key with their multiplicities and observable value sums. This state reproduces every finite generation distribution without reconstructing the original residual vector. If the centered terminal readout is nonzero, finitely many single-pass interventions identify the state from the probability ratio of one fixed token pair, even for a rank-one readout. Equality of the constructed states is therefore equivalent to equality of observations under all admissible controlled continuations. Controlled cache insertion separates the attention numerator and denominator. We show that residual resets are unnecessary and characterize the information retained when all executed queries lie in specified subspaces. These identification results yield minimum state dimensions for continuous and affine encoders on the stated local domains, and a sharp first-order approximation error under a specified observation metric. Checkpoint analyses examine ranks and sensitivity, while controlled FP64 experiments support fixed-token-pair reconstruction and its dependence on the available interventions, but reveal numerical instability in cache inversion. A separate quantized-checkpoint study finds equal candidate-scalar readings with unequal outputs and intervention effects, distinguishing causal influence from state sufficiency.