Spectral Signatures And Emergent Equivariance In Looped Transformers For In-Context Dynamical-System Prediction

We investigate the internal representations and computations learned by looped Transformers for in-context prediction of stochastic linear dynamical systems. Pretrained on diverse next-state prediction tasks, the models develop approximate spectral degeneracies and shared operator structure without explicit equivariance constraints. We explain these observations through a shared hidden representation of orthogonal coordinate transformations, in which compact core matrices organize the model's operators. Projecting onto this structure yields a computational skeleton that largely preserves the original model's outputs and predictive performance. With a specified first-loop routing rule, the fixed cores support prediction at new input dimensions and longer contexts without retraining. This compact representation makes the learned computation easier to interpret and analyze, connecting its hidden organization to its predictive function and reuse.