Tracing the Formation of Alternative Internal Solutions in Transformers

How do transformers arrive at alternative internal solutions to the same state-tracking task? We study cyclic state tracking, where distinct Fourier modes can each encode the complete state. Under standard training on $C_5$, runs with balanced two-mode representations converge rapidly, whereas single-mode-dominant runs converge slowly. Encouraging balanced mode energy yields balanced endpoints across runs; most converge rapidly, but a minority remain slow. Tracing the modes through the final MLP reveals a difference in how these representations are formed: in slow runs, one mode depends directionally on another, as shown by removal and rotation interventions. This intervention signature, identified in discovery runs, also separates the convergence regimes in held-out runs. Extending the analysis to the three-mode setting of $C_7$, we find that faster convergence is likewise associated with energy distributed across multiple modes, while differences in their causal contributions are associated with cross-mode dependence. Together, these findings show that coexisting representations need not reflect independent computational pathways, highlighting the importance of tracing how they arise within the network when interpreting a model’s internal solution.