Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does

Looped Transformers repeatedly apply the same set of Transformer layers, giving them a recurrent architecture for latent computation. Their strong performance on iterative reasoning and length-generalization tasks suggests an appealing explanation: recurrence may provide an inductive bias that lets the model reuse a learned algorithm across loops. However, weight sharing alone does not imply that every loop performs the same operation. This raises a basic question: is each loop actually repeating the same computation, and if not, what routes the shared parameters to different operations? We study this question using graph walks as a test case. In the model’s native trajectories, decoded predictions can advance by different numbers of graph steps or remain at a reached target, showing that recurrent progress need not follow a fixed one-loop-one-step pattern. We then show that the entering hidden state can steer a frozen loop toward different transitions: a learned affine map selects the desired transition without changing the shared Transformer layers. To test how this steering works, we use activation patching and find that attention patterns can recover its effects and switch the selected transition. Across five pairs of graph models differing only in intermediate supervision, \(J\) can select among computations supported by the frozen backbone but does not create arbitrary new ones. The controllers can be composed across two successive loops, but accuracy is order-dependent and drops with longer compositions. Together, these results show that the entering state can control shared computation, with attention routing as a causal pathway.