When Do Stored Facts Become Available in Looped Transformers?

Looped transformers do not acquire new knowledge during a forward pass, yet repeated applications of their shared block can change when and how stored knowledge becomes available. We study this process in looped transformers trained on synthetic tasks, focusing on factual retrieval and multi-hop reasoning. We first investigate single-hop factual retrieval. While prior work shows that looped and non-looped transformers store the same number of bits in their weights, it remains unclear how recurrence makes this knowledge available during inference. We find that stored facts become readable progressively across recurrences rather than through an immediate lookup. Moreover, the number of recurrences required to recover them increases with memorization load. Through representational analysis, we characterize this progression as a transition from representations of the query key to representations of its associated value. We then study multi-hop reasoning, where each retrieved entity serves as the key for the next relation. We find that intermediate entities emerge in logical order across recurrences, while the hidden-state geometry progressively shifts from the current frontier entity toward its successor. Comparing fixed- and dynamic-recurrence training, we find that dynamic recurrence exhibits faster state turnover and enables earlier solutions. We further show that these representations are functionally involved in the computation: counterfactual patches to the frontier entity redirect subsequent bridge entities and ultimately change the model’s answer. Together, these results show that stored knowledge becomes progressively available in representations that support subsequent computation. More broadly, these findings suggest that recurrence organizes when and how stored knowledge can be used, rather than increasing storage capacity, motivating training and early-exit strategies that account for the progressive availability of knowledge across recurrences.