- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the underlying mechanisms of looped transformers (LTs) when solving modular multiplication tasks (e.g., 2 × 3 × 6 mod 113). By training small-scale looped transformers on such tasks from scratch and evaluating their performance on problems beyond the training horizon (e.g., training on up to 8-hop problems and evaluating on 32-hop problems), the authors make several main contributions. First, they find that looped transformers can achieve length extrapolation by learning reusable updates across recurrences, and that operand-access patterns (e.g., attention masking at each recurrence to determine which factors to read) can affect extrapolation performance. Second, they find that even models with similar in-distribution performance can vary significantly in length generalization performance, and argue that this generalization failure is due to readout degradation rather than recurrent-state instability.
Critical strengths
- The paper studies the underlying mechanisms of looped transformers in solving reasoning tasks, which is a timely and interesting topic, given the growing interest in pretraining looped transformers as LLMs.
Critical weaknesses
-
The paper seems heavily written by AI, yet there is no AI USE STATEMENT section to disclose the AI usage, which breaks the ICLR AI policy for authors.
-
Some of the claimed contributions overlap with previously published work, and there is insufficient discussion comparing them with prior work or justifying the additional contributions. For example, the authors claim that their first contribution (line 53) is to show that looped transformers can perform length generalization, but this has already been systematically studied in [1, 2]. Similar analyses of readout margins and grokking in looped transformers have also been conducted in [2], which is not cited.
-
Some experimental setups need further justification. For example, Section 3.2 tests whether extrapolation depends on information being carried between recurrent applications by resetting the recurrent state to zero (lines 169–172). This seems to provide limited additional insight: since looped transformers are trained to use the recurrent states from earlier steps for deeper computations, resetting the state to zero would very likely lead to near-zero accuracy. The proposed Picard-style update (lines 173–174) also needs further justification to compare with existing residual connections. Additionally, the experimental setups seem to follow the curriculum learning regimes used in [2, 3] without proper references.
-
The writing is hard to follow, and some technical details are not clearly presented. It is unclear to me what model sizes are used across the main experiments, and the authors seem to adopt different model configurations (e.g., one shared Transformer block in Section 5.1 versus six shared Transformer blocks in Section 3.5). I would also suggest improving the writing of the introduction and experimental sections to better emphasize the key novelties over established work. The current experimental section includes many separate findings, making it difficult to identify the key takeaways. It would also be helpful to include figures in the introduction or methodology sections to make the paper easier to understand.
[1] Fan, Y., Du, Y., Ramchandran, K., & Lee, K. (2025, May). Looped transformers for length generalization. In International Conference on Learning Representations (Vol. 2025, pp. 14502-14520).
[2] Kohli, H., Parthasarathy, S., Sun, H., & Yao, Y. (2026). Loop, think, & generalize: Implicit reasoning in recurrent-depth transformers. arXiv preprint arXiv:2604.07822.
[3] Yao, Y., Du, Y., Zhu, D., Hahn, M., & Koller, A. (2025, November). Language models can learn implicit multi-hop reasoning, but only if they have lots of training data. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (pp. 9695-9713).
Questions for authors
- Do the authors have any insights into how the studied progressive or sequential schemes could be applied to realistic (pre)training tasks? Do these methods require prior knowledge of the problem structure, such as the positions of operand tokens in the tasks studied here?
Revisions
Summary of changes in the revised PDF
New experiments
Claims revised
Positioning and presentation