acceptodds
Under review as a conference paper at ICLR 2027

Operand Access and State Dynamics in Looped Transformers

Abstract

Looped Transformers increase computational depth by repeatedly applying shared parameters across recurrent steps, yet it remains unclear how their internal representations support multi-step computation and extrapolation beyond the training horizon. We study modular product learning in looped Transformers, including progressively masked models and Fixed-Point Reasoning Models, using mechanistic interpretability to examine how operand access shapes their computation and extrapolation. We find that the way operands are exposed across recurrent steps, rather than recurrent depth alone, determines how well models extrapolate. Progressive operand exposure extends the compute horizon for commutative composition but fails for non-abelian composition, while among models that learn the task, sequential exposure generalizes farther beyond the training range than causal access. In one successful model, accuracy decreases from 100.0% at sequence length 8 to 10.0% at length 24, with failure emerging gradually through declining readout margins rather than instability or drift in the recurrent state. Representation analyses reveal distinct patterns of information encoding across successful models, suggesting that task success and length generalization alone do not uniquely determine the internal representation of a solution. Together, these results show that operand-access constraints shape recurrent extrapolation without uniquely determining how the underlying computation is represented.

Primary area:
interpretability and explainable AI
Keywords:
Loop transformers, Modular products, Discrete log clock, non-abelian multiplication, fixed-point reasoning model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uJjpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the underlying mechanisms of looped transformers (LTs) when solving modular multiplication tasks (e.g., 2 × 3 × 6 mod 113). By training small-scale looped transformers on such tasks from scratch and evaluating their performance on problems beyond the training horizon (e.g., training on up to 8-hop problems and evaluating on 32-hop problems), the authors make several main contributions. First, they find that looped transformers can achieve length extrapolation by learning reusable updates across recurrences, and that operand-access patterns (e.g., attention masking at each recurrence to determine which factors to read) can affect extrapolation performance. Second, they find that even models with similar in-distribution performance can vary significantly in length generalization performance, and argue that this generalization failure is due to readout degradation rather than recurrent-state instability.

Critical strengths
  • The paper studies the underlying mechanisms of looped transformers in solving reasoning tasks, which is a timely and interesting topic, given the growing interest in pretraining looped transformers as LLMs.
Critical weaknesses
  • The paper seems heavily written by AI, yet there is no AI USE STATEMENT section to disclose the AI usage, which breaks the ICLR AI policy for authors.

  • Some of the claimed contributions overlap with previously published work, and there is insufficient discussion comparing them with prior work or justifying the additional contributions. For example, the authors claim that their first contribution (line 53) is to show that looped transformers can perform length generalization, but this has already been systematically studied in [1, 2]. Similar analyses of readout margins and grokking in looped transformers have also been conducted in [2], which is not cited.

  • Some experimental setups need further justification. For example, Section 3.2 tests whether extrapolation depends on information being carried between recurrent applications by resetting the recurrent state to zero (lines 169–172). This seems to provide limited additional insight: since looped transformers are trained to use the recurrent states from earlier steps for deeper computations, resetting the state to zero would very likely lead to near-zero accuracy. The proposed Picard-style update (lines 173–174) also needs further justification to compare with existing residual connections. Additionally, the experimental setups seem to follow the curriculum learning regimes used in [2, 3] without proper references.

  • The writing is hard to follow, and some technical details are not clearly presented. It is unclear to me what model sizes are used across the main experiments, and the authors seem to adopt different model configurations (e.g., one shared Transformer block in Section 5.1 versus six shared Transformer blocks in Section 3.5). I would also suggest improving the writing of the introduction and experimental sections to better emphasize the key novelties over established work. The current experimental section includes many separate findings, making it difficult to identify the key takeaways. It would also be helpful to include figures in the introduction or methodology sections to make the paper easier to understand.

[1] Fan, Y., Du, Y., Ramchandran, K., & Lee, K. (2025, May). Looped transformers for length generalization. In International Conference on Learning Representations (Vol. 2025, pp. 14502-14520).

[2] Kohli, H., Parthasarathy, S., Sun, H., & Yao, Y. (2026). Loop, think, & generalize: Implicit reasoning in recurrent-depth transformers. arXiv preprint arXiv:2604.07822.

[3] Yao, Y., Du, Y., Zhu, D., Hahn, M., & Koller, A. (2025, November). Language models can learn implicit multi-hop reasoning, but only if they have lots of training data. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (pp. 9695-9713).

Questions for authors
  • Do the authors have any insights into how the studied progressive or sequential schemes could be applied to realistic (pre)training tasks? Do these methods require prior knowledge of the problem structure, such as the positions of operand tokens in the tasks studied here?
1 comment

Summary of changes in the revised PDF

New experiments

  • Frozen linear decoder on the three progressively masked multiplication models of Table 1 (Sec. 5.4).
  • Matched sequential vs causal analysis on S4 and A4, three seeds each (Sec. 5.8, Appendix C).
  • Looped-GPT patching with a stated unit definition and selection rule (Sec. 3.5), 1,000 base/source pairs, a block-matched random control and a dose curve over 5, 10, 20 and 50% of units (Sec. 5.9).
  • Grokking timing over three seeds per model (Sec. 5.9).

Claims revised

  • Extrapolation failure is now attributed to drift in the content of the recurrent state, not to readout loss with a stable state (abstract, Sec. 5.4, conclusion).
  • The Looped-GPT result is now reported as distributed, not localized, computation (Sec. 5.9).
  • The original representation comparison is labelled illustrative (Sec. 5.7).

Positioning and presentation

  • New Appendix D comparing against Fan et al. (2025), Kohli et al. (2026), Yao et al. (2025) and other related work, with a pointer in Related Work (Sec. 2.4).
  • Curriculum citations added (Sec. 4).
  • AI Use Disclosure section added before the references.
  • The key matched-analysis figure is now in the main text (Fig. 1, Sec. 5.8).
Reviewer_as1COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper looks on how operand-access schedules affect length generalization in looped Transformers on modular multiplication and non-abelian group composition. The main finding is that sequential access extrapolates farther than causal access, while successful models can develop different internal representations.

Critical strengths

The most important results are:

  1. controlling parameter count and recurrent updates provides evidence that access restrictions affect extrapolation, rather than simply changing the amount of computation.
  2. separating final-product accuracy from mean-prefix accuracy is also interesting since the latter can substantially overstate extrapolation performance
Critical weaknesses
  1. While the paper claims explanation for extrapolation failure this is not clarified. Because the norm measures the hidden-state vector’s length, not its direction or what it encodes, as stated in Section 5.4 a constant hidden-state norm does not rule out drift in the represented information. At the same time, declining top-two logit gaps do not establish the cause of failure.

  2. The mechanistic experiments are not connected enough to the extrapolation results. In particular Sections 5.7–5.10 provide useful observations about representations and causal influence, but their connection to the operand-access extrapolation results remains unclear. In Sections 5.7 models differ in more than their access schedule. In particular the comparison, the task, training length and the number of recurrent steps also change. Therefore, the observed representation differences cannot be used to explain its extrapolation advantage. In the experiment from Sections 5.10 does not test which mechanism lets a model correctly compose longer sequences or it does not compare that mechanism between sequential and causal access.

Questions for authors
  1. what evidence beyond the state norm rules out representational drift? Do correct-class margins support the proposed explanation across seeds?

  2. does progressive access still fail on non-abelian groups when operand order is explicitly provided?

  3. section 5.10 says 20% of neurons are patched, but Table 3 reports 461/9,216, approximately 5%. Which is correct?

  4. how does the effect compare with random neurons matched by layer and count?

1 comment

Summary of changes in the revised PDF

New experiments

  • Frozen linear decoder on the three progressively masked multiplication models of Table 1 (Sec. 5.4).
  • Matched sequential vs causal analysis on S4 and A4, three seeds each (Sec. 5.8, Appendix C).
  • Looped-GPT patching with a stated unit definition and selection rule (Sec. 3.5), 1,000 base/source pairs, a block-matched random control and a dose curve over 5, 10, 20 and 50% of units (Sec. 5.9).
  • Grokking timing over three seeds per model (Sec. 5.9).

Claims revised

  • Extrapolation failure is now attributed to drift in the content of the recurrent state, not to readout loss with a stable state (abstract, Sec. 5.4, conclusion).
  • The Looped-GPT result is now reported as distributed, not localized, computation (Sec. 5.9).
  • The original representation comparison is labelled illustrative (Sec. 5.7).

Positioning and presentation

  • New Appendix D comparing against Fan et al. (2025), Kohli et al. (2026), Yao et al. (2025) and other related work, with a pointer in Related Work (Sec. 2.4).
  • Curriculum citations added (Sec. 4).
  • AI Use Disclosure section added before the references.
  • The key matched-analysis figure is now in the main text (Fig. 1, Sec. 5.8).
Reviewer_cpkjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper analyzes looped transformers and how their depth generalizes beyond training depths on a synthetic task of modular multiplication. The architecture is one shared transformer block with no positional encoding. The authors consider different operand access levels such as progressive access, sequential access, and causal access. They also explore different update styles. The key observations are that progressive masking somewhat extrapolates in length, while sequential access degrades more slowly than causal access beyond the training horizon.

Critical strengths

I liked that the authors report the per-seed spread and the gap between mean-prefix and final accuracy for the fixed-point model (FPRM), which help understand the gains and the variance more appropriately. Overall, I like the problem setup and the approach used to study iterative computations in looped transformers.

Critical weaknesses

The key observation that transformers find it challenging to compose to lengths beyond training is well established in prior works. While its important to replicate the findings specifically for looped transformers, this significantly reduces the impact of the work. The finding that operand access patterns are important seems locally relevant to the specific problem, and there is no attempt to generalize this importance to other task setups beyond modular multiplication.

In the progressive setting, the failure on non-abelian groups is likely due to lack of positional information in the setup by design. This essentially prevents the curriculum training to take over as the initial accuracy threshold is never reached. Please clarify if this not the case and how.

Please consider moving the figures in the main text instead of appendix. They are much more useful to understand the results. Also, it'd help to have more descriptive figures of each of the variant, so its easier to understand the nuances.

Questions for authors

NA

1 comment

Summary of changes in the revised PDF

New experiments

  • Frozen linear decoder on the three progressively masked multiplication models of Table 1 (Sec. 5.4).
  • Matched sequential vs causal analysis on S4 and A4, three seeds each (Sec. 5.8, Appendix C).
  • Looped-GPT patching with a stated unit definition and selection rule (Sec. 3.5), 1,000 base/source pairs, a block-matched random control and a dose curve over 5, 10, 20 and 50% of units (Sec. 5.9).
  • Grokking timing over three seeds per model (Sec. 5.9).

Claims revised

  • Extrapolation failure is now attributed to drift in the content of the recurrent state, not to readout loss with a stable state (abstract, Sec. 5.4, conclusion).
  • The Looped-GPT result is now reported as distributed, not localized, computation (Sec. 5.9).
  • The original representation comparison is labelled illustrative (Sec. 5.7).

Positioning and presentation

  • New Appendix D comparing against Fan et al. (2025), Kohli et al. (2026), Yao et al. (2025) and other related work, with a pointer in Related Work (Sec. 2.4).
  • Curriculum citations added (Sec. 4).
  • AI Use Disclosure section added before the references.
  • The key matched-analysis figure is now in the main text (Fig. 1, Sec. 5.8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.