- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper recasts the recurrent transitions of looped Transformers as optimization steps, typically gradient steps with weight decay or modifications thereof, with respect to implicit local objectives. In this view, the recurrent state itself is the variable being optimized (a fast weight), and each transition is derived from a projection of the state, a local objective and an optimizer update rule.
By mapping existing loop transitions into this framework, the authors identify mismatches between the updates predicted by the optimization view and the actual looping mechanisms. They modify the transitions accordingly and successfully train the resulting looped Transformers, which match the downstream performance of the models they were derived from. This shows the viability of their theoretical view. Notably, the framework provides a way to design new and more powerful looped Transformers in the future by designing new objectives, projections and update rules.
Critical strengths
S1. The modifications derived from the optimization view preserve the downstream performance. Without knowledge of the underlying optimization objective, the input-map alignment would appear to be a rather non-trivial change to existing transitions. Nevertheless, the modified models match the downstream performance of their original counterparts, which shows that the theoretical view is viable in practice.
S2. The paper provides a new way to design looped transformers. Decomposing a loop transition into a projection, an objective and an update rule turns the design of a transition into the choice of these components, which can guide the development of future looped transformers.
Critical weaknesses
W1. The theoretical support for the optimization view is weak. Although the paper recasts looped transformers as optimizers, the theoretical evidence for this view is limited. The transitions are traced back to objectives that merely relate the target to the projection : either a negative inner product or, for the proposed OperLoop, the squared distance (the delta objective). Since is itself computed from , but held fixed in the derivation/differentiation, these are not optimization objectives in the usual sense, which would be functions of alone. Moreover, the paper offers no intuitive interpretation of these objectives in the context of looped transformers. This has two unfortunate consequences: the framework provides little new understanding of these architectures, and it gives no intuition about which new objectives could, in turn, lead to better looped Transformers. The optimization perspective is further weakened by the fact that typically only three to six recurrent steps are performed. As the optimization view is the central contribution of the paper, this point is central to our assessment.
W2. The experiments show no qualitatively new behaviour, and some conclusions are overstated. The modified models match the performance of the original methods from which they were derived, but they do not change their behaviour in an interesting way. The statement in line 366 that the results "support the benefits of its transition design for both language modeling and generation" overstates these benefits. Lines 371 and 372 conclude that looped Transformers have benefits over non-looped baselines; this likely has already been shown by the modified works themselves and is not the main claim of the paper. Hence, the experiments demonstrate the viability of the framework, but not a clear advantage resulting from it.
Questions for authors
Q1. I am not familiar with the literature on looped transformers and I am not aware of its exact current state, so we ask this question for clarification. Are the four methods in Table 1 (Ouro, Huginn, Parcae and HyperLoop) the most important examples of looped transformers? There seem to be other relevant works that are not discussed, for example Mixture-of-Recursions (Bae et al., NeurIPS 2025) or the Hierarchical Reasoning Model (Wang et al., 2025), which updates two coupled recurrent states. What is the reason for selecting these specific methods and not others?
Q2. In Section 3.1 (Vanilla State Transition), the recurrent state is , i.e. -dimensional, whereas the hidden state in Section 2 is . Could you clarify this change? Moreover, the text currently reads as if the random initialization could be combined with a zero bias , which would make the prelude output irrelevant. Is this combination intended?
Q3. What is the interpretation of the objective in line 218, and of the differentiation with a fixed target ?
Q4. Similarly, what is the interpretation of the delta objective in Section 4?
4 comments
Response to Reviewer vzef (2/3)
Q1: I am not familiar with the literature on looped transformers and I am not aware of its exact current state, so we ask this question for clarification. Are the four methods in Table 1 (Ouro, Huginn, Parcae and HyperLoop) the most important examples of looped transformers? There seem to be other relevant works that are not discussed, for example Mixture-of-Recursions (Bae et al., NeurIPS 2025) or the Hierarchical Reasoning Model (Wang et al., 2025), which updates two coupled recurrent states. What is the reason for selecting these specific methods and not others?
RQ1: Thank you for suggesting these relevant works. Our focus is how to design and improve loop transitions within looped Transformers. We therefore select representative works on loop-transition design for the comparison in Table 1: Huginn, Parcae [3], and HyperLoop [2]. Under our abstraction, they introduce input injection, structured state transformation, and an expanded multi-stream state, respectively, while Ouro serves as the vanilla-loop reference.
Mixture-of-Recursions [4] and HRM [5] primarily emphasize complementary, higher-level architectural choices: Mixture-of-Recursions combines parameter sharing, adaptive recursive depth, and KV caching; HRM organizes two coupled recurrent modules at different time scales. We will add a discussion of Mixture-of-Recursions and HRM to the related work.
Q2: In Section 3.1 (Vanilla State Transition), the recurrent state is , i.e. -dimensional, whereas the hidden state in Section 2 is . Could you clarify this change? Moreover, the text currently reads as if the random initialization could be combined with a zero bias , which would make the prelude output irrelevant. Is this combination intended?
RQ2: Thank you for identifying these ambiguities. We will clarify the notation and explicitly state the per-token convention. We omit the token index to make the loop-state dimensions easier to compare. collects the hidden states of tokens, whereas the recurrent (loop) state describes the state of one token. The expanded recurrent (loop) state likewise describes one token, with parallel streams.
This combination is not intended. We should have distinguished the architecture-specific initialization/injection pairs more clearly. For Ouro, the state is initialized from the Prelude output, , with no input injection (). Huginn uses random initialization, , together with input injection, . Both configurations retain a path from the input. We will specify these initialization/injection pairs in Section 3.1.
Response to Reviewer vzef (3/3)
Q3: What is the interpretation of the objective in line 218, and of the differentiation with a fixed target ?
Q4: Similarly, what is the interpretation of the delta objective in Section 4?
RQ3–RQ4: Thank you for asking about the objectives and the fixed-target derivative. Both loop transitions and optimizer update rules describe iterative state updates. Our framework connects them by treating the looped state as a fast weight and identifying an implicit target and a compatible local objective.
-
Differentiation with a fixed target. The implicit target is the shared blocks’ prediction, used as a reference for the current local state update. Holding it fixed is a convention for deriving the local loop transition algebraically; end-to-end training still backpropagates through target generation without detaching gradients. A related perspective appears in self-supervised learning: Kaiming et al [6] interpret SimSiam through an approximate alternating-optimization hypothesis, where a network-generated target is held fixed within an optimization subproblem. This illustrates how a predicted target can serve as a fixed reference for a particular update.
-
Negative inner product. For the negative inner product objective, the write term is driven by the predicted target, without an explicit correction for the current projection. It rewards a larger inner product with the target. Its projection-space gradient is , giving a write proportional to . For a fixed nonzero target, the objective is unbounded below in projection space.
-
Delta objective. The delta objective adds an explicit correction based on the difference between the predicted target and the current projection. It penalizes projection–target mismatch, giving the negative fast-weight gradient . The objective is bounded below by zero and minimized when the projection matches the target.
Moreover, we analyze and visualize the local objectives in the anonymous supplementary material to make their behavior more intuitive. Supplementary Figure 1 illustrates their geometry, while Supplementary Figures 2–3 visualize their values across loop iterations.
Anonymous supplementary figures (https://anonymous.4open.science/r/loop_rebuttal-5F77/README.md)
Thank you again for your careful review. We hope the local-gradient formulation, diagnostics, and controlled comparisons address your concerns about the objectives and their empirical value. If so, we would appreciate it if you could consider raising your rating in light of this evidence. Please let us know if a substantive concern remains.
The authors
References
[1] Songlin Yang et al. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. NeurIPS, 2024.
[2] Abbas Zeitoun et al. Hyperloop Transformers. COLM, 2026.
[3] Hayden Prairie et al. Parcae: Scaling Laws For Stable Looped Language Models. COLM, 2026.
[4] Sangmin Bae et al. Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation. NeurIPS, 2025.
[5] Guan Wang et al. Hierarchical Reasoning Model. arXiv:2506.21734, 2025.
[6] Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. CVPR, 2021.
Dear Reviewer vzef,
Thank you again for your time and thoughtful feedback. As the discussion period draws to a close, we would appreciate your feedback on whether our responses and additional analyses address your concerns.
If these clarifications and additional results resolve your principal concerns, we would be grateful if you could reconsider your assessment. If any questions remain, we would be happy to clarify them.
Best regards,
The authors
Response to Reviewer vzef (1/3)
Thank you for your careful review and for recognizing that the framework offers a useful decomposition of loop transitions and a route to new designs. We also appreciate your observation that input-map alignment is a non-trivial modification. Your questions help us clarify the mathematical meaning of the local objectives and the evidence supporting the framework’s design value.
R1: Thank you for highlighting the mathematical interpretation. The fixed-target convention defines a local optimization problem conditional on the current prediction. The shared blocks predict tl from the current projection and use it as the update’s fixed reference. For example, with the current map and target fixed, Ll(Y)=21∣HlinY−tl∣22 is a function of the candidate fast weight Y. The map and target are fixed for this partial derivative, recomputed at each loop, and differentiated through during training (Section 3.2; Appendix E). RQ3–RQ4 provide the detailed interpretation.
To provide an intuitive interpretation of these objectives, we visualize their geometry and behavior in the anonymous supplementary figures. Figure 1 contrasts the constant descent direction of the fixed-target negative inner product with delta-rule error correction [1]. Figures 2–3 show measured token-wise objective values across loops on the illustrated GSM8K example: later-loop values are generally lower, with Parcae often rebounding from loop 2 to loop 3.
Our optimization view targets effective finite-step updates under a fixed compute budget; it does not require many loops or convergence to a minimizer. HyperLoop [2] also uses three loops in its main experiments. In our matched-FLOP setting, with training tokens, width, and Prelude/Coda fixed, increasing the loop count requires a smaller shared stack, hence fewer distinct Transformer blocks and parameters. The loop count therefore reflects a trade-off between repeated computation and unique parameters.
Anonymous supplementary figures (https://anonymous.4open.science/r/loop_rebuttal-5F77/README.md)
R2: Thank you for the constructive suggestion to examine qualitative behavior and for encouraging us to clarify the scope of the empirical claims. Figures 4–5 illustrate differences following transition changes. Figure 4 shows generally lower local objective values after alignment. Figure 5 shows that OperLoop’s cross-token state similarity does not increase monotonically across loops, whereas Parcae has high similarity at loop 3. These observations concern the illustrated GSM8K example.
The statement in line 366 is intended to describe controlled comparisons with looped baselines. We use the same backbone, loop layout, training recipe, and matched training FLOPs; the architectural variable is the loop transition. OperLoop has the lowest PPL and highest LL and EM averages among the evaluated looped models at both scales (Table 3). At the 5B setting, LL Avg. is 59.77%, compared with 58.93% for HyperLoop and 59.14% for Parcae; EM Avg. is 25.58%, compared with 22.67% and 24.08%. We will explicitly state these comparison conditions in the claim.
We agree that the general benefits of looping should be distinguished from our transition-design contribution. Lines 371–372 describe the parameter–performance trade-off against non-looped baselines. OperLoop improves EM Avg. by 1.69 and 1.04 at the two scales with 0.56× and 0.48× the parameters, while PPL and LL remain worse (Table 3). We will clarify the role of this reference and emphasize the framework-derived transition changes and their controlled validation.
To address the distinction between viability and improvement, we highlight the performance gains in the controlled comparisons. Relative to HyperLoop, OperLoop improves EM Avg. by 2.91 and 0.89 at the two scales under matched training FLOPs (Table 3). The separately trained six-loop comparison provides additional evidence: EM Avg. is 31.90% versus 29.75% at matched training FLOPs (Table 4). The controlled alignment study (Table 2) and component ablations (Tables 5–6) further connect the performance gains to specific framework-derived changes.
Anonymous supplementary figures (https://anonymous.4open.science/r/loop_rebuttal-5F77/README.md)