acceptodds
Under review as a conference paper at ICLR 2027

Looped Transformers as Optimizers

37 views

Abstract

Looped Transformers provide a parameter-efficient approach to depth scaling by repeatedly applying shared Transformer blocks. Recent reasoning models have likewise highlighted the value of scaling test-time computation through longer computation trajectories. However, the principles for designing effective loop transitions remain poorly understood. We view the looped hidden state as a fast weight that is updated throughout the depth. We formulate loop transitions as local gradient-based updates, with recurrent blocks predicting implicit targets at each depth. Our framework derives loop transitions in closed form from a projection, a local objective and an optimizer update rule. Mapping representative loop transitions into this framework reveals mismatches between their transitions and projections. We first align the input maps of existing transitions. We then derive OperLoop, which combines explicit weight decay, adaptive step size and a delta objective. The aligned variants reduce training loss and improve average commonsense accuracy. OperLoop improves average generative performance over the compared looped and non-looped baselines under matched training FLOPs. These results support the framework's usefulness for loop design. We extend the analysis to additional loop models and outline a roadmap for future loop transition design.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Transformers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vzefOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

The paper recasts the recurrent transitions of looped Transformers as optimization steps, typically gradient steps with weight decay or modifications thereof, with respect to implicit local objectives. In this view, the recurrent state itself is the variable being optimized (a fast weight), and each transition is derived from a projection of the state, a local objective and an optimizer update rule.

By mapping existing loop transitions into this framework, the authors identify mismatches between the updates predicted by the optimization view and the actual looping mechanisms. They modify the transitions accordingly and successfully train the resulting looped Transformers, which match the downstream performance of the models they were derived from. This shows the viability of their theoretical view. Notably, the framework provides a way to design new and more powerful looped Transformers in the future by designing new objectives, projections and update rules.

Critical strengths

S1. The modifications derived from the optimization view preserve the downstream performance. Without knowledge of the underlying optimization objective, the input-map alignment would appear to be a rather non-trivial change to existing transitions. Nevertheless, the modified models match the downstream performance of their original counterparts, which shows that the theoretical view is viable in practice.

S2. The paper provides a new way to design looped transformers. Decomposing a loop transition into a projection, an objective and an update rule turns the design of a transition into the choice of these components, which can guide the development of future looped transformers.

Critical weaknesses

W1. The theoretical support for the optimization view is weak. Although the paper recasts looped transformers as optimizers, the theoretical evidence for this view is limited. The transitions are traced back to objectives that merely relate the target tl=Blocksθ(ol)t_l = \text{Blocks}_\theta(o_l) to the projection olo_l: either a negative inner product −⟨ol,tl⟩-\langle o_l, t_l \rangle or, for the proposed OperLoop, the squared distance 12∣ol−tl∣22\frac{1}{2}\\|o_l - t_l\\|_2^2 (the delta objective). Since tlt_l is itself computed from olo_l, but held fixed in the derivation/differentiation, these are not optimization objectives in the usual sense, which would be functions of olo_l alone. Moreover, the paper offers no intuitive interpretation of these objectives in the context of looped transformers. This has two unfortunate consequences: the framework provides little new understanding of these architectures, and it gives no intuition about which new objectives could, in turn, lead to better looped Transformers. The optimization perspective is further weakened by the fact that typically only three to six recurrent steps are performed. As the optimization view is the central contribution of the paper, this point is central to our assessment.

W2. The experiments show no qualitatively new behaviour, and some conclusions are overstated. The modified models match the performance of the original methods from which they were derived, but they do not change their behaviour in an interesting way. The statement in line 366 that the results "support the benefits of its transition design for both language modeling and generation" overstates these benefits. Lines 371 and 372 conclude that looped Transformers have benefits over non-looped baselines; this likely has already been shown by the modified works themselves and is not the main claim of the paper. Hence, the experiments demonstrate the viability of the framework, but not a clear advantage resulting from it.

Questions for authors

Q1. I am not familiar with the literature on looped transformers and I am not aware of its exact current state, so we ask this question for clarification. Are the four methods in Table 1 (Ouro, Huginn, Parcae and HyperLoop) the most important examples of looped transformers? There seem to be other relevant works that are not discussed, for example Mixture-of-Recursions (Bae et al., NeurIPS 2025) or the Hierarchical Reasoning Model (Wang et al., 2025), which updates two coupled recurrent states. What is the reason for selecting these specific methods and not others?

Q2. In Section 3.1 (Vanilla State Transition), the recurrent state is yl∈Rdy_l \in \mathbb{R}^d, i.e. dd-dimensional, whereas the hidden state in Section 2 is H∈RN×dH \in \mathbb{R}^{N \times d}. Could you clarify this change? Moreover, the text currently reads as if the random initialization y0∼N(μ,σ)y_0 \sim \mathcal{N}(\mu, \sigma) could be combined with a zero bias b=0b = 0, which would make the prelude output ee irrelevant. Is this combination intended?

Q3. What is the interpretation of the objective Ll=−⟨ol,tl⟩\mathcal{L}_l = -\langle o_l, t_l \rangle in line 218, and of the differentiation with a fixed target tlt_l?

Q4. Similarly, what is the interpretation of the delta objective Ll=12∣ol−tl∣22\mathcal{L}_l = \frac{1}{2}\\|o_l - t_l\\|_2^2 in Section 4?

4 comments
AuthorsOpenReview ↗

Response to Reviewer vzef (1/3)

Thank you for your careful review and for recognizing that the framework offers a useful decomposition of loop transitions and a route to new designs. We also appreciate your observation that input-map alignment is a non-trivial modification. Your questions help us clarify the mathematical meaning of the local objectives and the evidence supporting the framework’s design value.


W1: The theoretical support for the optimization view is weak.

R1: Thank you for highlighting the mathematical interpretation. The fixed-target convention defines a local optimization problem conditional on the current prediction. The shared blocks predict tl\boldsymbol t_l from the current projection and use it as the update’s fixed reference. For example, with the current map and target fixed, Ll(Y)=12∣HlinY−tl∣22\mathcal L_l(\mathbf Y)=\frac12\\|\mathbf H_l^{\mathrm{in}}\mathbf Y-\boldsymbol t_l\\|_2^2 is a function of the candidate fast weight Y\mathbf Y. The map and target are fixed for this partial derivative, recomputed at each loop, and differentiated through during training (Section 3.2; Appendix E). RQ3–RQ4 provide the detailed interpretation.

To provide an intuitive interpretation of these objectives, we visualize their geometry and behavior in the anonymous supplementary figures. Figure 1 contrasts the constant descent direction of the fixed-target negative inner product with delta-rule error correction [1]. Figures 2–3 show measured token-wise objective values across loops on the illustrated GSM8K example: later-loop values are generally lower, with Parcae often rebounding from loop 2 to loop 3.

Our optimization view targets effective finite-step updates under a fixed compute budget; it does not require many loops or convergence to a minimizer. HyperLoop [2] also uses three loops in its main experiments. In our matched-FLOP setting, with training tokens, width, and Prelude/Coda fixed, increasing the loop count requires a smaller shared stack, hence fewer distinct Transformer blocks and parameters. The loop count therefore reflects a trade-off between repeated computation and unique parameters.

Anonymous supplementary figures (https://anonymous.4open.science/r/loop_rebuttal-5F77/README.md)


W2: The experiments show no qualitatively new behaviour, and some conclusions are overstated.

R2: Thank you for the constructive suggestion to examine qualitative behavior and for encouraging us to clarify the scope of the empirical claims. Figures 4–5 illustrate differences following transition changes. Figure 4 shows generally lower local objective values after alignment. Figure 5 shows that OperLoop’s cross-token state similarity does not increase monotonically across loops, whereas Parcae has high similarity at loop 3. These observations concern the illustrated GSM8K example.

The statement in line 366 is intended to describe controlled comparisons with looped baselines. We use the same backbone, loop layout, training recipe, and matched training FLOPs; the architectural variable is the loop transition. OperLoop has the lowest PPL and highest LL and EM averages among the evaluated looped models at both scales (Table 3). At the 5B setting, LL Avg. is 59.77%, compared with 58.93% for HyperLoop and 59.14% for Parcae; EM Avg. is 25.58%, compared with 22.67% and 24.08%. We will explicitly state these comparison conditions in the claim.

We agree that the general benefits of looping should be distinguished from our transition-design contribution. Lines 371–372 describe the parameter–performance trade-off against non-looped baselines. OperLoop improves EM Avg. by 1.69 and 1.04 at the two scales with 0.56× and 0.48× the parameters, while PPL and LL remain worse (Table 3). We will clarify the role of this reference and emphasize the framework-derived transition changes and their controlled validation.

To address the distinction between viability and improvement, we highlight the performance gains in the controlled comparisons. Relative to HyperLoop, OperLoop improves EM Avg. by 2.91 and 0.89 at the two scales under matched training FLOPs (Table 3). The separately trained six-loop comparison provides additional evidence: EM Avg. is 31.90% versus 29.75% at matched training FLOPs (Table 4). The controlled alignment study (Table 2) and component ablations (Tables 5–6) further connect the performance gains to specific framework-derived changes.

Anonymous supplementary figures (https://anonymous.4open.science/r/loop_rebuttal-5F77/README.md)

AuthorsOpenReview ↗

Response to Reviewer vzef (2/3)

Q1: I am not familiar with the literature on looped transformers and I am not aware of its exact current state, so we ask this question for clarification. Are the four methods in Table 1 (Ouro, Huginn, Parcae and HyperLoop) the most important examples of looped transformers? There seem to be other relevant works that are not discussed, for example Mixture-of-Recursions (Bae et al., NeurIPS 2025) or the Hierarchical Reasoning Model (Wang et al., 2025), which updates two coupled recurrent states. What is the reason for selecting these specific methods and not others?

RQ1: Thank you for suggesting these relevant works. Our focus is how to design and improve loop transitions within looped Transformers. We therefore select representative works on loop-transition design for the comparison in Table 1: Huginn, Parcae [3], and HyperLoop [2]. Under our abstraction, they introduce input injection, structured state transformation, and an expanded multi-stream state, respectively, while Ouro serves as the vanilla-loop reference.

Mixture-of-Recursions [4] and HRM [5] primarily emphasize complementary, higher-level architectural choices: Mixture-of-Recursions combines parameter sharing, adaptive recursive depth, and KV caching; HRM organizes two coupled recurrent modules at different time scales. We will add a discussion of Mixture-of-Recursions and HRM to the related work.


Q2: In Section 3.1 (Vanilla State Transition), the recurrent state is yl∈Rdy_l\in\mathbb R^d, i.e. dd -dimensional, whereas the hidden state in Section 2 is H∈RN×dH\in\mathbb R^{N\times d}. Could you clarify this change? Moreover, the text currently reads as if the random initialization y0∼N(μ,σ)y_0\sim\mathcal N(\mu,\sigma) could be combined with a zero bias b=0b=0, which would make the prelude output ee irrelevant. Is this combination intended?

RQ2: Thank you for identifying these ambiguities. We will clarify the notation and explicitly state the per-token convention. We omit the token index to make the loop-state dimensions easier to compare. H∈RN×d\mathbf{H}\in\mathbb{R}^{N\times d} collects the hidden states of NN tokens, whereas the recurrent (loop) state yl∈Rd\boldsymbol{y}_l\in\mathbb{R}^d describes the state of one token. The expanded recurrent (loop) state Yl∈Rr×d\mathbf{Y}_l\in\mathbb{R}^{r\times d} likewise describes one token, with rr parallel streams.

This combination is not intended. We should have distinguished the architecture-specific initialization/injection pairs more clearly. For Ouro, the state is initialized from the Prelude output, y0=e\boldsymbol{y}_0=\boldsymbol{e}, with no input injection (b=0\boldsymbol{b}=0). Huginn uses random initialization, y0∼N(μ,σ)\boldsymbol{y}_0\sim\mathcal{N}(\mu,\sigma), together with input injection, b=e\boldsymbol{b}=\boldsymbol{e}. Both configurations retain a path from the input. We will specify these initialization/injection pairs in Section 3.1.

AuthorsOpenReview ↗

Response to Reviewer vzef (3/3)

Q3: What is the interpretation of the objective Ll=−⟨ol,tl⟩\mathcal L_l=-\langle o_l,t_l\rangle in line 218, and of the differentiation with a fixed target tlt_l?

Q4: Similarly, what is the interpretation of the delta objective Ll=12∣ol−tl∣22\mathcal L_l=\frac12\\|o_l-t_l\\|_2^2 in Section 4?

RQ3–RQ4: Thank you for asking about the objectives and the fixed-target derivative. Both loop transitions and optimizer update rules describe iterative state updates. Our framework connects them by treating the looped state as a fast weight and identifying an implicit target and a compatible local objective.

  1. Differentiation with a fixed target. The implicit target is the shared blocks’ prediction, used as a reference for the current local state update. Holding it fixed is a convention for deriving the local loop transition algebraically; end-to-end training still backpropagates through target generation without detaching gradients. A related perspective appears in self-supervised learning: Kaiming et al [6] interpret SimSiam through an approximate alternating-optimization hypothesis, where a network-generated target is held fixed within an optimization subproblem. This illustrates how a predicted target can serve as a fixed reference for a particular update.

  2. Negative inner product. For the negative inner product objective, the write term is driven by the predicted target, without an explicit correction for the current projection. It rewards a larger inner product with the target. Its projection-space gradient is −tl-\boldsymbol t_l, giving a write proportional to (Hlin)⊤tl(\mathbf H_l^{\mathrm{in}})^{\top}\boldsymbol t_l. For a fixed nonzero target, the objective is unbounded below in projection space.

  3. Delta objective. The delta objective adds an explicit correction based on the difference between the predicted target and the current projection. It penalizes projection–target mismatch, giving the negative fast-weight gradient (Hlin)⊤(tl−ol)(\mathbf H_l^{\mathrm{in}})^{\top}(\boldsymbol t_l-\boldsymbol o_l). The objective is bounded below by zero and minimized when the projection matches the target.

Moreover, we analyze and visualize the local objectives in the anonymous supplementary material to make their behavior more intuitive. Supplementary Figure 1 illustrates their geometry, while Supplementary Figures 2–3 visualize their values across loop iterations.

Anonymous supplementary figures (https://anonymous.4open.science/r/loop_rebuttal-5F77/README.md)


Thank you again for your careful review. We hope the local-gradient formulation, diagnostics, and controlled comparisons address your concerns about the objectives and their empirical value. If so, we would appreciate it if you could consider raising your rating in light of this evidence. Please let us know if a substantive concern remains.

The authors


References

[1] Songlin Yang et al. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. NeurIPS, 2024.

[2] Abbas Zeitoun et al. Hyperloop Transformers. COLM, 2026.

[3] Hayden Prairie et al. Parcae: Scaling Laws For Stable Looped Language Models. COLM, 2026.

[4] Sangmin Bae et al. Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation. NeurIPS, 2025.

[5] Guan Wang et al. Hierarchical Reasoning Model. arXiv:2506.21734, 2025.

[6] Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. CVPR, 2021.

Dear Reviewer vzef,

Thank you again for your time and thoughtful feedback. As the discussion period draws to a close, we would appreciate your feedback on whether our responses and additional analyses address your concerns.

If these clarifications and additional results resolve your principal concerns, we would be grateful if you could reconsider your assessment. If any questions remain, we would be happy to clarify them.

Best regards,

The authors

Reviewer_yTJ8OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes viewing the looped hidden states as fast weights and uses this lens to describe popular, existing looped designs. This sometimes leads to slightly different terms, which are empirically validated to improve performance. Additionally, this view inspires a novel looped formulation, OperLoop, which further reduces training loss and improves downstream metrics.

Critical strengths
  • Optimization View: I like the motivation of using a fast weight perspective and the formulation of existing looped designs in this framework. The derivations and explanations in Section 3 are concise and clear.

  • Empirical Validation: I really appreciate the extensive, large-scale, and honest empirical validation. Regarding size, this is the largest matched comparison of different looped designs (up to 10B MoE and 500B tokens) that I am aware of, and it includes, in my opinion, the subset of the most relevant looped designs for the empirical comparison. Additionally, benchmarks are organized in a sensible way and compared to the appropriate baselines (inference FLOPs matched to the baseline) so that improvements or gaps are both convincing and interesting. I especially like the distinction between NLL/PPL, LL, and EM measured tasks, and the validation that looped designs improve upon the baseline, especially in generation tasks (Exact Match).

Critical weaknesses

None

Questions for authors

I do not have any major concerns, but I do not fully understand the motivation behind the proposed design OperLoop (Section 4). Concretely, I don't know whether the previous section actually inspired this design, and if so, in what way.

  • Could you please explain in either more detail or make the connection clearer, how this design is motivated by the previous section/framework, its connection to Delta Linear Attention, and if there is any motivation for the parameterization of equations 13-15?

I need to mention that I am not deeply familiar with the (Gated) Delta Linear Attention literature, but I think that, compared to the previous section (Section 3), this could be made clearer, and I would appreciate further clarification. Overall, I have no additional questions for the empirical validation. My questions are mostly about the theoretical/framework motivation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer yTJ8 (1/2)

Thank you for your careful and encouraging review. We appreciate your recognition of the fast-weight perspective and the effort behind the large-scale matched comparisons. Your attention to the distinction between PPL, LL, and EM evaluation, including both improvements and remaining gaps, is especially valuable to us. Your questions identify the connection between the framework and OperLoop as the point needing further explanation. We address that connection below.


Q1: I do not have any major concerns, but I do not fully understand the motivation behind the proposed design OperLoop (Section 4). Concretely, I don't know whether the previous section actually inspired this design, and if so, in what way.

RQ1: Thank you for pinpointing the connection between the framework and OperLoop. Sections 3.2–3.3 first derive input-map alignment for Parcae and HyperLoop. Section 5.1 shows that alignment improves performance, providing empirical support for the framework. We then use the framework to derive new loop transitions from optimization-inspired choices of the local objective and optimizer update rule. Section 4 applies this approach to derive OperLoop through two design choices:

  1. Local objective. Because the negative inner product objective is unbounded below for a fixed nonzero target, we adopt the delta objective [1, 2]. It is bounded below by zero and penalizes projection–target mismatch. Its gradient yields the error-correction update in Eq. (12). Table 5 tests this objective choice.

  2. Step size. The mapped Parcae and HyperLoop transitions use unit step sizes (Section 3.3). Viewing loops as local optimization steps motivates exploring learning-rate scheduling. We introduce a monotonically decreasing adaptive step size across loops (Eq. 13), evaluated in Table 6.

We will clarify how the framework motivates these objective and step-size choices at the start of Section 4 to make the design rationale easier to follow.


Q2: Could you please explain in either more detail or make the connection clearer, how this design is motivated by the previous section/framework, its connection to Delta Linear Attention, and if there is any motivation for the parameterization of equations 13-15?

RQ2: Thank you for asking us to distinguish the design rationale from the parameterization choices in Eqs. (13)–(15). RQ1 explains the framework motivation; RQ3 details the connection to Delta Linear Attention. The parameterizations of OperLoop are adapted from HyperLoop [3]; our framework determines how these quantities enter the transition.

  1. Eq. (13): Step size. RMS normalization of the flattened state, a learned linear projection, and a sigmoid produce the scalar factor γl\gamma_l. We introduce ηl=γlηl−1\eta_l=\gamma_l\eta_{l-1}. Table 6 evaluates this schedule.

  2. Eq. (14): Decay. We use the same diagonal sigmoid parameterization as HyperLoop’s Hlres\mathbf H_l^{\mathrm{res}} to parameterize the decay operator Λl\mathbf\Lambda_l.

  3. Eq. (15): Input map. We use the same input-map parameterization as HyperLoop.

The pseudocode in Appendix B shows how the two methods use similar parameterizations but different transition updates.

AuthorsOpenReview ↗

Response to Reviewer yTJ8 (2/2)

Q3: I need to mention that I am not deeply familiar with the (Gated) Delta Linear Attention literature, but I think that, compared to the previous section (Section 3), this could be made clearer, and I would appreciate further clarification.

RQ3: Thank you for this constructive suggestion. Section 3 expresses loop transitions as fast-weight updates, connecting them to Delta Linear Attention and motivating the use of its delta rule. Delta Linear Attention updates memory across tokens, while OperLoop updates its state across loop depth.

The delta rule corrects readout–target mismatch [1, 2]. In OperLoop, the input map forms the projection, and the shared blocks predict the implicit target. A squared-error local objective measures this mismatch; its gradient yields the error-correction update in Eq. (12). Delta Linear Attention instead obtains its target from a value projection. In gated variants, a decay gate additionally controls memory retention.

For this local derivative, read maps and targets are held fixed; training remains end to end (Section 3.2 and Appendix E). We will introduce this explanation before the equations in Section 4. As shown in Table R1, the two methods share the error-correction structure despite their different recurrence axes and target sources.

Table R1. Correspondence between the basic delta rule and OperLoop.

ComponentDelta Linear Attention (basic delta rule)OperLoop (Eq. 12)
Recurrencet−1→tt-1\rightarrow tl→l+1l\rightarrow l+1
StateSt−1∈Rdk×dv\mathbf{S}_{t-1}\in\mathbb{R}^{d_k\times d_v}Yl∈Rr×d\mathbf{Y}_l\in\mathbb{R}^{r\times d}
Read mapkt∈R1×dk\boldsymbol{k}_t\in\mathbb{R}^{1\times d_k}Hlin∈R1×r\mathbf{H}_l^{\mathrm{in}}\in\mathbb{R}^{1\times r}
Local predictionv^t=ktSt−1\widehat{\boldsymbol{v}}_t=\boldsymbol{k}_t\mathbf{S}_{t-1}ol=HlinYl\boldsymbol{o}_l=\mathbf{H}_l^{\mathrm{in}}\mathbf{Y}_l
Targetvt\boldsymbol{v}_ttl=Blocksθ(ol)\boldsymbol{t}_l=\mathrm{Blocks}_{\theta}(\boldsymbol{o}_l)
Local prediction objectiveℓt(St−1)=12∥ktSt−1−vt∥22\ell_t(\mathbf{S}_{t-1})=\frac12\lVert\boldsymbol{k}_t\mathbf{S}_{t-1}-\boldsymbol{v}_t\rVert_2^2ℓl(Yl)=12∥HlinYl−tl∥22\ell_l(\mathbf{Y}_l)=\frac12\lVert\mathbf{H}_l^{\mathrm{in}}\mathbf{Y}_l-\boldsymbol{t}_l\rVert_2^2
Correctionkt⊤(vt−v^t)\boldsymbol{k}_t^{\top}(\boldsymbol{v}_t-\widehat{\boldsymbol{v}}_t)(Hlin)⊤(tl−ol)(\mathbf{H}_l^{\mathrm{in}})^{\top}(\boldsymbol{t}_l-\boldsymbol{o}_l)
Step sizeβt\beta_tηl=γlηl−1\eta_l=\gamma_l\eta_{l-1}
Original recurrent dynamicsSt=St−1+βtkt⊤(vt−ktSt−1)\mathbf{S}_t=\mathbf{S}_{t-1}+\beta_t\boldsymbol{k}_t^{\top}(\boldsymbol{v}_t-\boldsymbol{k}_t\mathbf{S}_{t-1})Yl+1=(I−ηlΛl)Yl+ηl(Hlin)⊤(tl−HlinYl)\mathbf{Y}_{l+1}=(\mathbf{I}-\eta_l\mathbf{\Lambda}_l)\mathbf{Y}_l+\eta_l(\mathbf{H}_l^{\mathrm{in}})^{\top}(\boldsymbol{t}_l-\mathbf{H}_l^{\mathrm{in}}\mathbf{Y}_l)

Q4: Overall, I have no additional questions for the empirical validation. My questions are mostly about the theoretical/framework motivation.

RQ4: Thank you for your positive assessment of the empirical validation. Your specific questions help us focus the revision on the connection between Section 3 and OperLoop. Section 4 will first explain the objective and step-size choices, then distinguish the parameterization adapted from HyperLoop from the update derived through our framework.


Thank you again for the care you have taken with both the framework and the experiments. Your questions help us make the path from the optimization view to OperLoop easier to follow. We hope these explanations resolve your questions about the design motivation. If so, we would appreciate it if you could consider a more positive rating in light of the clarified contribution.

The authors


References

[1] Songlin Yang et al. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. NeurIPS, 2024.

[2] Kimi Team. Kimi Linear: An Expressive, Efficient Attention Architecture. arXiv:2510.26692, 2025.

[3] Abbas Zeitoun et al. Hyperloop Transformers. COLM, 2026.

Thank you for the clear and detailed response.

Your explanation resolves my questions regarding Section 4. Separating the framework-motivated design choices (the bounded delta objective and the monotonically decreasing step-size schedule) from the parameterizations adapted from HyperLoop in equations 13-15 makes the rationale behind OperLoop much clearer. The comparison with Delta Linear Attention and Table R1 are also very helpful. Please make sure to include these clarifications at the start of Section 4 in the revised manuscript, as they make the connection to Section 3 much easier to follow.

With these points addressed, I am updating my score from 3 to 4 and recommend acceptance, as the paper offers both an interesting optimization lens on looped Transformers and a strong, large-scale empirical contribution.

Thank you for your thoughtful follow-up and recommendation for acceptance. We are glad that our response resolved your questions. Your feedback has been very valuable in helping us clarify the motivation behind OperLoop, and your positive assessment is truly encouraging.

We will incorporate these clarifications at the start of Section 4, as suggested. In particular, we will make the connection to Section 3 explicit, distinguish the framework-motivated design choices from the parameterizations adapted from HyperLoop, and include the comparison with Delta Linear Attention and Table R1. Thank you again for your careful review and constructive suggestions.


With sincere thanks and respect,

The authors

Reviewer_5566OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a framework for understanding state transitions in Looped Transformers. The authors interpret recurrent hidden states as fast weights and formulate their updates using a projection, a local objective, and an optimizer update rule. They claim mismatches between the read and write mappings in existing loop architectures and propose input map alignment. They further introduce OperLoop, which extends HyperLoop with a delta objective, a monotonically decreasing adaptive step size, and a state dependent decay to control hidden state updates across recurrent steps. Experiments compare the proposed methods with existing looped and standard Transformers under matched training computation, evaluating language modeling, commonsense reasoning, mathematical reasoning, and code generation.

Critical strengths

The paper tries to have a common framework to explain existing loop transformer architectures.

Critical weaknesses

The paper is poorly written and lacks clarity. The research questions and overall organization are presented in an unusual and confusing manner. Despite my efforts to understand the paper’s core contributions, several important questions remain unresolved, as outlined below.

To me the claimed contribution of this paper is to reinterpret loop transitions as gradient based optimization. However, vanilla loop transformers alreay perform iterative hidden state refinement through residual connections and are trained end to end. The proposed framework interprets the forward state update of a looped Transformer as a gradient descent step. By treating the hidden state as a fast weight and each loop transition as one gradient step on a local objective, it requires the write map to be the transpose of the read map, which for HyperLoop means H_post must equal H_pre. However, this requirement follows from an assumption rather than from empirical evidence. The paper does not measure whether looped Transformers actually behave like gradient descent, for example by tracking whether the local objective decreases across loop iterations or whether the actual state updates align with the predicted gradient direction. Moreover, the paper does not show that enforcing this constraint brings benefits beyond the regularization effect of reducing the number of parameters. The paper doesn’t identify failure of existing loop transformers. For example, there is no evidence that HyperLoop or Parcae suffer from instability issues, so the proposed fixes are not tied to observed problem in the first place?

I am not sure what additional values this framework adds to the state of the loop transformer research. The framework is too flexible to be falsifiable, because any update that is affine in the state with a block output can be mapped into it. It doesn’t predict what designs will work better if it just describes existing designs.

The results should have variance estimates or confidence interval because many improvements over the baselines are within one point.

Questions for authors

The paper argues that read and write mappings should satisfy a transpose relationship but there is no specific reason. I still don’t understand why gradient consistency should be preferred over independently learned read and write mappings?

The paper only evaluates the MoE backbone under the middle loop layout. How would the method perform with fully looped or dense backbones? Can it generalize to these alternative architectures?

The ablations seem incomplete by running only at 5B with three loops, and there is no ablation of read and write tying within OperLoop, different learned decay, and no test of whether the rank one delta term matters beyond simple weight tying.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5566 (1/4)

Thank you for your detailed review and for recognizing our effort to develop a common framework for loop transitions. We clarify the design rationale, supporting evidence, and remaining limitations below.


W1: The paper is poorly written and lacks clarity. The research questions and overall organization are presented in an unusual and confusing manner. Despite my efforts to understand the paper’s core contributions, several important questions remain unresolved, as outlined below.

R1: Thank you for this feedback. Reviewer yTJ8 describes the derivations and explanations in Section 3 as “concise and clear,” while Reviewer vzef rates the presentation as “Clear and easy to follow.” We nevertheless take your difficulty in following the paper seriously and should make the connection between analysis, design, and experiments more explicit.

We propose a unified optimization framework for analyzing and designing loop transitions. Section 3 develops the unified optimization framework and derives input-map alignment. Section 5.1 shows that alignment improves performance. Section 4 uses the optimization view to derive OperLoop by changing the local objective and step-size rule. Sections 5.2–5.3 demonstrate its performance benefits through downstream comparisons and component ablations. Section 6 extends the analysis to other loop transitions and future designs.

We will add this roadmap to the Introduction to make the paper easier to follow.


W2: To me the claimed contribution of this paper is to reinterpret loop transitions as gradient based optimization. However, vanilla loop transformers alreay perform iterative hidden state refinement through residual connections and are trained end to end.

R2: Thank you for raising this point. Residual connections update hidden states within the shared blocks; the loop transition determines how the state is read, retained, and updated between applications of those shared blocks. The attention and FFN residual connections defined in Section 2 remain present in our models. Our framework studies the loop transition: for example, Parcae transforms the block input, while HyperLoop reads from and writes to an expanded multi-stream state.

Iterative refinement and end-to-end training alone do not determine which loop transition works better or how to improve it. This motivates our framework in the Introduction. R3 and RQ1 show the derivation and design rationale; R7 summarizes the performance improvements.


W3: The proposed framework interprets the forward state update of a looped Transformer as a gradient descent step. By treating the hidden state as a fast weight and each loop transition as one gradient step on a local objective, it requires the write map to be the transpose of the read map, which for HyperLoop means H_post must equal H_pre. However, this requirement follows from an assumption rather than from empirical evidence.

R3: Thank you for this question. The framework leads to an alignment condition, and the submitted manuscript provides empirical evidence for its performance benefits. Section 3.2 formulates the optimization interpretation under the specified local differentiation assumptions. Section 3.3 applies the resulting gradient form to existing loop transitions and derives input-map alignment. Section 5.1 then provides empirical evidence: alignment improves training loss, perplexity, and average commonsense accuracy for both Parcae [1] and HyperLoop [2] (Figure 2 and Table 2; Parcae: LAMBADA PPL falls from 7.71 to 7.41, commonsense Avg. rises from 59.14% to 59.29%; HyperLoop: LAMBADA PPL falls from 7.68 to 7.35, commonsense Avg. rises from 58.93% to 59.47%). These results support the framework-derived design hypothesis in the tested settings.


W4: The paper does not measure whether looped Transformers actually behave like gradient descent, for example by tracking whether the local objective decreases across loop iterations or whether the actual state updates align with the predicted gradient direction.

R4: Thank you for suggesting this diagnostic. We track token-wise local objectives across loop iterations and observe overall reductions in all three models on the same GSM8K example (Supplementary Figure 3). HyperLoop and OperLoop show generally decreasing objective values; Parcae’s values decrease relative to loop 1 but often rebound from loop 2 to loop 3. These measurements illustrate different local descent behaviors across loop transition mechanisms.

Supplementary Figure 3 reports the negative inner product objectives for Parcae and HyperLoop and the delta objective for OperLoop. Supplementary Figure 2 maps these measured values onto two-dimensional local objective loss landscape contours. We will add these figures and their interpretation to a diagnostic subsection of the revised appendix.

Anonymous supplementary figures (https://anonymous.4open.science/r/loop_rebuttal-5F77)

AuthorsOpenReview ↗

Response to Reviewer 5566 (2/4)

W5: Moreover, the paper does not show that enforcing this constraint brings benefits beyond the regularization effect of reducing the number of parameters.

R5: Thank you for raising this alternative explanation. The gains cannot be explained solely by reducing the number of parameters. Input-map alignment is derived from the optimization framework (Section 3.3). Table 2 of the submitted manuscript reports the following improvements:

  1. Parcae after alignment: LAMBADA PPL decreases from 7.71 to 7.41, and average commonsense accuracy increases from 59.14% to 59.29%, with unchanged parameter count.

  2. HyperLoop after alignment: LAMBADA PPL decreases from 7.68 to 7.35, and average commonsense accuracy increases from 58.93% to 59.47%, with fewer parameters.

Parcae uses the transpose of its existing input map for writing; HyperLoop removes the independent write map. Thus, parameter reduction alone cannot explain both improvements, although structural regularization may contribute. Supplementary Figure 4 on the anonymous page also shows generally lower token-wise local objective values after alignment on the illustrated input.

Anonymous supplementary figures (https://anonymous.4open.science/r/loop_rebuttal-5F77)


W6: The paper doesn’t identify failure of existing loop transformers. For example, there is no evidence that HyperLoop or Parcae suffer from instability issues, so the proposed fixes are not tied to observed problem in the first place?

R6: Thank you for raising this concern. We test whether transition modifications derived from the optimization framework improve performance. Section 3.3 identifies input-map mismatches in Parcae and HyperLoop relative to the specified local gradient form and derives aligned transitions. This mismatch does not imply instability. Section 5.1 shows that alignment lowers training loss and improves perplexity and average commonsense accuracy in the tested settings.


W7: I am not sure what additional values this framework adds to the state of the loop transformer research. The framework is too flexible to be falsifiable, because any update that is affine in the state with a block output can be mapped into it. It doesn’t predict what designs will work better if it just describes existing designs.

R7: Thank you for this question. We propose a framework that connects projections, local objectives, and optimizer update rules to analyze and derive loop transitions. The experiments test whether the resulting modifications improve performance.

  1. Sections 3.2–3.3 derive input-map alignment under the optimization framework. Section 5.1 demonstrates improvements in training loss, PPL, and average commonsense accuracy (Figure 2 and Table 2).

  2. Section 4 derives OperLoop with the delta objective in place of the negative inner product objective. Its monotonically decreasing adaptive step size is inspired by learning-rate scheduling. Performance gains are demonstrated in both main comparisons and component ablations (Sections 5.2–5.3, Tables 3–6).

  3. Section 6 extends the framework to additional loop transitions and identifies further designs informed by fast-weight learning and optimization.

The framework thus goes beyond describing existing designs: it guides improvements to existing transitions and the design of new loop transitions, with performance gains demonstrated through controlled comparisons in the tested settings.


W8: The results should have variance estimates or confidence interval because many improvements over the baselines are within one point.

R8: Thank you for highlighting uncertainty estimates. The magnitude of the EM Avg. gains depends on the baseline, with most exceeding one point. Against non-looped baselines, OperLoop gains 1.69 and 1.04 at the two scales while using 0.56× and 0.48× the parameters. Against looped baselines with similar parameter budgets, gains range from 0.89 to 2.91 (Table 3). In the separately trained six-loop setting, OperLoop achieves 31.90 versus HyperLoop’s 29.75 at matched training FLOPs (Table 4).

All models use the same fixed task configurations and evaluation protocol in lm-eval-harness. Training also uses a fixed seed for controlled comparisons. Adding two training seeds for four architectures at both scales would cost approximately 36,864 H800 GPU-hours, assuming 2,304 GPU-hours per run (32 H800 GPUs for three days), beyond our rebuttal budget.

AuthorsOpenReview ↗

Response to Reviewer 5566 (3/4)

Q1: The paper argues that read and write mappings should satisfy a transpose relationship but there is no specific reason. I still don’t understand why gradient consistency should be preferred over independently learned read and write mappings?

RQ1: Thank you for this question. The transpose relationship follows from the chain rule, and empirical evaluations show lower training loss and improved downstream performance after input-map alignment.

As in Section 3.2 (Eq. 10), let the fast weight be Yl∈Rd1×d2\mathbf Y_l\in\mathbb R^{d_1\times d_2} and the input map be Hlin∈Rm×d1\mathbf H_l^{\mathrm{in}}\in\mathbb R^{m\times d_1}. The projection is ol=HlinYl+bl∈Rm×d2\boldsymbol o_l=\mathbf H_l^{\mathrm{in}}\mathbf Y_l+\boldsymbol b_l\in\mathbb R^{m\times d_2}, with L=f(ol)\mathcal L=f(\boldsymbol o_l). For the local derivative, hold the input map, bias, and implicit target fixed. The chain rule gives:

=\underbrace{(\mathbf H_l^{\mathrm{in}})^\top}_{\in\mathbb R^{d_1\times m}}\underbrace{\frac{\partial\mathcal L}{\partial\boldsymbol o_l}}_{\in\mathbb R^{m\times d_2}} =(\mathbf H_l^{\mathrm{in}})^\top f^{\prime}(\boldsymbol o_l) \in\mathbb R^{d_1\times d_2}.$$ The gradient matches the shape of the fast weight $\mathbf Y_l$. Section 5.1 shows that alignment lowers training loss and perplexity and improves average commonsense accuracy, including Parcae at unchanged parameter count (R5). Within OperLoop, removing tying while retaining the delta term lowers Overall Avg12 from 42.68% to 41.96% (RQ3). Independent mappings remain valid alternatives. --- > **Q2:** The paper only evaluates the MoE backbone under the middle loop layout. > How would the method perform with fully looped or dense backbones? > Can it generalize to these alternative architectures? **RQ2:** Thank you for raising this question. **Additional experiments show performance gains for OperLoop in a fully looped dense nanoGPT setting.** We apply the entire stack of shared blocks twice, giving an effective depth of 24. We trained Vanilla Loop, Parcae, HyperLoop, and OperLoop at the same width and effective depth, using the same training recipe, 50B tokens, a 4K context length, and the same evaluation protocol. Within this fully looped dense nanoGPT setting, the loop transition is the architectural variable being compared. As shown in Table R1, OperLoop achieves the lowest perplexity on both datasets. Compared with HyperLoop, WikiText word PPL decreases from 23.73 to 23.30 (1.81%), and LAMBADA PPL decreases from 27.28 to 26.11 (4.30%). These results support OperLoop’s benefits beyond the original middle-loop MoE setting. We will add the full comparison and training settings to the appendix of the revised manuscript. *Table R1. Fully looped dense nanoGPT models trained on 50B tokens.* | Architecture | WikiText word PPL ↓ | LAMBADA PPL ↓ | | --- | ---: | ---: | | Vanilla Loop2 | 24.12 | 27.98 | | Parcae Loop2 | 23.66 | 27.45 | | HyperLoop2 | 23.73 | 27.28 | | OperLoop2 (Ours) | **23.30** | **26.11**

Comments on the paper

2 comments
Area_Chair_yv1LOpenReview ↗

Few hours left to clear your doubts

Dear reviewers vzef and 5566,

Please, consider the authors' response and see if you still have questions left unresolved. We have a few hours to resolve minor issues and explain what is still left out.

your AC

AuthorsOpenReview ↗

Global Response

Dear Area Chair and Reviewers,

Thank you for your careful reviews and constructive feedback. Reviewer yTJ8 highlights the fast-weight perspective and large-scale matched comparisons. Reviewer vzef recognizes input-map alignment as a non-trivial modification and the decomposition into a projection, an objective, and an update rule as a new way to design looped Transformers.

We have provided point-by-point responses, additional experiments, and supplementary analyses. Reviewer yTJ8 confirmed that our explanations resolved the design-motivation questions and raised the rating from 3 to 4, recommending acceptance. As the author discussion closes, we summarize the evidence addressing the remaining concerns.

1. Framework and design contribution. We treat the recurrent state as a fast weight and derive loop transitions from a projection, a local objective, and an optimizer update rule. We study how the looped state is read, retained, and updated between applications of shared blocks, which retain their internal residual connections. This decomposition guides input-map alignment and OperLoop, with performance benefits tested through controlled comparisons.

2. Local objective interpretation. The shared blocks predict an implicit target as a reference for the current local state update. The input map and target are held fixed for the local derivative; training remains end to end without detaching gradients. The negative inner product objective gives a target-driven write; the delta objective penalizes projection–target mismatch and yields an error-correction update. Supplementary figures illustrate objective geometry and token-wise local objective values across loops.

Anonymous supplementary figures (https://anonymous.4open.science/r/loop_rebuttal-5F77/README.md)

3. Performance gains from framework-derived changes. Alignment lowers training loss and improves average commonsense accuracy; LAMBADA PPL decreases from 7.71 to 7.41 for Parcae and from 7.68 to 7.35 for HyperLoop (Section 5.1, Table 2). Parcae’s parameter count is unchanged, excluding parameter reduction as the sole explanation. OperLoop improves EM Avg. over HyperLoop by 2.91 and 0.89 at the two scales under matched training FLOPs (Table 3). Tables 5–6 separately test the rank-one delta term and step-size schedule. These results support the framework’s design value beyond establishing viability.

4. Additional generalization and component experiments. We trained Vanilla Loop, Parcae, HyperLoop, and OperLoop in a fully looped dense nanoGPT setting on 50B tokens, with the same width, effective depth, training recipe, and evaluation protocol. OperLoop achieves the lowest PPL on both datasets: compared with HyperLoop, WikiText word PPL decreases from 23.73 to 23.30 and LAMBADA PPL from 27.28 to 26.11. In the extended ablations, Overall Avg12 (the mean of the 12 downstream scores) is 42.68% for OperLoop, versus 42.10% without the rank-one delta term, 41.96% without read/write tying, 41.03% with fixed decay, and 41.95% without decay. The rank-one ablation retains tying, testing its contribution beyond tying alone. Tables R1–R2 in our response to Reviewer 5566 identify the result sources.

We will incorporate these clarifications, figures, and additional experiments into the revised manuscript, including the design motivation at the start of Section 4. We hope these clarifications and additional results address the principal concerns. We would appreciate your consideration of the paper based on these clarifications and results.

Thank you again for your time and constructive feedback. We sincerely appreciate the care you have taken in evaluating our work and helping us clarify its contributions.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.