acceptodds
Under review as a conference paper at ICLR 2027

LT2: Linear-Time Looped Transformers

Abstract

Looped Transformers (LT) reuse the same layers several times before producing each token, gaining computational depth without adding parameters. However, this *parameter efficiency* does not directly translate into *computational efficiency*: with full softmax attention, the quadratic cost is incurred at every loop, making both training and inference slow. We therefore study **LT2 (Linear-Time Looped Transformers)**, a family of looped architectures that replace quadratic softmax attention with linear-time attention, enabling fast looping over a fixed-size recurrent memory. We first show, both theoretically and empirically, that looping and linear attention are strongly synergistic: on recall—linear attention's known weakness—looping helps linear attention more than it helps full attention, improving natural-text recall by +8.9 points versus +4.8 for full attention. We then propose **historical KV replay**, a simple, parameter-free mechanism that co-designs *recurrence in depth* (looping) and *recurrence in time* (linear attention): at every loop, each token replays its key–value pairs from all earlier loops, in reverse order, into the current loop's state, so that each update draws on historical information as well as the current loop. Applied to a vanilla looped Gated DeltaNet (GDN), historical KV replay improves natural-text recall from 34.6 to 41.1, strengthens needle-retrieval extrapolation, and raises performance on every downstream benchmark, by 2.7 points on average. Building on these insights, we further propose a cheaper variant that replays only the first loop's key–value pair instead of the full history. It runs at the cost of plain looping yet recovers about 70% of the full replay's gain, offering the best efficiency–quality trade-off in practice. Finally, we provide a simple recipe for converting a pretrained looped Transformer into a hybrid LT2. With only 1.3B tokens of continued training, our *Ouro-LT2-1.4B* retains most of its teacher's quality, letting practitioners reuse existing checkpoints rather than train from scratch.

Primary area:
foundation or frontier models, including LLMs
Keywords:
looped transformers, linear-time attention, deep learning architectures

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_ChAaOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes linear-time looped transformer with kv replay. A looped transformer reuses a transformer block TT times, trading additional computation for better performance. When the token-mixing layer is a linear attention layer, the model is a linear-time looped transformer. The paper first proposes reverse replay: at loop tt, the linear-attention state is updated with the kv pairs that the same block generated at loops t,t−1,…,1t, t-1, \dots, 1. It then proposes first-loop replay, a variant of LT2 that updates the state only with the kv pair generated in the first loop, regardless of the current loop index. The authors claim that first-loop replay retains most of the performance gain of reverse replay while saving the FLOPs of the key and value projections in later loops.

Critical strengths

The method is simple yet effective. In the first-loop replay variant of LT2, the kv pairs generated in later loops are simply replaced by the ones generated in the first loop. This introduces no additional computation. in fact, it saves the FLOPs used to project the keys and values in later loops. These properties make the method highly promising for practical applications.

Critical weaknesses
  1. The authors claim that looping can turn compute into context and it can update the memory update from rank-1 in DPLR models to rank-TT at most. However, it is not looping that can turn compute into context. It is historical KV replay that turns compute into context. The description in the introduction is inaccurate and misleading.

  2. It seems that most of the performance gain comes from reusing the kv pair generated in the first loop. This is an interesting observation, but the paper gives no explicit explanation of why the first-loop kv pair matters so much.

  3. As far as I understand, the method is simple and can therefore be adapted to other linear attention variants such as KDA. Although KDA shows no noticeable improvement over GDN in the paper, experiments on LT2-KDA with both first-loop replay and reverse replay are necessary to demonstrate the generality of the method.

Questions for authors
  1. Do you think the increase from rank-1 to rank-TT is the main source of the performance gain?
1 comment

Thank you for the detailed responses, additional experiments, and clarification of the attribution of the rank-TT result.

The first three explanatory experiments confirm or reconfirm that (1) the trained models rely more heavily on KV pairs from earlier loops, (2) reverse full replay produces higher effective-rank updates, and (3) stopping gradients through replayed KV reduces the gains. However, I believe the mechanisms underlying the method’s effectiveness remain incompletely understood.

The first experiment demonstrates the importance of early-loop KV under the current design, but does not explain why it is particularly useful. The third experiment shows that the additional gradient pathway contributes to performance. However, stopping this gradient omits contributions from some uses of the shared KV to the gradient of the original forward computation. Some performance degradation is therefore unsurprising, and this result provides limited explanation of why first-loop KV is especially beneficial.

Similarly, the second experiment confirms an increase in effective rank without establishing a causal relationship between higher rank and better performance, as the authors themselves acknowledge. This distinction is particularly relevant because forward full replay admits the same rank bound and may also produce higher effective-rank updates, yet performs substantially worse. Its effective rank has not been reported.

Thus, both the origin of the early-loop KV advantage and the mechanism behind reverse full replay’s additional gains over first-loop replay remain unresolved.

An incomplete mechanistic explanation does not diminish the method’s demonstrated practical usefulness. The fourth experiment strengthens the evidence that replay makes additional loops useful, while the fifth experiment substantially increases my confidence in its applicability beyond GDN.

I find the method, especially first-loop replay, strikingly simple. Given its demonstrated effectiveness and the stronger evidence for broader applicability, I am raising my recommendation to weak acceptance. A more complete and convincing theoretical analysis explaining the early-loop KV advantage and the additional benefits of reverse full replay would make me consider raising my recommendation further to clear acceptance.

Reviewer_hNe4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies looped transformer models with linear attention. It conducts a comprehensive analysis on different variants of looped transformers and proposed two novel variants that co-designs linear attention with looping: first-loop replay (analogous to YOCO, which reuse the key and value of the first loop for every subsequent loop), and full-replay (which take the chained product of state transition matrices of all previous loops for the current loop). The authors also showed that full-attention looped transformers can be also distilled into linear attention variants.

It demonstrated that these modifications can significantly boost the performance beyond naive looped transformers linear attention with minor training overhead. I find the modifications simple yet effective, and the empirical evidences are well-supported. However, I am not convinced by the theoretical (or intuitive) explanation on why such modifications work better than the naive linear attention variants (see weakness). However the shaky explanations does not completely offset the empirical contributions.

In summary, I lean toward acceptance conditioned on major narrative reframing on the explanations. I will raise my current score to reflect acceptance if the authors address my concern about the narratives.

Critical strengths
  1. The looping co-designed linear attention variants are elegant and effective. I find it to be a strong empirical work.
  2. The empirical validations are comprehensive and convincing.
Critical weaknesses
  1. The authors explained the performance gain by claiming the variants "turn compute into context" (line 69-72). While the expressivity result that a chaining of T rank-1 updates gives a rank-T update is factually correct, the connection between the expressivity change and performance change is largely unjustified. In particular, much of the performance gain can be already attained by first-loop replay, which is still using a rank-1 update that is actually arguably less expressive compared to the canonical looped GDN variant, making people questioning what is the actual source of improvement (is it simply caused by St(1)S^{(1)}_t appears in all loops giving better learning signal?). I recommend the authors to amend the explanatory narratives unless more supportive evidence are provided.
Questions for authors
  1. Figure 2a showed better loop-count scaling property for the linear attention variants against full-attention variants. Does the linear trend extrapolate beyond T=4 to say 6 or 8 (i.e. looped GDN surpassing looped full attn by a larger margin)?

  2. For the full-replay 4x looped GDN, which yields the largest gain, can the authors measure how heterogeneous are state transition matrices at the same layer but for different loops? In particular, can the authors measure, during inference of variable tasks used in the paper, what is the averaged effective rank of Pt(τ)−I\mathbf{P}_t^{(\tau)}-I for τ=2,3,4\tau = 2,3,4? If the updates are still effectively rank-1, the explanatory claims may need to be modified.

  3. Can the authors provide a more comprehensive analysis as to why first-loop replay, despite being less expressive, can also attain a significant performance gain? In particular, I am curious of the following setup: if we take the 4x looped GDN trained with first-loop replay but only inference with T=1, does it also perform better than the standard 4x looped GDN inferenced with T=1?

I will raise my score if either additional evidence are provided to support the explanatory claims or the authors amend the explanatory claims.

Reviewer_8pMSOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LT2: Linear-Time Looped Transformers, combining the state-of-the-art looped model architecture with linear-time attention. The motivation is compelling. The experiments show that looped linear-attention models are more memory- and compute-efficient than Looped Transformers (Table 5). Building on LT2, the paper proposes historical KV replay and experimentally compares forward replay, reverse replay, and first-loop replay in detail. Both reverse replay and first-loop replay effectively improve LT2. The experiments are thorough, covering model quality, computational efficiency, hybrid architectures, and the conversion of a pre-trained Looped Transformer into LT2. These results provide a valuable reference for future work.

Critical strengths
  1. The paper studies the combination of Looped Transformers and linear-time attention. This is a direct and effective idea that has not been sufficiently explored in prior research, making the research question valuable.
  2. The proposed reverse replay and first-loop replay are both effective improvements and achieve strong results. The experimental evaluation is thorough, making this work a valuable reference.
Critical weaknesses

None.

Questions for authors

Why does looping linear-attention layers improve long-context recall? Although the authors argue that multiple writes increase the rank of the recurrent-state update, historical information is written through these higher-rank updates. Is the recall gain due to greater storage capacity, as the updates make better use of the state, or to stronger retrieval, as multiple queries read from the state during generation? Could both contribute, and how do they interact? I could not find an analysis of this distinction in the main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.