acceptodds
Under review as a conference paper at ICLR 2027

LoopLA: Causal State Refinement for Looped Linear Attention

Abstract

Looped Transformers iteratively refine hidden representations at the cost of additional computation. Linear attention can reduce this cost through subquadratic scaling with sequence length. However, the role of its recurrent memory state in looped algorithms has not been systematically studied. We introduce LoopLA, which refines this memory along both token and loop dimensions. To preserve causality without serial token-by-token computation, LoopLA uses a Jacobi-style draft-and-consolidate procedure that approximates state-dependent coefficients for parallel scans. Chunk-wise state commits refresh the memory used by subsequent drafts. We train approximately 1B-parameter Gated DeltaNet models from scratch for 100B tokens. At matched parameter count, LoopLA outperforms baselines without state carry overall and improves the average RULER score by 2.89 points over the no-loop baseline. Under matched FLOPs, it remains comparable, though slightly weaker overall, to the no-loop baseline, with better long-context performance on RULER 2k/4k. This work demonstrates the importance of state refinement along both token and loop dimensions for looped linear attention.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Loop Transformers, Large Language Model, Linear Attention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PiAYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LoopLA, applying the idea of looped transformer to Linear Attention, as a looped linear-attention architecture that carries and refines recurrent memory across both tokens and loop passes. To preserve causality while avoiding fully sequential computation, it uses a Jacobi-style draft-and-consolidate procedure, optionally with chunk-wise memory commits. Experiments train approximately 1B-parameter Gated DeltaNet models on 100B tokens. LoopLA improves RULER performance over parameter-matched baselines.

Critical strengths
  1. A meaningful architectural extension with careful causal handling. While previous works, such as LT2, MixerLoop, and DeltaProduct, have explored related ideas in looped linear attention, LoopLA provides a meaningful extension by combining network-level looping with recurrent-state updates. In particular, the method carefully handles the causal alignment of recurrent states across loops, ensuring that information from future tokens is not incorrectly introduced. It also approximates the state-dependent update coefficients to preserve parallel scan operations, allowing the model to benefit from iterative computation while maintaining efficient training.

  2. The paper evaluates LoopLA across several benchmarks and tasks, providing evidence of its effectiveness in different settings. The experiments also investigate how model performance changes as the number of loops increases, showing the potential benefits of allocating more computation through repeated iterations.

Critical weaknesses

The main quality experiments use two loop passes, whereas Figure 5 primarily evaluates the latency of increasing the loop count. Could the authors report task performance across several loop counts, such as (K=4,8), using a smaller-scale controlled setting if possible? I wonder the scaling performance in LoopLA. And these comparisons with LT2-style looping and MixerLoop would help establish whether explicit state carry provides sustained benefits as recurrent depth increases.

Questions for authors
  1. As I wonder in above, quality–compute trade-off and test-time scaling is suggested to be further disscussed. Could the authors provide results with different loop counts if possible? Reporting quality against measured latency and compute, including state-consolidation overhead, would clarify whether additional passes offer a favorable trade-off against other baselines.

  2. Could the authors elaborate on the rationale for choosing the middle eight layers as the recurrent core? It would be helpful to understand how sensitive LoopLA’s performance is to the placement and number of looped layers. If feasible, a small ablation with alternative layer configurations could clarify whether the benefits of state refinement depend on this particular design.

3 comments
AuthorsOpenReview ↗

Kind Reminder from the Authors

Dear Reviewer PiAY,

Thank you again for your valuable comments. As the reviewer--author discussion phase is approaching its end within 24 hours, we would be grateful to know whether our response has addressed your concerns or whether there are any remaining questions we could clarify. Thank you for your time and feedback, and look forward to your reply.

Best regards from the authors.

Thank you for the author's response, which has largely addressed my concerns. I will maintain my current positive opinion.

Thank you for the feedback! We're glad that our response addressed your concerns. Thank you again for your valuable time and the positive assessment of our work!

Reviewer_qDQxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LoopLA, a looped linear-attention (LA) architecture that refines recurrent memory across token and loop dimensions. It ensures causal cross-pass state carry via token-first/pass-second updates. Parallel training is enabled with a Jacobi-style draft-and-consolidate procedure, parallel prefix scans, and chunk-wise state commits. Using ~1B-parameter Gated DeltaNet models trained for 100B tokens, chunked LoopLA improves average RULER at 1k/2k/4k by 2.89 points over a no-loop baseline at matched parameters. Ablations cover information carriers, state advancement, state refinement, and chunk size. Under approximate iso-FLOPs, it remains competitive and improves RULER 2k/4k.

Critical strengths

S1. A meaningful and technically well-motivated formulation of cross-loop memory refinement.

The paper identifies recurrent linear-attention memory as an explicit information carrier that can be refined across loop passes, complementing the hidden-state feedback used by conventional looped architectures. It also correctly identifies causal leakage as a central obstacle to naïve state carry. The token-first/pass-second causal recurrence provides a clear architectural constraint, and the leakage diagnostics in Appendix D help establish why this constraint matters.

S2. A coherent algorithmic design supported by substantial training and targeted ablations.

The Jacobi-style draft-and-consolidate procedure addresses the dependency between recurrent states and state-dependent update coefficients while retaining parallel prefix scans. The authors train approximately 1B-parameter models for 100B tokens and evaluate both task performance and latency. Useful evidence for the design choices is provided by ablations on information carriers, state-advancement rules (per-pass decay and convolution append), refined reads versus refined-state commits, and Jacobi chunk size. In particular, the chunked model improves the average RULER score by 2.89 points over the iso-parameter no-loop baseline. At B=1024 and K=2, reported training latency is 1.07× that of the approximate iso-FLOPs 40-layer no-loop reference.

Critical weaknesses

W1. The overall advantage under matched computation remains limited and task-dependent.

The strongest results are concentrated in long-context RULER evaluation. Under the approximate iso-FLOPs comparison, the deeper no-loop baseline performs better on language modeling and the reasoning average, while LoopLA is better on RULER, especially at 2k and 4k contexts. The chunked variant also does not uniformly outperform the no-loop baseline across all downstream metrics. Thus, the experiments support the value of cross-loop state refinement for particular tasks, but provide more limited evidence that it is a generally superior allocation of computation. A clearer discussion of the intended operating regime and the practical trade-off against deeper no-loop models would strengthen the contribution.

W2. The Jacobi approximation lacks error analysis and direct comparison to exact sequential recurrence.

The central parallelization relies on approximating the state-dependent update coefficients from the preceding draft, while the subsequent prefix scan is exact only for those fixed coefficients. However, the paper does not quantify the discrepancy between this draft-and-consolidate approximation and the exact sequential recurrence it targets, nor does it provide an error bound or a systematic empirical comparison. It also does not characterize how the approximation gap varies with loop count K, Jacobi chunk size B, or model scale. Since exact sequential recurrence is the conceptual reference, such analysis is important for assessing whether the reported gains come from the intended causal state refinement rather than from the approximation. A direct exact-vs-Jacobi comparison, even on smaller models or selected layers, and clarification of whether training and inference use the same approximation, would materially strengthen the central algorithmic claim.

Questions for authors

Q1. Approximation quality of the Jacobi procedure.

Can the authors quantify the discrepancy between the proposed Jacobi draft-and-consolidate procedure and the exact token-major sequential recurrence, for example through state error, hidden-state error, or output divergence? How does this discrepancy vary with loop count and chunk size? This would help clarify the accuracy–parallelism trade-off underlying the method.

Q2. Computation-matched comparison.

Since the 40-layer no-loop reference matches the effective layer-application count but excludes additional state-consolidation work from its matching criterion, can the authors provide a more explicit accounting of total training FLOPs and end-to-end inference cost? How should the observed RULER gains be interpreted when all additional computation is included?

Q3. Chunk-size sensitivity and the choice B=1024.

The chunk-size ablation shows that B=1024 yields the strongest overall long-context performance, whereas B=512 and B=256 degrade markedly, especially on RULER. What explains this non-monotonic behavior? Is it primarily due to state-refresh frequency, increased sequential dependencies, implementation overhead, or optimization difficulty? Does the optimal chunk size depend on sequence length, loop count, or model scale?

Q4. Scaling to larger models and longer training budgets.

Can the authors verify the observed trends among LoopLA, the iso-parameter no-loop baseline, and the 40-layer approximate iso-FLOPs no-loop reference at larger model scales or longer training budgets? In particular, do the long-context RULER advantages persist or grow, and does the gap on language modeling and reasoning narrow? Such evidence would clarify whether cross-loop state refinement is broadly beneficial or mainly effective in the studied regime.

5 comments
AuthorsOpenReview ↗

Kind Reminder from the Authors

Dear Reviewer qDQx,

Thank you again for your valuable comments. As the reviewer--author discussion phase is approaching its end within 24 hours, we would be grateful to know whether our response has addressed your concerns or whether there are any remaining questions we could clarify. Thank you for your time and feedback, and look forward to your reply.

Best regards from the authors.

Thank you for the detailed rebuttal, including the new exact-versus-Jacobi appendix, the FLOPs accounting, and the additional K=3 results. I accept that the approximation is faithful at inference — max argmax disagreement 0.547% and |ΔNLL| < 0.0016 — but that comparison is necessarily run on a checkpoint that was itself trained under the same approximation, so it establishes implementation fidelity rather than validating the proposed recurrence. The counterfactual of exact-serial training remains untested, and no error bound or convergence statement is offered. My reservation is therefore not about the cost but about demonstrated benefit: the authors' own accounting places training FLOPs within 1.007× of the baseline, yet at matched computation LoopLA is reported as "comparable, though slightly weaker overall" than the 40-layer dense baseline, with gains confined to RULER at 2k/4k while the no-loop model still leads on language modeling and reasoning. The remaining questions — a single seed with no variance estimates, and a chunk size validated only at one context length — likewise remain open. I therefore keep my rating at 2 (weak rejection).

Thank the reviewer for the reply. We appreciate your recognition of our rebuttal efforts and faithful approximation on exact-versus-Jacobi analysis.

However, we would like to clarify your misunderstanding and unreasonable new concerns raised during your rebuttal reply at the end day of the ICLR's reviewer-author discussion phase:

  1. The counterfactual of exact-serial training remains untested.

We have already stated in the rebuttal comments that, exact sequential looping without Jacobi parallelization is computationally prohibitive, with 2,381 times slower during prefill phase. For example, if we spend 1 hour training several steps in Jacobi-chunk method, we need anthoer 2,381 hours (~ 3 months) to train a exact sequential ablation for error estimation, not to mention that we actually train ~ 1 days for each ablation setting. And this infeasibility is just our motivation for jacobi-style estimation, which has shown end-to-end performance gain during our 100B tokens experiment. So there're still misunderstanding in your statement "My reservation is therefore not about the cost but about demonstrated benefit" because our Jacobi-chunk method showed real model performance benefits, not to mention that it changes an infeasible training method into a trainable method.

  1. Training FLOPs within 1.007× of the baseline, yet at matched computation LoopLA is reported as "comparable, though slightly weaker overall".

We have already stated in the rebuttal comments that, our main claim is about performance gain over the common no-loop baseline but not the strong iso-FLOPs baseline, with the latter quesiton is a common-sense problem and has not been fully resolved in looped-transformer area of research (refering to all the referenced papers above). Nor should it serve as an explanation for the reviewer's statement “My reservation is therefore not about the cost but about demonstrated benefit:,” because we have never presented the advantages of iso-FLOPs as a primary claim in our paper.

  1. The remaining questions — a single seed with no variance estimates, and a chunk size validated only at one context length — likewise remain open.

We would like to respectfully clarify that, these experimental designs significantly increase the overall budget for the experiment, requiring 5 to 6 times the GPU computing resources currently allocated in this work. The reviewer explicitly identified strength 2 of our paper as “supported by substantial training and targeted ablations,” but then went on to suggest that we should significantly expand our training resources at the end of rebuttal phase, which is contradictory. Futhermore, these questions are not "remaining", because the reviewer didn't mention any of these questions in the initial review comment, but choose to raise these issues nearly a week after we submitted our response and less than 9 hours before the rebuttal deadline. We consider this to be irresponsible.


In light of the above, we respectfully request that the reviewer reconsider the final decision, and we also kindly ask the Area Chairs to take into account the fact that this reviewer raised additional questions in an unreasonable and extremely untimely manner, despite having acknowledged the clarifications we provided during the rebuttal phase.

Reviewer_AYyMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LoopLA, which refines the recurrent state of looped linear attention. The proposed architecture LoopLA addresses the hidden state carry of the looped linear attention. It also addresses the parallel training and inference. The experiment is comprehensive, covering iso-params comparisons, an approximate iso-flops reference, a causal info leak diagnostic, training/inference latency. LoopLA is competitive overall. Against the approximate iso-FLOPs reference, it remains competitive overall while retaining advantages on longer-context RULER.

Critical strengths
  1. The paper considers the causal order, to prevent info leak, and provides a dedicated causal-leakage diagnostic.
  2. The experiments provide comparison in two dimensions, iso-params and iso-flops. The iso-params comparison shows that the proposed architecture is competitive overall.
  3. The paper proposes a Jacobi-style draft-and-consolidate procedure to enable parallel training. An optional chunk-wise state commits is proposed to shorten the feedback interval and improve the long context performance.
Critical weaknesses
  1. I am not fully convinced of the soundness of the problem setup. From a function perspective, the computation order (1, p1) -> ... -> (1, pk) -> (2, p1) -> ... (2, pk) -> ... -> (T, pk) suggests that looping makes the per-token information-processing function more complex, effectively changing (t, p1) to (t, p1) -> ... -> (t, pk). However, in the experimental section, I could only find results for K = 2. This raises the question: is the performance gain driven by the more complex info processing function induced by looping K = 2, or specifically by looped information processing itself? That is, what is the performance curve of K = 1, 2, 3, ..., k. Is the gain monotonically increasing, or existing a turning point?
Questions for authors

See Weakness.

1 comment
AuthorsOpenReview ↗

Kind Reminder from the Authors

Dear Reviewer AYyM,

Thank you again for your valuable comments. As the reviewer--author discussion phase is approaching its end within 24 hours, we would be grateful to know whether our response has addressed your concerns or whether there are any remaining questions we could clarify. Thank you for your time and feedback, and look forward to your reply.

Best regards from the authors.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Rebuttal Summary From the Authors

We would like to thank all reviewers and Area Chairs for the valuable time they have dedicated to this project so far. As the author-reviewer discussion phase is closing soon, we would like to briefly summarize this phase:

  • Reviewer AYyM

The reviewer acknowledged our analysis of causal order and causal leakage and considered our LoopLA model to be competitive overall, not only in terms of iso-params but also in terms of iso-FLOPs. The reviewer also treated the Jacobi-style draft-and-consolidate procedure to enable parallel training as a critical strength.

The reviewer’s main concern was not fully convincing of the soundness of the problem setup. We clarified two points: First, regarding the misunderstanding about whether the performance gain is driven by a more complex function or by looped information processing itself, we explained that our ablation experiments have already made this clear, and we further emphasized this in Section 4.2. Second, regarding the discussion on the additional performance-k gain curve and whether there is a turning point, we supplemented the relevant discussion during the rebuttal phase, demonstrating that it does not affect the main claim of our paper, and updated the relevant content in Appendix E. Unfortunately, by the end of the phase, this reviewer had still not responded to our rebuttal.

  • Reviewer qDQx:

This reviewer acknowledged the motivation behind the structure of our LoopLA paper and recognized the recurrent linear-attention memory as an explicit information carrier that can be refined across loop passes. The reviewer also acknowledged our algorithm design and considered our extensive training setups and ablation experiments to be one of the paper’s key strengths.

The reviewer’s concerns primarily centered on the still-limited performance of our LoopLA vs. strong iso-FLOPs baseline (not the standard iso-params baseline), the insufficiently thorough explanation of the exact-vs-Jacobi distinction, chunk-size sensitivity and larger training budgets. We provided a detailed response midway through the rebuttal process and updated the PDF. It is worth noting that this reviewer did not provide further feedback until 10 hours within the end of the review period with additional questions (please refer to the discussion below for more information).

  • Reviewer PiAY:

This reviewer acknowledged our LoopLA model’s handling of causality, as well as its meaningful extension of information propagation at the loop level. The reviewer also acknowledged that we evaluated LoopLA across several benchmarks and tasks, providing evidence of its effectiveness in different settings.

The reviewer’s main concern also centered on the curve effect of the number of loops on performance (similar to Reviewer AYyM’s concern) and included additional questions regarding the selection of loop-level positions. In our rebuttal, we provided a detailed explanation of these experimental results and design choices, and updated the corresponding sections of the manuscript and the appendix accordingly. In subsequent discussions, the reviewer acknowledged our response, noting that we had largely addressed the concerns, and maintained the positive rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.