- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LoopLA, applying the idea of looped transformer to Linear Attention, as a looped linear-attention architecture that carries and refines recurrent memory across both tokens and loop passes. To preserve causality while avoiding fully sequential computation, it uses a Jacobi-style draft-and-consolidate procedure, optionally with chunk-wise memory commits. Experiments train approximately 1B-parameter Gated DeltaNet models on 100B tokens. LoopLA improves RULER performance over parameter-matched baselines.
Critical strengths
-
A meaningful architectural extension with careful causal handling. While previous works, such as LT2, MixerLoop, and DeltaProduct, have explored related ideas in looped linear attention, LoopLA provides a meaningful extension by combining network-level looping with recurrent-state updates. In particular, the method carefully handles the causal alignment of recurrent states across loops, ensuring that information from future tokens is not incorrectly introduced. It also approximates the state-dependent update coefficients to preserve parallel scan operations, allowing the model to benefit from iterative computation while maintaining efficient training.
-
The paper evaluates LoopLA across several benchmarks and tasks, providing evidence of its effectiveness in different settings. The experiments also investigate how model performance changes as the number of loops increases, showing the potential benefits of allocating more computation through repeated iterations.
Critical weaknesses
The main quality experiments use two loop passes, whereas Figure 5 primarily evaluates the latency of increasing the loop count. Could the authors report task performance across several loop counts, such as (K=4,8), using a smaller-scale controlled setting if possible? I wonder the scaling performance in LoopLA. And these comparisons with LT2-style looping and MixerLoop would help establish whether explicit state carry provides sustained benefits as recurrent depth increases.
Questions for authors
-
As I wonder in above, quality–compute trade-off and test-time scaling is suggested to be further disscussed. Could the authors provide results with different loop counts if possible? Reporting quality against measured latency and compute, including state-consolidation overhead, would clarify whether additional passes offer a favorable trade-off against other baselines.
-
Could the authors elaborate on the rationale for choosing the middle eight layers as the recurrent core? It would be helpful to understand how sensitive LoopLA’s performance is to the placement and number of looped layers. If feasible, a small ablation with alternative layer configurations could clarify whether the benefits of state refinement depend on this particular design.
3 comments
Thank you for the author's response, which has largely addressed my concerns. I will maintain my current positive opinion.
Thank you for the feedback! We're glad that our response addressed your concerns. Thank you again for your valuable time and the positive assessment of our work!
Kind Reminder from the Authors
Dear Reviewer PiAY,
Thank you again for your valuable comments. As the reviewer--author discussion phase is approaching its end within 24 hours, we would be grateful to know whether our response has addressed your concerns or whether there are any remaining questions we could clarify. Thank you for your time and feedback, and look forward to your reply.
Best regards from the authors.