- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ARLoop, a looped transformer architecture that incorporates Attention Residuals to improve information reuse across recurrent depths. Instead of relying on standard residual connections that accumulate intermediate representations with fixed weights, ARLoop uses depth-wise attention to selectively aggregate information from previous residual blocks. The authors further investigate sliding-window depth attention, finding that attending only to recent residual blocks can match or slightly outperform full-history attention in looped transformers. Controlled experiments on 140M and 370M models show consistent improvements in validation perplexity over existing residual designs, including Parcae and DeepLoop. The authors also scale ARLoop to 0.6B and 1.3B parameters, demonstrating improvements on mathematical reasoning and code generation benchmarks.
Critical strengths
- Interesting findings on residual redundancy in looped transformers. The observation that sliding-window depth attention can achieve comparable performance to full-history attention is interesting, especially given its different behavior in standard transformers. The analysis of attention weights and representation similarity provides insights into how information is reused across recurrent depths.
Critical weaknesses
-
Concern on methodology contribution. Both Attention Residuals and sliding-window depth attention have been proposed in prior work, as acknowledged in the paper. The main contribution of this paper is applying these existing techniques to looped transformers and systematically evaluating their effectiveness, rather than introducing a fundamentally new method. Therefore, I would see this paper more as an analysis paper.
-
The effectiveness of sliding-window is not sufficiently demonstrated. First, the maximum number of recurrences tested is only 4, which is probably insufficient to demonstrate the advantages of sliding-window attention (since it is specifically designed for deep looping scenarios). Second, as shown in Table 2, the perplexity differences between sliding-window and full-history attention are extremely small (0.02–0.10). Without any statistical significance analysis, these differences are not particularly convincing. Therefore, I think the paper overclaims the benefits of sliding-window depth attention because they claim in Line 275 that "sliding-window outperforms in perplexity at deeper recurrences".
-
Insufficient controlled comparisons at larger scales. The paper only provides a direct PreNorm/ARLoop comparison at the 0.6B scale with 180B training tokens. However, the heavily emphasized ARLoop-1.3B model on 500B tokens lacks a corresponding PreNorm baseline under the same training recipe and budget. Moreover, the paper does not compare against existing residual designs such as Parcae or DeepLoop under the realistic pretraining setup. It is difficult to determine how much of the strong performance of ARLoop-1.3B can be attributed to the proposed residual design.
Questions for authors
See weaknesses.
2 comments
References:
[1] Team, K., Chen, G., Zhang, Y., Su, J., Xu, W., Pan, S., ... & Zhou, X. (2026). Attention residuals. arXiv preprint arXiv:2603.15031.
[2] Zhu, R. J., Wang, Z., Hua, K., Zhang, T., Li, Z., Que, H., ... & Eshraghian, J. (2025). Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741.
[3] Wang, S., Zhang, G., Luo, K., Wu, Y., Liu, S., Liu, J., ... & Li, J. (2026). Smelt: Scaling laws for compute-matched moe looped transformers. arXiv preprint arXiv:2609.01343.
[4] Fishman, M., Sun, S., Acharya, S., Ladhak, F., Chmiel, B., Serano, S. A., ... & Puvvada, K. C. (2026). Dory1.0-4B: A Looped Normalized-GPT. arXiv preprint arXiv:2611.03998.
[5] Prairie H, Novack Z, Berg-Kirkpatrick T, et al. Parcae: Scaling laws for stable looped language models[J]. arXiv preprint arXiv:2604.12946, 2026.
[6] Li, S., Zhang, Y., Guo, J., Gu, Q., & Wang, M. (2026). DeepLoop: Depth Scaling for Looped Transformers. arXiv preprint arXiv:2607.13491.
[7] Zeitoun, A., Torroba-Hennigen, L., & Kim, Y. (2026). Hyperloop transformers. arXiv preprint arXiv:2604.21254.
We thank the reviewer for the constructive feedback. We address your concerns below.
[Scope and contribution of our work — Weakness 1] Although ARLoop builds on prior work on attention residuals [1], it uses sliding-window depth attention rather than the full-history attention adopted in the original work. This design is motivated by the high similarity between representations across recurrences in looped transformers. Neither our study (Table 9) nor prior work (Section 5.3 in [1]) demonstrates comparable benefits from sliding-window depth attention in standard transformers. To the best of our knowledge, our work is the first to show that sliding-window depth attention is particularly well suited to looped transformers. We would also appreciate pointers to prior work demonstrating its effectiveness within attention residuals.
Furthermore, we will release all model artifacts, including training recipes, intermediate checkpoints, and final models. Our strongest model outperforms Huginn, the strongest prior fully-open looped LLM, and matches Qwen3.5-2B on the evaluated benchmarks. We believe these resources can facilitate research on looped transformers, including interpretability, training dynamics, and improved methods for mid-training, continued pretraining, and fine-tuning.
We therefore view our contributions as a combination of a residual design tailored to looped transformers (ARLoop), systematic empirical analysis under both controlled settings and a realistic pretraining recipe, and the release of useful artifacts for the looped transformer community. Together, these contributions extend beyond analysis alone. We will clarify this scope in the final version of the paper.
[Effectiveness of sliding-window depth attention — Weakness 2] We acknowledge that our findings are limited to the recurrence counts evaluated, R<5. However, we do not consider this range insufficient, as most existing looped LLMs operate within a similar range, including Ouro [2] (R<5), SMELT [3] (R=2), and Dory1.0-4B [4] (R<5). We agree that investigating larger recurrences would be valuable for exploring the further benefits of sliding-window depth attention. Due to computational budget constraints, we leave this investigation to future work and will discuss it in the limitations section of the final paper.
Regarding the perplexity improvements in Table 2, our claim is not that sliding-window depth attention is substantially better than full-history depth attention. Rather, our point is that maintaining the full residual history may be unnecessary for looped transformers, making sliding-window depth attention a suitable alternative with lower overhead. The comparison between ARLoop-0.6B and PreNorm-0.6B in Table 3 further demonstrates that ARLoop with sliding-window depth attention remains effective on complex generative tasks, even after extensive pretraining on 180B tokens.
[Reproducing other baselines at the largest scale — Weakness 3] We primarily use ARLoop-1.3B to demonstrate that our design can produce a strong looped LLM at a larger parameter scale (1.3B) and with a larger training data budget (500B tokens). Reproducing PreNorm and other baselines at this scale would certainly be valuable. However, pretraining ARLoop-1.3B on 500B tokens already requires approximately two weeks on 32 H100 GPUs, at a cost of over $50,000 using Amazon AWS GPU services. Training additional baselines would multiply this cost, which cannot be afforded with our academic compute budget.
We would also like to note that prior work on residual designs for looped transformers, including Parcae [5], DeepLoop [6], and Hyperloop [7], does not validate their designs on the complex generative tasks, such as math and coding, on which we evaluate ARLoop-0.6B and PreNorm-0.6B. Prior studies generally focus on controlled FineWeb-Edu settings, similar to those in our Section 4. We therefore believe that our matched comparison at the 0.6B scale already extends substantially beyond the evaluations in these studies.