acceptodds
Under review as a conference paper at ICLR 2027

ARLoop: Strong Fully-Open Looped Transformers with Attention Residuals

16 views

Abstract

Looped transformers increase effective depth by repeatedly applying a shared set of transformer blocks, offering a parameter-efficient way to scale computation. However, standard residual connections aggregate information across recurrent depth into a single stream, providing limited control over which intermediate representations from preceding layers should be reused. In this paper, we introduce ARLOOP, a looped transformer architecture based on attention residuals, where the input of each layer is selectively aggregated through attention over model depth. Distinct from standard transformers, we find that looped transformers do not require access to the full residual history: sliding-window depth attention matches or outperforms full-history attention at larger recurrence depths. Across controlled experiments with multiple parameter scales (140M and 370M) and recurrence depths (–), ARLOOP consistently outperforms prior residual designs for looped transformers. More importantly, we further scale ARLOOP to 0.6B and 1.3B parameters under a realistic LLM pretraining recipe, demonstrating that the gains transfer to realistic pretraining, with particularly strong improvements on math and coding tasks. Our 1.3B model, pretrained on only 0.5T tokens with three recurrences, outperforms the fully-open looped LLM Huginn-3.5B across all evaluated tasks and is competitive with similar-sized non-looped models such as Qwen3.5-2B despite being trained on substantially less data. ARLOOP will be fully open: we will release its code, checkpoints, and training recipe.

Primary area:
foundation or frontier models, including LLMs
Keywords:
looped transformer, model architecture, language model pretraining

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vE9KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ARLoop, a looped transformer architecture that incorporates Attention Residuals to improve information reuse across recurrent depths. Instead of relying on standard residual connections that accumulate intermediate representations with fixed weights, ARLoop uses depth-wise attention to selectively aggregate information from previous residual blocks. The authors further investigate sliding-window depth attention, finding that attending only to recent residual blocks can match or slightly outperform full-history attention in looped transformers. Controlled experiments on 140M and 370M models show consistent improvements in validation perplexity over existing residual designs, including Parcae and DeepLoop. The authors also scale ARLoop to 0.6B and 1.3B parameters, demonstrating improvements on mathematical reasoning and code generation benchmarks.

Critical strengths
  1. Interesting findings on residual redundancy in looped transformers. The observation that sliding-window depth attention can achieve comparable performance to full-history attention is interesting, especially given its different behavior in standard transformers. The analysis of attention weights and representation similarity provides insights into how information is reused across recurrent depths.
Critical weaknesses
  1. Concern on methodology contribution. Both Attention Residuals and sliding-window depth attention have been proposed in prior work, as acknowledged in the paper. The main contribution of this paper is applying these existing techniques to looped transformers and systematically evaluating their effectiveness, rather than introducing a fundamentally new method. Therefore, I would see this paper more as an analysis paper.

  2. The effectiveness of sliding-window is not sufficiently demonstrated. First, the maximum number of recurrences tested is only 4, which is probably insufficient to demonstrate the advantages of sliding-window attention (since it is specifically designed for deep looping scenarios). Second, as shown in Table 2, the perplexity differences between sliding-window and full-history attention are extremely small (0.02–0.10). Without any statistical significance analysis, these differences are not particularly convincing. Therefore, I think the paper overclaims the benefits of sliding-window depth attention because they claim in Line 275 that "sliding-window outperforms in perplexity at deeper recurrences".

  3. Insufficient controlled comparisons at larger scales. The paper only provides a direct PreNorm/ARLoop comparison at the 0.6B scale with 180B training tokens. However, the heavily emphasized ARLoop-1.3B model on 500B tokens lacks a corresponding PreNorm baseline under the same training recipe and budget. Moreover, the paper does not compare against existing residual designs such as Parcae or DeepLoop under the realistic pretraining setup. It is difficult to determine how much of the strong performance of ARLoop-1.3B can be attributed to the proposed residual design.

Questions for authors

See weaknesses.

2 comments

We thank the reviewer for the constructive feedback. We address your concerns below.

[Scope and contribution of our work — Weakness 1] Although ARLoop builds on prior work on attention residuals [1], it uses sliding-window depth attention rather than the full-history attention adopted in the original work. This design is motivated by the high similarity between representations across recurrences in looped transformers. Neither our study (Table 9) nor prior work (Section 5.3 in [1]) demonstrates comparable benefits from sliding-window depth attention in standard transformers. To the best of our knowledge, our work is the first to show that sliding-window depth attention is particularly well suited to looped transformers. We would also appreciate pointers to prior work demonstrating its effectiveness within attention residuals.

Furthermore, we will release all model artifacts, including training recipes, intermediate checkpoints, and final models. Our strongest model outperforms Huginn, the strongest prior fully-open looped LLM, and matches Qwen3.5-2B on the evaluated benchmarks. We believe these resources can facilitate research on looped transformers, including interpretability, training dynamics, and improved methods for mid-training, continued pretraining, and fine-tuning.

We therefore view our contributions as a combination of a residual design tailored to looped transformers (ARLoop), systematic empirical analysis under both controlled settings and a realistic pretraining recipe, and the release of useful artifacts for the looped transformer community. Together, these contributions extend beyond analysis alone. We will clarify this scope in the final version of the paper.

[Effectiveness of sliding-window depth attention — Weakness 2] We acknowledge that our findings are limited to the recurrence counts evaluated, R<5R < 5. However, we do not consider this range insufficient, as most existing looped LLMs operate within a similar range, including Ouro [2] (R<5R < 5), SMELT [3] (R=2R=2), and Dory1.0-4B [4] (R<5R < 5). We agree that investigating larger recurrences would be valuable for exploring the further benefits of sliding-window depth attention. Due to computational budget constraints, we leave this investigation to future work and will discuss it in the limitations section of the final paper.

Regarding the perplexity improvements in Table 2, our claim is not that sliding-window depth attention is substantially better than full-history depth attention. Rather, our point is that maintaining the full residual history may be unnecessary for looped transformers, making sliding-window depth attention a suitable alternative with lower overhead. The comparison between ARLoop-0.6B and PreNorm-0.6B in Table 3 further demonstrates that ARLoop with sliding-window depth attention remains effective on complex generative tasks, even after extensive pretraining on 180B tokens.

[Reproducing other baselines at the largest scale — Weakness 3] We primarily use ARLoop-1.3B to demonstrate that our design can produce a strong looped LLM at a larger parameter scale (1.3B) and with a larger training data budget (500B tokens). Reproducing PreNorm and other baselines at this scale would certainly be valuable. However, pretraining ARLoop-1.3B on 500B tokens already requires approximately two weeks on 32 H100 GPUs, at a cost of over $50,000 using Amazon AWS GPU services. Training additional baselines would multiply this cost, which cannot be afforded with our academic compute budget.

We would also like to note that prior work on residual designs for looped transformers, including Parcae [5], DeepLoop [6], and Hyperloop [7], does not validate their designs on the complex generative tasks, such as math and coding, on which we evaluate ARLoop-0.6B and PreNorm-0.6B. Prior studies generally focus on controlled FineWeb-Edu settings, similar to those in our Section 4. We therefore believe that our matched comparison at the 0.6B scale already extends substantially beyond the evaluations in these studies.

References:

[1] Team, K., Chen, G., Zhang, Y., Su, J., Xu, W., Pan, S., ... & Zhou, X. (2026). Attention residuals. arXiv preprint arXiv:2603.15031.

[2] Zhu, R. J., Wang, Z., Hua, K., Zhang, T., Li, Z., Que, H., ... & Eshraghian, J. (2025). Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741.

[3] Wang, S., Zhang, G., Luo, K., Wu, Y., Liu, S., Liu, J., ... & Li, J. (2026). Smelt: Scaling laws for compute-matched moe looped transformers. arXiv preprint arXiv:2609.01343.

[4] Fishman, M., Sun, S., Acharya, S., Ladhak, F., Chmiel, B., Serano, S. A., ... & Puvvada, K. C. (2026). Dory1.0-4B: A Looped Normalized-GPT. arXiv preprint arXiv:2611.03998.

[5] Prairie H, Novack Z, Berg-Kirkpatrick T, et al. Parcae: Scaling laws for stable looped language models[J]. arXiv preprint arXiv:2604.12946, 2026.

[6] Li, S., Zhang, Y., Guo, J., Gu, Q., & Wang, M. (2026). DeepLoop: Depth Scaling for Looped Transformers. arXiv preprint arXiv:2607.13491.

[7] Zeitoun, A., Torroba-Hennigen, L., & Kim, Y. (2026). Hyperloop transformers. arXiv preprint arXiv:2604.21254.

Reviewer_CK4sOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates a new approach to looped transformers, focusing on residuals in looped transformers. The key idea is that residuals are essentially a sum over all prior layer outputs; AR (Attention Residuals) use a learned query vector (per layer) to weight the sum over all prior layers. The authors reference Kimi Team 2026, which developed this idea, and aim to apply it to looped transformers (since it can potentially be a useful tool when there's more redundancy). The authors conduct a number of experiments studying the effectiveness of AR under different degrees of recurrence, the size of a sliding window for residuals, and when scaling up to 1.3B parameters. They conclude that Attention Residuals are an effective approach for looped transformers. Ultimately, while the idea makes a lot of sense to apply to Looped Transformers, the novelty of the submission was unclear to me given that much of the ideas around Attention Residuals were based on a prior work.

Critical strengths
  • The paper explores and applies exciting new ideas for residuals
  • The paper gets to the point quickly, and provides a clear formal expression of their approach.
  • Experiments compare the Attention Residuals to a few other approaches with looped transformers
  • Paper includes experiments to analyze the importance of the recurrent context (by this I mean the number of previous layers) that are a part of the residual
  • The paper impressively scales the proposed methods to billion-parameter (1.3B) scale.
  • Paper includes experiments to analyze effectiveness of AR for different recurrence depths
  • Discussion of training complexity given the natural questions of how ARLoop would impact backpropagation.
  • The paper contains a number of extensive additional experiments and validations in the appendix. They also provide detail on their pretraining recipe.
Critical weaknesses
  • "Baselines" (ie, the other residuals) are not described in as much detail as PreNorm and AR
  • Related work and introduction seem to bury the lede that the work is primarily applying AR to looped transformers
  • Experiments on "depth" of layers going into the residual
  • A number of parameters throughout are discussed, but not clearly noted on figures. For instance what is the recurrent depth of the residual (B_W, I think, based on prior notation) for Table 2
  • More discussion of backprop complexity would be appreciated. The discussion in the paper is primarily empirical. I would have expected more of an impact considering that AR could grow quadratically; but on the other hand using the sliding window could ameliorate this.
  • I think that the paper would be significantly improved with better visualization to help us understand the relationship between all the different properties tested.
Questions for authors
  • Why doesn't AR simply learn the sliding window itself? Can't the depth / recency of a layer be provided as part of the "query" component (I believe its simply the residual itself, with no other information) just as we provide positional encoding for standard attention?
  • What is the impact of the Sliding window size?
  • How do residuals besides PreNorm and AR perform on different recurrence depths? Are Table 1 and Table 2 measuring the same thing? If so I would suggest combining them.
  • It would be helpful for Table 3 to have indicators for which models perform best on which tasks.
  • How do the models perform on an IsoParam or IsoFlop basis? Maybe a two axis plot with x=FLOPs/Params would be helpful; interpreting the table is challenging (and the table only includes training FLOPs, not inference)
  • My score factors in: 1) The novelty of the approach); 2) The clarity of the experimental details (can be improved); 3) The scale of the implementation and empirical validation. Two discussion points that would improve my score would be: better approaches to carry the small-scale insights to the scale up of the model (perhaps even including analysis for a ladder from small to 1.3B?), and results that specifically demonstrate the cases where ARLoop solves problems inherent in looped transformers.
3 comments

Response to reviewer 2

We thank the reviewer for the thoughtful comments and constructive suggestions. We address your questions below.

[Description of baselines — Weakness 1] We agree that the baseline descriptions should include more technical detail. We will expand the appendix to provide the formulations and implementation details of Sandwich Norm, DeepLoop, and Parcae using the notation from Section 3. Briefly:

  • Sandwich Norm adds output normalization to each PreNorm residual update: h(i)=Norm⁡(h(i−1)+fi(Norm⁡(h(i−1)))).h^{(i)} = \operatorname{Norm}\left(h^{(i-1)} + f_i\left(\operatorname{Norm}(h^{(i-1)})\right)\right). This normalizes the residual stream after every sublayer.

  • DeepLoop adapts DeepNorm to looped transformers by scaling the skip connection: h(i)=Norm⁡(αh(i−1)+fi(Norm⁡(h(i−1)))),h^{(i)} = \operatorname{Norm}\left(\alpha h^{(i-1)} + f_i\left(\operatorname{Norm}(h^{(i-1)})\right)\right), where α=(2LR)1/2\alpha=(2LR)^{1/2} for LL physical layers and RR recurrences. It also scales the initialization of selected residual-branch weight matrices by β=(8LR)−1/2\beta=(8LR)^{-1/2}. Unlike DeepNorm, which uses an exponent of 1/41/4, DeepLoop uses 1/21/2 to account for parameter sharing across recurrences.

  • Parcae introduces a stable state transition across recurrences. Let s(r)=h(2Lr)s^{(r)}=h^{(2Lr)} denote the output of recurrence rr. Its recurrent update takes the form s(r)=S(Aˉs(r−1)+Bˉe),s^{(r)} = S\left(\bar{A}s^{(r-1)}+\bar{B}e\right), where ee is the normalized input embedding. The transition matrices are parameterized as

    \bar{A}=\exp(\Delta A), \qquad \bar{B}=\Delta B,$$ where $a$, $\Delta$, and $B$ are learnable parameters, with $\Delta$ constrained to be positive. This ensures that the eigenvalues of $\bar{A}$ have magnitude less than one. The recurrent state is thus decayed by a learned diagonal matrix, with optional input injection before the next recurrence. We evaluate Parcae both with and without input injection, setting $\bar{B}=0$ in the latter case.

[Positioning of the paper — Weakness 2] We agree that ARLoop builds on attention residuals. Our broader goal, however, is to investigate how residual designs enable looped transformers to better exploit recurrent computation. Prior work, including DeepLoop and Parcae, leaves two important questions underexplored. First, comparisons primarily focus on standard PreNorm baselines, leaving limited evidence from apples-to-apples comparisons among different residual designs. Second, their pretraining primarily uses web documents from FineWeb-Edu, leaving the effectiveness of these designs on more complex generative tasks underexplored. We address these gaps through controlled comparisons and diverse downstream evaluations under a larger-scale pretraining recipe.

In addition, our study shows that sliding-window depth attention can match or outperform full-history attention in looped transformers, highlighting a promising design choice for recurrent architectures. This finding contrasts with the original attention residuals work (Section 5.3 in [1]), where sliding-window depth attention performed substantially worse than full-history attention in standard transformers.

Furthermore, we will release all model artifacts, including training recipes, intermediate checkpoints, and final models. Our strongest model outperforms Huginn, the strongest prior fully-open looped LLM, and matches Qwen3.5-2B on the evaluated benchmarks. We believe these resources can facilitate research on looped transformers, including interpretability, training dynamics, and improved methods for mid-training, continued pretraining, and fine-tuning.

Our contributions therefore combine a residual design tailored to looped transformers (ARLoop), systematic empirical validation under both controlled settings and a realistic pretraining recipe, and the release of useful artifacts for the looped transformer community. We will revise the introduction to foreground this scope and these contributions in the final version of the paper.

[Hyperparameter details, residual block size, and sliding-window size — Weaknesses 3–4 and Question 2] Unless otherwise specified in the window-size ablation, we retain three recent residual blocks in addition to the input embedding, as stated in Lines 238–239 and Line 772 of the appendix. We also examine the effect of sliding-window size in Appendix A.6, Table 11. Each residual block contains 6 sublayers for the 140M models and 12 sublayers for the 370M models, as described in Lines 234–237. We agree that these settings should be easier to locate and interpret. We will consolidate the relevant hyperparameters in Section 4, including moving the corresponding entries from Table 5 into the main text, and explicitly state the block and window sizes in the relevant table captions.

[1] Team, K., Chen, G., Zhang, Y., Su, J., Xu, W., Pan, S., ... & Zhou, X. (2026). Attention residuals. arXiv preprint arXiv:2603.15031.

[Efficiency overhead analysis — Weakness 5] We appreciate this suggestion and will include a detailed analysis of computational overhead in the final version of the paper. In terms of computation, full-history depth attention indeed scales quadratically with effective depth, as causal attention does with sequence length, and sliding-window attention can mitigate this by bounding the number of sources.

In practice, we observe that the additional overhead of attention residuals relative to PreNorm is primarily memory-bound rather than compute-bound. This is because the attention residual (AttnRes) operation at each sublayer introduces additional GPU memory I/O, increasing the time spent on forward and, especially, backward computation. Unlike PreNorm, each AttnRes operation computes attention weights for the residual sources and a weighted mixture of those sources. Backpropagation must differentiate through both the values and their attention weights, accumulate gradients for shared residual sources, and compute gradients for the AttnRes parameters. This introduces additional data movement through the GPU memory hierarchy, which is the primary contributor to the observed throughput decrease (Table 4). Let KK denote the number of available residual sources and DD the hidden dimension. For each sublayer and token, reading each residual source once and writing its gradient once already incurs an I/O cost of O(2KD)O(2KD), without accounting for gradient writes for the queries and RMSNorm weights.

[Why full-history attention does not learn the window - Question 1] Full-history attention can in principle learn to concentrate on recent residual sources, and our attention-weight visualization (Figure 2, left) suggests that it does so to a substantial extent. However, learning concentrated weights does not remove the cost of reading and scoring the full history, which is the main motivation to adopt sliding-window depth attention. We would also like to clarify that the depth-attention query is a learned vector associated with each effective sublayer, so the query parameterization already distinguishes the depth of each sublayer. We will make this clear in the final paper.

[Combining Tables 1 and 2 — Question 3] Table 1 compares ARLoop with PreNorm and other residual designs across recurrence settings. Although all evaluated designs improve in perplexity as the number of recurrences increases, ARLoop shows the strongest improvements across recurrences. Tables 1 and 2 report perplexity on the same validation set and can therefore be combined. Completing the merged table would require the currently missing R=5R=5 results for the configurations in Table 2. We will add these experiments and merge the tables in the revised paper.

[Table 3 presentation and IsoFLOP/IsoParam comparisons — Questions 4–5] We will use boldface and underlining to identify the best and second-best results for each task in Table 3. We also agree that adding IsoParam and IsoFlop plots is valuable, and will include the two-axis plot to visualize benchmark performance versus FLOPs/Params with extra space upon acceptance.

[Benefits of ARLoop and connecting controlled findings to larger-scale pretraining — Question 6 and discussion points] Our experiments provide two levels of evidence. At 140M and 370M, controlled comparisons isolate residual-design choices across recurrence settings. ARLoop-0.6/1.3B then validates whether the benefits transfer to a realistic and more diverse pretraining recipe. Since the 140M/370M and 0.6B/1.3B results are obtained under different experimental settings, they cannot be directly combined into a scaling ladder.

Regarding the cases where ARLoop improves over PreNorm-based looped transformers, we provide a starting point here by evaluating ARLoop-0.6B and PreNorm-0.6B on a set of specially designed synthetic tasks. Specifically, we consider three tasks: copying, preschool math, and variable assignment, motivated by [1]. Each task prompts the model with a prefix such as Fill in blank: {Program}. For copying, an example program is lip far man spy lip far kid man __. -> spy. For preschool math, an example is m=2, p=8, g=+m-p, g=__. -> → +2-8=-6. For variable assignment, we consider multi-hop setups, with hop counts ranging from 1 to 3. A 1-hop example is g=13, z=2, q=5, p=7, g=__. -> → 13, and a 2-hop example is g=13, z=2, q=5, p=7, w=g, w=__. -> → 13. For each task, we generated 512 examples, each accompanied by five-shot demonstrations, and report the task accuracies averaged over 4 random seeds below.

TaskPreNorm-0.6BARLoop-0.6B
Copy21.88%38.77%
Preschool math91.99%99.90%
Variable assignment: 1-hop31.93%70.51%
Variable assignment: 2-hop20.41%23.05%
Variable assignment: 3-hop20.70%20.02%

We can see that ARLoop substantially improves performance on copying, preschool math, and 1-hop variable assignment compared to PreNorm. We believe this suggests that ARLoop is better at associative recall tasks, which require retrieval from context. In contrast, we observe only a minor improvement on 2-hop variable assignment, and both PreNorm and ARLoop perform near the random baseline (i.e. 20%) on the 3-hop task. Note that despite the limited gains of ARLoop on the 2-hop and 3-hop tasks, the conclusions might change when considering CoT reasoning, instruction tuning, and diverse evaluation tasks. We will include this preliminary study as a starting point in a discussion section, and leave analysis across more diverse setups for future work.

Reviewer_YhFjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes using a sliding window attention residual architecture when looping, calling it ARLoop. In this variant, a per-layer query vector is used to compute attention scores to the output of previous layers. These scroes are in turn used to compute a weighted average which goes through a pre-norm and used as the input to the layer. In the sliding variant, the output of the last 3 layers and the input embedding are available for this attention. The authors show that this method performs better than the standard residuals and that sliding window is matching or sometimes better than full residual. This is established both in terms of short-horizon training on Fineweb-EDU as well as long horizon training.

Critical strengths

The ARLoop architecture is interesting, its cost doesn't necessarily increase with scale and therefore makes it possibly useful for larger models. The authors do both short-horizon and long-horizon tasks, comparing with other pre-trained models. There is also investigation on the other properties of the model, for example whether this architecture is more suitable for extrapolation.

Overall, the proposed method improves noticeably over standard pre-norm looping, increasing the gains from looping.

Critical weaknesses

Some numbers such as perplexity values in Table 2 are pretty close and there is no information on the confidence interval for these. Given that these numbers are also not reported on standard validation sets, it'd be useful to understand for example if the 0.02 gap between sliding window and full ARLoop in R=4 is meaningful or not.

Another concern is that in the experiments, the size of block for block residual is chosen such that there are only 2 blocks in each loop. The relation between this choice and the number of residuals needed to be attended to has not been studied. As the depth of the model grows, this might become more important.

Questions for authors
  1. can the reason sliding window works better for more loops be that the number of residuals remains constant across loops? Have you considered using a separate query per loop (and per layer)?

  2. Regarding training efficiency, do I understand correctly that the 10% slowdown is coming from storing 3 more activation vectors? That seems quite a noticeable drop. Does this change across scale?

  3. What is the effect of block residual on a non-looped model in experiments done in Table 3?

2 comments

Thank you for the positive feedback of our paper. We would like to address your questions below.

[Perplexity improvements from sliding-window depth attention — Weakness 1] We agree that a small perplexity difference (-0.02 at the 140M scale with R=4R=4) between sliding-window and full-history depth attention is not, on its own, distinguishable. Our claim is not that sliding-window depth attention significantly outperforms full-history attention, but that it can match the performance of full-history depth attention while reducing computational overhead, making it a design specifically suitable for looped transformers. We will clarify this point in the revised paper.

Related work, including Parcae, DeepLoop, and Hyperloop [1, 2, 3], also uses improvements in perplexity to evaluate their residual designs. Nevertheless, we agree that downstream evaluation provides important complementary evidence. This is exactly our motivation to evaluate ARLoop on complex generative tasks under a realistic pretraining recipe (Section 5). As shown in Table 3, ARLoop-0.6B, which uses sliding-window depth attention, substantially outperforms the PreNorm baseline on math and coding (+5.2% on GSM8K and +3.1% on MBPP), supporting the effectiveness of our overall design under a substantially larger pretraining budget.

[Residual block size and sliding-window size — Weakness 2] We agree that jointly studying residual block size and sliding-window size would provide a more complete picture. We did not conduct this joint study due to computational budget constraints at the time of submission, and will leave it for future investigation. We already provide preliminary results at the 140M scale that examine these two factors independently in Tables 10 and 11 of Appendix A.6. These results suggest that the residual block size does not significantly change the validation perplexity.

[Bounded history size and recurrence-specific queries — Question 1] Yes, bounding the number of directly accessible residual sources is a plausible explanation for the effectiveness of sliding-window depth attention. With full-history attention, the number of sources grows across recurrences, whereas a sliding window keeps it bounded. This may simplify source selection when earlier recurrences provide highly similar representations. Our observations that attention concentrates on recent sources and that representations exhibit high cross-recurrence similarity (Figure 2) are consistent with this explanation.

Regarding query parameterization, we already use a separate learned depth-attention query for each effective sublayer, including different recurrences of the same physical layer. The Transformer weights are shared across recurrences, but the depth-attention queries are not. We will clarify this distinction in the method section of the revised paper.

[1] Prairie H, Novack Z, Berg-Kirkpatrick T, et al. Parcae: Scaling laws for stable looped language models[J]. arXiv preprint arXiv:2604.12946, 2026.

[2] Li, S., Zhang, Y., Guo, J., Gu, Q., & Wang, M. (2026). DeepLoop: Depth Scaling for Looped Transformers. arXiv preprint arXiv:2607.13491.

[3] Zeitoun, A., Torroba-Hennigen, L., & Kim, Y. (2026). Hyperloop transformers. arXiv preprint arXiv:2604.21254.

[The overhead of training efficiency - Question 2] The 10% slowdown does not come from storing three additional activation vectors (e.g., residual blocks). In terms of memory usage, Table 4 shows that ARLoop does not require additional memory to store residual blocks in our setting. This is because, without activation checkpointing (as in our experiments), residual activations in transformers already need to be retained for gradient computation during the backward pass, regardless of whether attention residuals are used.

The 10% throughput reduction primarily arises because the attention residual (AttnRes) operation at each sublayer introduces additional GPU memory I/O, increasing the time spent on forward and, especially, backward computation. Unlike PreNorm, each AttnRes operation computes attention weights for the residual sources and a weighted mixture of those sources. Backpropagation must differentiate through both the values and their attention weights, accumulate gradients for shared residual sources, and compute gradients for the AttnRes parameters. This introduces additional data movement through the GPU memory hierarchy, which is the primary contributor to the observed throughput reduction. Let KK denote the number of available residual sources and DD the hidden dimension. For each sublayer and token, reading each residual source once and writing its gradient once already requires O(2KD)O(2KD) I/O cost, without accounting for writing gradients for the queries and RMSNorm weights. This overhead persists across scales, and its magnitude may vary depending on the number of AttnRes operations and available residual sources.

The measured slowdown also depends strongly on the kernel implementation of attention residuals. In our experiments, a straightforward implementation using torch.compile can incur approximately 30% slowdown. The implementation used in the submission instead builds on a fused Triton kernel to reduce kernel launches and redundant I/O. Since submission, we have further optimized gradient writes, reducing the slowdown to approximately 7% for the largest 1.3B model with the same test setting as Table 4. These improvements suggest space for further optimization (e.g. MoonShot reports below 4% cost with their optimizated implementations), which we leave to future work.

[Effect on non-looped models - Question 3] We want to clarify that non-looped models in Table 3 does not adopt attention residuals. Our Table 1 (140M and 370M, RR=1) provides results of non-looped models equipped with attention residuals under a controlled setup (trained on Fineweb-edu only). For results in Table 3, we did not train such a baseline due to computation budgets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.