- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper presents an empirical study of looped Transformers, examining how different looping strategies affect model performance under various parameter and training compute budgets. The authors systematically compare existing looping approaches, including full stack looping, middle stack looping, and layer wise looping, and investigate how model performance changes with the number of loops. They also explore loop specific low rank adapters to allow different loops to learn specialized transformations. Additionally, the paper studies Parallel Loop Transformers (PLTs) and proposes a modification that separately normalizes and combines the current token embedding and previous token hidden state to improve model performance while maintaining parallelizable inference. The experiments evaluate these approaches across different model sizes, training budgets, and language modeling benchmarks.
Critical strengths
The paper systematically investigates multiple looping strategies across different model sizes, training budgets, and loop counts. The experiments cover models ranging from 70M to 1B parameters and up to 32 loops.
The paper investigates different loop counts, looping locations, specialization methods, normalization strategies, and gating mechanisms.
Critical weaknesses
The paper provides an extensive empirical investigation of looped Transformers, but the methodological novelty appears limited. Several central components, including recurrent weight sharing, loop specific low rank adapters, and cross loop parallelism, have already been introduced in prior work.
The main technical modification to PLT consists of separately normalizing the current token embedding and recurrent hidden state, followed by gated mixing. While the empirical improvements are encouraging, these operations are well established, and the paper does not sufficiently establish why their combination constitutes a substantial methodological advance over existing recurrent architectures.
The systematic comparisons across looping strategies, model sizes, and resource budgets are potentially valuable. However, the authors should clarify which findings are genuinely new relative to existing scaling studies. Furthermore, the proposed PLT modifications should be evaluated using actual inference latency, throughput, and memory measurements to substantiate their practical efficiency claims.
Questions for authors
Could you provide actual inference latency, throughput, and memory measurements for the proposed modified PLT more specifically?
The proposed normalized PLT maintains separate KV caches for different loops, whereas the original PLT incorporates KV sharing. How does this design choice affect memory efficiency, particularly for long context lengths and larger loop counts?
Most evaluations focus on language modeling BPB and likelihood based downstream accuracy. Could the authors evaluate whether the proposed modifications improve actual autoregressive generation, particularly on mathematical reasoning, code generation, and longer generation tasks?
6 comments
Response to W1 - 2
Parallel looping. Parallel Loop Transformer for Efficient Test-Time Computation Scaling (Wu et al., 2025) introduces cross-loop parallelism, while LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling (Yang et al., 2026) reports declining performance beyond two loops. We investigate this limitation through ablations of the cross-token shift, normalization, state mixing, current-token access, and attention/cache mechanisms. We identify a substantial magnitude mismatch between the current-token embedding and shifted recurrent state. Separately normalizing the inputs, controlling their mixture, and preserving the current-token path recovers much of the lost quality. Our normalized shifted models generally improve or remain stable from two to four and eight loops (Section 3.4, Figure 7, Appendix G.1). Although the settings differ from LoopCoder-v2, this shows that useful parallel refinement need not end at two loops. Our 1B experiments also retain strong performance at two and four loops, with a remaining gap to the unshifted control at four loops (Figure 8, Appendix H).
We would like to note that we report measured decoding latency, throughput, and KV-cache memory in our response to the next question(s).
We therefore believe this study makes an important contribution by establishing when looping helps, which design choices matter, and how those choices depend on the available parameters, training data, and compute.
Response to Q1.and Q2.
Q1.and Q2. Could you provide actual inference latency, throughput, and memory measurements for the proposed modified PLT more specifically? The proposed normalized PLT maintains separate KV caches for different loops, whereas the original PLT incorporates KV sharing. How does this design choice affect memory efficiency, particularly for long context lengths and larger loop counts?
We address both questions in our response to Q3 by Reviewer HCDe, and summarize the main results here. Since the original PLT work already demonstrated inference benefits, our work focused on understanding which looping designs work and addressing limitations such as the degradation beyond two loops reported in LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling (Yang et al., 2026). Our normalization and input-mixing changes keep parallel looping useful at four and eight loops in the evaluated settings. Below we have benchmarked our modified PLT (20.0B-parameter model) in BF16 on a single NVIDIA GB300 GPU, using a 16k-token prompt.
Latency and Throughput: For a single sequence, standard four-loop execution takes 29.75 ms per generated token. Modified PLT reduces this to 10.86 ms with separate caches and 8.69 ms with a shared cache, close to the no-looping baseline of 8.24 ms. For a batch of eight sequences, total throughput increases from 226 tokens/s with standard looping to 489 with separate-cache PLT and 772 with shared-cache PLT, compared with 824 without looping. The shared-cache implementation also fuses attention across loops. Together, these changes provide approximately 3.4-fold improvements over standard looping while approaching the decoding performance of the no-looping baseline.
Cache memory savings. Standard looping and separate-cache PLT keep one KV cache for each loop. With four loops, this means storing four caches. Our modified PLT can also share a single cache across all loops, bringing cache memory back to the no-looping level. For the 20.0B BF16 model with one sequence containing 64,000 cached tokens, the calculated cache size is 5.86 GiB without looping. Four separate caches require 23.44 GiB, while four loops sharing one cache require only 5.86 GiB. Sharing therefore saves 75% of cache memory with four loops, or 50% with two. Note that the model weights occupy the same memory in all configurations.
We would also like to note that the speedups reported above apply to generating tokens after the input prompt has been processed. Processing the prefill, still takes longer with more loops. For a single 16k-token prompt, this takes 0.46 seconds without looping and 1.91 seconds with four-loop shared-cache PLT. Our response to Q3 provides the full timing and memory results.
Response to W1 - 1
We thank Reviewer xtRj for their careful review and constructive feedback, and for recognizing the breadth of our empirical investigation. Below we try to address some of the weaknesses and questions.
W1. The paper provides an extensive empirical investigation ... substantiate their practical efficiency claims. '
We agree that weight sharing, loop-specific adapters, and cross-loop parallelism are established mechanisms, and we acknowledge these precedents in Section 4 and Appendix A. Our contribution is the controlled empirical evidence about which looping designs to use, when specialization helps, and how parallel looping can remain useful beyond two loops. Establishing these findings required substantial experimental work across seven looping strategies, multiple loop depths, and different resource constraints. We compare all seven strategies at 70M–500M, with targeted experiments extending to 1B and training budgets up to 155B tokens. This breadth allows us to distinguish improvements in parameter efficiency from improvements in training-compute efficiency within a common model family. Our approach shares the motivation of Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., 2020) and Do Transformer Modifications Transfer Across Implementations and Applications? (Narang et al., 2021): systematic comparisons can establish which design choices work and under what conditions. We would like to note that the distinctions between our study and prior work are outlined in Section 4 and discussed in more detail in Appendix A. We summarize them again below to clarify how our experiments and findings differ from prior work.
Looping schedules and scaling. We compare alternative looping schedules within a common model family and training setup to identify architectural effects that prior scaling studies do not directly resolve. For example, our comparison of Full-Stack and Per-Layer Looping keeps the parameters and number of layer applications fixed while changing their execution order.
This complements Parcae: Scaling Laws for Stable Looped Language Models (Prairie et al., 2026), which studies the allocation of compute between recurrence and training data, and How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models (Schwethelm et al., 2026), which compares shared recurrence with independently parameterized depth. Hyperloop Transformers (Zeitoun et al., 2026) instead develops middle-stack recurrence with hyper-connections. These recently accepted COLM 2026 and NeurIPS 2026 papers are already discussed in Section 4 and Appendix A, and we view them as potentially concurrent work. Our experiments show why the architectural choices matter: Full-Stack Looping generally gives the strongest quality at fixed parameter and token budgets, while Middle-Stack Looping often offers a better return when training FLOPs are constrained. Thus, our comparisons establish how both the looping schedule and the resource constraint affect which design is preferable.
Loop-specific specialization. Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA (Bae et al., 2025) uses depth-specific adapters to recover performance when converting a pretrained Transformer into a compact recursive model. Our work is different as we train the shared weights and adapters together from initialization and study how their benefit changes with loop depth. We find small gains at two loops and clearer gains at four and eight, with relatively few additional parameters (Section 3.3, Figure 5, Appendix F). We also compare against added physical depth, which can achieve better BPB at comparable training FLOPs when additional parameter storage is available. This establishes when specialization helps over exact sharing while distinguishing its storage benefits from compute efficiency.
[Continue Below]