acceptodds
Under review as a conference paper at ICLR 2027

Loop-PP: Pipeline Parallelism for Looped Transformers

Abstract

Looped Transformers increase effective model depth and capacity by recursively applying shared blocks, but their recurrent execution is poorly matched to conventional pipeline parallelism (PP), which assumes a one-pass feed-forward execution order. When recurrent blocks span multiple pipeline stages, outputs from later stages feed back to earlier ones, disrupting conventional PP's fixed compute and communication orders. Supporting recurrence therefore requires globally rescheduling computation and communication. To this end, we present Loop-PP, the first pipeline-parallel training system designed for Looped Transformers. Its core, the Loop-Aware Pipeline Compiler, first makes repeated logical visits and recurrent dependencies explicit in a scheduling-independent Semantic Graph, then synthesizes a Global Compute Order under these semantic constraints, and finally compiles cross-stage dependencies into Per-Rank Action Streams that overlap communication with computation. Across 8B-60B models, Loop-PP maintains broadly comparable throughput to compute-matched baseline with interleaved 1F1B while reducing memory by up to 32.8%. Under an equal resource budget, it improves the best measured throughput by up to 12.7%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Pipeline Parallelism, Looped Transformers, Depth Recurrence, Distributed Training, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UXrWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper presents Loop-PP, a pipeline-parallel training system for looped/depth-recurrent transformers, whose shared layers are revisited across pipeline stages and therefore break the assumptions of standard pipeline schedules. Loop-PP unrolls the recurrence into a dependency graph of logical visits to shared layers, schedules it using interleaved 1F1B and compiles cross-stage transfers into static per-rank action streams in order to overlap communication with compute. On 8B-60B MoE models with two recurrent steps on 32 GPUs, the paper reports throughput comparable to an unrolled baseline with up to 32.8% lower memory, and up to 12.7% higher throughput under an equal resource budget.

Critical strengths
  1. The paper addresses the important current problem of increasing popularity of looped transformers while systems support for training them at scale is insufficient. It presents a clean and general compiler design and preserves the exact model structure with only one copy of the weights while supporting prelude/coda layouts and injection.
Critical weaknesses

My largest concern is that important baselines are missing:

(1) the most important one is standard interleaved pipeline parallelism with shared weights. For the full-loop layouts used in all main experiments, Loop-PP at VPP=V produces the same chunk placement, execution order and transfers as standard interleaved 1F1B at VPP=T*V on the unrolled model.

As a simple example with 4 unique layers, T=2 and PP2, compare: (a) standard interleaved PP with VPP2, where GPU0 holds unrolled layers 0-1 and 4-5, GPU1 holds layers 2-3 and 6-7, reuse the parameter tensors of layers 0-3, and layers 4-7 reuse the parameter tensors of layers 0-3. (b) Loop-PP with zero prelude and coda layers and VPP1, where GPU0 holds layer 0-1, GPU1 holds layers 2-3 and execution goes back and forth between the two GPUs twice.

Because the shared copies are on the same GPU, (a) trains exactly the same looped model with a single copy of the weights and needs no cross-stage synchronization, and it differs from the "Tied" baseline only in aliasing the parameters instead of copying them.

Please compare against (a) at the scale of the main experiments. The numbers in the manuscript suggest that (a) may be faster, because at PP8 on the 30B model, Loop-PP at VPP1 reaches ~70k tokens/s versus ~100k for "Unrolled" at VPP2. If (a) matches Loop-PP in memory and "Unrolled" in throughput, the benefit of your proposed method would be limited to layouts with a prelude/coda or injection even though its results are weakest for these configurations (see Figure 6).

(2) You also do not compare against non-pipeline strategies which is needed for showing that pipeline parallelism and hence Loop-PP is a good choice for training models at this scale. The manuscript motivates PP by memory, but looped models store only their unique parameters while their compute grows with T, so the memory case for PP is weaker than for standard models. The paper's own PP1 run in Appendix H.3 trains the 8B looped model on only 4 GPUs without recomputations. Please compare Loop-PP against data/expert parallelism with sharded optimizer or FSDP, as well as against tensor/sequence and context parallelism on the same 32 GPUs. Alternatively please show results at a scale where PP is clearly required, e.g. hundreds of GPUs across many nodes where PP's lower cross-node communication matters.

(3) Please compare against stronger pipeline schedules, because the "Unrolled" baseline only runs interleaved 1F1B while Zero Bubble and DualPipe schedules are the current state of the art.

(4) the "Tied" baseline is missing from the equal resource comparison in Table 1 which compares only against "Unrolled".

Questions for authors
  1. How does interleaved 1F1B at VPP=T*V compare to Loop-PP in throughput and peak memory at the scale of your main experiments?

  2. In the "Tied" baseline, are the two copies of each layer placed on the same GPU? How much of the difference between "Tied" and Loop-PP comes from duplicated weights and the copy/synchronization step?

  3. What throughput does training without pipeline parallelism achieve on the same scale using data and expert parallelism with a sharded optimizer or FSDP, for the 8B and 30B models? At what model size or GPU count do you expect pipeline parallelism to become necessary for looped models and can you provide an experiment at sufficient scale that shows this?

16 comments
AuthorsOpenReview ↗

Rebuttal-Part1

We appreciate the reviewer's thoughtful review and recognition of our efforts. We hope that the following responses can help clarify the reviewer’s concerns:

W1. The most important missing baseline is standard interleaved PP with shared weights; under full-loop the two are equivalent, so the gains are confined to prelude/coda/injection, which are exactly the weakest cases

The shared-weight interleaved baseline the reviewer points to is the same as the one in Reviewer 2's W1, and so is our response: it is not simple to implement and its range is narrow, which is why the original paper did not include the comparison. We have nevertheless implemented this baseline and compared against it, which further confirms our advantage.

Stock Megatron cannot make replicas that land on the same device share a single copy of the weights after unrolling. Making it run requires reusing part of the Loop-PP implementation, for example packing and forwarding the recurrent state along every hop. Even then it only covers the aligned full-loop case. Chunk c lands on rank c mod P, so only when each iteration occupies exactly a multiple of P chunks do all T uses of a layer sit on the same GPU and allow aliasing. A loop body shallower than PP, an unaligned prelude or coda, or an unequal number of chunks per stage all push the replicas onto different devices, leaving no option but to fall back to Tied. Loop-PP, Tied and Lockstep have no such restriction and accommodate any loop layout.

The full loop in Figure 1 does support this scheme. We implemented Alias and compared it against Loop-PP and Tied under the same compute-matched setting as Figure 1. The three tables below all use compute-matched fixed configurations, reported at each PP point with no configuration search, whereas the table in Q3 reports the best throughput found by searching PP / VPP / MBS / recomputation under a common memory budget. The two settings differ and the numbers cannot be compared directly. Alias uses less memory than Tied and still more than Loop-PP. Its throughput is close to Tied's, and Loop-PP is faster at every PP2 and PP4 point across all three scales, falling behind only at PP8. At PP8 Alias is close to Tied and slightly faster than Loop-PP, because under a common VPP protocol Loop-PP's chunks are only half as thick as the others', which makes overlapping computation with communication harder (see our response to Reviewer 1's W3). We realised this shared-weight interleaved baseline was missing, so we have added Alias to Appendix G.11 (Table 19) and to the baseline description in §4.1. Alias only applies to the aligned full structure and does not cover most prelude / core / coda partitions, where it degenerates structurally into Tied.

8B

PPLoop tok/sLoop MemAlias tok/sAlias MemTied tok/sTied Mem
2342k52.3309k55.8306k63.5
4282k50.3280k57.1278k63.6
8248k58.3257k58.5259k64.5

30B

PPLoop tok/sLoop MemAlias tok/sAlias MemTied tok/sTied Mem
2135k69.2119k74.3110k97.6
4115k67.0112k73.5108k89.7
8102k72.2110k78.9110k92.1

60B

PPLoop tok/sLoop MemAlias tok/sAlias MemTied tok/sTied Mem
296.3k83.883.6k88.3OOM—
483.9k73.882.7k80.582.5k111.3
872.6k79.676.3k80.676.5k110.8

As for the reviewer's point that "if Alias catches up with Loop-PP, the gains are confined to prelude/coda/injection, which are exactly the weakest cases", these two kinds of layout are not weak corner cases.

First, the full experiment in the main figure is itself a layout with injection: extra state such as the residual has to be carried between iterations, which is precisely the extra hops and extra residual fields that any looped layout pays, and Unrolled does not (the loop tax in our response to Reviewer 1's W3). Even after paying that communication cost, Loop-PP is still clearly better than Tied in Figure 1, and better than Alias at most points. In other words, injection is already the default setting of the main results, and the advantage we report is what remains after that communication has been paid for.

AuthorsOpenReview ↗

Rebuttal-Part2

Second, under a prelude/coda layout the loop structure brings back edges and extra hops. When the loop body spans several PP stages, the model has to go around the pipeline T−1 more times, adding roughly (T−1)P hops. This comes from the loop structure and no schedule can remove it. Comparing only against Unrolled is therefore unfair, since Unrolled has no loop and pays none of this tax. The fair comparison is Tied under the same loop function. We added a run on 16 GPUs with an 8B model and a 12+8×4+12 layout (T=4), which has a thinner body and more iterations than the T=2 full-loop. Unrolled is faster because it pays no loop tax. Tied, which uses the same function, is still slower than Loop-PP at PP2 and PP4, close at PP8, and Loop-PP uses markedly less memory. The table is in Appendix G.12 (Table 20).

PPLoop-pp tok/sLoop MemTied tok/sTied MemUnroll tok/sUnroll Mem
2149.6k56.9145.3k74.0154.9k69.0
4148.7k54.8139.9k80.8150.4k70.5
8117.4k57.8120.5k92.2136.4k70.8

We realised that prelude/coda and injection could be read as weak corner cases, and that the original paper did not report a thinner body with more iterations, so we have added the T=4 12+8×4+12 run to Appendix G.12 (Table 20). The Alias comparison is in Appendix G.11 (Table 19) and in the baseline description in §4.1.

W2. No comparison against non-PP strategies (DP/EP + FSDP, TP/SP, CP); looped models have fewer parameters, so the memory argument is weak

We thank the reviewer for raising the non-PP comparison. FSDP and the distributed optimizer are indeed important, and they are orthogonal dimensions to PP. We have run the corresponding comparison to show at what scale PP becomes necessary.

We built Loop-PP precisely because the mainstream recipe at large scale, and especially for MoE, is already PP+EP (plus data parallelism / ZeRO-1), and PP is exactly the dimension looped models lack. Megatron-LM assigns cross-node scaling to PP, since the allreduce of TP is latency-sensitive and only suited to within a node [1]. Later large MoEs fixed this into PP+EP: DeepSeek-V2 uses 16-way zero-bubble PP with 8-way EP [2], and DeepSeek-V3 uses 16-way PP with 64-way EP and ZeRO-1, builds DualPipe specifically for cross-node EP, and essentially avoids TP during training [3]. Kimi K2 (1T) in 2025 is still 16-way PP (with virtual pipelining), 16-way EP and ZeRO-1 [4], and NVIDIA's 2026 Megatron Core MoE report trains DeepSeek-V3 and Qwen3-235B with the same PP+EP recipe, recommending PP8–16 with EP for large cross-node MoEs [5]. Production systems at the ten-thousand-GPU scale also rely on PP to split depth [6]. Even for dense Llama 3 405B, cross-node is still 16-way PP, with TP8 + FSDP used only within a node [7]. EP splits experts and FSDP splits parameters, and neither splits layers along depth. Once a model spans nodes layer by layer, PP is almost always in play. A looped model that cannot connect to PP cannot enter the PP+EP combination that MoE training most commonly uses today.

All of our experiments already follow this combination: PP + EP + distributed optimizer. Loop-PP only supplies the missing piece, the pipeline schedule for looped models. Adding TP within a node, or layering FSDP on top, is orthogonal to PP, and the main experiments do not use them as baselines. Following the reviewer's request, however, we measured all three classes of non-PP strategy on the same 32 GPUs, with results in the two tables below.

The reviewer suggests that looped models have fewer parameters and therefore less memory pressure, which weakens the case for PP. What looping saves is weights and optimizer state, not activations. EP splits expert weights, and FSDP/ZeRO splits parameters, gradients and optimizer state [8], and neither touches activations. TP/SP and CP do shard activations, but along the hidden and sequence dimensions [9], and no dimension among them is split along depth. Activations grow roughly with the effective depth T·L and shrink with the PP degree, and the only things that can split them along depth are PP and recomputation. Looping multiplies the effective depth by T, so without PP every device has to hold the activations of all T·L layers. A looped model therefore needs PP more than an ordinary model does, not less.

This only becomes visible beyond a certain scale. The table below is the DP/EP comparison we added, with a sharded optimizer and with FSDP (32 GPUs, seq=4096, every cell is the best throughput that strategy reaches after independently searching batch size and recomputation policy under the same memory budget, from the same set of experiments as Q3):

AuthorsOpenReview ↗

Rebuttal-Part3

ScaleEP8 + distributed optimizerEP8 + FSDPLoop-PP
8B475.8k / 107.8 GiB438.2k / 101.9 GiB428k / 94.4 GiB
30B171.2k / 119.3 GiB154.2k / 115.3 GiB170.7k / 117.7 GiB
60B100.3k / 118.9 GiB88.5k / 119.8 GiB108.4k / 120.0 GiB
110BOOM27.8k / 103.0 GiB44.2k / 93.0 GiB

At 8B, running without PP is actually faster, and at 30B the two are essentially level. We include these two scales in the paper to show that the method holds across scales, not to claim that they need pipeline parallelism. The turning point appears at 60B: without PP, even with MBS reduced to 1, the configuration still runs out of memory unless recomputation is used, so it has to fall back to recomputing the entire MoE block in order to fit into a single node, which drags throughput down to 100.3k, below Loop-PP's 108.4k. The scale we actually care about is beyond a hundred billion parameters, where a single node cannot even hold the activations. At 110B, EP8 with a distributed optimizer goes straight to OOM, and while FSDP does run, it too must recompute the whole MoE block and reaches only 27.8k. This shows that beyond a hundred billion parameters PP is not optional but required, and PP is exactly the dimension looped models are missing.

The tensor/sequence parallelism and context parallelism the reviewer named behave consistently with this at 110B (again 32 GPUs, with each row taking the best configuration found for that strategy):

Non-PP strategyParameter shardingThroughputPeak memory
EP8distributed optimizerOOM—
EP8 + FSDPdistributed optimizer + FSDP27.8k103.0 GiB
TP8 + SP + EP8distributed optimizer15.1k111.1 GiB
CP2 / CP4 + EP8distributed optimizerOOM—
Loop-PP (PP4 VPP2)distributed optimizer44.2k93.0 GiB

The distributed optimizer shards optimizer state only, while FSDP additionally shards the bf16 weights and gradients.

TP8 uses 18 GiB more memory than Loop-PP and still reaches only a third of its throughput. Even backing off to the point where its memory matches Loop-PP's (94.3 GiB), it reaches only 14.8k. The reason is the communication accounting above: the collectives of TP/SP are paid at every layer visit, and looping multiplies the number of layer visits by T. CP does not fit on this model at all, because the same configuration without CP (that is, the EP8 row of the table) already runs out of memory, and CP only shards activations along the sequence, so it cannot reach the parameter side where the overflow actually is. Sharding weights and gradients as well, which is what FSDP does, is what makes that configuration run, but its throughput is still far below PP.

The above has been added to the paper, in the new §4.3, "When Is Pipeline Parallelism Necessary?", and Table 2.

References

[1] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, et al. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. SC, 2021.

[2] DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434, 2024.

[3] DeepSeek-AI. DeepSeek-V3 Technical Report. arXiv:2412.19437, 2024.

[4] Kimi Team. Kimi K2: Open Agentic Intelligence. arXiv:2507.20534, 2025.

[5] NVIDIA. Scalable Training of Mixture-of-Experts Models with Megatron Core. arXiv:2603.07685, 2026.

[6] Ziheng Jiang, Haibin Lin, Yinmin Zhong, et al. MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs. NSDI, 2024. arXiv:2402.15627.

[7] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The Llama 3 Herd of Models. arXiv:2407.21783, 2024.

[8] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. SC, 2020.

[9] Vijay Korthikanti, Jared Casper, Sangkug Lym, et al. Reducing Activation Recomputation in Large Transformer Models. MLSys, 2023.

W3. Stronger pipeline schedules should be compared against: Unrolled only ran interleaved 1F1B, whereas Zero Bubble and DualPipe are the current state of the art

We thank the reviewer for the suggestion. Zero Bubble and DualPipe are indeed important recent works on pipeline and parallel scheduling, but the overheads they target are not the same. Zero Bubble reclaims pipeline bubbles by splitting the backward pass, while DualPipe's main benefit comes from hiding the all-to-all of cross-node expert parallelism behind computation. We discuss each of these overheads under our setting below, and why neither schedule was included in the comparison.

Reviewer_DFT2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Loop-PP, a pipeline-parallel training system for looped (depth-recurrent) Transformers built on Megatron-LM. When a recurrent block spans multiple pipeline stages, outputs from later stages must feed back to earlier ones, which conventional PP schedules do not support. Loop-PP compiles the model in three steps. It first unrolls the recurrence into a Semantic Graph of logical visits that share one copy of the parameters. It then orders computation by constraining Megatron's interleaved 1F1B schedule with the graph's dependencies. Finally, it splits each transfer into nonblocking POST/WAIT actions and produces a static action stream for every rank, so that communication overlaps with computation. On synthetic 8B–60B MoE models looped twice, Loop-PP reduces peak memory by up to 32.8% compared with an unrolled baseline using independent parameters, with throughput ranging from faster at PP2 to slower at PP8. Under an equal 32-GPU budget, its best measured throughput is up to 12.7% higher.

Critical strengths
  1. Timely and relevant problem. Looped Transformers are an active research direction, but systems support for training them at scale is lacking. The paper clearly identifies why conventional PP breaks when a recurrent block spans multiple stages: the feedback from later stages to earlier ones does not fit the fixed one-pass schedule.

  2. Solid engineering that preserves semantics with a single parameter copy. The compiler keeps Megatron's interleaved 1F1B cadence, overlaps communication with computation through POST/WAIT actions, and reuses a single copy of the shared parameters. This yields up to 32.8% lower peak memory and enables the 60B PP2 configuration, where all baselines run out of memory.

  3. Thorough correctness validation and honest analysis. Loop-PP matches a PP1 oracle on per-microbatch losses, event tensors, gradients and SGD updates. The paper also openly analyzes where it loses to the baseline, such as the PP8 slowdown and less favorable recurrent layouts.

Critical weaknesses
  1. A simple unroll-based baseline is missing. A straightforward approach is to unroll the loop and apply standard interleaved PP, placing the copies of each layer from different iterations on the same GPU. With cyclic virtual-stage placement this happens naturally, and the copies can then share a single set of weights. The paper's Tied baseline instead keeps separate parameter copies and synchronizes them, but that is a limitation of Megatron's implementation, not of the simple approach. All main experiments use a full-loop layout (the whole layer stack looped twice), which this simple approach can already handle, so the necessity of the proposed compiler is not established. This baseline should be included, even if it requires modifying Megatron.

  2. The evaluated models may not reflect the architectures that matter in practice, and Loop-PP's performance depends strongly on the configuration. Large-scale looped Transformers are not yet established, so it is unclear whether the synthetic models (a full stack of 32–48 layers looped twice) represent what eventually needs to be optimized. Existing looped models vary widely: Huginn loops a 4-layer body about 32 times, while Ouro loops a 24–48-layer stack up to 4 times. Meanwhile, Loop-PP's speedup is highly sensitive to the configuration. It is faster than Unrolled at PP2 but slower at PP8, and also slower with T=4 at PP4 and with 25%/75% recurrent layouts. The small-body, many-iteration regime, where Loop-PP's overhead would likely be largest, is not evaluated.

Questions for authors

How does Loop-PP compare with a simple unroll-based baseline that applies standard interleaved PP to the unrolled model, places the copies of each layer from different iterations on the same GPU, and shares a single set of weights among them? What concrete advantages does the proposed compiler provide over this approach for the full-loop layouts evaluated in the paper?

7 comments
AuthorsOpenReview ↗

Rebuttal-Part1

We appreciate the reviewer's thoughtful review and recognition of our efforts. We hope that the following responses can help clarify the reviewer’s concerns:

W1. A simple unrolled baseline is missing (unroll + standard interleaved PP + cyclic placement so that the replicas share weights)

The baseline the reviewer proposes looks simple. It cannot be done in stock Megatron, it is not simple, and its range of applicability is narrow, which is why the original paper did not include the comparison. We have nevertheless implemented this baseline and compared against it, which further confirms our advantage.

Concretely, the scheme has to reuse part of what we wrote for Loop-PP, such as packing and forwarding the recurrent state along every hop. Even with all of that in place, the range in which it holds is still narrow, covering only aligned full-loop layouts. Sharing weights requires that all T uses of a layer sit on the same GPU. Under cyclic placement, chunk c lands on rank c mod P, so only when each iteration occupies exactly a multiple of P chunks does the same layer in the second iteration land on the same device as in the first, which is what lets the replicas alias into a single parameter. All of the following common partitions put the replicas of a layer on different devices, so aliasing is impossible and the scheme falls back to Tied's replication plus cross-stage synchronisation:

  • a loop body shallower than PP (for example a 4-layer body looped many times, placed on PP8);
  • prelude and coda not aligned to chunk boundaries, or an uneven number of layers per stage;
  • an unequal number of logical chunks per stage in the first place.

Loop-PP, Tied and Lockstep have no such restriction. They keep a single instance of each physical layer, the scheduler routes the computation back to the same device, and any partition still has only one copy of the parameters. They can therefore accommodate any loop layout. This is why the original paper did not include this baseline (Alias).

The full-loop layout used in Figure 1 does support the scheme, so we implemented this baseline, which we call Alias, and compared it against Loop-PP and Tied under aligned settings. The three tables below use compute-matched fixed configurations and report each PP point separately. This is a different setting from the table in Reviewer 3's Q3, which reports the best throughput found by searching under a common memory budget. Under matched compute, Alias does use less memory than Tied, but still more than Loop-PP. Its throughput is close to Tied's, and Loop-PP is faster at every PP2 and PP4 point across all three scales, falling behind only at PP8. At PP8 Alias is close to Tied and slightly faster than Loop-PP, because under a common VPP protocol Loop-PP's chunks are only half as thick as the others', which makes overlapping computation with communication harder (see our response to Reviewer 1's W3). Alias only applies to this kind of aligned full structure, and it does not cover most prelude / core / coda partitions, where it degenerates structurally into Tied.

8B

PPLoop tok/sLoop MemAlias tok/sAlias MemTied tok/sTied Mem
2342k52.3309k55.8306k63.5
4282k50.3280k57.1278k63.6
8248k58.3257k58.5259k64.5

30B

PPLoop tok/sLoop MemAlias tok/sAlias MemTied tok/sTied Mem
2135k69.2119k74.3110k97.6
4115k67.0112k73.5108k89.7
8102k72.2110k78.9110k92.1

60B

PPLoop tok/sLoop MemAlias tok/sAlias MemTied tok/sTied Mem
296.3k83.883.6k88.3OOM—
483.9k73.882.7k80.582.5k111.3
872.6k79.676.3k80.676.5k110.8

We realised this shared-weight interleaved baseline was missing from the original comparison, so we have added Alias to Appendix G.11 (Table 19) and to the baseline description in §4.1.

AuthorsOpenReview ↗

Rebuttal-Part2

W2. The evaluated models may not represent real architectures; performance is sensitive to configuration; a small body with many loop iterations was not tested

We evaluate the loop structures that are actually used, and the analysis experiments cover most loop structures. The full-loop setting in the main experiments takes its structure directly from Ouro [5]: the whole stack is looped, with a small number of iterations. This is currently the most complete and most widely cited looped language model, not a synthetic task. The 8B / 30B / 60B models correspond to scaling this structure up. Prelude and coda were also tested (see Figure 6(b)(c)), with the extra vectors that different loop structures need at the boundaries injected and packed accordingly. This already covers most uses of the form prelude + loop body + coda, with a small number of iterations and state carried across the boundaries.

We did not build a Huginn-style model with a very shallow body looped dozens of times, because real deployments will mostly not run that many iterations. More and more work places the useful number of iterations at T=2–4 rather than in the Huginn range, because the additional iterations bring diminishing returns while the computation grows with every one of them [1, 2, 3, 4]. Inference serving also has to pay for latency and cost, so the number of iterations will not be large. To look at the smaller-body, more-iterations regime anyway, we added a run on 16 GPUs with an 8B model and a 12+8×4+12 layout (T=4), which has a thinner body and more iterations than the T=2 full-loop. Unrolled is faster because it pays no loop tax. Tied, which uses the same function, is still slower than Loop-PP at PP2 and PP4, close at PP8, and Loop-PP uses markedly less memory. The table is in Appendix G.12 (Table 20).

PPLoop-pp tok/sLoop MemTied tok/sTied MemUnroll tok/sUnroll Mem
2149.6k56.9145.3k74.0154.9k69.0
4148.7k54.8139.9k80.8150.4k70.5
8117.4k57.8120.5k92.2136.4k70.8

There are two explainable reasons for the configuration sensitivity, and neither is random variation.

  • The first is load imbalance, compounded by fallback scheduling. In Figure 6, the effective work per stage at 25% and 75% under PP4 is not even (from the appendix: [12,8,8,12] at 25% and [12,16,16,12] at 75%), and the busy stage holds up the whole pipeline. Re-partitioning the layers by effective work rather than by nominal layer count would improve this.
  • The second is the communication cost of the loop structure itself: the extra hops and extra residual fields that any looped layout pays, and Unrolled does not (the loop tax in our response to Reviewer 1's W3). When the loop body spans several PP stages, larger T and P mean more transfers and more data per hop, which is harder to hide behind computation. Alias and Tied pay the same cost under an aligned layout, while Unrolled does not, which is why the gap to Unrolled widens as T and P grow. It does not mean the method only works at one hyperparameter point. The loop tax is identical between Loop-PP and Tied / Alias, so what it explains is the gap to Unrolled. Our gap to Tied and Alias at PP8 has a different cause, namely that at the same nominal VPP Loop-PP's chunks are only half as thick as theirs (see our response to Reviewer 1's W3).

We realised the original experiments did not show a thinner body with more iterations, so we have added the T=4 12+8×4+12 run to Appendix G.12 (Table 20).

I thank the authors for implementing the Alias baseline, which is exactly the comparison I requested. The results are informative. Alias already achieves memory close to Loop-PP (e.g., 74.3 vs. 69.2 GiB for 30B PP2), so most of the memory saving comes from weight sharing rather than from the proposed compiler. On throughput, Loop-PP leads at PP2, is roughly on par at PP4, and is slower at PP8.

However, I share Reviewer UXrW’s concern that this comparison is not granularity-matched. In the response to Reviewer y5wv, the authors acknowledge that the number of pipeline steps scales with VPP for Tied/Alias but with VPP×T for Loop-PP, so a common nominal VPP does not control execution granularity. Yet the Alias tables use the same nominal VPP for all methods, and the response to UXrW’s Q2 states that the pipeline hops are aligned, which contradicts this. The granularity difference is invoked only to explain the PP8 loss, but it may equally account for the PP2 gain, since Loop-PP then runs twice as many, finer chunks with a smaller bubble. For the full-loop layouts in the main experiments, Loop-PP at VPP = V and Alias at VPP = T·V have the same chunk placement, dependency chain and transfer count. The native interleaved table already satisfies the dependencies, so it is unclear what the scheduler would change at matched granularity. Relatedly, the question of why Loop-PP at VPP1 is about 28% slower than Unrolled at VPP2 at PP8 (71.3k vs. 99.7k tokens/s), despite identical chunk structures, remains unanswered. The equal-budget search in Table 1 does vary VPP, but it does not include Alias, so it does not isolate the scheduler from weight sharing either.

I would therefore ask the authors to compare Loop-PP at VPP = V against Alias at VPP = T·V, or to include Alias in the equal-budget search. I also note that Loop-PP’s native-table path requires equal logical chunk counts per stage (App. B.1), so the alignment restriction is not specific to Alias.

Regarding W2, I find the argument that practical looped models use T = 2–4 reasonable, and I appreciate the additional T = 4 run.

I will keep my score for now. If the authors can show that Loop-PP retains a clear advantage over Alias at VPP = T·V, together with a reasonable analysis of where this advantage comes from, I am willing to raise my score.

Reviewer_y5wvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Loop-PP is a pipeline-parallel training system for looped (depth-recurrent) Transformers. It works in three steps:

  1. It unrolls the recurrence into a "Semantic Graph".
  2. It orders the forward and backward work with a priority-guided Kahn topological sort, using Megatron's interleaved 1F1B order as the template.
  3. It compiles the remote dependencies into per-rank streams of nonblocking send/receive actions.

It is evaluated on 8B, 30B and 60B MoE models with T=2 on 32 GPUs.

Critical strengths
  1. The problem is real: standard 1F1B assumes each layer is visited once, which loops break.
  2. Correctness is validated carefully: the 5-step losses match a single-stage (PP1) recurrent reference bit for bit, and a 1,000-step run tracks it.
  3. The paper reports failures (OOM runs, the PP8 slowdown) and diagnoses them.
Critical weaknesses
  1. Modest novelty: Unrolling a finite recurrence gives a DAG. The method then takes a topological order with 1F1B priority and inserts nonblocking communication. This is standard DAG pipeline scheduling (compare GraphPipe), applied to an unrolled loop. The components are renamed with new terminology (Semantic Graph, Global Compute Order, Per-Rank Action Streams), but the core technical contribution is thin.
  2. The headline gains mostly come from not duplicating parameters, not from the scheduler.
  3. Most experiments use T = 2. The relative throughput drops as T grows, falling to 0.906 at T = 4, and it degrades with finer pipelines (PP8).
Questions for authors

See weaknesses.

6 comments

We appreciate the reviewer's thoughtful review and recognition of our efforts. We hope that the following responses can help clarify the reviewer’s concerns:

W1. Limited novelty: unroll into a DAG, then apply topological ordering, 1F1B priorities and non-blocking communication, and this is just an off-the-shelf DAG pipeline scheduler of the GraphPipe family

The operator graph after unrolling is a DAG, but that does not mean the physical stage graph of the pipeline is also a DAG. Methods such as GraphPipe are designed for stage DAGs, where stages have no cycles among them and each micro-batch passes through each stage exactly once, and they cannot be applied directly to our layout. Loop-PP does give the operators of different loop iterations distinct names, so the unrolled operator graph is acyclic, but once it is mapped onto GPUs, the same stage is entered several times within the same micro-batch, and the call relation among stages is cyclic. Keeping 1F1B fully occupied and keeping communication non-blocking on such a cyclic stage layout requires rebuilding the scheduling and the communication from end to end. It is not a matter of feeding a new input to an existing method.


There are in fact two kinds of graph in a training system. The reviewer may treat them as one.

  • Operator graph: the naming the scheduler uses to tell executions apart, so that it knows which pass comes first and which comes later. Loop-PP names the operators of layer-in-iteration-1 and layer-in-iteration-2 separately, so after unrolling there is no back edge between operators and the graph is a DAG.
  • Stage graph: which GPUs those operators actually land on, and in what order a micro-batch visits those GPUs. Pipeline scheduling policies really act on the stage graph, not on the operator graph.
    • GraphPipe, and Megatron 1F1B, assume this graph is acyclic: stage 0 forward → stage 1 forward. For a given sample, each stage runs forward once and backward once.
    • Loop-PP: stage 0 forward (loop 1) → stage 1 forward (loop 1) → back to stage 0 for another forward (loop 2) → stage 1 forward again (loop 2). The same stage is visited several times within one micro-batch, so a cycle exists.

GraphPipe-style methods only handle the case where the stage graph is a DAG. The difficulty of looping lies in the second graph. The operators carry different names, but they share the same set of weights and land on the same set of physical stages, so the stage invocations are cyclic. Unrolling the operator graph into a DAG does not restore GraphPipe's assumption.


A cyclic stage graph is not something that can be attached to an ordinary pipeline by adjusting an interface. Ordinary PP and looping are incompatible at almost every point here. Conversely, as long as the stage graph stays acyclic, the changes GraphPipe makes to topological order and priorities are in fact quite small, precisely because it still keeps a stage DAG.

Once cycles appear, the original 1F1B schedule simply stops working. The 1F1B timetable is drawn under the assumption that every stage does one forward and one backward per micro-batch, and it has no room for a stage that runs forward on the same micro-batch more than once. In the steady state of ordinary 1F1B, each stage performs one forward and one backward per slot, backward passes are given priority for earlier micro-batches, and the bubble is absorbed over two stages. With a cycle, the second-iteration forward of stage 0 on micro-batch m has to wait until stage 1 finishes the first iteration and sends the result back, and it competes for the same compute and communication slot as the backward that stage 0 owes to the earlier micro-batch m-1. Native 1F1B priorities only distinguish forward from backward, and have no notion of a first versus a second iteration of the same sample, so they will place a second-iteration forward at a point where its dependency has not arrived, or insert a backward between two forwards so that the activations no longer match. The warmup depth is no longer P-1 either. The first iteration has to traverse the whole pipeline and come back before the second can start, so the number of slots grows with T and P together. Running such a model against the acyclic timetable does not make it slower, it makes it wrong or deadlocked.

AuthorsOpenReview ↗

rebuttal-part2

Communication overlap and non-blocking communication cannot be carried over either. The non-blocking point-to-point primitives already in Megatron are built on acyclic stages. A stage only talks to its two neighbours, each direction performs roughly one send and one receive per slot, each transfer carries a single activation, and even and odd ranks post their operations in an interleaved order so that they do not wait on each other. Overlap means sending the previous activation while computing the next block. Once cycles appear, these assumptions no longer hold. First, the end of every iteration carries a forward edge from the last stage back to the first, a channel that simply does not exist in the forward pass of ordinary PP. Second, the same pair of ranks exchanges data repeatedly across iterations within a single sample, and carries extra fields such as the residual, so it is not one tensor per pair per slot. Third, the second-iteration compute of stage 0 depends on the whole pipeline completing the first iteration and sending the result back, so the window available for overlap is nothing like neighbour-to-neighbour hiding. Posting sends and receives early under the original rules leaves the peer out of order, and the result is a deadlock, or worse, a run that appears healthy while the data is already misaligned. Adding non-blocking communication is therefore not a switch to be flipped in a toolbox. It requires specifying afresh, under cycles, multiple iterations and multiple fields, who posts a send first, who posts a receive first, how many outstanding transfers are allowed at a time, and how to guarantee that the schedule does not lock against itself.


What Loop-PP does is therefore a redesign aimed at cyclic stages, not a new front end bolted onto GraphPipe's pipeline.

  • On the scheduling side, we no longer assume that each sample reaches each device only once. We generate a timetable from the loop semantics that allows the same stage to be visited several times. We still keep the warmup / steady-state / cooldown shape and the bubble profile of ordinary 1F1B as far as possible, but the dependencies around every single visit have to be written into the table. When a request conflicts with the loop semantics we raise an error, rather than quietly falling back to running T times locally on each device, which would compute the wrong thing.
  • On the communication side, we separate posting from waiting, pair the operations by rank parity and by the order of each iteration, limit the number of outstanding sends and receives, and pack the hidden state together with the residual into a single transfer. The appendix gives a conditional deadlock-freedom argument for this, together with compile-time checks. At runtime we also have to supply what ordinary PP never needed, namely support for the same module being entered several times on the same sample: every visit keeps its own record for the backward pass; the backward pass completes a full layer per visit, instead of caching one deferred gradient per layer where the visits would overwrite each other; the gradient counts for data parallelism, the parameters seen by the optimizer and the layer indices used at checkpointing are all aligned to multiple visits; and the routing statistics and auxiliary losses of the MoE are accumulated per visit, not recorded only for the last iteration.

Taken together, this is what allows a looped model with a single copy of the weights to run on a pipeline at a concurrency close to ordinary 1F1B. Going the route the reviewer has in mind, unrolling plus an off-the-shelf DAG scheduler, corresponds to Tied, which replicates the parameters to turn the stage graph back into an acyclic one, at the cost of giving up exactly what the looped model was supposed to save: parameters, optimizer state and memory. Loop-PP chooses not to pay for that replication, and so it has to deal with a cyclic stage graph. The operator graph being a DAG does not remove that obligation.

We realise that our earlier version may have lacked an explanation of this point and so misled the reviewer. Related Work now adds a short note locating GraphPipe: unrolling can make the operator graph a DAG, but the stage graph stays cyclic, which is the setting GraphPipe does not cover.

AuthorsOpenReview ↗

rebuttal-part3

W2. The headline gains come mainly from not replicating parameters, not from the scheduler

We agree that the memory saving comes from parameter reuse, and that this is the selling point of looped models to begin with. The issue is that no existing pipeline can run a single copy of the weights with the same stage visited several times by the same sample, as we explain in our response to W1. For a single copy of the parameters to run at native concurrency, the scheduler first has to rebuild 1F1B and non-blocking communication into something that supports looping, and do so without replicating parameters. Without that step, the memory advantage never turns into throughput.

This is why the only off-the-shelf baselines the paper can compare against directly are compute-matched Unrolled and Tied, both of which use replication to turn the stage graph back into a DAG. These two measure the overall effect of a recurrent function with one copy of the weights against unrolling and replicating. They do not isolate the contribution of the scheduler. The scheduler should be judged at a fixed parameter count. Two such reference points exist. The conservative Lockstep keeps one copy of the weights and is 1.9× slower than Loop-PP at 30B with PP2 and more than 6× slower at PP8, and Alias, on an aligned layout, keeps one copy of the weights while running the native interleaved schedule. Evaluating the scheduler against Unrolled, which carries T times the parameters, measures something else.

One further point: Figure 1 uses compute-matched, fixed configurations and does not saturate memory at small PP, so it is not a case of trading the saved memory for lighter recomputation or a shallower pipeline. Under that setting Loop-PP is still no worse than Unrolled and Tied at small PP, and better at several points, which shows that the benefit is not confined to the memory-saturated regime.

W3. Most experiments use T = 2. The relative throughput drops as T grows, falling to 0.906 at T = 4, and it degrades with finer pipelines (PP8).

The main experiments focus on T=2, and the analysis experiments cover T=2–4, because this is the range in which looped models are actually useful.

Several independent works point to the same observation: the second loop contributes the most, and the returns diminish, or even reverse, beyond it.

  • SMELT finds that, with compute, parameters and KV matched, looping the middle layers twice is best, and going to three or four requires making the model narrower, at which point the benefit stops growing and can decline [1].
  • PLT and LoopCoder-v2 show that two iterations improve substantially over a non-looped baseline, while three or four regress noticeably, because the second iteration does most of the refinement and the later updates become small and start to oscillate [2, 3].
  • A monitoring study of existing looped LMs [4] reports that the Ouro base model peaks at roughly three to four iterations and degrades after that.

We observed the same diminishing returns when training our own models, with two to three loops working best. The T=2–4 range used in the paper therefore covers the range that looping work actually uses.

As for T=4 being slower than Unrolled, the reason is that a looped model pays a loop tax that an ordinary model does not, and that tax grows with both T and PP.

  • The first part of the tax is the back edges and the extra hops. When the loop body spans several PP stages, any model with a loop structure has to go around the pipeline T−1 more times, which adds roughly (T−1)P hops.
  • The second part is the communication cost of the extra inputs. Many looped models re-inject additional inputs at the start of each iteration, for example the previous residual, so the transfers between PP stages carry not just the hidden state but those extra inputs as well.

These costs do not come from the scheduler, they come from the loop structure, and no schedule can remove them. The model used by Unrolled is an entirely ordinary model, with no looping and no extra residual transfers, so it pays no loop tax at all. The genuinely fair comparison is against Tied, which has the same loop structure.

We compared Loop-PP, Tied and Unrolled on 16 GPUs with an 8B model and a 12+8×4+12 layout (T=4). Unrolled is indeed faster. But comparing against Tied, which pays the same loop tax, Tied is still slower than Loop-PP at PP2 and PP4, the two are close at PP8, and Loop-PP uses markedly less memory, which shows that our schedule still has the advantage. This is a new compute-matched, fixed-configuration run; the table is in Appendix G.12 (Table 20).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.