- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper presents Loop-PP, a pipeline-parallel training system for looped/depth-recurrent transformers, whose shared layers are revisited across pipeline stages and therefore break the assumptions of standard pipeline schedules. Loop-PP unrolls the recurrence into a dependency graph of logical visits to shared layers, schedules it using interleaved 1F1B and compiles cross-stage transfers into static per-rank action streams in order to overlap communication with compute. On 8B-60B MoE models with two recurrent steps on 32 GPUs, the paper reports throughput comparable to an unrolled baseline with up to 32.8% lower memory, and up to 12.7% higher throughput under an equal resource budget.
Critical strengths
- The paper addresses the important current problem of increasing popularity of looped transformers while systems support for training them at scale is insufficient. It presents a clean and general compiler design and preserves the exact model structure with only one copy of the weights while supporting prelude/coda layouts and injection.
Critical weaknesses
My largest concern is that important baselines are missing:
(1) the most important one is standard interleaved pipeline parallelism with shared weights. For the full-loop layouts used in all main experiments, Loop-PP at VPP=V produces the same chunk placement, execution order and transfers as standard interleaved 1F1B at VPP=T*V on the unrolled model.
As a simple example with 4 unique layers, T=2 and PP2, compare: (a) standard interleaved PP with VPP2, where GPU0 holds unrolled layers 0-1 and 4-5, GPU1 holds layers 2-3 and 6-7, reuse the parameter tensors of layers 0-3, and layers 4-7 reuse the parameter tensors of layers 0-3. (b) Loop-PP with zero prelude and coda layers and VPP1, where GPU0 holds layer 0-1, GPU1 holds layers 2-3 and execution goes back and forth between the two GPUs twice.
Because the shared copies are on the same GPU, (a) trains exactly the same looped model with a single copy of the weights and needs no cross-stage synchronization, and it differs from the "Tied" baseline only in aliasing the parameters instead of copying them.
Please compare against (a) at the scale of the main experiments. The numbers in the manuscript suggest that (a) may be faster, because at PP8 on the 30B model, Loop-PP at VPP1 reaches ~70k tokens/s versus ~100k for "Unrolled" at VPP2. If (a) matches Loop-PP in memory and "Unrolled" in throughput, the benefit of your proposed method would be limited to layouts with a prelude/coda or injection even though its results are weakest for these configurations (see Figure 6).
(2) You also do not compare against non-pipeline strategies which is needed for showing that pipeline parallelism and hence Loop-PP is a good choice for training models at this scale. The manuscript motivates PP by memory, but looped models store only their unique parameters while their compute grows with T, so the memory case for PP is weaker than for standard models. The paper's own PP1 run in Appendix H.3 trains the 8B looped model on only 4 GPUs without recomputations. Please compare Loop-PP against data/expert parallelism with sharded optimizer or FSDP, as well as against tensor/sequence and context parallelism on the same 32 GPUs. Alternatively please show results at a scale where PP is clearly required, e.g. hundreds of GPUs across many nodes where PP's lower cross-node communication matters.
(3) Please compare against stronger pipeline schedules, because the "Unrolled" baseline only runs interleaved 1F1B while Zero Bubble and DualPipe schedules are the current state of the art.
(4) the "Tied" baseline is missing from the equal resource comparison in Table 1 which compares only against "Unrolled".
Questions for authors
-
How does interleaved 1F1B at VPP=T*V compare to Loop-PP in throughput and peak memory at the scale of your main experiments?
-
In the "Tied" baseline, are the two copies of each layer placed on the same GPU? How much of the difference between "Tied" and Loop-PP comes from duplicated weights and the copy/synchronization step?
-
What throughput does training without pipeline parallelism achieve on the same scale using data and expert parallelism with a sharded optimizer or FSDP, for the 8B and 30B models? At what model size or GPU count do you expect pipeline parallelism to become necessary for looped models and can you provide an experiment at sufficient scale that shows this?
16 comments
Rebuttal-Part2
Second, under a prelude/coda layout the loop structure brings back edges and extra hops. When the loop body spans several PP stages, the model has to go around the pipeline T−1 more times, adding roughly (T−1)P hops. This comes from the loop structure and no schedule can remove it. Comparing only against Unrolled is therefore unfair, since Unrolled has no loop and pays none of this tax. The fair comparison is Tied under the same loop function. We added a run on 16 GPUs with an 8B model and a 12+8×4+12 layout (T=4), which has a thinner body and more iterations than the T=2 full-loop. Unrolled is faster because it pays no loop tax. Tied, which uses the same function, is still slower than Loop-PP at PP2 and PP4, close at PP8, and Loop-PP uses markedly less memory. The table is in Appendix G.12 (Table 20).
| PP | Loop-pp tok/s | Loop Mem | Tied tok/s | Tied Mem | Unroll tok/s | Unroll Mem |
|---|---|---|---|---|---|---|
| 2 | 149.6k | 56.9 | 145.3k | 74.0 | 154.9k | 69.0 |
| 4 | 148.7k | 54.8 | 139.9k | 80.8 | 150.4k | 70.5 |
| 8 | 117.4k | 57.8 | 120.5k | 92.2 | 136.4k | 70.8 |
We realised that prelude/coda and injection could be read as weak corner cases, and that the original paper did not report a thinner body with more iterations, so we have added the T=4 12+8×4+12 run to Appendix G.12 (Table 20). The Alias comparison is in Appendix G.11 (Table 19) and in the baseline description in §4.1.
W2. No comparison against non-PP strategies (DP/EP + FSDP, TP/SP, CP); looped models have fewer parameters, so the memory argument is weak
We thank the reviewer for raising the non-PP comparison. FSDP and the distributed optimizer are indeed important, and they are orthogonal dimensions to PP. We have run the corresponding comparison to show at what scale PP becomes necessary.
We built Loop-PP precisely because the mainstream recipe at large scale, and especially for MoE, is already PP+EP (plus data parallelism / ZeRO-1), and PP is exactly the dimension looped models lack. Megatron-LM assigns cross-node scaling to PP, since the allreduce of TP is latency-sensitive and only suited to within a node [1]. Later large MoEs fixed this into PP+EP: DeepSeek-V2 uses 16-way zero-bubble PP with 8-way EP [2], and DeepSeek-V3 uses 16-way PP with 64-way EP and ZeRO-1, builds DualPipe specifically for cross-node EP, and essentially avoids TP during training [3]. Kimi K2 (1T) in 2025 is still 16-way PP (with virtual pipelining), 16-way EP and ZeRO-1 [4], and NVIDIA's 2026 Megatron Core MoE report trains DeepSeek-V3 and Qwen3-235B with the same PP+EP recipe, recommending PP8–16 with EP for large cross-node MoEs [5]. Production systems at the ten-thousand-GPU scale also rely on PP to split depth [6]. Even for dense Llama 3 405B, cross-node is still 16-way PP, with TP8 + FSDP used only within a node [7]. EP splits experts and FSDP splits parameters, and neither splits layers along depth. Once a model spans nodes layer by layer, PP is almost always in play. A looped model that cannot connect to PP cannot enter the PP+EP combination that MoE training most commonly uses today.
All of our experiments already follow this combination: PP + EP + distributed optimizer. Loop-PP only supplies the missing piece, the pipeline schedule for looped models. Adding TP within a node, or layering FSDP on top, is orthogonal to PP, and the main experiments do not use them as baselines. Following the reviewer's request, however, we measured all three classes of non-PP strategy on the same 32 GPUs, with results in the two tables below.
The reviewer suggests that looped models have fewer parameters and therefore less memory pressure, which weakens the case for PP. What looping saves is weights and optimizer state, not activations. EP splits expert weights, and FSDP/ZeRO splits parameters, gradients and optimizer state [8], and neither touches activations. TP/SP and CP do shard activations, but along the hidden and sequence dimensions [9], and no dimension among them is split along depth. Activations grow roughly with the effective depth T·L and shrink with the PP degree, and the only things that can split them along depth are PP and recomputation. Looping multiplies the effective depth by T, so without PP every device has to hold the activations of all T·L layers. A looped model therefore needs PP more than an ordinary model does, not less.
This only becomes visible beyond a certain scale. The table below is the DP/EP comparison we added, with a sharded optimizer and with FSDP (32 GPUs, seq=4096, every cell is the best throughput that strategy reaches after independently searching batch size and recomputation policy under the same memory budget, from the same set of experiments as Q3):
Rebuttal-Part3
| Scale | EP8 + distributed optimizer | EP8 + FSDP | Loop-PP |
|---|---|---|---|
| 8B | 475.8k / 107.8 GiB | 438.2k / 101.9 GiB | 428k / 94.4 GiB |
| 30B | 171.2k / 119.3 GiB | 154.2k / 115.3 GiB | 170.7k / 117.7 GiB |
| 60B | 100.3k / 118.9 GiB | 88.5k / 119.8 GiB | 108.4k / 120.0 GiB |
| 110B | OOM | 27.8k / 103.0 GiB | 44.2k / 93.0 GiB |
At 8B, running without PP is actually faster, and at 30B the two are essentially level. We include these two scales in the paper to show that the method holds across scales, not to claim that they need pipeline parallelism. The turning point appears at 60B: without PP, even with MBS reduced to 1, the configuration still runs out of memory unless recomputation is used, so it has to fall back to recomputing the entire MoE block in order to fit into a single node, which drags throughput down to 100.3k, below Loop-PP's 108.4k. The scale we actually care about is beyond a hundred billion parameters, where a single node cannot even hold the activations. At 110B, EP8 with a distributed optimizer goes straight to OOM, and while FSDP does run, it too must recompute the whole MoE block and reaches only 27.8k. This shows that beyond a hundred billion parameters PP is not optional but required, and PP is exactly the dimension looped models are missing.
The tensor/sequence parallelism and context parallelism the reviewer named behave consistently with this at 110B (again 32 GPUs, with each row taking the best configuration found for that strategy):
| Non-PP strategy | Parameter sharding | Throughput | Peak memory |
|---|---|---|---|
| EP8 | distributed optimizer | OOM | — |
| EP8 + FSDP | distributed optimizer + FSDP | 27.8k | 103.0 GiB |
| TP8 + SP + EP8 | distributed optimizer | 15.1k | 111.1 GiB |
| CP2 / CP4 + EP8 | distributed optimizer | OOM | — |
| Loop-PP (PP4 VPP2) | distributed optimizer | 44.2k | 93.0 GiB |
The distributed optimizer shards optimizer state only, while FSDP additionally shards the bf16 weights and gradients.
TP8 uses 18 GiB more memory than Loop-PP and still reaches only a third of its throughput. Even backing off to the point where its memory matches Loop-PP's (94.3 GiB), it reaches only 14.8k. The reason is the communication accounting above: the collectives of TP/SP are paid at every layer visit, and looping multiplies the number of layer visits by T. CP does not fit on this model at all, because the same configuration without CP (that is, the EP8 row of the table) already runs out of memory, and CP only shards activations along the sequence, so it cannot reach the parameter side where the overflow actually is. Sharding weights and gradients as well, which is what FSDP does, is what makes that configuration run, but its throughput is still far below PP.
The above has been added to the paper, in the new §4.3, "When Is Pipeline Parallelism Necessary?", and Table 2.
References
[1] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, et al. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. SC, 2021.
[2] DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434, 2024.
[3] DeepSeek-AI. DeepSeek-V3 Technical Report. arXiv:2412.19437, 2024.
[4] Kimi Team. Kimi K2: Open Agentic Intelligence. arXiv:2507.20534, 2025.
[5] NVIDIA. Scalable Training of Mixture-of-Experts Models with Megatron Core. arXiv:2603.07685, 2026.
[6] Ziheng Jiang, Haibin Lin, Yinmin Zhong, et al. MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs. NSDI, 2024. arXiv:2402.15627.
[7] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The Llama 3 Herd of Models. arXiv:2407.21783, 2024.
[8] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. SC, 2020.
[9] Vijay Korthikanti, Jared Casper, Sangkug Lym, et al. Reducing Activation Recomputation in Large Transformer Models. MLSys, 2023.
W3. Stronger pipeline schedules should be compared against: Unrolled only ran interleaved 1F1B, whereas Zero Bubble and DualPipe are the current state of the art
We thank the reviewer for the suggestion. Zero Bubble and DualPipe are indeed important recent works on pipeline and parallel scheduling, but the overheads they target are not the same. Zero Bubble reclaims pipeline bubbles by splitting the backward pass, while DualPipe's main benefit comes from hiding the all-to-all of cross-node expert parallelism behind computation. We discuss each of these overheads under our setting below, and why neither schedule was included in the comparison.
Rebuttal-Part1
We appreciate the reviewer's thoughtful review and recognition of our efforts. We hope that the following responses can help clarify the reviewer’s concerns:
W1. The most important missing baseline is standard interleaved PP with shared weights; under full-loop the two are equivalent, so the gains are confined to prelude/coda/injection, which are exactly the weakest cases
The shared-weight interleaved baseline the reviewer points to is the same as the one in Reviewer 2's W1, and so is our response: it is not simple to implement and its range is narrow, which is why the original paper did not include the comparison. We have nevertheless implemented this baseline and compared against it, which further confirms our advantage.
Stock Megatron cannot make replicas that land on the same device share a single copy of the weights after unrolling. Making it run requires reusing part of the Loop-PP implementation, for example packing and forwarding the recurrent state along every hop. Even then it only covers the aligned full-loop case. Chunk
clands on rankc mod P, so only when each iteration occupies exactly a multiple ofPchunks do allTuses of a layer sit on the same GPU and allow aliasing. A loop body shallower than PP, an unaligned prelude or coda, or an unequal number of chunks per stage all push the replicas onto different devices, leaving no option but to fall back to Tied. Loop-PP, Tied and Lockstep have no such restriction and accommodate any loop layout.The full loop in Figure 1 does support this scheme. We implemented Alias and compared it against Loop-PP and Tied under the same compute-matched setting as Figure 1. The three tables below all use compute-matched fixed configurations, reported at each PP point with no configuration search, whereas the table in Q3 reports the best throughput found by searching PP / VPP / MBS / recomputation under a common memory budget. The two settings differ and the numbers cannot be compared directly. Alias uses less memory than Tied and still more than Loop-PP. Its throughput is close to Tied's, and Loop-PP is faster at every PP2 and PP4 point across all three scales, falling behind only at PP8. At PP8 Alias is close to Tied and slightly faster than Loop-PP, because under a common VPP protocol Loop-PP's chunks are only half as thick as the others', which makes overlapping computation with communication harder (see our response to Reviewer 1's W3). We realised this shared-weight interleaved baseline was missing, so we have added Alias to Appendix G.11 (Table 19) and to the baseline description in §4.1. Alias only applies to the aligned full structure and does not cover most prelude / core / coda partitions, where it degenerates structurally into Tied.
8B
30B
60B
As for the reviewer's point that "if Alias catches up with Loop-PP, the gains are confined to prelude/coda/injection, which are exactly the weakest cases", these two kinds of layout are not weak corner cases.
First, the full experiment in the main figure is itself a layout with injection: extra state such as the residual has to be carried between iterations, which is precisely the extra hops and extra residual fields that any looped layout pays, and Unrolled does not (the loop tax in our response to Reviewer 1's W3). Even after paying that communication cost, Loop-PP is still clearly better than Tied in Figure 1, and better than Alias at most points. In other words, injection is already the default setting of the main results, and the advantage we report is what remains after that communication has been paid for.