acceptodds
Under review as a conference paper at ICLR 2027

PipeTempo: Stagewise Accumulation for Asynchronous Pipeline-Parallel Training

Abstract

In GPU-rich language-model pretraining, large synchronous batches can improve hardware utilization but leave fewer optimizer updates within a fixed token budget. We show that more frequent asynchronous updates can improve model quality in this regime, despite using stale gradients. We introduce PipeTempo, which sets each pipeline stage's accumulation interval from its local delay. Later stages use smaller batches and update more often, enabling more frequent updates than PipeDream-2BW while limiting local staleness to one update and retaining at most two weight versions per stage. We generalize the convergence analysis of stochastic conditional-gradient updates with momentum to coupled pipeline stages with arbitrary fixed accumulation intervals. Under an expected -Kurdyka–Łojasiewicz condition, we prove convergence for both PipeTempo and PipeDream, preserving the synchronous method's leading stochastic complexity for expected loss gap . Language-model pretraining experiments with Muon on FineWeb-Edu show lower validation loss than synchronous training, PipeDream, and PipeDream-2BW at matched token budgets in the large-batch regime.

Primary area:
optimization
Keywords:
pipeline parallelism, asynchronous optimization, gradient accumulation, large-batch training, Muon, conditional gradient methods, PipeDream, PipeDream-2BW

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DCqtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether more frequent asynchronous updates can improve language-model pretraining when large synchronous batches leave too few optimizer steps within a fixed token budget. It introduces PipeTempo, which sets each pipeline stage’s accumulation interval according to its local delay. Later stages consequently use smaller update batches, while the schedule limits local staleness to one update and retains at most two weight versions per stage. The paper extends the convergence analysis of stochastic conditional-gradient updates with momentum to coupled pipeline stages with arbitrary fixed accumulation intervals. Experiments with Muon on FineWeb-Edu demonstrate lower validation loss in the large-batch regime.

Critical strengths

The accumulation rule has a clear justification in terms of update frequency, staleness, and weight storage. PipeTempo exploits differences in local delay instead of imposing a common accumulation window across all stages. Proposition 1 establishes that the proposed intervals are the shortest fixed choices that limit local staleness to one update while requiring at most two weight versions. This provides a principled and simple extension of existing asynchronous pipeline schedules.

Critical weaknesses
  1. The evaluation is limited in scale and lacks essential system measurements. Experiments use only 135M and 360M parameter models, leaving the applicability to substantially larger models insufficiently validated. More importantly, the paper reports validation loss at matched token budgets but provides no measurements of throughput, peak GPU memory, or training time. These quantities are central to assessing a distributed training method, especially because more frequent optimizer updates and weight prediction may introduce additional overhead.

  2. The synchronous pipeline comparison is incomplete. The evaluation lacks comparisons with stronger synchronous schedules, suc as Zero Bubble Pipeline Parallelism. These schedules may change the relationship between pipeline utilization and the required update batch. Without comparisons under common hardware and memory constraints, it remains unclear whether PipeTempo’s advantage persists against a well-configured synchronous baseline or primarily mitigates the large batch penalty of the selected configuration.

Questions for authors

None

1 comment

We thank the reviewer for their time and detailed review. Below, we respond to the concerns raised in the review.

  1. The evaluation is limited in scale and lacks essential system measurements. [...]

We agree that system measurements and experiments with larger models are important. We ran them; please see the general response and Appendix I. In short: we implemented the methods in NVIDIA Megatron-LM and measured a 1.09B model on up to 64 H100 GPUs. On our main 32-GPU mesh, PipeTempo+EF runs within 2% of PipeDream-2BW, faster than Megatron's synchronous schedule and than Zero Bubble; its peak memory is within 0.7GB of the synchronous one at the same microbatch; and at budgets set for equal wall-clock time it reaches a lower validation loss than synchronous training at every batch size we tested.

  1. The synchronous pipeline comparison is incomplete. The evaluation lacks comparisons with stronger synchronous schedules, suc as Zero Bubble Pipeline Parallelism. These schedules may change the relationship between pipeline utilization and the required update batch. Without comparisons under common hardware and memory constraints, it remains unclear whether PipeTempo’s advantage persists against a well-configured synchronous baseline or primarily mitigates the large batch penalty of the selected configuration.

We would first like to clarify what the synchronous baseline represents in the matched-token experiments. Section 5.1 defines it as synchronous training with an update batch of BB sequences, without prescribing how these updates are executed across devices. It can therefore represent any synchronous pipeline, including Zero Bubble, or even data-parallel training without pipeline parallelism (PP=1\mathrm{PP}=1). A different execution schedule can change throughput, but does not change the underlying synchronous update rule at a fixed batch size.

This freedom to choose a synchronous configuration does not, however, make the large-batch penalty merely an artifact of our selected setup. Four recent training reports (lines 94–100) describe the tension between using a large GPU cluster efficiently and keeping update batches small enough for effective optimization. It is reasonable to expect that these practitioners carefully configured their training systems, yet they still encountered this tradeoff. Mitigating it is precisely the motivation of our work. PipeTempo aims to address it through more frequent updates with smaller batches, accepting some gradient staleness while keeping the same concurrent batch size as the baseline (lines 104–105). We do not expect a benefit when synchronous training can already use a near-optimal update batch (line 431).

Our rationale for comparing at matched tokens is that, in the pipeline setting, keeping the same concurrent batch should allow approximately the same utilization, while avoiding pipeline flushes can improve throughput. However, as the reviewer and the other reviewers pointed out, more frequent optimizer steps may offset this benefit. We therefore measured their overhead and compared PipeTempo with tuned Zero Bubble under common hardware and memory constraints; these wall-clock measurements are reported in the general response and Appendix I.

We would be glad to continue the discussion and clarify any remaining concerns. For now, we believe we have addressed all of the reviewer’s concerns, none of which points to a serious issue with our approach. Therefore, we would kindly ask the reviewer to reconsider their score.

Reviewer_CKhJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PipeTempo, an asynchronous pipeline-training method that lets different stages update at different rates. Later stages accumulate smaller microbatches because their gradients arrive earlier. The method limits gradients to at most single local update of staleness and requires at most two weight versions per stage. The authors provide a convergence analysis for SCG and experiments with Muon showing improved training quality at large batches.

Critical strengths

I like the simple approach yet well-motivated design of this work. The paper also provided useful theoretical contribution. The analysis handles coupled pipeline stages with different accumulation intervals, rather than treating each stage as an independent optimization problem.

This paper also provided careful optimization experiments. Learning rates and baseline momentum are tuned, and the evaluation includes two model sizes (though same familiy--see below weakness), two pipeline depths, and different training budgets. The zero-delay reference helps examine the cost of stale gradients. The weight-prediction ablations also provide useful evidence that applying prediction to every stage can hurt training.

Critical weaknesses
  • The practical efficiency benefit is not demonstrated. Efficient use of a large GPU cluster is central to the motivation, but the reported comparisons focus on validation loss at matched token budgets. The paper does not report wall-clock training time, throughput, or measured memory usage. More frequent optimizer updates and weight prediction introduce additional work, so matching the number of sequences in flight does not establish comparable training speed. Time-to-target-loss measurements would make the practical benefit much clearer. I feel like this is kinda important missing piece and would surely bump the score if we have this. > Author updated with empirical speedup report.

  • Experiments use 135M and 360M models from the same family, with Muon as the main optimizer. These results support the optimization idea, but its usefulness for large-scale pretraining remains uncertain. Larger models may have different communication, memory, and optimizer costs, which could change the overall trade-off. (esp with the claim of "optimizer-independent") > The clarification resolves the concern, but it has to be updated clearly on this scope to avoid reader's misunderstanding.

Questions for authors

Prior work already adjusts learning rates / decay or uses gradient accumulation to mitigate staleness. Could the authors compare PipeTempo against a baseline that uses smaller learning rates for stages with greater delay, while keeping update frequency unchanged? This would help establish the benefit of changing accumulation intervals beyond existing delay-aware update adjustments.

3 comments

We appreciate the reviewer's time and careful assessment of our work. We address their concerns and questions below.

The practical efficiency benefit is not demonstrated. [...] The paper does not report wall-clock training time, throughput, or measured memory usage. [...] Time-to-target-loss measurements would make the practical benefit much clearer.

We have added the systems evaluation requested by the reviewer and the others; the general response and Appendix I present the full results. Using NVIDIA Megatron-LM, we measured throughput and memory on up to 64 H100 GPUs for a 1.09B MobileLLM model, three times larger than the largest model in the submission. On the main 32-GPU mesh, PipeTempo+EF takes at most 2% more time than PipeDream-2BW and outperforms both Megatron's synchronous schedule and Zero Bubble. At the same microbatch size, its peak memory exceeds the synchronous baseline by only 0.7 GiB. The quality comparisons also favour PipeTempo+EF at every tested batch size, with token budgets set using estimated time ratios.

[...] with Muon as the main optimizer. [...] (esp with the claim of "optimizer-independent")

By "optimizer-independent" we meant that the schedule treats the stage-local optimizer as a black box: any optimizer can be plugged in without modification. We made this explicit in Section 6. To check whether our Muon results also carry over to other optimizers, we additionally repeated the main 135M comparison with SOAP, AdamW and Adan (Appendix J.1). SOAP and Adan broadly share the trend seen with Muon, but synchronous SOAP remains competitive over a wider batch-size range. PipeDream+EF does not outperform synchronous SOAP anywhere in the tested range. AdamW is substantially less tolerant of delay, consistent with the findings of [1] and [2].

Could the authors compare PipeTempo against a baseline that uses smaller learning rates for stages with greater delay, while keeping update frequency unchanged?

We ran this baseline (Appendix J.2): PipeDream with unchanged update frequency and learning rates reduced on stages with larger delay, using fixed profiles (γ,ds−1/2\gamma\\,d_s^{-1/2}, γ,ds−1\gamma\\,d_s^{-1}) and the annealed discount of PipeMare [3], which improves PipeDream in [4]. We tested each profile with and without Error Feedback and tuned the base learning rate for every cell. Without a correction, the delay-aware rates help PipeDream only at large batches; with Error Feedback, the strongest PipeDream variant in the paper, they do not help. PipeTempo+WP achieves lower loss than every tested delay-aware PipeDream variant across all five configurations. At S=8S=8, its advantage over the best delay-aware variant is 0.011 and 0.029 at B=2048B=2048 and 4096, respectively. Changing the accumulation intervals therefore gives a benefit that these learning-rate adjustments do not recover.

The reviewer indicated that evidence of practical efficiency would warrant a higher score. We have now added throughput and memory measurements, time-budgeted quality comparisons, and the requested delay-aware learning-rate experiments. We hope these additions address the concerns raised in the review and would appreciate a reconsideration of the score. We remain happy to discuss any outstanding questions.

[1] Jung et al. Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotation. ICML 2026.

[2] Zmushko et al. One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining. ICML 2026.

[3] Yang et al. PipeMare: Asynchronous Pipeline Parallel DNN Training. arXiv:1910.05124, 2019.

[4] Ajanthan et al. Nesterov Method for Asynchronous Pipeline Parallel Optimization. ICML 2025.

Dear reviewers, thanks for the additional experiments. The practical speed-up report is especially the one I think will benefit this work. I'm raising my score by 1.

Thank you for increasing your score and for your feedback, which helped us improve the paper

Reviewer_ucciOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper changes previous asynchronous pipeline-training method to let each stage update at a different frequency, and show that doing this allows it to have overall higher update frequency while only having one step staleness and require two weight copies. They prove convergence for specific choice of optimizer and lr/momentum scaling, and show strong results at large batches.

Critical strengths

The motivation is really nice: they explain the promise of asynchronous pipeline parallel approaches, then explain that the key benefit (frequent updates with high GPU utilization) only works with a large amount of staleness and weight copies, and efforts to reduce these reduce the benefit. They propose a compromise: somewhat more frequent updates with bounded (and low) staleness and weight copies.

The proposed method is also motivated theoretically quite nicely too, given the paper’s execution order, latest-weight forwards, and exact weight stashing (which are all reasonable choices), they choose the smallest fixed accumulation interval at each stage that guarantees both at most one update of staleness and at most two weight copies.

Critical weaknesses

One of the main things I find puzzling is that the paper produce a non-trivial convergence proof building upon Islamov et al. but then experimentally seems to depart from it? The proof is done for SCG following Islamov et al., and uses linear lr scaling and a specific momentum scaling across stages. However the experiments use Muon with Nesterov, sqrt lr scaling with batch size, and no momentum scaling. The use of Muon is fine, SCG and Muon are highly related and SCG makes sense to use as a theoretical proxy for Muon. Nesterov is motivated by earlier work showing it helps staleness such as Zmushko et al. and Ajanthan et al., but there is no theory or even discussion of this departure from theory? And then why the use of sqrt lr scaling with batch size and no momentum scaling, this is not explained at all?

Another major weakness is that there is no measured throughput or time-to-quality results, given the motivation is faster training on abundant hardware. The current experiments (which show val loss at matched data budgets and concurrency) make sense but I would have expected these type of results as well.

Questions for authors

Where do you think the focus should be next on improving asynchronous approaches? I'm confused about whether you think the results with weight prediction support it is a promising approach or not.

Notes:

  • Most of the paper is very well written (easy to follow) but the method section is then very concise and not intuitive to follow. The figure in the diagram helps but more diagrams (in the appendix) explaining how the schedule looks compared to pipedream and pipedream 2BW would make it a lot easier to understand (especially with labels for where opt updates happen and weight copies change).
  • Another thing I found confusing was the description of results as being for large batch sizes which makes it sound like its testing your method in the regime where it doesn't make many optimizer steps. I think this should be worded as in the large batch concurrency regime (in the large batch regime your method has smaller batch sizes but the same concurrency as the large batch).
7 comments
AuthorsOpenReview ↗

Theory and future directions

Thank you for the positive assessment of our paper and for your constructive comments. We have substantially extended the theory and expanded the explanation of the schedule in response.

Please note that this is not yet our complete response: we are still working on the throughput and wall-clock comparisons you requested, and will share them as soon as they are available.

Changes within existing sections are marked in blue. Entirely new appendices have blue section titles labelled “(new section)”, with black body text for readability. We address your comments below.

One of the main things I find puzzling is that the paper produce a non-trivial convergence proof building upon Islamov et al. but then experimentally seems to depart from it? The proof is done for SCG following Islamov et al., and uses linear lr scaling and a specific momentum scaling across stages. However the experiments use Muon with Nesterov, sqrt lr scaling with batch size, and no momentum scaling. The use of Muon is fine, SCG and Muon are highly related and SCG makes sense to use as a theoretical proxy for Muon. Nesterov is motivated by earlier work showing it helps staleness such as Zmushko et al. and Ajanthan et al., but there is no theory or even discussion of this departure from theory? And then why the use of sqrt lr scaling with batch size and no momentum scaling, this is not explained at all?

Our experimental choices were guided by preliminary experiments. We use Muon with Nesterov momentum, which Zmushko et al. found robust to one-step gradient delay; Ajanthan et al. find that Nesterov momentum improves robustness to gradient staleness in asynchronous pipeline training. For the stagewise parameters, we compared linear learning-rate scaling with stage-scaled momentum, as in Theorem 1, against square-root learning-rate scaling with constant momentum, and found the latter slightly better (new Appendix G, Figures 10 and 11). Square-root scaling is also a common transfer rule for Adam-type optimizers (Malladi et al., NeurIPS 2022), and for Muon-type (LMO) optimizers with fixed momentum it follows from minimizing a convergence upper bound at a fixed token budget (Shulgin et al., 2026). For momentum, Ajanthan et al. also find a constant coefficient slightly better than stage-dependent momentum.

Theorem 1 uses its parameter configuration because it yields a particularly clean convergence guarantee. However, the PipeTempo schedule also admits other learning-rate and momentum choices, and in the revision we extend the analysis to the stagewise parameter configuration used in our experiments, showing that it retains the leading O~(ε−3)\widetilde O(\varepsilon^{-3}) stochastic complexity.

We have extended the analysis in the new Appendix E. Theorem 2 allows general admissible stagewise learning rates and momentum parameters. Corollary 1 recovers Theorem 1, and Corollary 2 additionally analyzes linear LR scaling with the same momentum coefficient across stages (i.e., no stagewise momentum scaling). Corollary 3 combines this momentum choice with square-root LR scaling, matching the stagewise parameter profile used in our experiments. Under the stated assumptions and appropriate scalar tuning, all three configurations retain the leading stochastic complexity O~(ε−3)\widetilde O(\varepsilon^{-3}) at fixed pipeline geometry and problem constants.

We have also extended the theory to Nesterov momentum in the new Appendix F. Theorem 3 establishes the general stagewise result for the Nesterov variant of SCG, followed by specializations for the same three parameter profiles. The Nesterov extension preserves the leading O~(ε−3)\widetilde O(\varepsilon^{-3}) stochastic complexity. These additions address both the Nesterov structure and the experimental stagewise scaling rules, while retaining SCG as the theoretical proxy for Muon.

Where do you think the focus should be next on improving asynchronous approaches? I'm confused about whether you think the results with weight prediction support it is a promising approach or not.

We believe weight prediction is a promising direction: applied to later stages, it already lowers PipeTempo's validation loss further (Figure 3). The picture is not yet complete, however: somewhat surprisingly, applying it to early stages increases the loss, unlike similar delay corrections for PipeDream and PipeDream-2BW (Zmushko et al.).

We think an important next step is a theoretical analysis that accounts jointly for prediction, stage-dependent accumulation, and optimizer dynamics. Such an analysis could explain when prediction helps and when it can be harmful, and guide corrections tailored to each stage. Our current theory does not include weight prediction; extending it in this direction is a natural follow-up to the experimental findings.

AuthorsOpenReview ↗

Evaluation and presentation

Another major weakness is that there is no measured throughput or time-to-quality results, given the motivation is faster training on abundant hardware. The current experiments (which show val loss at matched data budgets and concurrency) make sense but I would have expected these type of results as well.

We deliberately focused on the algorithmic side and compared methods at matched token budgets. A token budget is comparable across setups, whereas wall-clock time depends on the implementation and on the hardware and parallelism setup. Beyond the engineering effort required for an efficient implementation, timings obtained on the few nodes and model sizes available to us may not carry over to the large-scale setting that motivates our work.

We also expect matched-token comparisons to be conservative for PipeTempo relative to synchronous training: at matched time, synchronous training additionally pays for fill/drain bubbles, which PipeTempo avoids (Figure 2), while PipeTempo's additional optimizer steps are typically cheap enough to be omitted from standard estimates of training cost (Ren et al., 2021). Even so, PipeTempo achieves lower validation loss than synchronous training at the same concurrency and token budget (Figure 1).

We agree, however, that for matrix optimizers such as Muon the optimizer step is more expensive, and that this overhead should be accounted for. We are preparing throughput and matched-time comparisons and plan to share them as soon as possible.

Most of the paper is very well written (easy to follow) but the method section is then very concise and not intuitive to follow. The figure in the diagram helps but more diagrams (in the appendix) explaining how the schedule looks compared to pipedream and pipedream 2BW would make it a lot easier to understand (especially with labels for where opt updates happen and weight copies change).

We revised Section 3 to explain pipeline execution, weight stashing, and stage-dependent accumulation more explicitly, with changes marked in blue.

We also expanded Figure 2 to compare synchronous 1F1B directly with PipeTempo. This shows how synchronous training drains the pipeline between update batches, while PipeTempo updates stages with other microbatches still in flight. Section 3 connects this difference to the need for weight stashing, explaining why each backward uses the weights retained from its matching forward.

The new Appendix D and Figure 9 provide a detailed comparison of PipeDream, PipeDream-2BW, and PipeTempo. The diagrams explicitly mark optimizer updates and show retained weight versions and buffer transitions. The accompanying text explains accumulation windows, version selection, update frequency, and weight storage.

Another thing I found confusing was the description of results as being for large batch sizes which makes it sound like its testing your method in the regime where it doesn't make many optimizer steps. I think this should be worded as in the large batch concurrency regime (in the large batch regime your method has smaller batch sizes but the same concurrency as the large batch).

We now use “large concurrent batch sizes” and explicitly distinguish this from the batch size per optimizer update.

For SS stages processing microbatches of bb sequences, the concurrent batch size is B=SbB=Sb. Synchronous training updates on all BB sequences, whereas PipeTempo’s stage ss updates on msb=msB/Sm_sb=m_sB/S sequences, where msm_s is its accumulation interval. PipeTempo therefore uses smaller update batches at the same concurrency and token budget.

We have clarified this distinction in the abstract, experimental setup, and figure captions, with revisions marked in blue.


We believe these revisions substantially strengthen both the theory and the presentation of our paper. We will add the throughput and matched-time results as soon as they are available. We wanted to share the revised manuscript now so that you can assess the new analyses while we work on these experiments. We welcome any further questions or comments, particularly on the theoretical extensions, and would be happy to address them during the discussion.

AuthorsOpenReview ↗

Throughput and peak-memory measurements

As promised, we have now added throughput and peak-memory measurements, together with quality comparisons using estimated time budgets, on a 1.09B model. These results directly examine the overhead of frequent Muon updates; please see the general response and Appendix I for details. We would be happy to discuss any further questions.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response: wall-clock time, throughput and memory

We thank the reviewers for their constructive comments. We address their shared concern about systems evaluation here. We apologize for the late reply: rebuilding PipeTempo, PipeDream and PipeDream-2BW in NVIDIA Megatron-LM [5] took most of the discussion period. We measured throughput and memory against synchronous baselines on up to 64 H100 GPUs; see details in App. H and I.

Why we compared at matched tokens, and why that needed checking

Our reasoning in the submission was that matched tokens would be a conservative protocol for PipeTempo. Avoiding flushes removes the steady-state synchronous pipeline bubble, while PipeTempo takes fewer optimizer steps than PipeDream. We therefore expected comparable or better throughput at the same concurrent batch, so that a matched-token comparison would understate its advantage.

The reviews raised a concern about optimizer-step overhead. This cost is usually neglected for AdamW [6] and reported at around 1% for large-batch Muon training [1, 2]. But PipeTempo's late stages update after each logical microbatch, rather than once per accumulated batch as in synchronous training. A 1% cost paid tens of times more often is no longer obviously smaller than a bubble. To our knowledge, the combined overhead of asynchronous pipelines and Muon has not been measured: prior systems studies use Adam-type optimizers [4], while the asynchronous Muon comparison in [3] reports iterations rather than time.

Setup and throughput

The GPU-rich regime. The paper is motivated by training with more GPUs than a small update batch can keep busy (Section 1). We study this regime by increasing the number of data-parallel replicas DD at a fixed pipeline depth SS. PipeDream-2BW has no steady-state bubble and steps as rarely as synchronous training, making it a natural reference. For the synchronous schedules considered here, the idealized relative bubble overhead is cβc\beta, where cc is schedule-specific and β=(S−1)Db/B\beta=(S-1)Db/B is shared. Here bb is the synchronous microbatch per replica and BB the global synchronous update batch (App. I.1). This formula illustrates our motivation: as DD grows, keeping β\beta small requires increasing the update batch BB or shrinking bb below what the hardware runs efficiently.

What we ran. A 1.09B model with the shape of MobileLLM-1B on meshes up to 8×88\times8 (D×SD\times S), with Muon and ZeRO-1, at BB = 1024–8192. This is also the largest model on which we can afford quality runs, which is why we studied its throughput in detail. We chose a deep, narrow model because we expected 54 layers to give more flexibility in balancing eight stages than the 16–28 layers common at this model size. We tuned microbatch size separately for every method and every BB, as well as Megatron's communication-overlap flags for the synchronous baselines. More details in App. I.2.

We make two small changes to the PipeTempo implementation used in Section 5. First, we use EF instead of WP: our 135M comparisons show similar quality, while EF can be faster (App. I.3, I.6). Second, for both PipeTempo and PipeDream, large logical microbatches can exceed GPU memory. We introduce a schedule that executes them as smaller pieces without changing the update rule (App. H).

Table 1. D=4D=4, S=8S=8. Time per sequence of PipeDream-2BW, and the extra time of every other method relative to it.

BB=1024204840968192
PipeDream-2BW (ms per sequence)0.6360.6210.6170.618
Sync 1F1B+42%+25%+17%+9%
Sync ZB1P+26%+13%+6%+2%
PipeDream+EF+12%+8%+4%+2%
PipeTempo+EF+2%+1%+1%0%

Several observations follow from this table.

Synchronous training pays for its bubble exactly where the paper says it does. At small BB the bubble is more costly; the tuned 1F1B even prefers a microbatch of 16 over the more efficient 32, because halving bb halves β\beta. Zero Bubble [10] removes between a third and three quarters of this gap, not all of it.

The concern about frequent steps is justified for PipeDream. It steps every stage after every microbatch; its first stage, which holds the most parameters, becomes the slowest, and PipeDream+EF is 12% behind PipeDream-2BW at B=1024B=1024.

It does not materialize for PipeTempo. PipeTempo+EF stays within 2% of PipeDream-2BW at every BB, i.e. it runs essentially at the speed of a pipeline that has neither a bubble nor frequent steps. Unlike PipeDream, PipeTempo updates frequently only on its late stages, and in our partition these stages hold one layer fewer than the early ones, so most of an optimizer step fits into this slack instead of delaying the pipeline.

How the configuration affects overhead

Using otherwise idle GPU time for optimizer work is a form of co-design: the model partition, parallelism and training algorithm are chosen together. Changing the setup can expose more of this cost. For example, removing four layers to balance the stages reduces the slack and raises PipeTempo's overhead over PipeDream-2BW from 2% to 9% at B=1024B=1024 (App. I.3).

Across the tested meshes, PipeTempo remains faster than the synchronous alternatives. On 32 and 64 GPUs it is 1.06–1.49× faster than 1F1B and 1.02–1.32× faster than Zero Bubble. We do not claim that either side is optimal; however, the synchronous baseline is NVIDIA's implementation with its microbatch tuned per cell, which we consider a reasonably strong one.

Memory

Peak memory on the most loaded stage, microbatch 32, D=4D=4, S=8S=8:

1F1BPipeDream-2BWPipeTempo+EFPipeDream+EF
GiB76.977.677.680.3

Activations dominate and are the same for all methods at the same microbatch. One source of overhead is extra compute-weight copies (the bf16 weights used in the forward and backward pass; the fp32 master weights and optimizer state are sharded by ZeRO-1 and are not duplicated). PipeTempo keeps at most two copies per stage, as PipeDream-2BW does; its total measured overhead is 0.7 GiB. DualPipe also keeps two [7]. PipeDream keeps as many copies as the stage has pending microbatches, up to SS on the first stage. Its total measured overhead is 3.4 GiB here; its weight storage grows with model size and pipeline depth. For a model and pipeline of the size of DeepSeek-V3 (16 stages) we estimate that the second copy costs about 3.4 GB per GPU, whereas PipeDream would need about 26 GB more on average and about 51 GB more on the first stage, on 80 GB GPUs (App. I.4). Unlike for PipeDream, the memory overhead is therefore not a concern for PipeTempo.

Quality at equal wall-clock time

Several reviews asked for a larger scale and time-to-quality results. We report a closely related quantity, the loss reached in a fixed wall-clock budget (min loss | time rather than min time | loss). Time-to-target-loss is awkward to measure with a decaying learning-rate schedule: the loss at an intermediate point of a run is not the loss a shorter run would reach (Hoffmann et al., 2022 [8]; Hägele et al., 2024 [9]), so every target needs its own tuned run, whose length is not known in advance. Fixing the time budget instead incorporates throughput in the same way and needs one run per configuration.

We train on half of the Chinchilla-optimal token count: the comparison has to be repeated for every batch size and requires some learning-rate tuning, so a full budget was too costly. The synchronous token budgets were set from our earlier estimate of Zero Bubble time (details in App. I.5): 1.25 / 1.17 / 1.09 / 1.04 times fewer tokens at BB = 1024 / 2048 / 4096 / 8192.

Table 2. Final validation loss at budgets set using estimated time ratios.

BB=1024204840968192
Synchronous2.4352.4382.4522.511
PipeTempo+EF2.4292.4262.4392.477
Difference−0.006−0.012−0.013−0.034

PipeTempo+EF reaches a lower loss at every batch size, which supports PipeTempo as a promising approach for scaling. One caveat follows directly from the throughput study: the time ratio we plug in belongs to one mesh (D=4D=4, S=8S=8). With fewer GPUs it would be closer to one, with more GPUs larger, and by choosing the mesh one could move the equal-time comparison in either direction. This is why we still regard the matched-token comparison of the paper as a good protocol: it is the limit in which no method pays for idle GPUs, and it does not depend on a hardware configuration.

If the reviewers have particular models or configurations in mind for which they would like to see the throughput and memory of PipeTempo, we would be glad to measure them.

[1] Liu et al. Muon is Scalable for LLM Training. arXiv:2502.16982, 2025.

[2] Poolside Team. Laguna M.1/XS.2 Technical Report. arXiv:2605.27605, 2026.

[3] Jung, Shin, Lee. Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotation. ICML 2026.

[4] Chen, Wu, Yu. AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training. ICML 2026.

[5] Narayanan et al. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. SC 2021.

[6] Ren et al. ZeRO-Offload: Democratizing Billion-Scale Model Training. USENIX ATC 2021.

[7] DeepSeek-AI. DeepSeek-V3 Technical Report. arXiv:2412.19437, 2024.

[8] Hoffmann et al. Training Compute-Optimal Large Language Models. NeurIPS 2022.

[9] Hägele et al. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations. NeurIPS 2024.

[10] Qi et al. Zero Bubble Pipeline Parallelism. ICLR 2024.

Dear Reviewers,

Thank you again for your feedback, which has helped us strengthen the paper. With the discussion period ending soon, we would appreciate hearing whether our responses and revisions have addressed your concerns. We would be happy to clarify any remaining points.

Thank you for your time and consideration.

AuthorsOpenReview ↗

Summary of Revisions and Discussion

Dear Area Chair and Reviewers,

Thank you for your feedback and the helpful discussion, which helped us improve the paper. We summarize the outcome of the discussion and the main changes below.

Reviews. Reviewer ucci (weak accept) wrote that we addressed their concerns and that they are also satisfied with our responses to the other reviewers. Reviewer CKhJ raised their score to weak accept (2->3), highlighting the measured speed-up. Reviewer DCqt (weak reject) did not respond to our reply; the two points of their review, system measurements at a larger scale and a comparison with Zero Bubble, are covered by item 1.

  1. Measured throughput and memory (App. I; requested by all three reviewers). We implemented PipeTempo, PipeDream and PipeDream-2BW in NVIDIA Megatron-LM and measured a 1.09B model on up to 64 H100 GPUs. On the main 32-GPU mesh PipeTempo+EF runs within 2% of PipeDream-2BW, and on 32 and 64 GPUs it is faster than both Megatron's synchronous 1F1B schedule and Zero Bubble (each with its microbatch size tuned), with nearly the same peak memory as 1F1B at the same microbatch size.

  2. Gradient accumulation for PipeDream and PipeTempo that is mathematically equivalent to the original algorithm (App. H). Prior work explicitly reports a limitation of asynchronous pipeline parallelism: in PipeDream the update batch is limited by the largest microbatch that fits in memory [1] (Section 4.2.1), and gradient accumulation is avoided "since it changes the level of delay" [2] (App. D.2). We show that this limitation can be removed without changing the algorithm, for both PipeDream and PipeTempo: one microbatch can be executed as rr smaller pieces, each reading the stored weight version of its microbatch rather than the latest local weights. The iterates, the delays and the number of stored weight versions are unchanged, while the activations in flight shrink by the factor rr. To our knowledge, this is the first gradient accumulation for PipeDream-type schedules that decouples the update batch from activation memory while remaining mathematically equivalent to the original algorithm.

  3. Theory and experimental parameter choices (App. E–G, K). We extended the SCG analysis to general stagewise learning rates and momentum, and to Nesterov momentum. The corollaries give explicit parameter choices with the same leading stochastic O~(ε−3)\widetilde{O}(\varepsilon^{-3}) complexity at fixed depth, with different depth dependence. Linear rates with stage-scaled momentum give the best depth dependence among these choices. These are worst-case SCG bounds and do not imply that linear scaling must perform better with Muon. We used square-root rates with shared momentum in the language-model experiments because this combination performed slightly better in our comparison (App. G). We also added controlled SCG experiments that use the theoretical parameters without tuning (App. K). In the controlled setting, linear learning rates with stage-scaled momentum reach the target fastest at every tested depth, with a larger advantage at higher depths.

  4. Quality at equal wall-clock time (App. I.5). At the request of Reviewers ucci and CKhJ, we also compared the methods at equal wall-clock time on the 1.09B model, with token budgets set using estimated time ratios: PipeTempo+EF reaches a lower validation loss than synchronous training at every tested batch size.

Additional changes. At the request of Reviewer CKhJ, we repeated the main comparison with SOAP, AdamW and Adan (App. J.1) and added a PipeDream baseline with delay-aware learning rates (App. J.2). We also added diagrams comparing PipeTempo, PipeDream and PipeDream-2BW, explaining their update timing, weight-version selection and storage requirements (App. D). We revised the writing for clarity, following the reviewers' suggestions.

We believe the revisions address all the reviewers' concerns. Thank you again for your time and consideration.


[1] Li & Hoefler. Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines. SC 2021.

[2] Jung et al. Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotation. ICML 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.