- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates whether more frequent asynchronous updates can improve language-model pretraining when large synchronous batches leave too few optimizer steps within a fixed token budget. It introduces PipeTempo, which sets each pipeline stage’s accumulation interval according to its local delay. Later stages consequently use smaller update batches, while the schedule limits local staleness to one update and retains at most two weight versions per stage. The paper extends the convergence analysis of stochastic conditional-gradient updates with momentum to coupled pipeline stages with arbitrary fixed accumulation intervals. Experiments with Muon on FineWeb-Edu demonstrate lower validation loss in the large-batch regime.
Critical strengths
The accumulation rule has a clear justification in terms of update frequency, staleness, and weight storage. PipeTempo exploits differences in local delay instead of imposing a common accumulation window across all stages. Proposition 1 establishes that the proposed intervals are the shortest fixed choices that limit local staleness to one update while requiring at most two weight versions. This provides a principled and simple extension of existing asynchronous pipeline schedules.
Critical weaknesses
-
The evaluation is limited in scale and lacks essential system measurements. Experiments use only 135M and 360M parameter models, leaving the applicability to substantially larger models insufficiently validated. More importantly, the paper reports validation loss at matched token budgets but provides no measurements of throughput, peak GPU memory, or training time. These quantities are central to assessing a distributed training method, especially because more frequent optimizer updates and weight prediction may introduce additional overhead.
-
The synchronous pipeline comparison is incomplete. The evaluation lacks comparisons with stronger synchronous schedules, suc as Zero Bubble Pipeline Parallelism. These schedules may change the relationship between pipeline utilization and the required update batch. Without comparisons under common hardware and memory constraints, it remains unclear whether PipeTempo’s advantage persists against a well-configured synchronous baseline or primarily mitigates the large batch penalty of the selected configuration.
Questions for authors
None
We thank the reviewer for their time and detailed review. Below, we respond to the concerns raised in the review.
We agree that system measurements and experiments with larger models are important. We ran them; please see the general response and Appendix I. In short: we implemented the methods in NVIDIA Megatron-LM and measured a 1.09B model on up to 64 H100 GPUs. On our main 32-GPU mesh, PipeTempo+EF runs within 2% of PipeDream-2BW, faster than Megatron's synchronous schedule and than Zero Bubble; its peak memory is within 0.7GB of the synchronous one at the same microbatch; and at budgets set for equal wall-clock time it reaches a lower validation loss than synchronous training at every batch size we tested.
We would first like to clarify what the synchronous baseline represents in the matched-token experiments. Section 5.1 defines it as synchronous training with an update batch of B sequences, without prescribing how these updates are executed across devices. It can therefore represent any synchronous pipeline, including Zero Bubble, or even data-parallel training without pipeline parallelism (PP=1). A different execution schedule can change throughput, but does not change the underlying synchronous update rule at a fixed batch size.
This freedom to choose a synchronous configuration does not, however, make the large-batch penalty merely an artifact of our selected setup. Four recent training reports (lines 94–100) describe the tension between using a large GPU cluster efficiently and keeping update batches small enough for effective optimization. It is reasonable to expect that these practitioners carefully configured their training systems, yet they still encountered this tradeoff. Mitigating it is precisely the motivation of our work. PipeTempo aims to address it through more frequent updates with smaller batches, accepting some gradient staleness while keeping the same concurrent batch size as the baseline (lines 104–105). We do not expect a benefit when synchronous training can already use a near-optimal update batch (line 431).
Our rationale for comparing at matched tokens is that, in the pipeline setting, keeping the same concurrent batch should allow approximately the same utilization, while avoiding pipeline flushes can improve throughput. However, as the reviewer and the other reviewers pointed out, more frequent optimizer steps may offset this benefit. We therefore measured their overhead and compared PipeTempo with tuned Zero Bubble under common hardware and memory constraints; these wall-clock measurements are reported in the general response and Appendix I.
We would be glad to continue the discussion and clarify any remaining concerns. For now, we believe we have addressed all of the reviewer’s concerns, none of which points to a serious issue with our approach. Therefore, we would kindly ask the reviewer to reconsider their score.