- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies pipeline parallelism, specifically aiming to understand the convergence of PipeDream (PD) method. To simplify the analysis, it proposes Randomized PipeDream (RPD) - a method that replaces deterministic scheduling with random stage and minibatch sampling while keeping the stale block-gradient structure. The paper derives the convergence guarantee for RPD and connects it to PD by analyzing how staleness grows with the number of stages. The paper also compares PD with a pipeline-parallel version of LocalSGD and shows that PD performs better on the tested quadratic and language-modelling tasks, while LocalSGD achieves lower loss on logistic regression.
Critical strengths
-
The paper is nicely written, it explains the motivation clearly and provides enough context to follow the main ideas. It first introduces the basics of pipeline parallelism, then explains naive scheduling and how PD differs from it, with Figure 1 providing a clear visual comparison. The RPD abstraction and the pipeline-parallel LocalSGD are also explained carefully.
-
The paper addresses a valid question for the pipeline-parallel training: how the number of stages and scheduling choices affect the optimization progress. Its analysis connects increasing pipeline depth to greater staleness in PD and weaker convergence guarantee for RPD. The comparison with LocalSGD examines the trade-off between gradient staleness and synchronization overhead. This is useful for evaluation of pipeline-parallel methods since high hardware utilization (a common target for such methods) does not guarantee faster convergence.
Critical weaknesses
The experiments are somewhat limited. PD and RPD are compared on synthetic quadratic and logistic-regression objectives, but it remains unclear whether RPD is still a useful approximation of PD in the nonconvex language-modelling setting. Further, synthetic experiments use only one seed so it's difficult to access how reliable the observed trends are.
Questions for authors
In Figure 2b, the loss for PD and RPD doesn't change much as the number of stages grow. Is it because the number of block updates actually increases even though the simulated time budget stays the same, offsetting the effect of greater staleness?
2 comments
Thank you for the clarifications. I'm keeping my acceptance score.
Response to Reviewer fcWG: Fixed-Time Scaling and Empirical Scope
Thank you for the positive assessment of the presentation and the scheduling–optimization question. Your observation about Figure 2b identifies an important distinction between simulator time and block-update count.
1. Does increasing the stage count increase the update budget? Yes. Figure 2b fixes a simulator-time target, not the number of block updates across stage counts. Appendix G.6 specifies a target of 3684 pipeline ticks. For each S, the PD and LocalSGD schedules use the smallest number of microbatches whose timelines reach that target. RPD is assigned the number of backward block updates in the corresponding PD timeline. Thus PD and RPD are matched in block-update count at a given S, but that count is not fixed as S changes.
Under the idealized even-S steady-state schedule, S/2 backward operations occur per tick. Ignoring startup and drain, a horizon Tsim therefore gives approximately K≈STsim/2 block updates. Increasing S consequently changes both the available update budget and the delay scale. Moreover, the synthetic dimension remains d=512, so the blocks become smaller as S grows, and learning rates are tuned separately.
Your proposed compensation mechanism is therefore consistent with the experimental design. However, the plot does not isolate how much of the observed behavior is caused by additional updates versus staleness, block size, and retuning. It should not be interpreted as a fixed-K test or as showing monotonic deterioration of PD/RPD. The supported conclusion is that LocalSGD attains a lower final objective in the larger-stage regime of this logistic-regression experiment. This remains a simulated-time comparison, not a memory-normalized hardware benchmark: LocalSGD uses R=S replicas.
Theorem 4.1 instead bounds average squared gradient norm at a specified block-update budget. Its worsening upper bound does not require the measured final objective in a fixed-time experiment to worsen. The stronger wording in Sections 5 and 7 should be narrowed accordingly.
2. Does the language-model experiment validate RPD as a proxy? The current evidence separates two questions. Figure 2 compares PD and RPD on synthetic objectives. Figure 4c compares deterministic PD with LocalSGD on the 11.1M-parameter NanoChat-style Transformer; it does not include an RPD curve. It therefore supports the observation that synchronization is not uniformly preferable, but does not establish that RPD predicts PD's nonconvex language-modeling trajectory.
Appendix F explains the structural connection: replaying the same stage order, batch order, and forward-pass weight snapshots reproduces the PD update template. Retaining that template while randomizing stage/batch selection is a further modeling step whose empirical fidelity on language modeling remains to be tested. The nonconvex theorem applies to RPD under its assumptions; this alone is not empirical validation of the proxy for Transformers.
3. Single-seed reliability. Appendix G.6 explicitly states that the reported synthetic plots use one seed. Zero initialization and common data make the comparisons controlled within that run, but do not quantify variation across generated problems, sampling sequences, or gradient noise. The existing plots should therefore be treated as illustrative, not as statistically established rankings or uncertainty estimates. The reported framework supports additional seeds, but implementation support is not a substitute for reporting those results.
The scheduler-level result has a different status: Appendix C derives both the mean and maximum global-history delays analytically, and Figure 6 checks them against reconstructed timelines. This supports the quadratic delay law in the stated model independently of the optimization seed, without resolving the uncertainty in the empirical performance comparisons.
We appreciate the distinction you raise: the paper's contribution is to make the interaction between execution and optimization explicit, while the current experiments establish objective-dependent examples rather than a universal performance ordering.