- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies inter-stage communication compression for pipeline parallelism over low-bandwidth networks. Authors proposed CoverPipe, which transmits input-adaptive sparse coordinates with norm-preserving rescaling and index coding, and sends the remaining coordinates during idle communication windows. Experiments train Llama-style models on OpenWebText under 10-hour wall-clock and Chinchilla budgets.
Critical strengths
- Under a fixed token budget, CoverPipe outperforms both bottleneck-compression baselines across compression ratios from 8× to 128× even without residual overlap. This indicates that the gain does not come solely from extra transmission.
- The analysis and engineering are careful. The derivation in Appendix B is correct, and at the same budget the index coding transmits 1.44×–1.54× more activation values than a naive representation.
Critical weaknesses
- The wall-clock results are insufficiently supported. The time axis is converted from average throughput rather than measured, it is unclear how the 10-hour endpoint is computed, and the hardware.
- The comparison with baselines does not control for traffic or tuning. The baselines use a fixed ρ = 64 while CoverPipe sends extra residual-overlap traffic.
- The evidence for the proposed mechanism is weaker than the claims.
Questions for authors
- Are the trajectories in Fig. 3 from actual runs with one GPU per stage? Why is time derived from average throughput rather than from measured timestamps?
- Could you tune the baselines' compression ratio at each bandwidth and report the actual total traffic per token, including residual overlap?
- Is the gradient of the rescaling factor α backpropagated? In which basis are the SSN gradients in Fig. 7/12 plotted?
3 comments
[W2, Q2] Comparison not controlled for traffic or tuning
The core idea of residual overlap is to utilize network links that would otherwise remain idle during computation to transmit additional information.
We applied the same bandwidth constraint to all methods, and CoverPipe adjusts overlapped data to fit within each bandwidth's communication idle time. Consequently, although the total transmitted volume increases, this stems from utilizing idle link capacity while maintaining the same critical-path budget and throughput as the baselines. Baselines share this idle time and could in principle transmit extra data. However, low-rank baselines face structural hurdles in directly applying residual overlap, from defining residuals to modifying model architectures for re-compression.
We provide results at equal total communicated bits and total traffic per token in Fig. 3.
Nevertheless, we agree that Fig. 3 does not separate critical-path parity gains from residual overlap gains. To address this concern, we report 10-hour training loss for CoverPipe without residual overlap, matching the baselines' exact communication volume.
| Model | Method | 60 Mbps | 100 Mbps | 200 Mbps | 500 Mbps |
|---|---|---|---|---|---|
| Dense | SOTA compression | 4.253 | 4.211 | 4.175 | 4.149 |
| CoverPipe (w/o Overlap) | 4.013 | 3.977 | 3.947 | 3.926 | |
| CoverPipe (w/ Overlap) | 3.932 | 3.896 | 3.797 | 3.748 | |
| MoE | SOTA compression | 4.185 | 4.160 | 4.137 | 4.097 |
| CoverPipe (w/o Overlap) | 3.970 | 3.943 | 3.915 | 3.892 | |
| CoverPipe (w/ Overlap) | 3.821 | 3.782 | 3.680 | 3.616 |
Both the w/o Overlap performance and overlap gains align with Fig. 4. Per-token communication volumes (bytes) in Fig. 3 are detailed below.
| Method | Critical path (bytes) | Residual overlap (bytes) | Total (bytes) |
|---|---|---|---|
| Uncompressed | 2048 | – | 2048 |
| SSN, ResBM | 32 | – | 32 |
| CoverPipe without overlap | 32 | – | 32 |
| CoverPipe η=8 (Dense: 60 and 100 Mbps) | 32 | 16 | 48 |
| CoverPipe η=16 (Dense: 200 Mbps; MoE: 60 and 100 Mbps) | 32 | 32 | 64 |
| CoverPipe η=32 (Dense: 500 Mbps; MoE: 200 Mbps) | 32 | 64 | 96 |
| CoverPipe η=64 (MoE: 500 Mbps) | 32 | 128 | 160 |
We provide 10-hour wall-clock loss estimates across different bandwidths and compression ratios for each method.
In Fig. 3, we fixed the compression ratio at across all bandwidths to isolate the effect of decreasing bandwidth and observe the performance gains from residual overlap under a controlled compression ratio. However, we agree with the reviewer that the optimal compression ratio can vary depending on the available bandwidth. Due to time and resource constraints during the rebuttal period, we were unable to re-tune or re-run 8-GPU training across all combinations. Instead, we estimated 10-hour losses from Fig. 4 across 60–500 Mbps and – compression. Bold marks each method's optimal ratio, and the last column highlights the best method per bandwidth.
| Bandwidth | Method | 8× | 16× | 32× | 64× | 128× | ║ | Best (ratio) |
|---|---|---|---|---|---|---|---|---|
| 60 Mbps | SSN | 4.148 | 4.017 | 4.051 | 4.215 | 4.398 | ║ | 4.017 (16×) |
| ResBM | 4.196 | 3.986 | 4.010 | 4.158 | 4.349 | ║ | 3.986 (16×) | |
| CoverPipe (w/o Overlap) | 4.022 | 3.858 | 3.857 | 3.990 | 4.161 | ║ | 3.857 (32×) | |
| 100 Mbps | SSN | 3.944 | 3.938 | 4.005 | 4.181 | 4.387 | ║ | 3.938 (16×) |
| ResBM | 3.983 | 3.908 | 3.961 | 4.070 | 4.371 | ║ | 3.908 (16×) | |
| CoverPipe (w/o Overlap) | 3.832 | 3.786 | 3.817 | 3.910 | 4.182 | ║ | 3.786 (16×) | |
| 200 Mbps | SSN | 3.843 | 3.853 | 3.918 | 4.175 | 4.355 | ║ | 3.843 (8×) |
| ResBM | 3.860 | 3.825 | 3.878 | 4.070 | 4.332 | ║ | 3.825 (16×) | |
| CoverPipe (w/o Overlap) | 3.713 | 3.715 | 3.740 | 3.934 | 4.155 | ║ | 3.713 (8×) | |
| 500 Mbps | SSN | 3.803 | 3.810 | 3.909 | 4.137 | 4.347 | ║ | 3.803 (8×) |
| ResBM | 3.790 | 3.785 | 3.864 | 4.052 | 4.327 | ║ | 3.785 (16×) | |
| CoverPipe (w/o Overlap) | 3.662 | 3.672 | 3.738 | 3.889 | 4.143 | ║ | 3.662 (8×) |
CoverPipe maintains its advantage even at each method's optimal compression ratio. We will add full 8-GPU results across ratios for Dense and MoE models in the revision.
After considering the author's response and other comments, I keep the score unchanged.
We thank the reviewer for the constructive feedback, which strengthens the fairness and transparency of our evaluation. Please find our responses below.
[W1] Insufficiently supported wall-clock results
Figure 3 uses measured mean throughput to convert training curves from eight RTX 3090s (one stage per GPU) to wall-clock time.
We calculated the number of tokens each method could process over 10 hours using its average throughput and derived the corresponding training loss at that token count. This approach was adopted to mitigate the impact of hardware execution variability when comparing methods. Even under identical configurations, system factors such as GPU/CPU resource contention, thermal or power throttling, OS scheduling, and cache states can affect throughput, and we indeed observed minor run-to-run variations. In long-running executions, system load and hardware states fluctuate over time, making it difficult to isolate algorithmic performance differences from environmental fluctuations if each method is trained independently for 10 hours at different times. Therefore, we measured and averaged the throughput of each method across multiple epochs. This allows us to compare methods based on their representative processing speeds rather than transient environmental variations during individual 10-hour runs.
[W3] Evidence for the mechanism is weaker than the claims
Our claim is grounded in both prior work and our empirical results.
Prior work has shown that identity paths significantly improve optimization in deep residual networks by providing direct pathways for gradient propagation, and that altering these paths can hinder optimization [1,2]. Our analysis demonstrates that prior pp compression methods restrict these identity paths. Providing a deeper fundamental theoretical explanation of why identity paths improve learning beyond this established understanding is outside the scope of this work. Furthermore, we empirically observe that the absence of the identity term leads to vanishing gradients (Figs. 7 and 12).
[Q3] Gradient details
Backpropagation includes the gradient of , and SSN gradients are projected onto an orthonormal basis of its preserved and orthogonal subspaces.
Specifically, we construct an orthonormal matrix $Q=
U,U⊤$ and compute Q⊤∇xL, so that the leading h coordinates correspond to SSN's preserved subspace and the remaining coordinates correspond to its orthogonal complement. In the revised manuscript, we will explicitly detail this basis transformation and gradient computation procedure in Appendix D.1.
References
[1] Identity Mappings in Deep Residual Networks
[2] The Shattered Gradients Problem: If ResNets Are the Answer, Then What Is the Question?