COVERPIPE: A NEW COMMUNICATION PROTOCOL FOR LOW-BANDWIDTH PIPELINE PARALLELISM

Pipeline parallelism partitions large models across multiple accelerators to overcome single-device memory limits, but inter-stage communication poses a severe bottleneck under bandwidth-constrained networks. While recent extreme compression techniques show promise in reducing communication volume, we uncover a fundamental structural limitation: they project representations into a static, data-oblivious subspace, which severely restricts representational capacity and starves orthogonal latent dimensions of gradient updates. We formalize this failure mode under a unified analytical framework, demonstrating both theoretically and empirically that fixed projections impair model trainability. To address this issue, we propose CoverPipe, a new pipeline-parallel training scheme based on input-adaptive sparsification. CoverPipe dynamically preserves salient features per sample on the critical path, while asynchronously transmitting residual components during idle communication windows to restore full representational capacity and gradient flow. Comprehensive evaluations on standard and MoE Transformer workloads across 60–500 Mbps networks demonstrate that CoverPipe outperforms both uncompressed and compressed baselines in wall-clock convergence while maintaining high throughput. Under Chinchilla-optimal training budgets, CoverPipe consistently outperforms existing compression methods across compression ratios from 8$\times$ to 128$\times$, demonstrating robust long-horizon optimization. Experiments on models up to 1B parameters further confirm the robustness of CoverPipe, establishing it as an effective and practical solution for pipeline parallelism over commodity networks.