- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work addresses the compute bottlenecks in NCCL that arise from using streaming multiprocessors (SMs) for both communication and computation. The authors propose HostCCL which precomputes the dependency graph on the CPU that enforces task sequential correctness and manages task completion signals from the GPU. HostCCL improves on NCCL by reducing the SM budget and generally has better throughput in most tested settings.
Critical strengths
Experimental results generally show HostCCL improves over NCCL, in particular on the small SM budget settings on which the paper focuses.
Critical weaknesses
Discussion of results are generally sparse and superficial, reporting only the numerical results and comparisons without further elaboration. Figure 2 shows NCCL generally outperforms HostCCL on smaller tensor sizes and it is briefly mentioned at the end of Section 5.2, but no discussion is provided of why this occurs. Similarly, the paper lacks a limitations section in the main body (Nor in the appendix) to acknowledge potential concerns, e.g. higher CPU usage (Figure 7).
HostCCL is only compared against NCCL and not against any of the related work mentioned in Section 6, weakening its significance. The paper only gives a brief one-sentence description of the difference between HostCCL and the various related works but does not clarify HostCCL’s position relative to those works, i.e., whether their methodologies are orthogonal to HostCCL or are competing baselines. Furthermore, additional experimental results are needed to compare against those related works.
Questions for authors
See weaknesses
2 comments
Thank you again for your feedback. Following your comments, we added comparisons with VCCL and MSCCL++, expanded the explanation of small-message performance and CPU overhead, and clarified HostCCL’s differences from related systems. The revision also includes end-to-end training results and a discussion of CPU requirements and platform coverage.
As the discussion period is coming to a close, we would appreciate your thoughts on whether these revisions address your concerns. If any points still need clarification, we would be happy to discuss them.
If you find that the revisions have addressed your concerns, we would be grateful if you could consider raising your rating. Thank you for your time and for helping us improve the paper.
Thank you for the review. We added VCCL and MSCCL++ comparisons and revised the explanations of the message-size and compute-overlap results.
W2. Comparisons and relation to prior work.
Figure 2 now compares all four collectives on four and eight L40S GPUs, covering eight sizes from 4 KiB to 256 MiB under the same measurement protocol. At every tested size from 64 to 256 MiB, HostCCL has the highest AllGather, ReduceScatter, and AllReduce throughput among the four libraries, using zero SMs for AllGather and one for the reductions. For example, eight-GPU AllReduce at 256 MiB reaches 17.10 GB/s with HostCCL, compared with 14.41 for NCCL, 14.39 for VCCL, and 5.38 for MSCCL++. VCCL leads AllToAll at that size. Figure 4 also includes both new baselines in the compute-overlap experiments.
MSCCL++ uses its default AllReduce selector, Fullmesh2 AllGather, and DSL ReduceScatter/AllToAll. Appendix G gives the versions, algorithm selections, and adapters.
We clarified the contribution in Sections 3.1 and 6. Throughout each collective call, the CPU executor observes tile arrivals, submits ready reductions, and releases forwarding as those reductions complete, while later reductions continue. Several transfers can remain in flight with a fixed worker SM budget because the CPU maintains their dependencies and copy engines move the data.
This differs from MSCCL++'s PortChannel, where a CPU proxy submits transfers but the GPU program issues requests and executes the associated waits. ResCCL optimizes schedules and block assignments within GPU communication kernels. These GPU-managed execution approaches are alternatives to CPU-managed collective progress. Host-controlled transport, such as UCCL-Tran, and collective algorithm optimizations can complement our executor. VCCL's CE paths directly overlap with our movement-only operations, which makes it a relevant baseline in the new comparison.
W1. Explaining the results and discussing limitations.
Small calls have little transfer or reduction work over which to amortize CPU dispatch and completion feedback. This helps explain why NCCL and VCCL are faster for AllGather and the reductions at 4 KiB-4 MiB (Section 5.2). At larger sizes, transfer and reduction work can overlap across tiles. The worker experiment in Section 5.3 also shows why a small SM budget is sufficient on these paths: adding SMs greatly increases ready-input reduction capacity but brings little improvement to full-collective throughput.
Section 5.4 explains how this affects concurrent computation. AllGather uses no communication worker, and GEMM stays within 3% of its standalone time. Joint completion takes 8.69-8.70 ms, versus 14.92-15.21 ms with default NCCL. With ReduceScatter, computation finishes before communication, so communication still determines joint completion despite the lower compute interference.
We brought the CPU cost, small-message overheads, and platform and scale coverage together in Section 5.6. We also added full-step ViT-L/16 DDP training on eight L40S GPUs, with active data loading and the same CPU allowance under both backends (Section 5.5). With matched batch size and a 256 MiB bucket cap, HostCCL improves throughput by 7.08% over NCCL using default communication settings. Summed CPU time, including DataLoaders, is 3.72 times as much.
We would appreciate your reassessment in light of these new comparisons and explanations.