acceptodds
Under review as a conference paper at ICLR 2027

HostCCL: Decoupling Collective Progress from GPU Execution

Abstract

GPU-driven collective communication uses streaming multiprocessors (SMs) for protocol progress as well as tensor processing. Peer-readiness waits therefore retain GPU resources even when data processing cannot proceed. We present HostCCL, a collective runtime that moves dependency progression to the CPU and uses copy engines for data movement. A CPU executor tracks tile dependencies and submits only ready reductions to a GPU worker, allowing transfer concurrency and reduction parallelism to be configured independently. Batched submission and ordered completion feedback let completed tiles begin forwarding while later reductions execute. On four and eight PCIe-connected L40S GPUs, HostCCL runs AllGather and AllToAll without steady-state GPU kernels, and ReduceScatter and AllReduce with one worker block per GPU. At the evaluated large FP32 reduction points, HostCCL uses 50-75% fewer thread blocks than default NCCL and improves throughput by up to 23.3%. In a controlled four-GPU experiment pairing 256 MiB AllGather with fixed GEMM work, HostCCL completes the joint work in 8 ms, compared with 14.9-15.2 ms for default NCCL.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Collective Communication, GPU Efficiency, Runtime Systems, Distributed Machine Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_wYwwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work addresses the compute bottlenecks in NCCL that arise from using streaming multiprocessors (SMs) for both communication and computation. The authors propose HostCCL which precomputes the dependency graph on the CPU that enforces task sequential correctness and manages task completion signals from the GPU. HostCCL improves on NCCL by reducing the SM budget and generally has better throughput in most tested settings.

Critical strengths

Experimental results generally show HostCCL improves over NCCL, in particular on the small SM budget settings on which the paper focuses.

Critical weaknesses

Discussion of results are generally sparse and superficial, reporting only the numerical results and comparisons without further elaboration. Figure 2 shows NCCL generally outperforms HostCCL on smaller tensor sizes and it is briefly mentioned at the end of Section 5.2, but no discussion is provided of why this occurs. Similarly, the paper lacks a limitations section in the main body (Nor in the appendix) to acknowledge potential concerns, e.g. higher CPU usage (Figure 7).

HostCCL is only compared against NCCL and not against any of the related work mentioned in Section 6, weakening its significance. The paper only gives a brief one-sentence description of the difference between HostCCL and the various related works but does not clarify HostCCL’s position relative to those works, i.e., whether their methodologies are orthogonal to HostCCL or are competing baselines. Furthermore, additional experimental results are needed to compare against those related works.

Questions for authors

See weaknesses

2 comments

Thank you for the review. We added VCCL and MSCCL++ comparisons and revised the explanations of the message-size and compute-overlap results.

W2. Comparisons and relation to prior work.

Figure 2 now compares all four collectives on four and eight L40S GPUs, covering eight sizes from 4 KiB to 256 MiB under the same measurement protocol. At every tested size from 64 to 256 MiB, HostCCL has the highest AllGather, ReduceScatter, and AllReduce throughput among the four libraries, using zero SMs for AllGather and one for the reductions. For example, eight-GPU AllReduce at 256 MiB reaches 17.10 GB/s with HostCCL, compared with 14.41 for NCCL, 14.39 for VCCL, and 5.38 for MSCCL++. VCCL leads AllToAll at that size. Figure 4 also includes both new baselines in the compute-overlap experiments.

MSCCL++ uses its default AllReduce selector, Fullmesh2 AllGather, and DSL ReduceScatter/AllToAll. Appendix G gives the versions, algorithm selections, and adapters.

We clarified the contribution in Sections 3.1 and 6. Throughout each collective call, the CPU executor observes tile arrivals, submits ready reductions, and releases forwarding as those reductions complete, while later reductions continue. Several transfers can remain in flight with a fixed worker SM budget because the CPU maintains their dependencies and copy engines move the data.

This differs from MSCCL++'s PortChannel, where a CPU proxy submits transfers but the GPU program issues requests and executes the associated waits. ResCCL optimizes schedules and block assignments within GPU communication kernels. These GPU-managed execution approaches are alternatives to CPU-managed collective progress. Host-controlled transport, such as UCCL-Tran, and collective algorithm optimizations can complement our executor. VCCL's CE paths directly overlap with our movement-only operations, which makes it a relevant baseline in the new comparison.

W1. Explaining the results and discussing limitations.

Small calls have little transfer or reduction work over which to amortize CPU dispatch and completion feedback. This helps explain why NCCL and VCCL are faster for AllGather and the reductions at 4 KiB-4 MiB (Section 5.2). At larger sizes, transfer and reduction work can overlap across tiles. The worker experiment in Section 5.3 also shows why a small SM budget is sufficient on these paths: adding SMs greatly increases ready-input reduction capacity but brings little improvement to full-collective throughput.

Section 5.4 explains how this affects concurrent computation. AllGather uses no communication worker, and GEMM stays within 3% of its standalone time. Joint completion takes 8.69-8.70 ms, versus 14.92-15.21 ms with default NCCL. With ReduceScatter, computation finishes before communication, so communication still determines joint completion despite the lower compute interference.

We brought the CPU cost, small-message overheads, and platform and scale coverage together in Section 5.6. We also added full-step ViT-L/16 DDP training on eight L40S GPUs, with active data loading and the same CPU allowance under both backends (Section 5.5). With matched batch size and a 256 MiB bucket cap, HostCCL improves throughput by 7.08% over NCCL using default communication settings. Summed CPU time, including DataLoaders, is 3.72 times as much.

We would appreciate your reassessment in light of these new comparisons and explanations.

Thank you again for your feedback. Following your comments, we added comparisons with VCCL and MSCCL++, expanded the explanation of small-message performance and CPU overhead, and clarified HostCCL’s differences from related systems. The revision also includes end-to-end training results and a discussion of CPU requirements and platform coverage.

As the discussion period is coming to a close, we would appreciate your thoughts on whether these revisions address your concerns. If any points still need clarification, we would be happy to discuss them.

If you find that the revisions have addressed your concerns, we would be grateful if you could consider raising your rating. Thank you for your time and for helping us improve the paper.

Reviewer_4TiwOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents HostCCL, a collective communication runtime designed to decouple protocol dependency tracking and progress orchestration from GPU SMs. Conventional GPU-driven collective communication libraries (such as NVIDIA NCCL) rely on GPU thread blocks to manage inter-rank dependencies, synchronization semaphores, and data reductions, which can retain valuable GPU compute resources even while waiting for remote peers.

Critical strengths

S1. Decoupling dependency progression from movement and reduction arithmetic is a principled design. The clear separation of transfer concurrency and reduction parallelism allows transport depth to scale without inflating the GPU SM budget.

S2. The asymmetric control queue design—combining host write-combined C2D FIFOs, fine-grained tile completion reporting via a monotonic D2C tail, and EndBatch generation sealing—allows pipelined task releases and early forwarding without waiting for full kernel exits or heavy per-task polling overheads.

S3. In the joint GEMM + AllGather benchmarks, HostCCL eliminates compute kernel resource contention, demonstrating near-zero compute stream degradation (~3% slowdown vs. ~76–79% slowdown under NCCL), translating directly to a ~42% reduction in total end-to-end joint latency.

S4. The paper provides exceptional diagnostic depth, including breakdown analysis of host observation delays vs. CUDA issue latencies, CPU provisioning requirements, SM budget sweeps, and prefilled worker reduction capacity vs. full collective bandwidth.

Critical weaknesses

W1. The evaluation is conducted strictly on PCIe Gen 4/5 connected L40S GPUs. High-performance distributed deep learning clusters predominantly utilize high-bandwidth interconnects (NVLink / NVSwitch providing 900 GB/s to 1.8 TB/s per GPU) and multi-node scale-out fabrics (InfiniBand/RoCE via GPUDirect RDMA). On ultra-high-bandwidth NVLink systems, CPU-GPU control queue polling latencies and host scheduling jitter could quickly become the primary throughput bottleneck.

W2. HostCCL trades GPU SM capacity for host CPU core allocation and CPU time. As documented in Section 5.3 and Appendices D/E, HostCCL consumes 5.6× to 5.8× more CPU time per call than default NCCL (e.g., ~1514 CPU-ms/call vs. ~259 CPU-ms/call for 256 MiB AllReduce). In production end-to-end training pipelines where host CPUs handle data loading, preprocessing, tokenization, dynamic scheduling, or CPU-offloaded optimizer states (e.g., ZeRO-Offload), dedicating 4–8 host CPU cores per GPU purely to collective progress polling is a major resource constraint.

W3. HostCCL exhibits noticeable latency/throughput degradation for small-to-medium payload sizes (4 KiB to 4 MiB) across all tested collectives compared to default NCCL (Figure 2). For workloads dominated by fine-grained or latency-sensitive communications (e.g., Tensor Parallelism in LLMs, pipeline bubble exchanges, or Mixture-of-Experts routing for small batch sizes), HostCCL's host dispatch latency overhead outweighs its SM savings.

W4. While the paper provides comparisons against default, efficiency, and fixed-CTA NCCL configurations, it omits experimental comparisons against recent open-source custom communication frameworks that also optimize CTA budgets or offload control/transport, such as MSCCL++ or ResCCL.

Questions for authors

D1. In an end-to-end model training run where CPU cores are actively engaged in PyTorch data loading or optimizer step processing, how sensitive is HostCCL to host thread preemption or CPU core frequency scaling? Have you evaluated performance under active CPU background load?

D2. What are the fundamental technical blockers or expected performance trade-offs when extending HostCCL to high-bandwidth NVLink systems or multi-node GPUDirect RDMA networks? Would CPU response latency become a bottleneck when link bandwidth exceeds 900 GB/s?

D3. Can the host executor dynamically fall back to a standard GPU-driven execution path for messages <4 MiB to avoid the latency penalty observed in Figure 2?

D4. How easily does the lightweight GPU worker support quantized or non-standard data types (e.g., BF16, FP8, INT8 with scaling factors) without incurring register pressure or alignment inefficiencies?

5 comments

Thank you for the review.

D1 / W2. CPU contention during training.

In the new ViT-L/16 experiment, training, communication, and two active DataLoader workers per rank share seven physical cores and their SMT siblings. With eager FP32 PyTorch DDP on eight L40S GPUs and Imagenette, HostCCL takes 150.78 ms/step compared with 161.45 ms for NCCL, a 7.08% throughput gain (Section 5.5, Figure 5). Both backends use the same CPU allowance, batch size, and 256 MiB bucket cap; NCCL retains its default communication settings. These are full-step measurements, including data loading and SGD, reported as medians of three independent launches.

We also measured CPU use for the whole training job, including the DataLoaders: 5.2204 CPU-s/step for HostCCL and 1.4042 for NCCL, or 3.72 times as much. We have not run separate preemption or frequency-scaling tests.

W4. Additional baselines.

Figure 2 now includes VCCL v2.0.0 and MSCCL++ on four and eight GPUs, covering all four collectives and eight sizes from 4 KiB to 256 MiB under the same measurement protocol. Figure 4 adds both libraries to the AllGather-GEMM and ReduceScatter-GEMM comparisons.

For example, eight-GPU AllReduce at 256 MiB reaches 17.10 GB/s with HostCCL, compared with 14.41 for NCCL, 14.39 for VCCL, and 5.38 for MSCCL++. VCCL is fastest for AllToAll at that size. Appendix G gives the configurations, including MSCCL++'s default AllReduce selector, Fullmesh2 AllGather, and DSL ReduceScatter/AllToAll.

D2 / W1. Higher-bandwidth architectures and networks.

For a fixed tile size, faster links leave less time for the CPU to process completions and submit ready work. Host progress can then limit throughput. Our next step is to study this on NVLink systems and across nodes, adjusting batching, the transfer window, and reduction capacity together. HostCCL lets us enlarge the transfer window without increasing the worker's SM budget, and scale the worker separately when reductions become the bottleneck.

D3 / W3. Size-based backend selection.

A prototype in our DDP adapter routes small buckets to NCCL and larger buckets to HostCCL. The reported training runs use a fixed backend per model: ResNet-50 stays on native NCCL, with a 0.20% step-time difference from native DDP, while ViT-L/16 uses HostCCL and delivers the 7.08% throughput gain above.

D4. Data types and quantization.

The worker already implements Sum for FP16, BF16, and INT8, although the measurements in the paper use FP32. FP8 needs a new implementation. For scaled quantization, the reduction kernel also needs to handle scale factors and choose an accumulation type. These kernels could use the existing task interface; their register pressure, alignment, and throughput would need evaluation.

It would be more interesting to see the performance measurements on FP16, BF16, and INT8, rather than the only FP32.

Thank you for this suggestion! We have added FP16, BF16, and INT8 measurements in Appendix J (Figure 12).

The INT8 result was particularly informative: FP16 and BF16 sustain high throughput with one SM per GPU, while INT8 reaches comparable bandwidth with two. Across these configurations, HostCCL achieves 14.2–17.9% higher throughput than default NCCL with a 50–75% smaller communication SM budget. NCCL uses four SMs per GPU in these measurements.

Your suggestion helped us show how the worker budget should adapt to different data types. If these additional results address your concern, we would be grateful if you could consider raising your rating. : )

Reviewer_jzrtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

HostCCL addresses GPU resource utilization during GPU-driven collective communication by offloading dependency tracking and communication coordination to the CPU. HostCCL divides collective communication into smaller tiles and uses the CPU to coordinate copy-engine transfers and GPU reductions, allowing receipt, reduction, and forwarding of different tiles to overlap. HostCCL also supports AllReduce by connecting ReduceScatter and AllGather at the tile level. For the evaluated large-message reductions, HostCCL achieves up to 23.3% higher throughput with a 50–75% lower SM budget than the default NCCL.

Critical strengths

S1. The paper presents a clear design that separates CPU coordination, copy-engine transfers, and GPU reductions. The tile-level dependency tracking allows transfers and reductions to overlap while keeping the GPU worker small.

S2. The performance result for HostCCL is strong. The paper shows that HostCCL achieve up to 23.3% higher reduction throughput with a one-SM budget. At a matched one-SM budget, it delivers 40–45% higher throughput than NCCL.

Critical weaknesses

W1: Weak fit for ICLR. The paper shows collective-level performance improvements, but there are no end-to-end ML results. It does not evaluate any training or inference workloads (e.g., data-parallel or tensor-parallel training) and does not show integration with PyTorch. Thus, how much the performance gain translates to ML training or inference is still not clear.

W2: The cost of CPU usage is concerning. HostCCL uses more CPU time per call than NCCL, and its throughput collapses with only one or two cores per rank. In real ML training, CPU cores are also used for data loading and framework overhead, so how this high CPU usage would affect end-to-end ML training remains unclear.

Questions for authors

Section 5.3 and Appendices D show that HostCCL requires more CPU time. Under what conditions would CPU processing become the critical path, especially when CPU resources are shared with an ML workload? Could you also compare CPU usage between HostCCL and default NCCL during full ML workloads?

2 comments

Thank you. We added PyTorch DDP experiments with active data loading to measure training performance and CPU cost together.

W1. ML workloads and PyTorch integration.

We trained ViT-L/16 and ResNet-50 with eager FP32 PyTorch DDP on eight L40S GPUs using Imagenette (Section 5.5, Figure 5). A DDP communication hook submits gradient buckets to HostCCL; native NCCL handles initialization and framework control collectives. Two DataLoader workers per rank decode and augment images. Step time includes loading, H2D transfers, forward/backward computation, SGD, and GPU completion.

At batch 4/GPU and the same 256 MiB bucket cap, ViT-L/16 takes 150.78 ms/step with HostCCL and 161.45 ms with NCCL, a 7.08% throughput gain. The communication-worker budgets are one and four SMs. With PyTorch's default bucket cap, NCCL takes 156.03 ms/step; HostCCL's throughput is 3.49% higher than that control.

For ResNet-50, the adapter leaves native DDP unchanged because the model's total gradients fall below the routing threshold. It takes 56.46 versus 56.34 ms/step, a 0.20% difference. Results are medians of three independent launches, each with ten warmup and fifty measured steps. Appendix G.3 gives the integration details and gradient/SGD-update checks.

W2 and Q: CPU usage during training.

At the matched bucket cap, summed ViT CPU time across all ranks and DataLoaders is 5.2204 CPU-s/step for HostCCL and 1.4042 for NCCL, or 3.72 times as much. For both libraries, each rank has access to seven GPU-local physical cores and their SMT siblings. Communication, training, and data loading share this set of CPUs. The 7.08% training gain is measured with data loading active under this allowance.

Host progress becomes critical when the CPU cannot dispatch ready work or process completions fast enough to keep transfers and reductions running. The core-allocation experiment shows this sensitivity: eight-GPU 128 MiB ReduceScatter reaches 16.45 GB/s with four local cores per rank, but 0.69/0.96 GB/s with one/two cores. Application work that delays the progress threads can cause the same problem. Training with a reduced CPU allowance remains to be evaluated.

We now report CPU time beside training throughput in Section 5.5 and discuss CPU provisioning in Section 5.6.

Thank you again for your feedback. We added end-to-end ViT-L/16 training measurements on eight L40S GPUs with active data loading and augmentation. Under matched training settings, HostCCL achieves 7.1% higher throughput than NCCL.

Both libraries receive the same CPU allowance, shared by training, data loading, and communication. We also report CPU time including the DataLoaders, allowing the training gain and additional host cost to be assessed together.

Before the discussion closes, could you let us know whether these results address your concerns about training performance and CPU usage? If they do, we would be grateful if you could consider raising your rating. Thank you for your time and helpful feedback.

Reviewer_mJkDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HostCCL, a collective communication runtime that decouples dependency progression and data movement from GPU reduction execution. A CPU executor manages tile dependencies, copy engines transfer tensor payloads between GPUs, and a small GPU worker executes ready reductions. This design allows transfer concurrency and reduction parallelism to be configured independently. On four and eight PCIe-connected L40S GPUs, HostCCL executes AllGather and AllToAll without steady-state GPU kernels and reducing collectives with a one-SM worker budget. The paper reports throughput improvements of up to 23.3% over default NCCL for large FP32 reductions and evaluates interference with concurrent GEMM computation

Critical strengths

The paper addresses a real and important problem: collective communication kernels can retain GPU resources while waiting for peer readiness, reducing the resources available to concurrent computation. The motivation for separating communication progress from reduction execution is convincing. The evaluation includes both default NCCL and comparisons under matched SM budgets, providing useful evidence that the approach can sustain competitive throughput with fewer communication blocks on the tested platform.

Critical weaknesses
  1. Insufficiently clear explanation of the core design.
    I found Section 3 difficult to follow, particularly the differences between NCCL’s communication execution and HostCCL’s approach. Although Figure 1 illustrates HostCCL’s tile pipeline, it does not directly contrast the two designs. The authors should more clearly explain how NCCL couples communication concurrency to GPU blocks, how HostCCL changes this relationship, and how additional transfers can remain in flight without increasing the reduction worker’s SM budget. A side-by-side execution diagram identifying CPU, copy-engine, and GPU responsibilities would substantially improve clarity.

  2. Limited platform coverage and missing comparison with NCCL’s CE path.
    All experiments use PCIe-connected L40S GPUs. As Section 5.1 acknowledges, NCCL already provides a conditional zero-CTA copy-engine path for supported reduction-free collectives on NVLink platforms. The current results therefore do not establish HostCCL’s advantage over this existing mechanism on hardware commonly used for large-scale training. Evaluation on an NVLink platform, with the relevant NCCL optimizations enabled, is needed. For reducing collectives, higher interconnect bandwidth may also change the worker budget required to sustain throughput.

  3. Unclear compatibility and performance with CUDA Graphs.
    HostCCL moves dependency tracking, transfer submission, and task dispatch to CPU threads. CUDA Graph replay reduces CPU launch overhead in NCCL-based workloads, making it an important execution mode for comparison. However, the paper neither explains whether HostCCL supports graph capture and replay nor compares it with graph-captured NCCL. This omission is particularly relevant because the paper already reports approximately 5.6–5.8 times NCCL’s CPU time for certain large-message reduction configurations. The practical trade-off between GPU resource savings and additional CPU work needs further evaluation under graph-based execution.

  4. Missing empirical comparisons with related communication systems.
    Section 6 discusses VCCL and UCCL-Tran, which share relevant ideas involving CPU progress or host-controlled transport, but the evaluation compares only against NCCL configurations. Comparisons with these systems, where their supported operations and hardware overlap, would help establish the benefit of HostCCL’s particular design. Any differences in functionality or deployment requirements should be explained so that the comparisons remain meaningful.

  5. No inter-node scalability evaluation.
    The four- and eight-GPU experiments are confined to one server. They do not establish how CPU-managed dependencies, notifications, and reduction forwarding behave across a network. Since the reported benefits are concentrated in large-message communication, their relevance to distributed training would be more convincing with inter-node evidence. Even a small two- or four-node experiment would help assess basic scalability and identify whether host progress becomes a bottleneck as the number of peers grows.

  6. Missing end-to-end training validation.
    The controlled GEMM–communication experiment is informative, but it does not establish improvements in actual training throughput or iteration time. Indeed, Figure 4 shows that reduced compute interference does not consistently improve joint completion: HostCCL’s ReduceScatter result is close to default NCCL and slower than one-SM NCCL. A representative training workload is needed to determine whether the communication benefits translate into an overall speedup under realistic dependencies and CPU contention.

Questions for authors
  1. Can you provide a direct NCCL–HostCCL execution comparison showing how transfer concurrency increases while the reduction worker’s SM budget remains fixed?
  2. How does HostCCL perform on NVLink hardware against NCCL’s applicable zero-CTA CE paths? What worker budget is required for reducing collectives on that platform?
  3. Does HostCCL support CUDA Graph capture and replay? How does it compare with graph-captured NCCL in latency, throughput, and CPU cost?
  4. Can you provide comparisons with VCCL or UCCL on commonly supported workloads ?
  5. Can you report results on two or four nodes to demonstrate basic inter-node scalability and characterize the associated performance?
  6. Can you evaluate a representative distributed training workload and report training throughput or iteration time?
3 comments

Thank you for the questions. We added eight-GPU training experiments and comparisons with VCCL and MSCCL++, and clarified how transfer concurrency is controlled.

Q6. End-to-end training.

We trained ViT-L/16 and ResNet-50 with eager FP32 PyTorch DDP on eight L40S GPUs using Imagenette (Section 5.5, Figure 5). Two DataLoader workers per rank decode and augment images. Step time includes loading, H2D transfers, forward/backward computation, SGD, and GPU completion.

At batch 4/GPU and the same 256 MiB bucket cap, ViT-L/16 takes 150.78 ms/step with HostCCL and 161.45 ms with NCCL, a 7.08% throughput gain. NCCL uses its default communication settings; the communication-worker budgets are one and four SMs. With PyTorch's default bucket cap, NCCL takes 156.03 ms/step, so HostCCL's throughput is also 3.49% higher than that control. ResNet-50 retains native NCCL through the adapter: 56.46 versus 56.34 ms/step, a 0.20% difference.

Results are medians of three independent launches, each with ten warmup and fifty measured steps. For both libraries, communication, training, and data loading share seven physical cores per rank and their SMT siblings. Summed ViT CPU time, including DataLoaders, is 5.2204 versus 1.4042 CPU-s/step at the matched bucket cap. In the ReduceScatter-GEMM case you noted, communication sets the joint completion time (Section 5.4).

Q1. Transfer concurrency with a fixed worker budget.

On the GPU-driven NCCL paths studied here, channel CTAs check peer readiness and perform reductions, with payload movement handled by the selected transport. A channel can pipeline multiple data ranges. In HostCCL, the CPU tracks arrival dependencies and outstanding transfers, copy engines move the data, and the GPU worker processes ready reductions.

Several tiles can therefore remain in flight while one worker block reduces a ready tile. Once it finishes, the CPU releases that tile's forwarding while other tiles arrive or undergo reduction. Increasing the copy window changes the number of outstanding transfers without changing the worker grid. Local GPU barriers and memory ordering remain necessary. Section 3.1 now explains this comparison alongside Figure 1.

Q4. Comparisons with VCCL and MSCCL++.

We added VCCL v2.0.0 and MSCCL++ to the four- and eight-GPU experiments: all four collectives, all eight sizes from 4 KiB to 256 MiB, and the same platform, CPU allowance, out-of-place buffers, and timing protocol. At 256 MiB, eight-GPU AllReduce reaches 17.10 GB/s with HostCCL, 14.41 with NCCL, 14.39 with VCCL, and 5.38 with MSCCL++. VCCL leads AllToAll at that size. Both systems also appear in the AllGather-GEMM and ReduceScatter-GEMM experiments (Figures 2 and 4).

Q3. CUDA Graph capture/replay.

We currently support eager execution. Application-level collective capture/replay remains to be implemented and evaluated against graph-captured NCCL. A complete replay must trigger host progress and advance per-run completion state; the cached tile DAG supplies the dependency structure. Appendix G.3 gives the eager setup used in our measurements.

Q2 and Q5. NVLink and inter-node scaling.

For higher-bandwidth GPU architectures and larger clusters, our next step is to adapt CPU progress and reduction capacity together. The copy window and worker grid can be tuned independently. We plan to study their settings as link bandwidth and peer count increase, with the goal of keeping the communication SM budget small. This work will compare against NCCL's applicable NVLink CE paths, determine the worker budget needed on each platform, and measure progress and forwarding across nodes (Section 5.6).

Thank you for the additional experiments. Q6 is substantively addressed, but the other five concerns are largely unresolved, so I am maintaining my weak reject rating.

Q1 (Core design) — not resolved. Figure 1 is unchanged and the added prose still does not show how SM budget stays fixed while transfer concurrency increases.

Q2 (NVLink / zero-CTA CE path) — deferred. NVLink is the dominant training platform and NCCL already provides a CE path there. Deferring this to future work leaves the main practical comparison unanswered.

Q3 (CUDA Graph) — not answered. The paper shows HostCCL uses roughly 3.7–5.8× NCCL's CPU time, but how it compares to graph-captured NCCL, is not addressed at all.

Q4 (VCCL / MSCCL++) — data added, no analysis. Numbers are reported but not explained. Without analysis I cannot tell whether HostCCL's gains come from the proposed design or from tuning differences. Q5 (Inter-node) — deferred. Still no multi-node evidence for a system whose claimed benefits grow with scale.

Summary. I would reconsider if the final version adds (a) an execution diagram with measurements, (b) an NVLink comparison, (c) a CUDA Graph discussion or experiment, and (d) analysis of the VCCL/MSCCL++ results.

Thank you for the suggestion. We have clarified the tile execution example in Section 3.1, added fixed-worker measurements in Appendix H and a CUDA Graph comparison in Appendix I, and expanded the baseline analysis in Section 5.2 and Appendix G.1.

Q1. We have added a concrete tile example in Section 3.1 and fixed-SM measurements with a bandwidth plot in Appendix H. Figure 1 remains focused on the overall architecture. While the worker reduces tile i, a CE can forward completed tile i-1, and the CPU can queue other ready copies. Copy requests go to the CE transport, while ready reductions enter the device FIFO. Increasing the copy window admits more submitted but incomplete transfers, while the independently configured worker grid maintains the same SM budget.

For 128 MiB ReduceScatter on four L40S GPUs, traces show the peak outstanding-copy count increasing from 1 to 8 with a fixed one-SM worker budget. Increasing the window from 1 to 2 raises bandwidth from 22.66 to 24.50 GB/s, an 8.1% improvement.

Q2/Q5. We apologize that we cannot provide the NVLink and inter-node measurements in this round, as we currently lack access to the required GPU resources. Newer GPU architectures and higher-bandwidth interconnects also require further study of how host progress and reduction capacity should match the transport rate. In future work, we will investigate completion batching, transfer granularity, and worker sizing on these platforms and larger clusters, aiming to exploit their bandwidth while retaining a small communication SM budget. This evaluation will include comparisons with NCCL's applicable zero-CTA CE paths and measurements of inter-node scalability.

Q3. We have added a CUDA Graph comparison in Appendix I and summarized the main result in Section 5.3. HostCCL currently uses eager submission and does not yet support full-collective CUDA Graph capture and replay. We compared eager HostCCL with both eager and graph-captured NCCL on four L40S GPUs. Each graph replay contains sixteen AllReduces; all modes synchronize after sixteen calls, and graph setup is excluded from timing.

At 256 MiB, HostCCL takes 15.69 ms per AllReduce, compared with 18.58 ms for graph-captured NCCL, corresponding to 25.67 versus 21.67 GB/s and an 18.5% throughput advantage. Eager NCCL achieves essentially the same throughput as graph-captured NCCL at this size. HostCCL's summed CPU time across ranks is 5.9 times that of graph-captured NCCL. NCCL remains faster at the tested smaller sizes of 4 KiB and 1 MiB.

Q4. We have added execution-path analysis for VCCL and MSCCL++ in Section 5.2, and expanded Appendix G.1 with trace observations and the MSCCL++ ReduceScatter parameter sweep. At 256 MiB, VCCL uses copy engines without GPU kernels for AllToAll, where it achieves the highest throughput. Its AllGather and reducing collectives use GPU kernels and perform close to NCCL. HostCCL extends host-managed progress and CE transfers to these operations, using GPU workers only for ready reductions.

MSCCL++'s default AllReduce selects Fullmesh, which exchanges partitions with every peer and performs channel synchronization on the GPU. HostCCL instead forwards tiles along a ring through copy engines. For MSCCL++ ReduceScatter, we tested 1, 4, and 16 instances and selected the fastest configuration for each GPU count; Appendix G.1 documents these choices.

To isolate the benefit of independent transfer depth, the experiment in Appendix H holds the algorithm, tile size, CPU allowance, and one-SM worker budget fixed. Increasing only the copy window from one to two improves 128 MiB ReduceScatter bandwidth by 8.1%. This directly measures a throughput gain from admitting more transfers while keeping reduction resources unchanged.

Thank you again for your careful review. We would appreciate it if you could reconsider your assessment and rating in light of the additional experiments and clarifications.

Comments on the paper

1 comment

Below are key measurements added during rebuttal, with manuscript locations for the full results. All performance values are medians of three independent launches on L40S GPUs. CPU allowances are matched between libraries within each experiment. Bandwidth is collective bus bandwidth in GB/s.

1. Additional communication systems — Section 5.2, Figure 2

Eight GPUs, 256 MiB per rank; FP32 Sum for reducing collectives.

\hline \text{Collective} & \text{NCCL} & \text{VCCL} & \text{MSCCL++} & \text{HostCCL} & \begin{array}{c}\text{HostCCL}\\\\\text{SMs/GPU}\end{array} \\\\ \hline \text{AllGather} & 14.35 & 14.72 & 5.29 & \mathbf{17.17} & 0 \\\\ \text{ReduceScatter} & 13.63 & 13.70 & 2.71 & \mathbf{16.80} & 1 \\\\ \text{AllReduce} & 14.41 & 14.39 & 5.38 & \mathbf{17.10} & 1 \\\\ \text{AllToAll} & 6.95 & \mathbf{7.52} & 5.22 & 6.65 & 0 \\\\ \hline \end{array}$$ NCCL uses two SMs/GPU at these points. NCCL and VCCL use default tuning; MSCCL++ uses default AllReduce, Fullmesh2 AllGather, and DSL ReduceScatter/AllToAll (Appendix G.1). Figure 2 covers four/eight GPUs and all eight sizes; Figure 4 also adds both systems to the compute-overlap comparison. **2. End-to-end training — Section 5.5, Figure 5** ViT-L/16, eight GPUs, FP32 PyTorch DDP with active Imagenette loading and augmentation. Batch size, bucket cap, and CPU allowance are matched; NCCL retains its default communication settings. $$\begin{array}{lrr} \hline \text{Metric} & \text{NCCL} & \text{HostCCL} \\\\ \hline \text{Step time (ms)} & 161.45 & \mathbf{150.78} \\\\ \text{Communication SMs/GPU} & 4 & \mathbf{1} \\\\ \begin{array}{l}\text{Summed CPU time, including}\\\\\text{DataLoaders (CPU-s/step)}\end{array} & 1.4042 & 5.2204 \\\\ \hline \end{array}$$ Training throughput improves by **7.08%**. An additional NCCL control with PyTorch's default bucket cap takes 156.03 ms/step; HostCCL's throughput is 3.49% higher than that control. **3. Transfer depth with a fixed SM budget — Appendix H** Four GPUs, 128 MiB FP32 ReduceScatter. The ring algorithm, tile size, and CPU allowance stay fixed. $$\begin{array}{lrrr} \hline \text{Copy window} & \begin{array}{c}\text{Traced peak}\\\\\text{outstanding copies}\end{array} & \begin{array}{c}\text{Worker}\\\\\text{SMs/GPU}\end{array} & \text{Bandwidth} \\\\ \hline 1 & 1 & 1 & 22.66 \\\\ 2 & 2 & 1 & 24.50 \\\\ 4 & 4 & 1 & 24.51 \\\\ 8 & 8 & 1 & 24.53 \\\\ \hline \end{array}$$ Outstanding copies are submitted but incomplete transfers. Increasing the window from one to two improves bandwidth by **8.1%**, with the worker budget unchanged. This measures the benefit of increasing transfer depth independently of reduction resources. **4. CUDA Graph comparison — Appendix I** Four GPUs, 256 MiB FP32 AllReduce. Each NCCL graph contains sixteen calls; all modes synchronize after sixteen calls, and graph setup is excluded from timing. $$\begin{array}{lrrr} \hline \text{Execution mode} & \begin{array}{c}\text{Latency}\\\\\text{(ms/call)}\end{array} & \text{Bandwidth} & \begin{array}{c}\text{Summed CPU time}\\\\\text{(CPU-ms/call)}\end{array} \\\\ \hline \text{NCCL eager} & 18.58 & 21.67 & 74.20 \\\\ \text{NCCL CUDA Graph} & 18.58 & 21.67 & 74.23 \\\\ \text{HostCCL eager} & \mathbf{15.69} & \mathbf{25.67} & 438.47 \\\\ \hline \end{array}$$ HostCCL achieves **18.5%** higher throughput at this size, with 5.9 times the CPU time. NCCL remains faster at the tested 4 KiB and 1 MiB sizes. HostCCL currently supports eager collective execution. **5. FP16, BF16, and INT8 — Appendix J, Figure 12** Eight GPUs; ReduceScatter uses 128 MiB input per rank and AllReduce uses 256 MiB, holding bytes fixed across types. Every pair below is **HostCCL / NCCL**. $$\begin{array}{lrrr} \hline \text{Type} & \text{SMs/GPU} & \begin{array}{c}\text{ReduceScatter}\\\\\text{bandwidth}\end{array} & \begin{array}{c}\text{AllReduce}\\\\\text{bandwidth}\end{array} \\\\ \hline \text{FP16} & 1\\,/\\,4 & 24.62\\,/\\,21.40 & 25.62\\,/\\,21.74 \\\\ \text{BF16} & 1\\,/\\,4 & 24.61\\,/\\,21.37 & 25.63\\,/\\,21.75 \\\\ \text{INT8} & 2\\,/\\,4 & 24.50\\,/\\,21.44 & 25.63\\,/\\,21.80 \\\\ \hline \end{array}$$ Across these six cases, HostCCL delivers **14.2–17.9%** higher throughput with a **50–75% smaller SM budget**. FP16/BF16 sustain this throughput with one SM; INT8 reaches comparable bandwidth with two. Thank you to the reviewers for the suggestions that led to these additional experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.