- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies AllReduce performance under partial network failures that create asymmetric bandwidth across GPU participants. It presents communication-time lower bounds for several failure configurations and proposes OptCC, a four-stage pipelined algorithm. OptCC partitions data into segments, redistributes communication work toward healthy GPUs, and overlaps healthy-node communication with transfers involving degraded participants. It further uses bubble filling to exploit available idle intervals. The evaluation uses SimAI to study different cluster sizes, message sizes, and failure configurations. For a single straggler retaining at least half of its original bandwidth, the authors report AllReduce completion times within 2–6% of fault-free NCCL Ring performance.
Critical strengths
This work addresses a practical production problem: partial network failures reduce the available NIC bandwidth, causing NCCL’s standard Ring AllReduce to be bottlenecked by degraded GPU network. OptCC recognizes that these participants need not carry the same communication volume as healthy GPUs and translates this insight into a four-stage pipeline that redistributes communication and overlaps transfers. This provides a constructive approach to reducing the collective-level impact of localized bandwidth degradation.
Critical weaknesses
-
Lack of real-system validation. The evaluation is entirely simulation-based. The authors provide reference implementations of the scheduling algorithms, but the publicly released artifact does not include a real-GPU communication prototype. So the real overheads and resource contention remain unvalidated on real hardware. This leaves uncertainty about whether the reported near no fault performance translates to actual training systems. A small-scale GPU testbed evaluation measuring complete AllReduce and training-iteration times would substantially strengthen the empirical claims. For example, R²CCL (the baseline method) evaluates its implementation on two physical GPU servers and uses SimAI to study larger-scale configurations; OptCC could also adopt a similar evaluation strategy.
-
Simplified network model and limited analysis of fabric contention. This network model does not consider the impact of congestion control and flow control algorithms, such as DCQCN/PFC. The model does not consider intra-host networking congestion as well, at least in the main body. These effects may disrupt the communication model on which OptCC relies. Please clarify the network assumptions and evaluate OptCC with background traffic and congestion.
Questions for authors
- Could you provide a real-world experiment, at least with 2 real GPU servers?
- Does the simplified network model accurately capture network behavior in real-world deployments? The authors should discuss and clarify it.