acceptodds
Under review as a conference paper at ICLR 2027

Bandwidth-Optimal AllReduce Under Network Failures

Abstract

Network failures are among the most frequent hardware faults in large-scale GPU clusters and a leading cause of training-job interruptions. Modern collective communication libraries such as NCCL mitigate network failures by rerouting traffic through surviving NICs on the same server, trading reduced inter-node bandwidth for uninterrupted training. However, the degraded server remains on the critical path of the standard ring algorithm, slowing the entire collective. We present the first information-theoretic lower bound on AllReduce completion time under asymmetric network bandwidth and show that when the straggler retains at least half of its original bandwidth, the unavoidable overhead relative to the fault-free optimum is only O(1/p) for p GPUs. We then design OptCC, a four-stage pipelined AllReduce algorithm that approaches this lower bound. Simulations on SimAI confirm that OptCC closes the gap left by existing fault-tolerant schemes: under practical network failures with up to 50% bandwidth loss, OptCC completes AllReduce within 2–6% of NCCL's fault-free ring performance, whereas the state-of-the-art incurs up to 57% overhead.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Collective communication, Fault-Tolerant Distributed Training, Bandwidth-Optimal Scheduling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dYbwOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies AllReduce performance under partial network failures that create asymmetric bandwidth across GPU participants. It presents communication-time lower bounds for several failure configurations and proposes OptCC, a four-stage pipelined algorithm. OptCC partitions data into segments, redistributes communication work toward healthy GPUs, and overlaps healthy-node communication with transfers involving degraded participants. It further uses bubble filling to exploit available idle intervals. The evaluation uses SimAI to study different cluster sizes, message sizes, and failure configurations. For a single straggler retaining at least half of its original bandwidth, the authors report AllReduce completion times within 2–6% of fault-free NCCL Ring performance.

Critical strengths

This work addresses a practical production problem: partial network failures reduce the available NIC bandwidth, causing NCCL’s standard Ring AllReduce to be bottlenecked by degraded GPU network. OptCC recognizes that these participants need not carry the same communication volume as healthy GPUs and translates this insight into a four-stage pipeline that redistributes communication and overlaps transfers. This provides a constructive approach to reducing the collective-level impact of localized bandwidth degradation.

Critical weaknesses
  1. Lack of real-system validation. The evaluation is entirely simulation-based. The authors provide reference implementations of the scheduling algorithms, but the publicly released artifact does not include a real-GPU communication prototype. So the real overheads and resource contention remain unvalidated on real hardware. This leaves uncertainty about whether the reported near no fault performance translates to actual training systems. A small-scale GPU testbed evaluation measuring complete AllReduce and training-iteration times would substantially strengthen the empirical claims. For example, R²CCL (the baseline method) evaluates its implementation on two physical GPU servers and uses SimAI to study larger-scale configurations; OptCC could also adopt a similar evaluation strategy.

  2. Simplified network model and limited analysis of fabric contention. This network model does not consider the impact of congestion control and flow control algorithms, such as DCQCN/PFC. The model does not consider intra-host networking congestion as well, at least in the main body. These effects may disrupt the communication model on which OptCC relies. Please clarify the network assumptions and evaluate OptCC with background traffic and congestion.

Questions for authors
  1. Could you provide a real-world experiment, at least with 2 real GPU servers?
  2. Does the simplified network model accurately capture network behavior in real-world deployments? The authors should discuss and clarify it.
Reviewer_BAASOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper considers the case of when NICs in a node (and sometimes in a few) fail, leading to degraded bandwidth between that node and other nodes. They show both that there is a bound on the time the operation can complete in this setting, and propose a method that achieves this in the limit and that is fairly practical. The approach has the healthy workers reduce-scatter their contributions, exchange partial results with the degraded worker to incorporate its contribution, and all-gather the completed results, pipelining these stages across data segments.

Critical strengths
  • The motivation and setting is both relevant and useful
  • A clear theoretical message: network degradation on one node does not have to proportionally degrade the entire collective operation (with exact bounds computed)
  • An algorithm that manages to get this result in the limit, so a practical application of the above message
  • Evaluation is ok, there are many settings properly looked into
Critical weaknesses
  • There should be proper discussion and comparison to ForestColl, which is cited. The paper claims to get the optimal result from what I can understand, though I think its expensive to actually construct practically. Even if it can't be compared to experimentally, a discussion of what the theoretical results in that paper are and how it relates to this paper should be there. For example, are the lower bounds a direct result of ForestColl's results, or are they a different setting.
  • The assumptions for the result seem to assume that a healthy worker communicating with a healthy worker cannot use the rest of their bandwidth allocation, do you think there is more to be gained if this is changed? Or whether it is more realistic this way?
  • Would be nice to have some real world (non-simulated) experiments
Questions for authors
  • I found the title a bit misleading (though it is properly explained in the abstract and main paper), "failures" sounds like the connection completely dropped out rather than degraded, so I'd prefer something like "Bandwidth-Optimal AllReduce Under Asymmetric Network Bandwidth" or "Bandwidth-Optimal AllReduce Under Network Bandwidth Degradation”.
  • Is there perhaps a more general setting this paper/algorithm should be targeting, e.g. AR in general low communication settings (heterogenous, identify slowest workers and distribute based on that)? Or maybe that should be an extension: can an extension to this be applied to any heterogenous network setting?
Reviewer_Yu98OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies AllReduce after a NIC failure has been handled by PXN rerouting, which leaves one or more GPUs with inter-node bandwidth reduced by a factor ℓ. The authors prove bandwidth-only lower bounds for three settings (one straggler, m stragglers, and g GPUs per server). These bounds show that the unavoidable overhead relative to the fault-free optimum is max{1+O(1/p), ℓ/2+O(1/p)}. They then propose OptCC, a closed-form four-stage pipelined schedule that runs the straggler’s slow-link transfers in parallel with a ring among the healthy GPUs. In SimAI simulations with a single straggler, OptCC stays within 2–6% of fault-free NCCL ring. NCCL on the degraded topology and R²CCL show much larger overheads.

Critical strengths

The key idea is simple and sound. The straggler only has to send and receive n elements over its slow link. These transfers use links that are separate from the healthy ring, so the two can overlap. Schemes that split data uniformly across all GPUs (NCCL ring, ForestColl-style RS+AG) stay close to ℓ·T0. OptCC instead reaches ℓn for ℓ≥2, which is provably tight, and roughly T0(1+1/(p−1)) for ℓ<2. This is a genuine conceptual contribution. The schedule needs no solver and no offline precomputation (under 1 ms to generate at p=1024), which makes online failover realistic. The paper also covers three topologies and releases code.

Critical weaknesses

The claims of tightness and optimality “even for small p” (Thms 6/13, Table 4, conclusion) require stronger justification. The tighter bounds appear to assume that a healthy NIC is occupied for ℓ time units per element when communicating with the straggler. Although Section 4.1 restricts each NIC to receiving at most one flow at a time, it is unclear whether this restriction applies to all algorithms considered in the lower-bound proofs, or whether the bounds remain valid when NICs can multiplex flows. Under the more general bandwidth model in Section 3, Thms 1–3 provide lower bounds, but the tighter optimality claims require additional justification. The authors should clarify the assumptions and qualify the optimality claims accordingly.

The evaluation is simulation-only, with no hardware validation at any scale. The fidelity of the ICCL and R²CCL baseline implementations is not sufficiently established. The reported 2–6% overhead depends on how SimAI models fixed overheads and synchronization costs. Appendix F shows that OptCC's best simulated runtime is approximately 15% above the bandwidth-only theoretical T0, while k is selected through offline profiling that is not clearly explained in the main text. There is also no ablation against a simpler baseline combining a ring among healthy GPUs with direct communication to the straggler. These limitations make it difficult to assess how well the reported gains would translate to real GPU clusters.

Questions for authors

Theoretical optimality: Does the time-weighted accounting used in Theorems 6 and 13 remain valid when healthy NICs can multiplex concurrent flows? Section 4.1 restricts each NIC to receiving at most one flow at a time, but it is unclear whether this restriction also applies to the lower-bound proofs. Please clarify the communication model and the scope of the optimality claims.

Performance comparison: The abstract attributes up to 57% overhead to existing approaches, while Section 5 states that OptCC “beats R²CCL/ICCL by 29%/57%.” Are these percentages calculated relative to the fault-free NCCL baseline or to the respective competing methods? Please provide the exact runtimes and clarify the metrics.

Parameter tuning and baseline fidelity: How was (k) selected for each configuration, and how were ICCL and R²CCL implemented and validated in SimAI? Appendix F describes offline profiling, while Section 4.2 assumes (k) is a multiple of four. How does the implementation handle other values of (k)?

Multiple stragglers: The paper describes multiple stragglers as the most common production scenario, yet evaluates only (m=2). Why is bubble filling not applied to this setting, and how does OptCC perform for (m>2), particularly when the slowdown factors are close to 1?

Related work and hardware validation: Devraj et al. (2025) appears to be closely related, but the paper categorizes it as an approach that excludes the failed server. Could the authors clarify this distinction? Is there any hardware-based evidence supporting the simulated performance gains?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.