acceptodds
Under review as a conference paper at ICLR 2027

Joint Communication-Computation Optimization For Multi-GPU Heterogeneous GNN Training

Abstract

Full-graph training of heterogeneous graph neural networks (GNNs) remains difficult to scale because irregular relation-specific workloads and cross-GPU aggregation jointly limit computation and communication efficiency. This bottleneck is particularly acute in learning-assisted large-scale supply chain production planning, where candidate-variable screening depends on preserving long-range heterogeneous dependencies, making sampling an unsuitable substitute for fullgraph execution. We propose a phase-aware multi-GPU framework that jointly optimizes workload organization and cross-GPU communication while preserving message-passing semantics. At preparation time, a lightweight surrogate selects the graph workload granularity by trading scheduling flexibility against one-time materialization overhead. During recurrent training, asynchronous aggregation overlaps cross-GPU reduction with local computation, while a net-benefit controller selects the communication granularity and reverts to blocking execution when overlap is not worthwhile. The resulting transformations preserve relation-specific full-graph aggregation up to floating-point reduction order. Experiments on five consecutive industrial supply-chain graphs show that the selected preparation configuration is within 0.709% of the empirical optimum on a held-out snapshot. The complete framework reduces mean steady-state epoch time by 62.50% and 50-epoch wall-clock time by 58.50%, with the additional preparation cost amortized after approximately four epochs.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
learning-to-optimize (L2O), heterogeneous graph neural network, multi-GPU optimization, communication computation integration

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_tZqROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Full graph training of heterogeneous GNNs remains an important and challenge problem, especially when there are long-range heterogeneous dependencies. This paper proposes a phase-aware multi-GPU framework that jointly optimizes workload organization and cross-GPU communication while preserving message-passing semantics. It mainly adopts two optimizations: first, at preparation time, a lightweight surrogate trading scheduling flexibility against one-time materialization overhead. Second, during recurrent training, asynchronous aggregation overlaps cross-GPU reduction with local computation. The results show substantial performance improvement.

Critical strengths
  1. Accelerating GNN via multi-GPU system is an important area.
Critical weaknesses
  1. I think the granularity selection is proposed by GNNAdvisor, though GNNAdvisor is designed for single GPU. But I do not find any challenge for extending GNNAdvisor to multi-GPU system.
  2. This paper has 10 pages, while the ICLR requirement says only 9 pages main content is allowed.
  3. The dataset is not public, the hardware is not specified. And in section 5.1, there are placeholders like [framework and version], [collective communication library and version]. I think this paper is even not finished or at least not proofread.
  4. Actually, the Appendix E lists 7 limitations (maybe they are generated by AI), and I think 2, 3, 5, and 7 are also my concerns.
Questions for authors

See weakness.

Reviewer_FVrfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a joint communication‑computation optimization framework for multi‑GPU full‑graph heterogeneous GNN training. However, several critical weaknesses exist regarding baseline comparisons, experimental design, dataset and novelty. The submitted experiments cannot sufficiently validate the core claims.

Critical strengths
  1. Targeting real‑world supply‑chain industrial scenarios, this work proposes a two‑phase optimization separating one‑time preparation cost from iterative runtime overhead, with practical problem significance.
  2. Semantic TARGETGROUP and physical SUPERBLOCK abstractions are designed, and mathematical proofs ensure heterogeneous GNN message‑passing semantics are preserved.
Critical weaknesses
  1. Experiments are incomplete: missing SOTA baselines, mini‑batch baseline, ablation studies and hyper‑parameter analysis; conclusions lack generalization due to single small‑scale private dataset.
  2. Core claims are insufficiently validated. The necessity of full‑graph training lacks experimental evidence; essential differences between heterogeneous and homogeneous GNN parallel training are not well analyzed, showing incremental novelty. Evaluations are limited to a single server and private temporal snapshots. No multi‑node cluster or public‑dataset scalability experiments, so the applicable boundary remains unclear.
Questions for authors
  1. The experiments only compare different variants of the authors' own framework. End‑to‑end evaluations against state‑of‑the‑art multi‑GPU GNN systems including MGG, Heta, CaPGNN, PipeGCN and P3 are absent. It is hard to demonstrate the advantages of the proposed approach over prior solutions.
  2. In industrial practice, full‑graph training is often prohibitively expensive. Mini‑batch training is widely adopted and can achieve comparable accuracy with properly tuned batch sizes. The authors claim that sampling cannot replace full‑graph execution because long‑range heterogeneous dependencies must be preserved for candidate variable screening. Nevertheless, mini‑batch training is capable of preserving long‑range heterogeneous dependencies. The paper provides no empirical evidence showing full‑graph training outperforms mini‑batch training, and mini‑batch is not included as a control group.
  3. Experiments are conducted on a single source of data with limited generalization. All five snapshots are temporally consecutive variants from one supply‑chain dataset. Evaluations under only one feature distribution weaken the persuasiveness of experimental results.
  4. Workload organization and communication‑computation overlap are well‑known optimization directions for multi‑GPU GNN systems. The presented work appears to be incremental. Furthermore, the essential differences between multi‑GPU heterogeneous GNN training and homogeneous GNN training are not clearly and deeply analyzed, making it difficult to assess the novelty compared with homogeneous GNN parallel systems.
  5. Graph scale is too small for validating parallel systems. With the given hardware setup, modern parallel GNN systems can handle graphs with tens or hundreds of millions of edges. Evaluations solely on million‑edge graphs are inadequate to verify the performance of a parallel GNN training system.
  6. No ablation studies are conducted. Without ablation experiments, individual performance gains brought by the two core components cannot be quantified. Hyper‑parameter sensitivity is not evaluated. All experiments are constrained to private data and fixed hardware. Observations only hold for this specific setup and cannot generalize to other graph datasets or multi‑node clusters.
Reviewer_4GEQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a two-stage framework for exact multi-GPU full-graph heterogeneous GNN training. It packs relation–destination groups into reusable SuperBlocks and overlaps destination-range reductions with computation. Surrogate models select preparation and communication granularities. On one held-out industrial snapshot, it reports 62.50% lower epoch time and 58.50% lower 50-epoch runtime against an internal direct-edge baseline.

Critical strengths
  1. The framework explicitly preserves relation-specific full-graph aggregation without stale representations.
  2. The evaluation includes preparation costs and uses a chronological calibration/test split, providing an initial assessment of practical amortization and temporal transfer.
Critical weaknesses
  1. Novelty and comparative effectiveness are insufficiently established. Heta already exploits relation-specific partial aggregation, and MG-GCN implements full-batch multi-GPU communication–computation overlap. Why are these approaches insufficient for the setting studied here? What specific limitations does the proposed method overcome, and what evidence demonstrates its advantages over appropriately adapted versions of these methods?

  2. The optimization procedure and measurements do not substantiate the central claim. Equation (2) minimizes preparation plus recurrent training time, whereas BB is selected using preparation time alone; the reported 0.709% regret also concerns preparation only. Table 4 contradicts the stated overlap definition: its timings imply 6.66% and 20.01% for C=2C=2 and C=8C=8, rather than 14.57% and 33.01%. Measured epoch times are also needed to establish why C=5C=5 is preferable to C=3C=3, which has lower reported exposed communication.

  3. The presentation lacks detailed illustrations of the core mechanisms. The overview figure does not adequately explain how TargetGroups are packed into SuperBlocks or how destination-range reductions overlap with computation across GPUs. A concrete partitioning example and an execution timeline would make the method and its differences from existing approaches substantially clearer.

Questions for authors
  1. How does selecting BB account for recurrent runtime and the training horizon NN?
  2. Can you reconcile Table 4 and provide measured overhead and epoch time for each CC?
  3. What exactly does the direct-edge baseline implement, and how much improvement comes from each component?
  4. What model and backward synchronization are used, and do numerical/gradient equivalence tests confirm exact training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.