- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper provides a scalable PDLP for the distributed multi-GPU system. The authors claim their work is the first work that design PDLP for multi-GPU acceleration and the results on eight GPU achieve 6 times speedup. Despite their results I have many concerns which need author reply.
Critical strengths
- Brand-new framework to scale PDLP to the mulit-GPU systems.
- Achieve remarkable performance improvement on multi-GPUs.
Critical weaknesses
-
Novelty is limit. I not expert in this field but in HPC. To my current understanding the bottleneck of this operation is just two SpMV operation and the author just tile the operation across GPUs. But this operation is a well-known and commonly used approach in HPC field. And there is many optimized method in SpMM and SpMV operations, which achieve efficient computation-communication overlap. For example https://arxiv.org/abs/1510.00844. But the authors only use global allreduce.
-
The authors introduce the THEORETICAL COST MODEL. But this section is somehow independent. This model is a performance model to predict the each step total runtime. But I don’t see it specific contribution. What is the contribution to your whole work? How to guide you design and optimization. As a reviewer, I prefer to see these questions. Please provide more claims.
-
No further system level optimization. Did the author leverage further kernel tailoring? My current understanding is that the author just invoke the official CUPDLPX. Did the single step optimization contain other operations you have designed? Please provide more details.
-
Provide more experiment insights. I m very happy to see detailed breakdown evaluations and the scalability evaluations when you scale more GPUs. But not just runtime evaluations.
Questions for authors
See weakness.
9 comments
Response to Reviewer 6kTG (Part 2/2)
3. Systems design with unchanged local primitives (W3)
We retain cuPDLPx's local computational primitives to isolate the gains from the distributed design, as clarified in Appendix L.1. The existing ablation (Table 10, Appendix J.1; Section 5.4) shows that block-wise permutation with nonzero-aware cuts reduces aggregate solving time by 13.8% relative to fully random permutation with the same cuts and by 4.6% relative to block-wise permutation with uniform cuts.
We also added a separate ten-setting permutation-seed sweep in Appendix M.2 to test whether the balance depends on a particular random permutation. These results demonstrate stable load balance and local SpMV performance across the tested seeds.
4. Component-level insights and larger systems (W4)
To explain performance beyond total solving time, we profiled the C/CUDA implementation on zib03 with Nsight Systems. Each window covers 999 complete Halpern-to-Halpern cycles, excluding initialization. We report the GPU timeline of rank 2 for four GPUs and rank 3 for both eight-GPU layouts. These measurements have been added to Appendix M.3.
Table R2. Per-cycle GPU-timeline breakdown for zib03, in milliseconds.
| Configuration | Sparse matrix–vector multiplication: all calls | NCCL collective kernels | Other GPU kernels | Copies and gaps | Total cycle time |
|---|---|---|---|---|---|
| 1 GPU, grid | 2.468 | 0.000 | 1.385 | 0.031 | 3.885 |
| 4 GPUs, grid | 0.836 | 0.767 | 0.716 | 0.037 | 2.356 |
| 8 GPUs, grid | 0.550 | 0.719 | 0.648 | 0.036 | 1.954 |
| 8 GPUs, grid | 0.478 | 0.704 | 0.484 | 0.043 | 1.710 |
The sparse matrix–vector multiplication column includes all local calls in the window, including periodic convergence checks; NCCL includes collective execution and synchronization waiting. These per-cycle profiles are separate from the grid-selection timing measurements in Table R1.
From one GPU to the layout, sparse matrix–vector multiplication improves by 5.16×, while the full profiled cycle improves by 2.27×. NCCL occupies 41.2% of the eight-GPU timeline, showing the importance of collective execution and synchronization alongside local computation.
The grid comparison is especially informative. Changing to reduces cycle time by 12.5% and other GPU kernel time from 0.648 to 0.484 milliseconds (25.3%), accounting for about two-thirds of the time saved. The model's dimension-dependent workload term decreases by 26.6% for this change. These corresponding reductions support optimizing the complete solver layout, including vector operations, rather than considering matrix–vector kernels alone.
To test larger systems, we added multi-node experiments in Appendix M.4, summarized in Section 7. The MetaX C550 system has four physical nodes with eight fully interconnected GPUs each, linked through rail-only PCIe Fabric. We measured solving time on one, two, and four nodes.
Table R3. Solving times on the MetaX C550 system.
| Instance | 8 GPUs / 1 node (seconds) | 16 GPUs / 2 nodes (seconds) | 32 GPUs / 4 nodes (seconds) | Speedup from 8 to 32 GPUs |
|---|---|---|---|---|
mcf_2500_100_500 | 1640 | 787 | 398 | 4.12× |
zib03 | 592 | 543 | 339 | 1.75× |
Scaling from eight to thirty-two GPUs gives 4.12× solving-time speedup on mcf_2500_100_500 and 1.75× on zib03, demonstrating acceleration across physical nodes on another accelerator platform.
We also scaled mcf_2500_100_500 on Hygon with eight GPUs per node, PCIe within nodes, and InfiniBand between nodes, using the same implementation and settings across GPU counts.
Table R4. Average iteration times for mcf_2500_100_500 on the Hygon cluster.
| Number of GPUs | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| Average time per iteration (milliseconds) | 38.4 | 23.6 | 10.3 | 5.1 | 2.8 | 1.8 | 1.5 | 1.28 |
Iteration time decreases from 38.4 to 1.28 milliseconds, giving approximately 30× per-iteration speedup across sixteen nodes. Crossing from one node to two reduces iteration time from 5.1 to 2.8 milliseconds. At 128 GPUs, the solver reaches optimal status in 427.13 seconds after 333,600 iterations. These results extend the evaluation from a single eight-H100 server to physical multi-node systems, while the component profiles explain how the distributed design affects solver execution.
Although the paper claims to be the first to scale the PDLP method, I still find its technical novelty and contributions limited.
-
the primary contribution lie in the NCCL-based communication design. However, the paper lacks sufficient technical details and in-depth explanations of the proposed system architecture, making it difficult to assess the significance of this contribution.
-
the performance-model-guided load-balancing strategy is a well-established technique in the HPC community. The authors do not provide sufficient technical depth or analysis to demonstrate how their approach differs from existing methods.
-
the paper does not present any substantial kernel-level optimizations, instead relying primarily on existing library implementations. This further limits the technical novelty and overall contribution of the work.
Response to Reviewer 6kTG (Part 1/2)
We sincerely thank the reviewer for the constructive HPC perspective. With local computational primitives held fixed, D-PDLP improves complete-solver execution through state layout, grid selection, load balancing, and communication organization. Changes are highlighted in blue.
1. Solver-level contribution and communication design (W1)
Building on established sparse-linear-algebra techniques, our contribution is multi-GPU acceleration of the complete PDLP solver. One layout supports both SpMV directions, primal–dual and Halpern updates, convergence checks, restarts, and primal-weight updates. Section 1 and Appendix L.1 clarify this contribution and discuss the cited SpGEMM work.
Core AllReduces use grid-axis communicators—groups of two and four GPUs on a 2×4 grid—while replication-aware scalar reductions support stopping and restart metrics (Section 3.2; Appendix E.2). The implementation is bulk-synchronous; finer-grained overlap offers a complementary optimization.
2. How the cost model guides execution (W2)
The model makes two decisions before execution. For a fixed GPU count, the dimension-dependent term n/∣C∣+m/∣R∣ guides the grid shape. The full model, including matrix work, vector work, collective payload, and latency, then compares GPU counts. Sections 3.3 and 4.1 describe how these decisions enter AUTO.
To validate the grid decision directly, we added an experiment with the hardware fixed at eight H100 GPUs. The model chooses before timing; we then compare its selection with the measured fastest and slowest candidates. This experiment is now in Section 5.3, “Direct Validation of Grid Selection.”
Table R1. Model-selected, measured fastest, and measured slowest process grids on eight H100 GPUs. All times are milliseconds per iteration.
mcf_2500_100_500zib03sdm_50k_500k_15_10For selected, fastest, and slowest times Ts,Tf,Tw, overhead is 100(Ts/Tf−1)% and speedup is Tw/Ts. The model selects the fastest grid on two instances and is only 5.41% slower on the third. Its 6.29× gain over the slowest layout on the same eight GPUs demonstrates both the importance of grid choice and the practical value of the model.
The choice has a clear structural interpretation.
mcf_2500_100_500has approximately 83.5 times as many variables as constraints. Its selected 1×8 layout makes each GPU's portion of A⊤y local and requires only a reduction of the shorter constraint-space vector in the core iteration. Thus, the model translates matrix shape into a favorable computation and communication layout.GPU-count selection is supported separately by the existing Table 1, Section 5.2: AUTO achieves the best aggregate shifted geometric means among the tested fixed counts, 18.38 seconds on Mittelmann and 7.31 seconds on MIPLIB. The new grid experiment and existing benchmark comparison therefore validate both configuration decisions. Point 4 below further explains their effects through measured execution components.