- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes a process for communication-aware layerwise weight offloading in distributed Diffusion Transformer inference. The method addresses contention between host-to-device parameter prefetching and inter-GPU collective communication on PCIe-only systems. The main idea is to divide each layer into fixed-size chunks, which allows prefetching to pause at chunk boundaries. The method also supports part of the model's parameters to remain on the GPU. This lowers the amount of data that must be prefetched when computation cannot fully hide transfer latency. The method is showcased using WanVideo, FLUX, and HunyuanVideo using two L40S and two H100 PCIe GPUs.
Critical strengths
- The paper targets a specific and relevant problem in high-resolution image and video generation systems.
- The proposed solution is simple and practical, and provides a configurable memory-latency trade-off.
- The evaluation is done for three large DiT models and two GPU platforms, and includes different aspects (chunk size analysis, and workload sizes).
Critical weaknesses
- The evaluation is done only on two-GPU PCIe nodes and Ulysses parallelism. It is not clear if the same benefits hold for larger GPU clusters and different topologies.
- Comparisons are limited to no offloading and SGLang's whole-layer offloading. Comparisons with other weight-offloading and prefetch-scheduling systems are needed.
- It is not clear how the analytical model can be used for different settings (e.g., different models, software implementations, parallel/simultaneous system activity).
- The experimental setup seems very simple. It uses one prompt and ten denoising steps. It is not clear how initialization cost, variation between different prompts, and CPU overhead affect the results.
Questions for authors
- Does the method provide similar benefits for larger GPU clusters (e.g., in the order of tens) and different model topologies?
- How does the method compare with other weight-offloading and prefetch-scheduling systems (beyond no offloading and SGLang's whole-layer offloading schemes)?
- What is the level of improvements observed when using different types of prompts, and a larger number of denoising steps?
- How does CPU overhead affect the results?
Thank you for the review: response to Reviewer sVmy
W1 and Q1 (two-GPU PCIe nodes and Ulysses only; larger clusters and other topologies). Two things decide where the benefit holds at larger scale.
The first is whether the collectives contend with the prefetch at a GPU's PCIe receive port. Inside a PCIe-only node they do for any number of GPUs, because each GPU receives the all-to-all through its own port. On NVLink they do not: on a node with two H100s connected by NVLink (new Appendix G, Table 5), Layerwise costs 6% over No offload at 41 frames and nothing at 81–161 frames, and ChunkFlow is up to 4% above it. Across nodes the bottleneck shifts to the network. We ran one Ulysses group across two L40S nodes on 100 Gb/s InfiniBand, and with two GPUs per node ChunkFlow gave no improvement over Layerwise. The GPUs share the link, so each receives the collective at no more than about 6 GB/s against 25 GB/s for its port, and the inter-node bandwidth limits the collective. Many DiT engines support a single node only, and xDiT crosses nodes with CFG or pipeline parallelism. A cluster of tens of commodity GPUs is therefore a set of nodes, and the mechanism applies inside each node.
The second is where the workload sits relative to F*. Under sequence parallelism each rank computes 1/N of the sequence against the same per-layer bytes, so a group of tens of GPUs sits far below F* for all but very long sequences. There the larger lever is partial residency (Section 3.3). The mechanism itself is per rank, so nothing in it grows with the number of GPUs. Model topology enters only through the block's FLOPs and bytes. The DiT block of WanVideo and the MM-DiT blocks of Flux and HunyuanVideo already span the two designs in use.
W2 and Q2 (comparison with other weight-offloading and prefetch-scheduling systems). The whole-layer prefetch we compare with is what production inference systems deploy. SGLang's diffusion runtime prefetches the next layer as one transfer while the current layer computes. vLLM's weight offloader, adapted from SGLang's, copies each offloaded layer back to back a fixed number of layers ahead. The other weight-offloading systems we know of (ZeRO-Infinity, FlexGen, MoE-Lightning) target LLM throughput and decide what to keep in GPU memory and when to transfer relative to compute. None of them schedules the transfer against the collective traffic of distributed inference, so on this question they would behave as whole-layer prefetch. We also implemented and measured two alternative prefetch schedules (see our reply to Reviewer PShp): issuing the chunks at fixed intervals, and pausing the prefetch for each attention's whole communication phase. The first stays within 2% of Layerwise at 41 and 81 frames. The second is slower than Layerwise at 41 frames and approaches ChunkFlow only at 81 frames.
W3 (using the analytical model in other settings). The model is meant to be used before deployment. Per platform it needs two constants read from the specification (P_peak, BW_h2d) and two calibration factors measured once per platform and software stack (η_comp, η_pref, Appendix C). Per model it needs the block's FLOPs (Appendix B) and its prefetch volume B_pref (Appendix C). The output is which side of F* the workload falls on, that is, whether layerwise offloading will cost latency. It is not a latency prediction. A different software implementation changes the realised fractions of peak compute and of PCIe bandwidth, which the two η absorb. Simultaneous activity on the same links lowers the bandwidth the prefetch sees, which the model represents as a lower η_pref and a higher F*. The NVLink control of Appendix G checks the criterion on a second H100 platform, where each GPU has the full PCIe Gen5 link for its prefetch instead of half of it. For this node the model predicts the WanVideo crossover at about 77 frames, down from 119. The measurement agrees: offloading costs 6–10% at 41 frames and at most 1% from 81 frames on.
W4, Q3 and Q4 (one prompt, ten steps; initialization cost, prompt variation, CPU overhead). The prompt does not affect the step time. The text is encoded once into a context of fixed length (512 tokens for WanVideo and Flux), so every block does the same work for every prompt. The number of denoising steps does not affect the step time either. It sets how much of a request the denoising loop is. With the 10 steps used in the paper, the loop is 51–56% of a request on L40S. With the 50 steps that these models usually run, it is 84–86%. The per-step changes we report translate to the request by this share. Initialization: offloading adds 7–10 s once per server start for pinning the DiT weights in host memory (L40S node). CPU overhead is one prefetch worker thread per rank, whose only work is to enqueue one copy per chunk. There is no polling thread, and Section 4.3 puts ChunkFlow's own runtime overhead at about 2% of the step time.