acceptodds
Under review as a conference paper at ICLR 2027

ChunkFlow: Communication-Aware Chunked Prefetching for Layerwise Offloading in Distributed Diffusion Transformer Inference

Abstract

Layerwise offloading reduces the GPU memory footprint of large diffusion transformer (DiT) inference by prefetching upcoming layers from host memory, but its effectiveness hinges on hiding prefetch latency behind per-layer computation. This assumption breaks down when the per-GPU compute workload is small. Moreover, on PCIe-only nodes, prefetch and inter-GPU collective communications such as all-gather and all-to-all contend on the shared PCIe path, exposing prefetch latency even when compute would otherwise hide it. We revisit layerwise offloading as a co-scheduling problem between prefetch and communication, guided by a first-order analytical model that predicts when prefetch can be hidden by computation. Building on this model, we design ChunkFlow, a communication-aware, chunk-granular offloading runtime that adaptively yields to collective communication and smoothly trades GPU memory for prefetch volume. On three representative diffusion transformers running on commodity L40S and H100 GPUs over PCIe with Ulysses sequence parallelism, ChunkFlow delivers up to step-time speedup over SGLang's existing layerwise offloading, reduces peak GPU memory by up to 49% over the no-offload baseline at near-identical step time once the workload is large enough, and exposes a tunable memory–latency tradeoff that recovers near-zero step-time overhead in the small-workload regime.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Diffusion Transformer, Distributed Inference, Parallelism, Offloading

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sVmyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a process for communication-aware layerwise weight offloading in distributed Diffusion Transformer inference. The method addresses contention between host-to-device parameter prefetching and inter-GPU collective communication on PCIe-only systems. The main idea is to divide each layer into fixed-size chunks, which allows prefetching to pause at chunk boundaries. The method also supports part of the model's parameters to remain on the GPU. This lowers the amount of data that must be prefetched when computation cannot fully hide transfer latency. The method is showcased using WanVideo, FLUX, and HunyuanVideo using two L40S and two H100 PCIe GPUs.

Critical strengths
  • The paper targets a specific and relevant problem in high-resolution image and video generation systems.
  • The proposed solution is simple and practical, and provides a configurable memory-latency trade-off.
  • The evaluation is done for three large DiT models and two GPU platforms, and includes different aspects (chunk size analysis, and workload sizes).
Critical weaknesses
  • The evaluation is done only on two-GPU PCIe nodes and Ulysses parallelism. It is not clear if the same benefits hold for larger GPU clusters and different topologies.
  • Comparisons are limited to no offloading and SGLang's whole-layer offloading. Comparisons with other weight-offloading and prefetch-scheduling systems are needed.
  • It is not clear how the analytical model can be used for different settings (e.g., different models, software implementations, parallel/simultaneous system activity).
  • The experimental setup seems very simple. It uses one prompt and ten denoising steps. It is not clear how initialization cost, variation between different prompts, and CPU overhead affect the results.
Questions for authors
  1. Does the method provide similar benefits for larger GPU clusters (e.g., in the order of tens) and different model topologies?
  2. How does the method compare with other weight-offloading and prefetch-scheduling systems (beyond no offloading and SGLang's whole-layer offloading schemes)?
  3. What is the level of improvements observed when using different types of prompts, and a larger number of denoising steps?
  4. How does CPU overhead affect the results?
1 comment
AuthorsOpenReview ↗

Thank you for the review: response to Reviewer sVmy

W1 and Q1 (two-GPU PCIe nodes and Ulysses only; larger clusters and other topologies). Two things decide where the benefit holds at larger scale.

The first is whether the collectives contend with the prefetch at a GPU's PCIe receive port. Inside a PCIe-only node they do for any number of GPUs, because each GPU receives the all-to-all through its own port. On NVLink they do not: on a node with two H100s connected by NVLink (new Appendix G, Table 5), Layerwise costs 6% over No offload at 41 frames and nothing at 81–161 frames, and ChunkFlow is up to 4% above it. Across nodes the bottleneck shifts to the network. We ran one Ulysses group across two L40S nodes on 100 Gb/s InfiniBand, and with two GPUs per node ChunkFlow gave no improvement over Layerwise. The GPUs share the link, so each receives the collective at no more than about 6 GB/s against 25 GB/s for its port, and the inter-node bandwidth limits the collective. Many DiT engines support a single node only, and xDiT crosses nodes with CFG or pipeline parallelism. A cluster of tens of commodity GPUs is therefore a set of nodes, and the mechanism applies inside each node.

The second is where the workload sits relative to F*. Under sequence parallelism each rank computes 1/N of the sequence against the same per-layer bytes, so a group of tens of GPUs sits far below F* for all but very long sequences. There the larger lever is partial residency (Section 3.3). The mechanism itself is per rank, so nothing in it grows with the number of GPUs. Model topology enters only through the block's FLOPs and bytes. The DiT block of WanVideo and the MM-DiT blocks of Flux and HunyuanVideo already span the two designs in use.

W2 and Q2 (comparison with other weight-offloading and prefetch-scheduling systems). The whole-layer prefetch we compare with is what production inference systems deploy. SGLang's diffusion runtime prefetches the next layer as one transfer while the current layer computes. vLLM's weight offloader, adapted from SGLang's, copies each offloaded layer back to back a fixed number of layers ahead. The other weight-offloading systems we know of (ZeRO-Infinity, FlexGen, MoE-Lightning) target LLM throughput and decide what to keep in GPU memory and when to transfer relative to compute. None of them schedules the transfer against the collective traffic of distributed inference, so on this question they would behave as whole-layer prefetch. We also implemented and measured two alternative prefetch schedules (see our reply to Reviewer PShp): issuing the chunks at fixed intervals, and pausing the prefetch for each attention's whole communication phase. The first stays within 2% of Layerwise at 41 and 81 frames. The second is slower than Layerwise at 41 frames and approaches ChunkFlow only at 81 frames.

W3 (using the analytical model in other settings). The model is meant to be used before deployment. Per platform it needs two constants read from the specification (P_peak, BW_h2d) and two calibration factors measured once per platform and software stack (η_comp, η_pref, Appendix C). Per model it needs the block's FLOPs (Appendix B) and its prefetch volume B_pref (Appendix C). The output is which side of F* the workload falls on, that is, whether layerwise offloading will cost latency. It is not a latency prediction. A different software implementation changes the realised fractions of peak compute and of PCIe bandwidth, which the two η absorb. Simultaneous activity on the same links lowers the bandwidth the prefetch sees, which the model represents as a lower η_pref and a higher F*. The NVLink control of Appendix G checks the criterion on a second H100 platform, where each GPU has the full PCIe Gen5 link for its prefetch instead of half of it. For this node the model predicts the WanVideo crossover at about 77 frames, down from 119. The measurement agrees: offloading costs 6–10% at 41 frames and at most 1% from 81 frames on.

W4, Q3 and Q4 (one prompt, ten steps; initialization cost, prompt variation, CPU overhead). The prompt does not affect the step time. The text is encoded once into a context of fixed length (512 tokens for WanVideo and Flux), so every block does the same work for every prompt. The number of denoising steps does not affect the step time either. It sets how much of a request the denoising loop is. With the 10 steps used in the paper, the loop is 51–56% of a request on L40S. With the 50 steps that these models usually run, it is 84–86%. The per-step changes we report translate to the request by this share. Initialization: offloading adds 7–10 s once per server start for pinning the DiT weights in host memory (L40S node). CPU overhead is one prefetch worker thread per rank, whose only work is to enqueue one copy per chunk. There is no polling thread, and Section 4.3 puts ChunkFlow's own runtime overhead at about 2% of the step time.

Reviewer_bJibOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits layerwise weight offloading for DiT inference as a prefetch–communication co-scheduling problem on PCIe-only multi-GPU nodes, where H2D prefetch and inter-GPU collectives contend for the same PCIe receive port. It derives a first-order overlap model with a critical compute workload F ⋆ F ⋆ , proposes communication-aware chunked prefetching that pauses and resumes at chunk boundaries when a collective is issued, and introduces chunk-granular partial parameter residency to trade GPU memory for prefetch volume.

Critical strengths

The problem is real. The paper identifies a concrete and previously under-modeled cost: on PCIe-only nodes, whole-layer prefetch contends with NCCL collectives at the GPU's PCIe receive port, and the measured contention stall is comparable to the collective's own time.

Critical weaknesses
  1. The contribution is narrowly scoped, and the core mechanism does not generalize beyond PCIe-only topologies. Application-wise, the method is evaluated only on DiT, and the paper itself argues in Appendix E that LLM inference sits deep in the prefetch-bound regime. System-wise, the central chunked-prefetch mechanism yields its benefit precisely because H2D prefetch and collectives share the PCIe fabric; on NVLink systems the collectives use a dedicated fabric, and the main benefit disappears. The experimental environment is also limited to 2×L40S and 2×H100 at SP=2 (plus a 4×L40S SP=4 appendix), with no 8-GPU PCIe node, no NVLink comparison, and no heterogeneous or multi-node setting.

  2. The system design is not evaluated systematically. Generality is argued but not measured—only three DiT models, one attention backend, one runtime, and one parallelism scheme are exercised, with no evidence on interaction with tensor parallelism, pipeline parallelism, CFG parallelism, or other offloading schedules. Scalability is probed only along SP 2→4, without examining pause-flag contention, copy-stream overhead, or load imbalance at larger degrees.

Questions for authors

Can you provide evidence on generality beyond the current setting—e.g., an 8-GPU PCIe node, an NVLink node as a negative control, or a different parallelism scheme—to show where the chunked-prefetch benefit persists and where it vanishes?

1 comment
AuthorsOpenReview ↗

Thank you for the review: response to Reviewer bJib

W1 and Question (where the benefit persists and where it vanishes). The benefit persists inside a PCIe-only node at any parallel degree. It vanishes on NVLink. Across nodes the bottleneck shifts from the PCIe port to the network, and the collectives are limited mainly by the inter-node bandwidth.

NVLink negative control (new Appendix G, Table 5). The node has two H100 SXM GPUs connected by NVLink, which carries the all-to-alls. Layerwise costs 6% over No offload at 41 frames and nothing at 81–161 frames, which is the exposed prefetch of Section 3.1 alone. ChunkFlow is up to 4% above Layerwise: with no contention to remove, what shows is the cost of the chunked transfer itself.

8-GPU PCIe node. We could not run one. NVIDIA's reference architecture for L40S servers is built on four GPUs per node (2-4-3-200), and the L40S nodes available to us have two or four. The one 8-GPU L40S system we can find is offline for a migration during the discussion period. At degree 4 (Appendix D) each GPU still receives the all-to-all through its own PCIe receive port, so the contention remains, and ChunkFlow is 1.10–1.33× faster than Layerwise at every frame count. The regime shifts, because each rank computes 1/N of the sequence against the same per-layer bytes. ChunkFlow comes within about 2% of No offload at 81 frames at degree 2 (Table 1) and only at 161 frames at degree 4. At degree 8 the crossover moves further out. Below it, partial residency (Section 3.3) recovers the latency.

Multiple nodes. Many DiT inference engines support a single node only: vLLM-Omni, diffusers and the official WanVideo launcher document no multi-node inference. xDiT does cross nodes (two 8×L40 nodes on 100 Gb/s Ethernet), but with CFG parallelism or pipeline parallelism between the nodes, not sequence parallelism. CFG parallelism exchanges the latent once per denoising step, and pipeline parallelism passes activations point to point between stages. Neither places a collective inside the DiT blocks, so our mechanism has nothing to act on there. We still ran one Ulysses group across two L40S nodes on 100 Gb/s InfiniBand. With two GPUs per node, ChunkFlow gave no improvement over Layerwise: the two are within 1% at 81 and 121 frames, and ChunkFlow is slower at 41 frames. Two GPUs sharing a 100 Gb/s link each receive at no more than about 6 GB/s, a quarter of what a PCIe Gen4 port carries (25 GB/s, Appendix C), so the inter-node bandwidth limits the collective. With one GPU per node, each GPU receives at up to 12 GB/s, about half of the port, and a gain of 1.05–1.10× remains at 41–121 frames.

DiT only; LLM inference. The scope of this paper is DiT inference, the workload that layerwise offloading itself suits best. Appendix E explains why, and why LLM inference is less favorable.

W2 (systematic evaluation). Attention backend and runtime. The mechanism touches the software stack at two points: the call sites of the collectives, where the pause flag is set and cleared, and the copy stream. Neither depends on the attention backend. A different backend changes the compute time, which η_comp absorbs (Appendix C). We have implemented the mechanism in SGLang's diffusion runtime only.

Tensor, pipeline and CFG parallelism. DiT inference engines generally avoid tensor parallelism because it is slow for DiTs. Sharding the weights adds a per-block all-reduce whose volume does not shrink with the parallel degree, whereas sequence parallelism moves only a 1/N share per rank (Section 2.1). xDiT reports tensor parallelism as the slowest of the methods it compares. Tensor parallelism can be seen as another way to save weight memory. It pays for the saving with this heavy communication, and the saving is fixed by the parallel degree. ChunkFlow keeps the cheap communication of sequence parallelism and adds a tunable trade-off between weight memory and extra latency (Section 3.3). Even if ChunkFlow were used under tensor parallelism, the regime would not change: both F_block and B_pref divide by the parallel degree, whereas sequence parallelism divides F_block alone. CFG parallelism and pipeline parallelism place no collective inside the DiT blocks, so they are decoupled from our design.

Other offloading schedules. We implemented fixed-interval chunk issue and pausing for each attention's whole communication phase (see our reply to Reviewer PShp). The first stays within 2% of Layerwise. The second is slower than Layerwise at 41 frames and approaches ChunkFlow only at 81 frames.

Scalability. The pause flag lives in device memory and is written by the compute stream and read by the copy stream of the same rank. Nothing is shared between ranks and nothing is polled. Each rank adds one copy stream and one prefetch worker thread, independent of the parallel degree, and Section 4.3 puts ChunkFlow's own overhead at about 2% of the step time. Ulysses partitions the sequence evenly, so the ranks reach each collective together.

Reviewer_PShpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the issue of PCIe bandwidth contention between weight prefetching and inter-GPU communication during Distributed Diffusion Transformer (DiT) inference. The authors propose ChunkFlow, which divides the transmission of an entire layer’s weights into small chunks, pauses prefetching at chunk boundaries and prioritises communication; in scenarios where the computational load is insufficient to mask the transmission, a portion of the weights is kept resident in video memory.

Critical strengths
  1. This paper clearly sets out its research motivation, and the experiments cover multiple models and two types of GPU.
  2. Experimental results show that this approach has some performance gains over the traditional approach.
Critical weaknesses
  1. The primary comparison is with SGLang’s full-layer prefetching. Although the paper conducted a scan of chunk sizes, it did not compare alternative approaches such as submitting prefetches at fixed intervals within the computation interval or pausing before coarse-grained communication.
  2. The experiment utilised a single fixed prompt and 10 denoising steps, primarily reporting the steady-state average single-step time; however, it did not specify the number of independent repetitions, the margin of error, or the specific warm-up protocol.
  3. With no data on total generation latency, the initial weight preparation cost, CPU and memory usage, or multi-request throughput, the actual deployment benefits remain incomplete.
  4. The description of the paper’s overall contribution is unclear; it is recommended that the challenges addressed and the innovative approaches taken be clarified further.
  5. The work is generally focused on engineering and lacks theoretical underpinnings. The core analysis consists solely of a first-order overlap model, in which two calibration coefficients are determined empirically, with no error bounds or sensitivity analysis.
Questions for authors

The core scheduling mechanism lacks sufficient detail, and the guarantee that at most one chunk will be blocked is not yet sufficient. The author needs to explain how the number of in-transit and committed chunks is limited, how the pause flag is synchronised with communication commits, and provide actual execution traces to verify this limit. This is an issue of implementation correctness that requires clarification; the implementation cannot be deemed incorrect based solely on the current description.

1 comment
AuthorsOpenReview ↗

Thank you for the review: response to Reviewer PShp

W1 (alternative schedulers). We implemented both schedulers and measured them on the 2×L40S node (WanVideo, 41 and 81 frames).

Fixed-interval issue: the worker issues one 16 MB chunk every millisecond. Its step time is within 2% of Layerwise at both frame counts. Even spacing does not keep the bytes away from the collectives, and more than 40% of the chunk bytes land inside them.

Pausing before coarse-grained communication: the prefetch is paused for each attention's whole communication phase and resumes only in the communication-free segments. This still needs a chunked, pauseable transfer, so it is the same mechanism at a coarser granularity. At 41 frames it is 4% slower than Layerwise. It gives up the attention compute inside the two phases, so more of the prefetch is left exposed at the end of the block. At 81 frames the remaining compute segments are long enough, and it performs close to ChunkFlow.

ChunkFlow pauses at every collective and uses every compute segment for prefetch. The only overlap left is the one chunk in flight when the pause takes effect, 20% of the bytes at 41 frames and 11% at 81.

W2 (repetitions, error margin, warm-up). The first denoising step includes kernel compilation and serves as the warm-up. Step time is the mean over the remaining steps. We repeated the WanVideo 41- and 81-frame configurations on the L40S nodes two to three times over several days. On the same node, repeated runs agree within 0.2% at 41 frames and within 1% at 81 frames. Across nodes of the same type, No offload varies by at most 1.8%. Each comparison runs on one node, and every speedup over Layerwise in Table 1 (1.02–1.32×) is above the same-node spread.

W3 (deployment metrics). Offloading changes only the denoising loop. On L40S, a request in a persistent server consists of text encoding (about 4 s), the denoising loop, and VAE decoding (7.4 s at 41 frames, 14.4 s at 81). With the 10 steps used in the paper, the loop is 51–56% of a request. With 50 steps, the usual default of these models, it is 84–86%. At server start, offloading adds 7–10 s once for pinning the DiT weights in host memory. Host memory holds one pinned copy of the DiT weights per rank (10 GB for WanVideo 5B). The CPU cost is one prefetch worker thread per rank. DiT inference engines execute one request at a time, so the throughput gain equals the latency gain.

W4 (contribution statement). We agree that the introduction states the two failure modes but not why a direct fix is unavailable, and the revision adds this. PCIe offers no priority, CUDA streams offer no preemption, and an issued DMA cannot be stopped, so a whole-layer prefetch delays the collectives it overlaps. The only control point is where the next piece of the copy is issued. This is why the transfer is chunked and the pause is enforced by the GPU itself at every collective. Below F* no schedule can hide the prefetch, and residency reduces the bytes instead.

W5 (the analytical model). The model is a first-order regime criterion, not a latency predictor. It tells a practitioner, before deploying, whether layerwise offloading will cost latency on a given GPU, model and parallel degree. It has two calibration factors per platform, shared by all three models (Appendix C). F* is proportional to η_comp/η_pref, so a calibration error of ε shifts the predicted crossover by at most ε in batch size or sequence length (stated in the revised Section 3.1). The NVLink control added in Appendix G checks the model on a second H100 platform, where each GPU has the full PCIe Gen5 link for its prefetch instead of half of it. For this node the model predicts the WanVideo crossover at about 77 frames, down from 119. The measurement agrees: offloading costs 6–10% at 41 frames and at most 1% from 81 frames on.

Question (in-flight chunks, flag synchronisation, traces). The pause is enforced by the GPU, not by a host thread (revised Section 3.2). The runtime enqueues a write of a device-memory flag on the compute stream immediately before each collective and a clear immediately after it. The copy stream waits on the flag before every chunk's copy. Chunks are enqueued ahead of time on the copy stream, at most 64 at once, but a queued copy cannot pass its wait while the flag is set. The only transfer that can overlap a collective is therefore the one chunk that had passed its wait when the flag was written, whatever the queue depth.

Traces (Nsight Systems, 2×L40S, WanVideo, 16 MB chunks) confirm the bound: at 41 and 81 frames at most one chunk overlaps each collective. Appendix F of the revision (Figure 7) shows one block under Layerwise and under ChunkFlow. Under Layerwise the first all-to-all is queued behind the whole-layer transfer. Under ChunkFlow the chunks stop at each all-to-all and resume after it, and the block takes 21 ms instead of 31 ms.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.