- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents ViSPI, a framework for distributed inference of sparse video diffusion transformers. It combines three techniques: a workload estimate based on Tensor Core MMA counts, greedy (LPT) assignment of attention heads and query blocks to GPUs, and asynchronous KV retrieval via NVSHMEM, with double buffering to overlap communication and computation. The evaluation covers two Wan2.1 model sizes (1.3B and 14B), sequence lengths from 76K to 364K tokens, and up to eight A800 or H20 GPUs. All distributed methods use the same SVG2 sparsity masks. ViSPI is reported to be 1.06–1.12× faster than DSV-CP, and its outputs exceed 45 dB PSNR relative to single-GPU SVG2.
Critical strengths
All distributed methods in the experiments use the same SVG2 masks. This is a useful control, because it separates execution efficiency from changes in the attention approximation. The design also has a plausible systems motivation, since its two mechanisms target different sources of overhead: workload-aware partitioning addresses load imbalance caused by uneven per-head sparsity, and asynchronous KV streaming addresses the communication and synchronization cost of KV exchange.
Critical weaknesses
-
The claimed advance in execution-aware partitioning is not established. With fixed block sizes, head dimension, and MMA tile sizes, Equation 1 and Algorithm 1 reduce to a constant multiple of the active-block count. Under these conditions, the two estimates produce identical scheduling orders and load comparisons, and therefore the same Spearman correlation. The manuscript does not explain how variable SVG2 cluster sizes or tile rounding would make the MMA estimate differ. Figure 2 also does not describe how its samples were constructed. For example, if all samples share one sequence length, Global Density is itself proportional to the active-block count, so its ρ = 0.243, compared with ρ = 0.652 for MMA Count, is hard to interpret. In addition, the paper does not clearly separate its mechanisms from closely related work. db-SP (Chen et al., MLSys 2026) already uses greedy partitioning at both the head and block levels, and SwiftFusion (Yang et al., CAIS 2026) already overlaps chunked communication with computation using NVSHMEM. These works do not make ViSPI redundant, but the current comparisons do not convincingly identify its additional contribution.
-
The reported latencies cannot be interpreted as complete denoising steps under standard execution assumptions. For Wan2.1-14B at 75,600 tokens, the two FFN matrix multiplications across 40 layers alone require about 856 TFLOP per forward pass, using the official hidden and FFN dimensions of 5,120 and 13,824. Eight A800 GPUs provide at most 2,496 TFLOP/s of dense BF16 matrix throughput, which gives an ideal FFN-only lower bound of about 343 ms. The ViSPI bar in Figure 4(c), however, is near 80 ms per step. This bound excludes attention, QKV and output projections, communication, and all other overhead. It would also double if classifier-free guidance requires two forward passes per step. The same gap appears at a smaller scale. Figure 4(a) shows about 100 ms per step for single-GPU Wan2.1-1.3B at 76K tokens. In contrast, my own measurement of this model on a single A800 took about 2.75 s per step, using dense FlashAttention at 480P and 81 frames (about 33K tokens) with CFG. A different timing scope (for example, attention only or a single layer), precision, or reuse mechanism could explain the discrepancy, but the paper specifies none of these. Without these timing boundaries and configuration details, the roughly 4% per-component gains in Figure 6 (1.12× cumulatively over DSV-CP) cannot support the claimed end-to-end improvement.
Questions for authors
- Which quantities vary across the samples in Figure 2 (for example, sequence length, head, layer, or denoising step)? How does the implemented cost estimate account for variable SVG2 cluster sizes and hardware tile rounding? Please give at least one concrete case where this estimate produces a different partition from simple active-block counting.
- Which operations are included in the timings in Figure 4? Please reconcile the Wan2.1-14B result (about 80 ms per step on eight A800 GPUs) with the FFN-only lower bound of about 343 ms. Please also report the precision, whether classifier-free guidance is used, the selected HP/CP configuration, and a timing breakdown for that configuration that includes mask generation, cost estimation, and redistribution.
- Which normalization statistics (for example, the row-wise maximum and log-sum-exp) are returned with each partial output in Algorithm 2, and how are they merged with the local KV contribution? Please give the merge recurrence or the relevant implementation details. Please also report an attention-level numerical comparison against single-GPU SVG2 under the same masks.
- Beyond the partitioning in db-SP and the communication scheme in SwiftFusion, which changes are specifically required for SVG2 workloads? Please state what contribution remains, and whether a matched comparison or ablation can isolate it.