acceptodds
Under review as a conference paper at ICLR 2027

ViSPI: Efficient 2D Parallelism for Sparse Long-Sequence Video DiT Inference

Abstract

Diffusion Transformers (DiTs) have become a dominant architecture for video generation. However, scaling DiTs to longer videos is challenging as rapidly growing sequence lengths exacerbate the quadratic cost of full self-attention. While sparse attention reduces computational overhead by selectively enabling token interactions, it fundamentally changes the execution characteristics of DiTs, introducing irregular workloads and communication patterns. Existing distributed parallel strategies designed for dense computation do not adapt well to these sparse workloads. To address this challenge, we present ViSPI, a distributed inference framework tailored for sparse video DiTs. Specifically, it models hardware-level execution costs to balance heterogeneous attention workloads across GPUs. Meanwhile, it overlaps irregular Key-Value (KV) communication with computation via asynchronous execution. Experiments on representative video DiT models demonstrate that ViSPI significantly improves inference efficiency over state-of-the-art distributed inference frameworks while preserving generation quality and scaling efficiently across multiple GPUs.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Video Diffusion Transformers, Sparse Attention, Distributed Inference, 2D Parallelism, Load Balancing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7XxhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents ViSPI, a framework for distributed inference of sparse video diffusion transformers. It combines three techniques: a workload estimate based on Tensor Core MMA counts, greedy (LPT) assignment of attention heads and query blocks to GPUs, and asynchronous KV retrieval via NVSHMEM, with double buffering to overlap communication and computation. The evaluation covers two Wan2.1 model sizes (1.3B and 14B), sequence lengths from 76K to 364K tokens, and up to eight A800 or H20 GPUs. All distributed methods use the same SVG2 sparsity masks. ViSPI is reported to be 1.06–1.12× faster than DSV-CP, and its outputs exceed 45 dB PSNR relative to single-GPU SVG2.

Critical strengths

All distributed methods in the experiments use the same SVG2 masks. This is a useful control, because it separates execution efficiency from changes in the attention approximation. The design also has a plausible systems motivation, since its two mechanisms target different sources of overhead: workload-aware partitioning addresses load imbalance caused by uneven per-head sparsity, and asynchronous KV streaming addresses the communication and synchronization cost of KV exchange.

Critical weaknesses
  1. The claimed advance in execution-aware partitioning is not established. With fixed block sizes, head dimension, and MMA tile sizes, Equation 1 and Algorithm 1 reduce to a constant multiple of the active-block count. Under these conditions, the two estimates produce identical scheduling orders and load comparisons, and therefore the same Spearman correlation. The manuscript does not explain how variable SVG2 cluster sizes or tile rounding would make the MMA estimate differ. Figure 2 also does not describe how its samples were constructed. For example, if all samples share one sequence length, Global Density is itself proportional to the active-block count, so its ρ = 0.243, compared with ρ = 0.652 for MMA Count, is hard to interpret. In addition, the paper does not clearly separate its mechanisms from closely related work. db-SP (Chen et al., MLSys 2026) already uses greedy partitioning at both the head and block levels, and SwiftFusion (Yang et al., CAIS 2026) already overlaps chunked communication with computation using NVSHMEM. These works do not make ViSPI redundant, but the current comparisons do not convincingly identify its additional contribution.

  2. The reported latencies cannot be interpreted as complete denoising steps under standard execution assumptions. For Wan2.1-14B at 75,600 tokens, the two FFN matrix multiplications across 40 layers alone require about 856 TFLOP per forward pass, using the official hidden and FFN dimensions of 5,120 and 13,824. Eight A800 GPUs provide at most 2,496 TFLOP/s of dense BF16 matrix throughput, which gives an ideal FFN-only lower bound of about 343 ms. The ViSPI bar in Figure 4(c), however, is near 80 ms per step. This bound excludes attention, QKV and output projections, communication, and all other overhead. It would also double if classifier-free guidance requires two forward passes per step. The same gap appears at a smaller scale. Figure 4(a) shows about 100 ms per step for single-GPU Wan2.1-1.3B at 76K tokens. In contrast, my own measurement of this model on a single A800 took about 2.75 s per step, using dense FlashAttention at 480P and 81 frames (about 33K tokens) with CFG. A different timing scope (for example, attention only or a single layer), precision, or reuse mechanism could explain the discrepancy, but the paper specifies none of these. Without these timing boundaries and configuration details, the roughly 4% per-component gains in Figure 6 (1.12× cumulatively over DSV-CP) cannot support the claimed end-to-end improvement.

Questions for authors
  1. Which quantities vary across the samples in Figure 2 (for example, sequence length, head, layer, or denoising step)? How does the implemented cost estimate account for variable SVG2 cluster sizes and hardware tile rounding? Please give at least one concrete case where this estimate produces a different partition from simple active-block counting.
  2. Which operations are included in the timings in Figure 4? Please reconcile the Wan2.1-14B result (about 80 ms per step on eight A800 GPUs) with the FFN-only lower bound of about 343 ms. Please also report the precision, whether classifier-free guidance is used, the selected HP/CP configuration, and a timing breakdown for that configuration that includes mask generation, cost estimation, and redistribution.
  3. Which normalization statistics (for example, the row-wise maximum and log-sum-exp) are returned with each partial output in Algorithm 2, and how are they merged with the local KV contribution? Please give the merge recurrence or the relevant implementation details. Please also report an attention-level numerical comparison against single-GPU SVG2 under the same masks.
  4. Beyond the partitioning in db-SP and the communication scheme in SwiftFusion, which changes are specifically required for SVG2 workloads? Please state what contribution remains, and whether a matched comparison or ablation can isolate it.
Reviewer_dPYHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ViSPI, a multi-GPU inference framework for sparse long-sequence video Diffusion Transformers. It uses an MMA-based cost estimate to balance workloads across attention heads and query blocks, and combines on-demand KV retrieval with asynchronous, double-buffered streaming to overlap communication and computation. Experiments on Wan 2.1 1.3B and 14B, using SVG2-generated sparsity masks on A800 and H20 GPUs, report 1.06–1.12× speedups in per-step denoising over DSV-CP while maintaining close numerical agreement with single-GPU SVG2 outputs.

Critical strengths

(1) A systematic approach to multi-GPU execution of sparse attention. ViSPI combines head-level and query-level load balancing with asynchronous KV streaming, addressing both uneven computation and communication stalls. Reducing attention computation alone does not guarantee efficient multi-GPU execution, and the proposed framework helps translate sparsity into practical performance gains.

(2) Additional speedups without retraining or further sparsification. ViSPI reuses pretrained weights and existing sparsity masks. Comparisons using the same SVG2 masks help isolate the benefit of the execution strategy, and the improvement over DSV-CP demonstrates gains over an already sparsity-aware baseline. The reported PSNR above 45 dB relative to single-GPU SVG2 also supports close numerical agreement without adding another approximation to the underlying sparse computation.

Critical weaknesses

(1) The dependence of the gains on the sparse workload distribution remains unclear. Load imbalance across heads and query partitions is a central motivation, but the paper does not quantify the residual imbalance after baseline scheduling or show how speedup varies with imbalance. For relatively balanced workloads, it is unclear whether load balancing still provides enough benefit to offset scheduling and redistribution costs. The evaluation also uses only SVG2-generated masks and two sizes of the same Wan 2.1 model family. DSV-CP uses those same masks, so it does not provide validation on another sparsity pattern. The results establish effectiveness on the evaluated workloads, but leave the broader applicability uncertain.

(2) The trade-off between computation balance and communication cost is not sufficiently examined. Algorithm 1 balances estimated computation cost without explicitly incorporating KV placement, overlap in KV requirements across query blocks, or data redistribution costs. Although Section 3.3 models communication and computation overlap, it remains unclear how the chosen partitioning affects communication volume. A more balanced assignment could reduce data locality or increase remote transfers. Measurements of communication volume, redistribution overhead, and exposed communication time would help establish whether the improved computational balance translates into lower overall latency.

Questions for authors

(1) How does performance vary with the degree of load imbalance? Could you quantify head-level and query-level imbalance after baseline scheduling and report ViSPI’s gains across different imbalance levels? In particular, when workloads are nearly balanced, do the load-balancing components still justify their scheduling and redistribution overhead? A controlled experiment that varies the distribution of active blocks while keeping sequence length and overall sparsity approximately fixed would help isolate this effect.

(2) How does ViSPI perform at different spatial resolutions? The current experiments fix the resolution at 720P and mainly vary the number of frames. Could you report performance at additional resolutions, together with the corresponding sparsity and load imbalance? Resolution changes may affect both the spatial structure of sparse connections and KV access patterns, so this would help assess whether the gains depend on the evaluated spatial workload.

(3) Does ViSPI generalize to other video DiT models? The current evaluation covers two model sizes within the Wan 2.1 family. Could you evaluate ViSPI on another model, such as HunyuanVideo, and report both its performance gains and the adaptations required? This would help clarify whether the framework and its benefits extend beyond the evaluated model family.

Reviewer_BDdSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ViSPI, a distributed inference framework specifically designed for sparse video DiTs, to address the mismatch between the irregular workloads and communication patterns of sparse attention and existing parallelization strategies for DiTs. ViSPI estimates sparse attention workloads by modeling hardware execution costs based on the triggered matrix multiply-accumulate (MAC) operations. It separately estimates the computational workloads of attention heads and Query blocks and employs the Longest Processing Time (LPT) algorithm for non-uniform task partitioning to mitigate load imbalance across GPUs. Furthermore, ViSPI introduces asynchronous KV Streaming to retrieve the KV blocks required by local sparse attention on demand and uses double buffering to hide the overheads associated with irregular KV computation and communication. Compared with single-GPU SVG2 sparse inference, ViSPI achieves speedups of up to 6.7× and 7.1× on Wan 2.1 1.3B and Wan 2.1 14B, respectively.

Critical strengths

ViSPI uses MMA Count as a proxy metric for sparse attention execution cost. Compared with workload estimation based solely on sparsity ratios, this metric has a more explicit basis in hardware execution characteristics. The authors report a Spearman rank’s correlation of 0.652 between MMA Count and measured GPU latency, higher than those of Global Density and Max Row Density, providing preliminary experimental evidence for hardware-aware workload estimation. Furthermore, ViSPI applies its hardware-cost-based scheduling strategy to both Head Parallelism (HP) and Context Parallelism (CP), performing non-uniform partitioning of attention heads and Query blocks based on their estimated costs, thereby mitigating computational load imbalance across both parallelism dimensions. Combined with asynchronous KV Streaming, this design improves the efficiency of distributed sparse attention execution from both computation allocation and data transfer perspectives.

Critical weaknesses

A key design of ViSPI is the introduction of MMA Count as a proxy metric for estimating hardware execution cost. However, the current evaluation primarily compares its Spearman rank’s correlation with those of Global Density and Max Row Density, without sufficiently accounting for factors such as memory access behavior, kernel implementation, tile padding, instruction issue efficiency, resource utilization, and concurrent execution under different hardware and workload conditions. To further strengthen the generalizability of this work, it is necessary to investigate whether MMA Count remains an appropriate proxy for hardware execution cost across different GPU architectures, block sizes, and sparsity patterns.

Questions for authors

ViSPI independently performs non-uniform partitioning by treating attention heads and Query blocks as scheduling units, but it does not explicitly address or account for the correlations between the sparsity patterns of heads and Query blocks. It would be valuable to further compare independent LPT assignment with a joint scheduling method that accounts for two-dimensional interaction costs, and to evaluate the actual execution-time distributions across GPUs, the maximum GPU latency, and the degree of load imbalance under the joint scheduling method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.