- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the computational cost of video generation, where attention over long spatiotemporal sequences is the main bottleneck. It proposes JetDiffusion, a system built on Video Sparse Quantized Attention (VSQA), which to the authors' knowledge is the first trainable attention method combining high sparsity (90%) with NVFP4 quantization, supported by custom training and inference kernels for the RTX 5090 and GB300; the full system additionally quantizes linear layers to NVFP4 and distills the model to three denoising steps, with a staged training recipe—layer-wise reconstruction warm-up, then regularized ODE initialization, then DMD refinement—to adapt the model to the combined effects of quantization, sparsity, and few-step denoising.
Critical strengths
-
The paper presents VSQA, a trainable video attention mechanism that jointly exploits high sparsity and native NVFP4 computation. Its hardware aware training and inference kernels address an important systems bottleneck in efficient video generation.
-
The staged training recipe is technically well motivated and consists of layer wise reconstruction warm up, sparse ODE initialization, reconstruction regularization, and DMD refinement. The training analyses and ablations support the role of each component in reducing quantization error and preserving teacher alignment.
-
JetDiffusion achieves a strong quality and efficiency trade off under aggressive compression. With 90% sparsity and three denoising steps, it preserves or improves aggregate VBench scores while achieving 17.2 to 28.5 times attention speedups and up to 282 times end to end acceleration.
Critical weaknesses
- The reported end to end speedups are not fully apples to apples. Table 6 uses TAEHV for JetDiffusion and the default VAE for the base model, so the 282 times and 144 times gains combine denoising, attention, linear quantization, and decoder differences.
- The evaluation scope is relatively narrow. Experiments focus on Wan2.1 models at two resolutions with text prompts from Wan Syn and VidProM, while the 14B results use only one seed, leaving generalization to other architectures, sequence lengths, and prompt distributions uncertain.
- The quality evaluation relies mainly on VBench scores and selected qualitative examples. The absence of human preference studies, confidence intervals, and systematic failure case analysis makes it difficult to assess perceptual degradation under 90% sparsity and three step generation.
- The contribution is difficult to disentangle from the combination of existing techniques. Although the ablations validate the staged training procedure, they do not fully separate the effects of sparse attention design, kernel fusion, linear layer quantization, and distillation, which makes the precise source of the gains less clear.
Questions for authors
- How does the reported end to end speedup change when JetDiffusion and the base model use the same video decoder? A component wise latency breakdown would clarify the contributions of attention, linear quantization, denoising step reduction, and TAEHV.
- What evidence supports generalization beyond Wan2.1 and the current prompt distributions? Evaluation on another video diffusion architecture, longer sequences, or additional datasets would strengthen this claim.
- What is the variance of the reported 14B results? Since these results use only one random seed, repeated evaluations or uncertainty estimates would be useful for assessing the reliability of the quality improvements.
- How can the individual contributions of VSQA, NVFP4 linear quantization, kernel fusion, and three step distillation be disentangled? Matched comparisons with recent sparse and low bit baselines would clarify the source of the overall gains.
4 comments
W2.1, Q2: Generalization across sequence lengths, architectures, and prompts
What evidence supports generalization beyond Wan2.1 and the current prompt distributions? Evaluation on another video diffusion architecture, longer sequences, or additional datasets would strengthen this claim.
Our experiments cover multiple sequence lengths on a representative DiT backbone, using established Wan-Syn and VidProM prompts for training and VBench for evaluation.
Sequence lengths. We evaluate generation quality at 61 and 77 frames (25,088 and 76,800 tokens). We focus on the base models' supported duration range, since extending beyond it can degrade generation quality independently of our acceleration method.
Architectures. Our experiments use Wan2.1, a representative open-source video generation model built on the widely adopted diffusion-transformer (DiT) architecture. Our method could also be applied to other video generation models, such as MiniMax H3; we leave this extension and its evaluation to future work.
Training data. We use only text prompts from Wan-Syn and VidProM, established public datasets used in video-generation research.
Evaluation. We use VBench to assess generation quality and additionally test its extended prompts:
| Model | Evaluation seeds | Original prompts, Total | Extended prompts, Total |
|---|---|---|---|
| Wan2.1-1.3B | 42–46 | 79.72 | 84.36 |
| JetDiffusion-1.3B | 42–46 | 82.72 | 85.21 |
| Wan2.1-14B | 42 | 82.59 | 84.78 |
| JetDiffusion-14B | 42 | 84.02 | 84.64 |
With extended prompts, JetDiffusion remains above the 1.3B base and close to the 14B base. We will add these results to the revised version.
W2.2, W3.1, Q3: Variance of the 14B results
... while the 14B results use only one seed ... What is the variance of the reported 14B results? Since these results use only one random seed, repeated evaluations or uncertainty estimates would be useful for assessing the reliability of the quality improvements.
We extended both the 14B base and JetDiffusion to five generation seeds.
| Model | Quality, mean ± SD | Semantic, mean ± SD | Total, mean ± SD |
|---|---|---|---|
| Wan2.1-14B base, shift 5 | 83.37 ± 1.32 | 75.63 ± 2.07 | 81.82 ± 1.21 |
| JetDiffusion-14B | 84.98 ± 0.40 | 80.32 ± 0.91 | 84.05 ± 0.21 |
JetDiffusion's mean Total is 2.23 points higher, with a 95% confidence interval for its mean Total of [83.79, 84.31]. SD measures variation across generation seeds. We will update the 14B results in the revised version.
W3.2: Human evaluation and failure analysis
The absence of human preference studies, confidence intervals, and systematic failure case analysis makes it difficult to assess perceptual degradation under 90% sparsity and three step generation.
We are conducting a human A/B study; collecting evaluations from human participants takes additional time.
We acknowledge that some generated videos lose fine detail. However, our observations suggest that this degradation is primarily associated with DMD and NVFP4 linear quantization. Our reliance on publicly available training datasets may also limit detail recovery. We will include paired visual comparisons with NVFP4 and BF16 linears in the revised appendix to illustrate the contribution of linear quantization. Improving fine-detail preservation through better distillation, quantization, and training data remains an important direction for future work.
Thanks the authors for the detailed response. My concerns are mostly solved, and I decide to remain my positive score.
Thank you for your positive feedback and valuable suggestions, which have greatly improved our work! Please feel free to raise any further questions.
We thank Reviewer QwdW for recognizing our trainable sparse-attention design, staged training recipe, and quality–efficiency trade-off. To address your concerns about component attribution and evaluation robustness, we have added latency and fusion ablations, prompt extension comparisons, and multi-seed evaluations, as detailed below.
W1, W4, Q1, Q4.1: End-to-end speedup and component contributions
We ablated the RTX 5090 14B pipeline cumulatively, including the decoder as the final change.
On top of three-step distillation, VSQA provides a further 2.88× acceleration with the decoder fixed. NVFP4 linears add another 2.27×. Switching to TAEHV adds 1.34×. With the same Wan decoder, the combined pipeline achieves 203× acceleration.
BF16 14B exceeds RTX 5090’s 32 GB memory. Estimates combine DiT times extrapolated from fewer layers with measured non-DiT times.
To isolate pooling–quantization fusion, we added a four-step ablation from FastVideo VSA to VSQA on GB300 at 77 × 768 × 1280, with identical inputs and selected cubes at 90% sparsity:
The final two rows use the same fine-attention kernel and differ only in pooling–quantization fusion. Fusion reduces preparation time from 1.20 to 0.73 ms (1.64×), yielding a 1.05× full-operator speedup. Together, the changes make VSQA 1.71× faster than FastVideo VSA. The 13.88 ms row is the sparse BF16 variant in the four-variant ablation.
Q4.2: Existing baselines and controlled attention ablations
We provide comparisons with SageAttention2++, SageAttention3, VSA, SLA2, and SageSLA in Table 5. To further clarify the contributions of sparsity and quantization, we added the following controlled ablation on identical inputs within each hardware group.
Since SLA2 has no publicly available implementation, we use our own reproduction.
On RTX 5090, sparsity gives 5.0× / 6.2×, quantization 3.6× / 3.7×, and their combination 20.6× / 29.9× attention speedup for 1.3B / 14B, respectively. On GB300-14B, the corresponding gains are 5.7×, 1.7×, and 8.0×. We will add these benchmarks to the revised version.
RTX 5090: dense BF16 uses PyTorch SDPA (FlashAttention-2); sparse BF16 uses FastVideo’s Triton block-sparse kernel with PyTorch pooling and combining. Both dense and sparse low-bit variants use NVFP4 for QK and PV. GB300: dense BF16 and low-bit use FA4 CuTe. Both dense and sparse low-bit variants use NVFP4 for QK and FP8 for PV. Sparse variants select identical cubes at 90% sparsity.