acceptodds
Under review as a conference paper at ICLR 2027

JetDiffusion: Pushing the Speed Limits of Video Generation with NVFP4 Sparse Attention

Abstract

Attention over long sequences is a major computational bottleneck in video generation, which can be alleviated through sparsity and low-bit quantization. We propose JetDiffusion, a video generation system built on trainable Video Sparse Quantized Attention (VSQA), with custom inference kernels that combine native NVFP4 computation and kernel fusion. At 90% sparsity, VSQA accelerates attention by 17-29x compared with FlashAttention on different hardwares. To further accelerate video generation, we combine VSQA with NVFP4 linear quantization and step distillation. Although each technique can individually preserve generation quality, jointly adapting to all three remains challenging, particularly under aggressive NVFP4 quantization. Through empirical studies, we develop an effective staged training recipe that enables stable adaptation to quantization, sparsity, and few-step denoising. On Wan2.1-14B, the resulting model achieves 282x end-to-end video generation speedup on the RTX 5090 while maintaining comparable video quality.

Primary area:
generative models
Keywords:
Video Generation, Diffusion Transformers, Few-Step Distillation, Sparse Attention, Quantization-Aware Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QwdWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the computational cost of video generation, where attention over long spatiotemporal sequences is the main bottleneck. It proposes JetDiffusion, a system built on Video Sparse Quantized Attention (VSQA), which to the authors' knowledge is the first trainable attention method combining high sparsity (90%) with NVFP4 quantization, supported by custom training and inference kernels for the RTX 5090 and GB300; the full system additionally quantizes linear layers to NVFP4 and distills the model to three denoising steps, with a staged training recipe—layer-wise reconstruction warm-up, then regularized ODE initialization, then DMD refinement—to adapt the model to the combined effects of quantization, sparsity, and few-step denoising.

Critical strengths
  1. The paper presents VSQA, a trainable video attention mechanism that jointly exploits high sparsity and native NVFP4 computation. Its hardware aware training and inference kernels address an important systems bottleneck in efficient video generation.

  2. The staged training recipe is technically well motivated and consists of layer wise reconstruction warm up, sparse ODE initialization, reconstruction regularization, and DMD refinement. The training analyses and ablations support the role of each component in reducing quantization error and preserving teacher alignment.

  3. JetDiffusion achieves a strong quality and efficiency trade off under aggressive compression. With 90% sparsity and three denoising steps, it preserves or improves aggregate VBench scores while achieving 17.2 to 28.5 times attention speedups and up to 282 times end to end acceleration.

Critical weaknesses
  1. The reported end to end speedups are not fully apples to apples. Table 6 uses TAEHV for JetDiffusion and the default VAE for the base model, so the 282 times and 144 times gains combine denoising, attention, linear quantization, and decoder differences.
  2. The evaluation scope is relatively narrow. Experiments focus on Wan2.1 models at two resolutions with text prompts from Wan Syn and VidProM, while the 14B results use only one seed, leaving generalization to other architectures, sequence lengths, and prompt distributions uncertain.
  3. The quality evaluation relies mainly on VBench scores and selected qualitative examples. The absence of human preference studies, confidence intervals, and systematic failure case analysis makes it difficult to assess perceptual degradation under 90% sparsity and three step generation.
  4. The contribution is difficult to disentangle from the combination of existing techniques. Although the ablations validate the staged training procedure, they do not fully separate the effects of sparse attention design, kernel fusion, linear layer quantization, and distillation, which makes the precise source of the gains less clear.
Questions for authors
  1. How does the reported end to end speedup change when JetDiffusion and the base model use the same video decoder? A component wise latency breakdown would clarify the contributions of attention, linear quantization, denoising step reduction, and TAEHV.
  2. What evidence supports generalization beyond Wan2.1 and the current prompt distributions? Evaluation on another video diffusion architecture, longer sequences, or additional datasets would strengthen this claim.
  3. What is the variance of the reported 14B results? Since these results use only one random seed, repeated evaluations or uncertainty estimates would be useful for assessing the reliability of the quality improvements.
  4. How can the individual contributions of VSQA, NVFP4 linear quantization, kernel fusion, and three step distillation be disentangled? Matched comparisons with recent sparse and low bit baselines would clarify the source of the overall gains.
4 comments

We thank Reviewer QwdW for recognizing our trainable sparse-attention design, staged training recipe, and quality–efficiency trade-off. To address your concerns about component attribution and evaluation robustness, we have added latency and fusion ablations, prompt extension comparisons, and multi-seed evaluations, as detailed below.

W1, W4, Q1, Q4.1: End-to-end speedup and component contributions

How does the reported end to end speedup change when JetDiffusion and the base model use the same video decoder? A component wise latency breakdown would clarify the contributions of attention, linear quantization, denoising step reduction, and TAEHV. How can the individual contributions of VSQA, NVFP4 linear quantization, kernel fusion, and three step distillation be disentangled?

We ablated the RTX 5090 14B pipeline cumulatively, including the decoder as the final change.

Cumulative configurationEnd-to-end latency
Base: 50 steps with CFG, dense BF16, optimized Wan VAE3,177 s
Remove CFG: 50 forwards1,591 s
Reduce to three forwards100.0 s
+ VSQA34.68 s
+ NVFP4 linears15.29 s estimated / 15.64 s measured
+ TAEHV decoder11.64 s measured

On top of three-step distillation, VSQA provides a further 2.88× acceleration with the decoder fixed. NVFP4 linears add another 2.27×. Switching to TAEHV adds 1.34×. With the same Wan decoder, the combined pipeline achieves 203× acceleration.

BF16 14B exceeds RTX 5090’s 32 GB memory. Estimates combine DiT times extrapolated from fewer layers with measured non-DiT times.

To isolate pooling–quantization fusion, we added a four-step ablation from FastVideo VSA to VSQA on GB300 at 77 × 768 × 1280, with identical inputs and selected cubes at 90% sparsity:

ConfigurationFull-operator latency (ms)Speedup over previous stepSpeedup over dense BF16
FastVideo VSA, FA4 CuTe BF16 fine kernel17.06—4.7×
+ Optimized pooling, mask packing, and branch combining; BF1613.881.23×5.7×
+ NVFP4 QK / FP8 PV, separate pooling and quantization10.461.33×7.6×
+ Fused pooling and quantization (VSQA)9.981.05×8.0×

The final two rows use the same fine-attention kernel and differ only in pooling–quantization fusion. Fusion reduces preparation time from 1.20 to 0.73 ms (1.64×), yielding a 1.05× full-operator speedup. Together, the changes make VSQA 1.71× faster than FastVideo VSA. The 13.88 ms row is the sparse BF16 variant in the four-variant ablation.

Q4.2: Existing baselines and controlled attention ablations

Matched comparisons with recent sparse and low bit baselines would clarify the source of the overall gains.

We provide comparisons with SageAttention2++, SageAttention3, VSA, SLA2, and SageSLA in Table 5. To further clarify the contributions of sparsity and quantization, we added the following controlled ablation on identical inputs within each hardware group.

Since SLA2 has no publicly available implementation, we use our own reproduction.

Hardware / model / video sizeLow-bit QK / PVDense BF16Dense low-bitSparse BF16Sparse low-bit (VSQA)
RTX 5090 / 1.3B / 61 × 448 × 896 (25,088 tokens)NVFP4 / NVFP417.594.86 (3.6×)3.53 (5.0×)0.855 (20.6×)
RTX 5090 / 14B / 77 × 768 × 1280 (76,800 tokens)NVFP4 / NVFP4546.95149.83 (3.7×)87.72 (6.2×)18.29 (29.9×)
GB300 / 14B / 77 × 768 × 1280 (76,800 tokens)NVFP4 / FP879.4445.58 (1.7×)13.88 (5.7×)9.98 (8.0×)

On RTX 5090, sparsity gives 5.0× / 6.2×, quantization 3.6× / 3.7×, and their combination 20.6× / 29.9× attention speedup for 1.3B / 14B, respectively. On GB300-14B, the corresponding gains are 5.7×, 1.7×, and 8.0×. We will add these benchmarks to the revised version.

RTX 5090: dense BF16 uses PyTorch SDPA (FlashAttention-2); sparse BF16 uses FastVideo’s Triton block-sparse kernel with PyTorch pooling and combining. Both dense and sparse low-bit variants use NVFP4 for QK and PV. GB300: dense BF16 and low-bit use FA4 CuTe. Both dense and sparse low-bit variants use NVFP4 for QK and FP8 for PV. Sparse variants select identical cubes at 90% sparsity.

W2.1, Q2: Generalization across sequence lengths, architectures, and prompts

What evidence supports generalization beyond Wan2.1 and the current prompt distributions? Evaluation on another video diffusion architecture, longer sequences, or additional datasets would strengthen this claim.

Our experiments cover multiple sequence lengths on a representative DiT backbone, using established Wan-Syn and VidProM prompts for training and VBench for evaluation.

Sequence lengths. We evaluate generation quality at 61 and 77 frames (25,088 and 76,800 tokens). We focus on the base models' supported duration range, since extending beyond it can degrade generation quality independently of our acceleration method.

Architectures. Our experiments use Wan2.1, a representative open-source video generation model built on the widely adopted diffusion-transformer (DiT) architecture. Our method could also be applied to other video generation models, such as MiniMax H3; we leave this extension and its evaluation to future work.

Training data. We use only text prompts from Wan-Syn and VidProM, established public datasets used in video-generation research.

Evaluation. We use VBench to assess generation quality and additionally test its extended prompts:

ModelEvaluation seedsOriginal prompts, TotalExtended prompts, Total
Wan2.1-1.3B42–4679.7284.36
JetDiffusion-1.3B42–4682.7285.21
Wan2.1-14B4282.5984.78
JetDiffusion-14B4284.0284.64

With extended prompts, JetDiffusion remains above the 1.3B base and close to the 14B base. We will add these results to the revised version.

W2.2, W3.1, Q3: Variance of the 14B results

... while the 14B results use only one seed ... What is the variance of the reported 14B results? Since these results use only one random seed, repeated evaluations or uncertainty estimates would be useful for assessing the reliability of the quality improvements.

We extended both the 14B base and JetDiffusion to five generation seeds.

ModelQuality, mean ± SDSemantic, mean ± SDTotal, mean ± SD
Wan2.1-14B base, shift 583.37 ± 1.3275.63 ± 2.0781.82 ± 1.21
JetDiffusion-14B84.98 ± 0.4080.32 ± 0.9184.05 ± 0.21

JetDiffusion's mean Total is 2.23 points higher, with a 95% confidence interval for its mean Total of [83.79, 84.31]. SD measures variation across generation seeds. We will update the 14B results in the revised version.

W3.2: Human evaluation and failure analysis

The absence of human preference studies, confidence intervals, and systematic failure case analysis makes it difficult to assess perceptual degradation under 90% sparsity and three step generation.

We are conducting a human A/B study; collecting evaluations from human participants takes additional time.

We acknowledge that some generated videos lose fine detail. However, our observations suggest that this degradation is primarily associated with DMD and NVFP4 linear quantization. Our reliance on publicly available training datasets may also limit detail recovery. We will include paired visual comparisons with NVFP4 and BF16 linears in the revised appendix to illustrate the contribution of linear quantization. Improving fine-detail preservation through better distillation, quantization, and training data remains an important direction for future work.

Thanks the authors for the detailed response. My concerns are mostly solved, and I decide to remain my positive score.

Thank you for your positive feedback and valuable suggestions, which have greatly improved our work! Please feel free to raise any further questions.

Reviewer_e2UyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper targets the cost of attention and low-precision inference in video diffusion. It proposes VSQA, a trainable NVFP4 sparse attention built on VSA's cube routing, and JetDiffusion, a three-stage recipe that combines quantization warm-up, sparse ODE initialization, and DMD for three-step generation. On Wan2.1-14B at 720p, it reports a 282x end-to-end speedup on an RTX 5090 and 144x on a GB300, with VBench scores above its dense base.

Critical strengths
  1. The systems work is solid and the paper documents it well. VSQA keeps the coarse routing, quantizes only the selected fine branch, uses a straight-through estimator, and saves an auxiliary unrounded-probability output for the softmax correction. The kernels cover two Blackwell parts, and the measured 17-29x attention speedups and 151-307x total DiT speedups are large against reasonable dense and sparse baselines.

  2. The training recipe is the stronger contribution. The paper explains why QAT/QAD warm-up makes a poor initialization for DMD: it recovers standalone quality while teacher alignment drifts. Layer-wise reconstruction keeps the student closer to the teacher (Figs. 3-5), and the final ablation holds together: reconstruction warm-up beats QAD warm-up, and removing ODE initialization or the reconstruction regularizer costs 0.2-0.8 total points. I expect this recipe to carry over to other compressed video diffusion models.

Critical weaknesses
  1. The paper times fast baselines but never measures their quality. Table 5 compares latency to TurboDiffusion, LightWan2.2, and SageSLA. Table 1 compares quality only to the 50-step dense base and to the authors' own ablations. FastWan/FastWan-QAD, which the related work describes as a direct mix of low-bit attention and DMD, is missing from the experiments. Without quality numbers for these systems at matched prompts, resolution, and step budget, the 282x figure tells me little about the quality-speed frontier. A three-step model that beats TurboDiffusion on speed but loses detail would still be a useful system, though the framing of the paper would need to change.

  2. The quality claim is not supported under the public Wan2.1 numbers, and the paper does not isolate the cost of its components. The Wan2.1 report gives 84.92/80.10/83.96 for Wan2.1-1.3B on the 16-dimension VBench, while Table 1 reports 83.52/64.50/79.72 for the same checkpoint. That 15.6-point semantic gap is too large to ignore. Under the report's numbers, the student's 85.28/72.47/82.72 is +0.36 Quality, −7.63 Semantic, and −1.24 Total, so "match or exceed" holds only against the paper's own base. The 14B case shows the same pattern: the report gives 86.67/84.44/86.22, the base in Table 1 is 83.91/75.41/82.21, and the student is 85.16/79.48/84.02. Table 1 also has no run that removes VSQA, the NVFP4 linear layers, or the TAEHV decoder, so I cannot attribute the quality delta to any one component. Appendix D reinforces the concern: several panels (Figs. 8, 9, 11, 12, 14, 17, 19) show the JetDiffusion output less detailed than the base, which VBench does not capture.

Questions for authors
  1. Could you add quality numbers for TurboDiffusion, LightWan2.2, and FastWan/FastWan-QAD at the same prompts, resolution, and step budget as JetDiffusion? If checkpoints or code are unavailable, a controlled reimplementation or a clear statement of what cannot be compared would help. With those numbers, the latency table becomes a quality-speed comparison.

  2. Could you ablate the final three-step model with (i) dense attention instead of VSQA, (ii) BF16 linear layers instead of NVFP4, and (iii) the default Wan VAE instead of TAEHV? A sparsity sweep at 70/80/90% would show where the quality cliff is. These runs would let readers see which compression step costs what.

  3. The base numbers need a clear reconciliation. The Wan2.1 report gives 84.92/80.10/83.96 for Wan2.1-T2V-1.3B on the 16-dimension VBench, while Table 1 gives 83.52/64.50/79.72. At 14B, the report gives 86.67/84.44/86.22, while Table 1 gives 83.91/75.41/82.21 for the base. The Wan2.1 model card recommends 832×480, CFG 6, and shift 8, with prompt extension off. Could you specify the VBench version and commit, resolution, frame count, sampling steps, CFG scale, negative prompt, shift, and whether prompt rewriting was used? It would help to add two rows: the official Wan2.1 checkpoint under your protocol, and your student under the official protocol. That would separate a protocol gap from a model gap.

  4. You report five seeds for 1.3B and one seed for 14B. Could you report standard deviations or confidence intervals, at least for the 1.3B comparisons? Several differences in Table 1 (e.g., Ours vs QAD warm-up, 0.5 total points) are small enough that variance matters.

  5. The 282x end-to-end number combines 50→3 steps, NVFP4 linear layers, VSQA, and TAEHV. Table 6 credits 5x of non-DiT speedup on the RTX 5090 but 34.4x on the GB300 for what looks like the same decoder swap. Could you explain that difference and give a clean decomposition? Do you also plan to release the custom kernels? They are the main artifact for reproducing the attention numbers.

9 comments

We thank Reviewer e2Uy for the constructive review and recognition of our systems work and teacher-alignment motivation. To address your concerns about evaluation fairness and quality preservation, we have added protocol and component ablations, baseline comparisons, and multi-seed evaluations, as detailed below.

W1, Q1: Additional baseline quality results

The paper times fast baselines but never measures their quality. ... Could you add quality numbers for TurboDiffusion, LightWan2.2, and FastWan/FastWan-QAD at the same prompts, resolution, and step budget as JetDiffusion? ...

We have added quality results for TurboDiffusion, LightWan/LightX2V, and FastWan/FastWan-QAD, using the same VBench prompts and original Wan VAE as Table 1. Results are pooled over five seeds (42–46) for both 1.3B and 14B.

For TurboDiffusion, we use our own reproduction with numerical precision settings, including low-bit formats, matched to the original implementation. For LightX2V-1.3B, we use our own reproduction with the released checkpoint and sampler. Both implementations are available in our anonymous repository.

ModelFrames × H × WStepsQualitySemanticTotal
JetDiffusion-14B77 × 768 × 1280384.9980.3284.06
TurboDiffusion-14B81 × 720 × 1280483.5173.8281.57
LightWan2.2-A14B81 × 720 × 1280483.9578.7882.92
JetDiffusion-1.3B61 × 448 × 896385.2872.4782.72
TurboDiffusion-1.3B81 × 480 × 832483.9468.5780.86
LightX2V-1.3B81 × 480 × 832484.8470.1981.91
FastWan-1.3B81 × 480 × 848385.1466.1481.34
FastWan-QAD-1.3B81 × 480 × 832381.6262.8977.87

JetDiffusion achieves the highest VBench Total among the compared methods at both 1.3B and 14B. We will add these results to Table 1 of the revised version.

W2.1, Q3.1: Evaluation settings and base-score differences

... The Wan2.1 report gives 84.92/80.10/83.96 for Wan2.1-1.3B on the 16-dimension VBench, while Table 1 reports 83.52/64.50/79.72 for the same checkpoint. ... Could you specify the VBench version and commit, resolution, frame count, sampling steps, CFG scale, negative prompt, shift, and whether prompt rewriting was used?

Our main-table evaluation uses the original VBench prompts without prompt extension, Wan VAE, and the following sampling settings:

ModelFrames × H × WSteps / samplerCFGShift
Wan2.1-1.3B base61 × 448 × 89650 / UniPC53
JetDiffusion-1.3B61 × 448 × 8963 / DMD18
Wan2.1-14B base77 × 768 × 128050 / UniPC55
JetDiffusion-14B77 × 768 × 12803 / DMD18

Evaluation uses all 16 VBench dimensions over 946 prompts, commit 45e79ec, with AVoCaDO replacing Tag2Text for scene scoring. Videos use 16 fps. Bases use Wan's negative prompt translated into English; students use none. The updated 14B base uses shift 5 for both Quality and Semantic.

For 1.3B, Wan recommends CFG 6, shift 8, and 81 × 480 × 832 videos, with a Chinese negative prompt. VBench's configuration uses CFG 6 and shift 3 at the same resolution and frame count. Our base evaluation uses CFG 5, shift 3, and 61 × 448 × 896 videos, without prompt extension. Since these configurations differ and the Wan report does not disclose its full evaluation protocol, we ablated the original checkpoint under different settings, using seed 42 throughout:

CFGShiftFrames × H × WPrompt extensionDecoderQualitySemanticTotal
5361 × 448 × 896OffWan VAE84.4266.1080.75
5361 × 448 × 896OffTAEHV84.2265.5380.48
5361 × 448 × 896OnWan VAE85.2981.0184.43
6881 × 480 × 832OffWan VAE84.1568.3480.99
6881 × 480 × 832OnWan VAE85.1783.8884.91
5381 × 480 × 832OffWan VAE84.1767.7980.89
5581 × 480 × 832OffWan VAE83.7768.7080.75
3.5861 × 448 × 896OffWan VAE84.6366.0780.92

Prompt extension has the largest observed effect in this ablation, raising Semantic by 14.91 points under our settings and 15.54 points under Wan's upstream settings. Changes to CFG, shift, and resolution have only modest effects in this ablation. These results suggest that whether prompt extension is enabled is the main contributor to the gap with Wan's reported VBench scores.

Q3.2: Base and student with and without prompt extension

It would help to add two rows: the official Wan2.1 checkpoint under your protocol, and your student under the official protocol. That would separate a protocol gap from a model gap.

We therefore tested prompt extension for both the base and JetDiffusion, using 5 seeds (seed = 42-46) for 1.3B and 1 seed (seed = 42) for 14B. This ablation changes only the prompts; each model retains its sampling settings above.

ModelPrompt extensionEvaluation seedsQualitySemanticTotal
Wan2.1-1.3BOff (main table)42–4683.5264.5079.72
On42–4685.2180.9584.36
JetDiffusion-1.3BOff (main table)42–4685.2872.4782.72
On42–4686.2081.2585.21
Wan2.1-14B (shift 5)Off4284.3875.4182.59
On4285.6081.5384.78
JetDiffusion-14BOff4285.1679.4884.02
On4284.8483.8584.64

With prompt extension enabled for both models, JetDiffusion scores higher on all three VBench metrics at 1.3B and remains comparable at 14B. We will add these ablations to the revised version.

We also conducted an additional experiment with prompt extension enabled during training. We will report the results in a follow-up response.

W2.2, Q2.1: Component quality ablations

... Table 1 also has no run that removes VSQA, the NVFP4 linear layers, or the TAEHV decoder, so I cannot attribute the quality delta to any one component. ... Could you ablate the final three-step model with (i) dense attention instead of VSQA, (ii) BF16 linear layers instead of NVFP4, and (iii) the default Wan VAE instead of TAEHV?

First, we swapped components of the final model at inference, without retraining:

Configuration1.3B Total, 5 seeds14B Total, seed 42
VSQA + NVFP4 linears + Wan VAE82.7284.02
Dense BF16 attention81.9383.40
BF16 linears82.7584.52
Dense BF16 attention + BF16 linears82.0683.63
TAEHV decoder82.7883.78

Second, we trained a separate 1.3B model for each attention and linear-layer precision (one training run and five evaluation seeds each):

TrainingAttentionLinearsTotal
Direct DMDDense BF16BF1682.69
Direct DMD90% sparse BF16BF1682.91
Direct DMD90% sparse NVFP4BF1682.32
Direct DMDDense BF16NVFP481.96
Direct DMD90% sparse FP8FP882.33
Direct DMD90% sparse NVFP4NVFP481.67
Staged recipe90% sparse NVFP4NVFP482.72

Restoring dense attention or BF16 precision does not consistently improve Total without retraining. With staged training, the fully compressed model achieves 82.72, comparable to the dense BF16 control (82.69) and 1.05 points above direct NVFP4 DMD.

The quality reference already uses Wan VAE; the TAEHV swap changes Total by only +0.06 / −0.24 points. We will add both ablations to the revised version.

Q2.2: Quality at 70–90% sparsity

A sparsity sweep at 70/80/90% would show where the quality cliff is.

Thank you for this helpful suggestion. We swept inference sparsity from 70% to 90% and measured DiT speedup on RTX 5090.

Sparsity1.3B Total, 5 seeds1.3B DiT speedup14B Total, seed 4214B DiT speedup*
70%82.824.13×84.205.44×
80%82.794.41×84.526.24×
90%82.724.87×84.027.26×

Relative to dense BF16 attention, with linears fixed; 14B timings are extrapolated.

Across 70–90% sparsity, Total varies by only 0.10 / 0.50 points for 1.3B / 14B while DiT speedup increases, showing no sharp quality drop in the tested range. We will add this sparsity sweep to the revised version.

W2.3: Fine-detail preservation

... Appendix D reinforces the concern: several panels (Figs. 8, 9, 11, 12, 14, 17, 19) show the JetDiffusion output less detailed than the base, which VBench does not capture.

We acknowledge that some generated videos lose fine detail. However, our observations suggest that this degradation is primarily associated with DMD and NVFP4 linear quantization and should not be attributed solely to our staged training strategy. Our reliance on publicly available training datasets may also limit detail recovery. We will include paired visual comparisons with NVFP4 and BF16 linears in the revised appendix to illustrate the contribution of linear quantization. Improving fine-detail preservation through better distillation, quantization, and training data remains an important direction for future work. We are conducting a human A/B study; collecting evaluations from human participants takes additional time.

Q4: Variance of the quality comparisons

Could you report standard deviations or confidence intervals, at least for the 1.3B comparisons? Several differences in Table 1 (e.g., Ours vs QAD warm-up, 0.5 total points) are small enough that variance matters.

We report mean ± sample SD across five generation seeds for the same training variants, and extend both 14B main rows to five seeds.

MethodWarm-upODE initializationQuality, mean ± SDSemantic, mean ± SDTotal, mean ± SD
1.3B base——83.50 ± 1.1964.50 ± 3.3879.70 ± 1.59
1.3B Direct DMDNoneNone84.32 ± 0.2171.05 ± 1.2181.67 ± 0.15
QADNone84.51 ± 0.4669.71 ± 2.5181.55 ± 0.43
Layerwise reconstructionNone84.71 ± 0.4073.54 ± 1.5482.47 ± 0.22
QADw/ reconstruction regularization85.08 ± 0.2970.54 ± 1.6182.17 ± 0.48
Layerwise reconstructionw/o reconstruction regularization84.82 ± 0.4170.33 ± 0.9481.92 ± 0.40
1.3B OursLayerwise reconstructionw/ reconstruction regularization85.25 ± 0.2872.47 ± 1.7482.69 ± 0.42
14B base, shift 5——83.37 ± 1.3275.63 ± 2.0781.82 ± 1.21
14B OursLayerwise reconstructionw/ reconstruction regularization84.98 ± 0.4080.32 ± 0.9184.05 ± 0.21

The 14B student's 95% confidence interval for mean Total is [83.79, 84.31]. The small 1.3B QAD warm-up gap is not established as statistically significant. SD measures variation across generation seeds. We will add these statistics to the revised version.

Q5.1: End-to-end component ablation

The 282x end-to-end number combines 50→3 steps, NVFP4 linear layers, VSQA, and TAEHV. ... Could you explain that difference and give a clean decomposition?

We ablated the RTX 5090 14B pipeline cumulatively, including the decoder as the final change.

Cumulative configurationEnd-to-end latency
Base: 50 steps with CFG, dense BF16, optimized Wan VAE3,177 s
Remove CFG: 50 forwards1,591 s
Reduce to three forwards100.0 s
+ VSQA34.68 s
+ NVFP4 linears15.29 s estimated / 15.64 s measured
+ TAEHV decoder11.64 s measured

On top of three-step distillation, VSQA provides a further 2.88× acceleration with the decoder fixed. NVFP4 linears add another 2.27× and switching to TAEHV adds 1.34×. With the same Wan decoder, the combined pipeline achieves 203× acceleration. We will add this breakdown to the revised version.

BF16 14B exceeds RTX 5090’s 32 GB memory. Estimates combine DiT times extrapolated from fewer layers with measured non-DiT times.

Q5.2: Different non-DiT speedups

Table 6 credits 5x of non-DiT speedup on the RTX 5090 but 34.4x on the GB300 for what looks like the same decoder swap. Could you explain that difference ...

The discrepancy mainly comes from different baseline decoder implementations: RTX 5090 used an optimized Wan VAE, whereas GB300 used the slower stock implementation. We remeasured GB300 at 77 frames using the same optimized Wan VAE implementation as RTX 5090 for the base, and TAEHV for VSQA:

TimingDense FA4, 50 steps with CFG, optimized Wan VAEVSQA, 3 steps, TAEHVSpeedup
Average DiT forward4.88 s1.28 s3.8×
Total DiT487.60 s3.83 s127×
Non-DiT1.376 s0.280 s4.9×
End-to-end488.97 s4.11 s119×

With the decoder implementation and frame count aligned, GB300 achieves a 4.9× non-DiT speedup, comparable to the approximately 5× on RTX 5090. We will update these measurements in the revised version.

Q5.3: Kernel release

Do you also plan to release the custom kernels? They are the main artifact for reproducing the attention numbers.

Our inference code and custom kernels are available in our anonymous repository. The repository also includes our reproductions of TurboDiffusion and LightX2V-1.3B. The TurboDiffusion reproduction matches the original implementation’s numerical precision settings, including low-bit formats. Our LightX2V reproduction uses the released checkpoint and sampler.

Reviewer_wMpBOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper combines sparse attention, NVFP4 quantization and few step distillation in one for Wan2.1-14B, a diffusion based video generation model. It includes kernel optimization, reconstruction training warm up, ODE initialization and DMD to keep the quality nice. Besides, this paper introduces Video Sparse Quantized Attention (VSQA), video sparse attention at low precision. It provides new forward and backward kernels. Experiments cover Wan2.1 models. Finally get huge speed over original Wan inference pipeline with similar Vbench results.

Critical strengths
  1. Kernel Optimizations. Kernels for NVFP4 low precision sparse attention is the crucial point of the method. It beats current dense flash attention and other sparse attentions. It performs faster than 14B Wan2.1 with SageSLA on RTX 5090 which speeds up from 42.5ms to 19.7ms.
  2. Training recipe. This paper try to figure out a better to combine low precision kernels, sparse attention and few step distillation during training and it does figure out a better way to sort them. Experiments shows the effective.
  3. This paper is clear written and easy to follow.
Critical weaknesses
  1. The evaluation does not sufficiently isolate the contributions of sparsity and NVFP4 quantization or establish how their benefits scale with video size. Table 5 usefully separates attention, single-step DiT, and total DiT timings, and includes relevant dense quantized and sparse baselines. However, these comparisons vary in kernel implementation and block size, while the complete system additionally changes linear-layer precision, sampling steps, and the decoder.
  2. Quality comparisons with accelerated alternatives are missing from the reported evaluation. Table 1 compares the base models and the authors' training variants, while TurboDiffusion and LightWan2.2 are included in the timing comparisons without corresponding quality scores. Consequently, the reported latency advantages over these systems do not establish an advantage at comparable quality.
Questions for authors
  1. Compare dense BF16, dense low-bit, sparse BF16, and sparse low-bit attention on identical inputs and hardware, keeping the routing, coarse branch, sparsity, and cube size matched between the sparse variants. Please state QK/PV precision separately for each hardware implementation and report attention and single-step DiT latency with non-attention settings fixed. A breakdown of pooling/routing, quantization/packing, and fine attention would help explain which changes account for the gain.
  2. Will the sparse attention or the NVFP4 kernel perform worse than dense attention or FP8 kernels when it comes to fewer token numbers? For example, in streaming generation scenarios.
  3. Which decoder and inference implementation produced Table 1? Do its quality scores correspond to the timed deployment configurations, including TAEHV? Could you provide quality alongside the reported latency for the closest accelerated alternatives, such as TurboDiffusion, using the same base model, prompts, resolution, and frame count where available?
  4. Is there any training stability problem in sparse + NVFP4 + DMD training?
6 comments

We thank Reviewer wMpB for recognizing our kernel optimizations, training recipe, and clear presentation. To address your concerns about efficiency attribution and quality comparisons, we have added controlled ablations, short-sequence benchmarks, and baseline quality evaluations, as detailed below.

W1.1, Q1.1: Ablation of sparsity and quantization

The evaluation does not sufficiently isolate the contributions of sparsity and NVFP4 quantization ... while the complete system additionally changes linear-layer precision, sampling steps, and the decoder. Compare dense BF16, dense low-bit, sparse BF16, and sparse low-bit attention on identical inputs and hardware ...

We benchmarked four attention variants of Wan2.1-1.3B and Wan2.1-14B on identical inputs on RTX 5090 and GB300. The table gives the latency of the full attention operator in ms, with the speedup over dense BF16 in parentheses.

Hardware / model / video sizeLow-bit QK / PVDense BF16Dense low-bitSparse BF16Sparse low-bit (VSQA)
RTX 5090 / 1.3B / 61 × 448 × 896 (25,088 tokens)NVFP4 / NVFP417.594.86 (3.6×)3.53 (5.0×)0.855 (20.6×)
RTX 5090 / 14B / 77 × 768 × 1280 (76,800 tokens)NVFP4 / NVFP4546.95149.83 (3.7×)87.72 (6.2×)18.29 (29.9×)
GB300 / 14B / 77 × 768 × 1280 (76,800 tokens)NVFP4 / FP879.4445.58 (1.7×)13.88 (5.7×)9.98 (8.0×)

On RTX 5090, sparsity gives 5.0× / 6.2×, quantization 3.6× / 3.7×, and their combination 20.6× / 29.9× attention speedup for 1.3B / 14B, respectively. On GB300-14B, the corresponding gains are 5.7×, 1.7×, and 8.0×.

RTX 5090: dense BF16 uses PyTorch SDPA (FlashAttention-2); sparse BF16 uses FastVideo’s Triton block-sparse kernel with PyTorch pooling and combining. Both dense and sparse low-bit variants use NVFP4 for QK and PV. GB300: dense BF16 and low-bit use FA4 CuTe. Both dense and sparse low-bit variants use NVFP4 for QK and FP8 for PV. Sparse variants select identical cubes at 90% sparsity.

We also timed one full DiT forward step with each attention variant in a separate benchmark:

Hardware / modelLinearsDense BF16Dense low-bitSparse BF16Sparse low-bit (VSQA)
RTX 5090 / 1.3BNVFP4748.0 ms258.1 ms (2.9×)400.2 ms (1.9×)153.1 ms (4.9×)
RTX 5090 / 14B*NVFP425,258 ms7,899 ms (3.2×)8,686 ms (2.9×)3,470 ms (7.3×)
GB300 / 14BBF166.132 s4.795 s (1.3×)3.331 s (1.8×)3.221 s (1.9×)

These attention speedups translate into 4.9× / 7.3× faster DiT steps on RTX 5090-1.3B / 14B with NVFP4 linears, and 1.9× on GB300-14B with BF16 linears. We will add these ablations to the revised version.

* RTX 5090 14B DiT times are extrapolated from models with fewer layers because the BF16 14B model exceeds its 32 GB GPU memory capacity. For the speedups reported in the paper, the base model uses CPU offloading, following the same benchmarking protocol as TurboDiffusion and related work.

Q1.2: Kernel breakdown.

A breakdown of pooling/routing, quantization/packing, and fine attention would help explain which changes account for the gain.

Thank you for this helpful suggestion. We profiled four VSQA stages at 90% sparsity on RTX 5090 and GB300.

StageRTX 5090, 1.3B (ms)RTX 5090, 14B (ms)GB300, 14B (ms)
Quantization/pooling and packing/tile0.195 (22.5%)1.93 (10.2%)4.518 (45.1%)
Routing0.033 (3.8%)0.18 (1.0%)0.257 (2.6%)
Fine attention0.485 (55.9%)15.66 (82.8%)4.538 (45.3%)
Combining branches / untile0.155 (17.9%)1.15 (6.1%)0.710 (7.1%)

Fine attention accounts for 55.9% / 82.8% / 45.3% of the stage totals; routing adds only 1.0–3.8%. We will add this stage-level profiling breakdown to the revised version.

W1.2, Q2: Performance at short sequence lengths

The evaluation does not ... establish how their benefits scale with video size. ... Will the sparse attention or the NVFP4 kernel perform worse than dense attention or FP8 kernels when it comes to fewer token numbers? ...

We swept the token count on RTX 5090, keeping each model's default cube, and report attention latency in ms:

ModelTokensDense BF16Dense NVFP4 QK / FP8 PVDense NVFP4 QK / NVFP4 PVVSQA
1.3B3920.0240.0250.0260.049
1.3B7840.0450.0300.0310.050
1.3B1,5680.0900.0420.0440.050
1.3B3,1360.3270.1050.1070.084
1.3B6,2721.2660.3690.3630.204
14B9600.1100.0540.0550.257
14B1,9200.4030.1500.1440.257
14B3,8401.5640.5250.5140.278
14B7,6805.7581.8151.6840.563

With default cubes, VSQA is fastest from 3,136 tokens for 1.3B and 3,840 for 14B among the tested sizes. Dense attention is preferable for shorter chunks; NVFP4 and FP8 PV differ by only a few microseconds there. We will add these benchmarks to the revised version.

W2, Q3.1: Additional baseline quality results

Quality comparisons with accelerated alternatives are missing from the reported evaluation. Could you provide quality alongside the reported latency for the closest accelerated alternatives ...

We have added quality results for TurboDiffusion and LightWan/LightX2V, using the original Wan VAE consistently with Table 1.

For TurboDiffusion, we use our own reproduction with numerical precision settings, including low-bit formats, matched to the original implementation. For LightX2V-1.3B, we use our own reproduction with the released checkpoint and sampler. Both implementations are available in our anonymous repository.

ModelFrames × H × WStepsQualitySemanticTotal
JetDiffusion-14B77 × 768 × 1280384.9980.3284.06
TurboDiffusion-14B81 × 720 × 1280483.5173.8281.57
LightWan2.2-A14B81 × 720 × 1280483.9578.7882.92
JetDiffusion-1.3B61 × 448 × 896385.2872.4782.72
TurboDiffusion-1.3B81 × 480 × 832483.9468.5780.86
LightX2V-1.3B81 × 480 × 832484.8470.1981.91

JetDiffusion achieves the highest VBench Total among the compared methods at both 1.3B and 14B. We will add these results to Table 1 of the revised version.

Q3.2: Decoder used for latency and quality evaluation

Which decoder and inference implementation produced Table 1?

For efficiency comparisons with other accelerated systems, we use the lightweight TAEHV decoder; main-table quality comparisons use the original Wan VAE. To isolate the decoder's effect, we compared both decoders with each model and its sampling settings fixed. The outputs are nearly indistinguishable by visual inspection; we will include paired examples in the revised paper. VBench Quality scores are reported below:

ModelSeedsWan VAE QualityTAEHV QualityChange
Wan2.1-1.3B base583.5383.31−0.22
JetDiffusion-1.3B585.2885.37+0.09
JetDiffusion-14B1 (42)85.1685.00−0.16

Changing the decoder affects VBench Quality by at most 0.22 points across these comparisons.

Q4: Training instability

Is there any training stability problem in sparse + NVFP4 + DMD training?

To assess how the two initialization stages affect DMD training, we compared different warm-up and ODE initialization settings on Wan2.1-1.3B, examining final quality and visual stability at fixed sparsity and precision.

MethodWarm-upODE initializationQualitySemanticTotal
Direct DMDNoneNone84.3371.0581.67
QADNone84.5169.7181.55
Layerwise reconstructionNone84.7373.5482.49
QADw/ reconstruction regularization85.1170.5482.19
Layerwise reconstructionw/o reconstruction regularization84.8370.3381.93
OursLayerwise reconstructionw/ reconstruction regularization85.2872.4782.72

Our full staged recipe achieves the highest VBench Total among these variants, improving over direct DMD by 1.05 points. In our visual inspection, direct DMD recovers overall image quality but can still produce severe color shifts and abrupt object appearances or disappearances. We will include this ablation and visual examples in the revised version.

Thank you for the detailed responses and additional experiments. These results address most of my concerns. I have two remaining questions:

  1. The new quality comparisons use different resolutions and frame counts. Why don't you provide a matched-setting comparison with the closest accelerated baseline, reporting both latency and quality using the decoder used for timing (TAEHV)?
  2. The training ablation demonstrates differences in final quality. Did you also observe numerical instability, such as NaNs, divergence, or failed runs, during sparse + low-bit + DMD training? What will help prevent these issues?

I also encourage the authors to incorporate the key results from the rebuttal into the revised paper, particularly the controlled ablations, short-sequence benchmarks, baseline quality comparisons, decoder evaluation, and training ablation, with their settings and limitations clearly stated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.