- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work identifies the high-sparsity trap failure regime, a problem that occur when velocity error accumulates at >= 90% sparsity, and they proposed a sparse warm up then trajectory-mixed distillation method to address this sparse attention conversion problem. They've shown the empirical evidence behind the velocity error accumulation, and shown SparkDiffusion as a better method than the baseline mentioned here.
Critical strengths
The exploration and empirical supports behind the high-sparsity trap failure are interesting, and the sparsity method motivation is sound, and the paper writing is really good.
Critical weaknesses
One potential concern is about model arch generalizations, does it generalize beyond wan attention arch?
On speedup baseline, PyTorch SDPA may be a better baseline (the later pytorch version, which select between the proper flash attention, cudnn, etc. based on env) for 5090.
Percentage sparsity itself is not as kernel friendly as fix budget sparse attention, even if it generalize better as context grow. Because fixed budget sparse allow fix shape for better kernel pre-compiling, and a potential improvement is find that, after the 90% or 97% sparsity conversion, under the model max context length, a fixed N tokens sparsity budget.
Questions for authors
- why only on WAN family, is this method generalize to other arch like HunyuanVideo and CogVideoX?
- For the high-sparsity trap failure discussion, can this just be a capacity or routing problem? Increasing the compensation rank doesn’t really rule it out, and fixing high-noise steps let us know the place where corrections help, but not why the model fails. Also, distillation also changes several things at once, so it is hard to fully justify the reasoning chain.
- This may not be as relevant, but sparsity becomes extremely beneficial when context grow up, but a common failure mode of sparsity method is it fail when context grow up and sparsity error accumulate to alternate the trajectory even with training. In my understanding, none of VBench, VBench-2.0, V-JEPA 2, VideoMAE V2 are long context video generation. Do sparkdiffusion have similar issue when applied on longer video generation model? I understand this may be beyond the this work, but I wanted to bring it up as an interesting point for further discussion.
We thank the reviewer for these careful points. They ask three things: does the method work beyond Wan, how we measured speed, and when the sparse model fails. We answer them below.
1. Does it generalize beyond Wan? (W1, Q1)
The Wan experiments are enough to answer this, and the study is large. We trained on Wan2.1-T2V-1.3B, Wan2.1-T2V-14B-720P, Wan2.1-I2V-14B-720P and Wan2.2-A14B-MoE. The study spans multiple model scales (1.3B/14B/A14B), two resolutions (480P/720P), both tasks (T2V/I2V), and dense and MoE architectures, at 90% and 97% sparsity.
Wan is where the field compares. FastWan, TurboDiffusion and SLA all run on Wan, so our numbers sit next to theirs on the same model.
What the method needs from a model. Sparsifiable attention with a small dense correction branch, and a diffusion model with a slow teacher. HunyuanVideo and CogVideoX both satisfy this. Sec. 5 now states these conditions.
Percent sparsity vs. fixed block count (W3). In Wan2.1 the number of tokens is fixed, so "97%" always means the same number of retained blocks. The kernel sees a fixed shape and is compiled once, as with a fixed budget.
The percentage keeps one recipe across 480P/720P and 1.3B/14B/MoE. A fixed block count matters when the sequence length changes, as in autoregressive decoding. We added this to the appendix.
2. Is FA2 a fair speedup baseline? (W2)
FA2 is the right reference for this comparison. FastWan and TurboDiffusion report speedups against FlashAttention-2, so FA2 keeps our table comparable to theirs.
We also ran the SDPA check you suggested. RTX 5090, NGC 25.10, bf16, Diffusers group offload, 14B-720P, 50 steps:
auto)The two differ by 0.2%, inside the 0.6% run-to-run noise. SDPA
autosimply picked the same flash kernel on this machine. All speedup ratios in the paper stay the same.3. Why does the sparse model fail? (Q2, Q3)
(a) The rank experiment was too strong a claim. We trained the same 14B-480P model with the dense branch at rank 64 and at rank 128. Terminal error went 0.124 → 0.121, and VBench 80.92 → 81.05. This rules out one capacity change. The earlier sentence said more than that, and Sec. 3.1 is now corrected.
(b) Fig. 4 localises the failure to the high-noise steps. We keep every weight and every routing decision, and only replace the model's velocity at the five noisiest steps with the teacher's. Most of the error disappears. This rules out routing or capacity problems outside that window: the later steps are untouched, yet the video recovers. Inside the window the substitution masks routing-, capacity-, and supervision-induced velocity errors together, so Fig. 4 localises the failure but cannot separate the three. We say so explicitly in the revision.
(c) Fig. 7 and two new controls explain the cause. Under the standard loss the model trains for 10,000 steps: validation loss keeps falling, video quality does not. Two extra signals, each a single change on the same VSA branch, recover part of the gap (14B-480P, 97%):
Each change is applied singly to the same VSA 97% checkpoint: high-noise reweighting reaches 80.61 (+0.89 over the 79.72 step-local baseline), teacher velocity on the student's own states reaches 81.34 (+1.62), and trajectory-mixed distillation reaches 82.45 (+2.73, Table 6). None closes the gap alone; the full RoLA recipe reaches 82.99, the extra 0.54 from the RoLA branch. The capacity question is handled by (a) and the training-signal question by Fig. 7.
(d) Longer videos (Q3). You are right that VBench, VBench-2.0, V-JEPA 2 and VideoMAE V2 are not long-context benchmarks, and our results are at 81 frames. We expect the same failure to grow with length: one bad high-noise step corrupts a block early, sparse attention copies it into later tokens, and three steps are too few to undo it. In bidirectional attention the damage stays inside one video; in causal autoregressive attention it passes into later chunks, which is why Sec. 1 argues the final alignment stage matters even more for autoregressive models. We state this as a limitation and a direction for future work.
4. Revision
We updated Sec. 3.1, Sec. 5, the appendix, and the limitations section. The new control rows will be added to Table 6 (Appendix D) and Sec. 5.3.