- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Video Sparse Attention 2 (VSA2), a trainable sparse attention framework for video DiTs, featuring three key improvements: (1) Fine-Grained Router, which uses smaller pooling regions to compute fine-grained attention scores before aggregating them into block-level scores for more accurate token selection; (2) Per-Sequence TopK, which selects important query–key block pairs globally, allowing adaptive computation allocation across queries under a fixed budget; and (3) Sparse Rebasing with Hard-to-Easy Curriculum, which replaces full attention with sparse attention during high-resolution pretraining and uses higher sparsity during training than inference to improve generation quality. Experiments demonstrate comparable or better video quality than full attention with 90–95% attention sparsity, achieving 8.9× attention speedup and 4.62× end-to-end acceleration at 720P.
Critical strengths
Clear motivation and well-justified design choices. The paper identifies concrete limitations of existing sparse attention methods, particularly the loss of important token interactions caused by coarse pooling and the inefficient allocation of fixed TopK budgets. The proposed designs directly address these limitations, and the ablation results demonstrate that more accurate routing can substantially reduce the required attention budget.
Critical weaknesses
- Missing comparisons with recent trainable sparse attention methods. The experimental comparisons primarily focus on VSA and selected NSA variants, while overlooking recent relevant methods such as MoGA (ICLR 2026). In particular, MoGA employs learnable token-level routing to avoid coarse blockwise attention estimation, providing an alternative solution to the routing accuracy problem addressed by VSA2. A comparison or more detailed discussion of their respective accuracy–efficiency trade-offs would help better establish the advantages and limitations of VSA2.
- VSA2 largely retains the coarse-to-fine architecture of the original VSA. Its two main architectural modification, finer-grained pooling for routing and global TopK selection, are relatively straightforward extensions. While effective, the paper could better clarify the conceptual novelty of these modifications beyond improving routing accuracy and computation allocation.
- The Hard-to-Easy Curriculum is presented as an important finding, but its underlying mechanism remains unclear. The authors attribute the improved motion quality to a regularization effect similar to structured dropout, without providing direct evidence. Moreover, Table 2 shows that motion improvements can come at the expense of prompt following and aesthetics. It remains unclear whether the gains originate from sparse training itself or from the increased attention budget during inference.
- The paper motivates VSA2 through the computational challenges of long-video generation and highlights its potential for minute-long videos. However, the main quantitative evaluations are conducted on 10-second videos. Although Figure 11 provides qualitative examples of 30-second generation, there is no systematic evaluation of temporal consistency, motion quality, or computational efficiency at longer durations.
Questions for authors
Please refer to the weakness part.
2 comments
I appreciate the author's rebuttal, part of my concerns have beed addressed. However, the concerns about the Hard-to-Easy mechanism I think needs more conprehensive and solid experiments to claim its contribution, as the author also says "the hard-to-easy effect is specific to T2V in our data", which is not very convincing. After reading other reviewers' comment, I lean towards a score between 2 and 3, but remain original 2.
We thank the reviewer for the detailed and constructive review.
W1 (MoGA). Thank you for the pointer; we will add this discussion to the related work. We cannot run additional experiments during the rebuttal period because the original training infrastructure and model checkpoints are no longer accessible to us. MoGA and VSA2 solve the routing problem in different ways:
The two papers also test different things. MoGA targets multi-shot long video. We test a drop-in replacement for full attention across pretraining, RL and inference, against a full-attention model trained in the same way.
W2 (novelty). We agree that the two changes are simple to state. However, their effectiveness and efficient implementation at scale are nontrivial. We will make two aspects more visible in the paper:
We believe that demonstrating the effectiveness of these methods at scale, together with an efficient kernel implementation, constitutes a meaningful contribution.
W3 (mechanism of Hard-to-Easy Curriculum). We agree that the mechanism is not proven. The paper presents structured dropout only as a hypothesis (Sec. 3.2). On the source of the gain, Table 2 gives some separation for T2V:
Two limits of this evidence:
W4 (longer videos). We agree that a systematic long-video evaluation would make the paper stronger, and we cannot add one now. Our evaluation is on 10-second videos because the training clips are 5–12 seconds. For longer durations, the paper provides:
The paper describes this only as "early evidence". We will make sure the introduction does not overstate it, and we will list the missing systematic long-video evaluation as a limitation.