acceptodds
Under review as a conference paper at ICLR 2027

Improving Video Sparse Attention with Fine-grained Router and Sparse Rebasing

Abstract

We present VSA2, a frontier trainable sparse attention for video DiTs. VSA2 includes a variety of new architectural features and training procedures that we apply across all stages of DiT development cycle – including pretraining, RL, and inference – to produce a 5.8B DiT with comparable or better quality than a full attention counterpart. Architecturally, VSA2 introduces a fine-grained router that improves the precision of identifying critical tokens and supports dynamic computation by allowing each query to attend to a variable number of key–value pairs. On training, we identify a Hard-to-Easy Curriculum, where models trained under high sparsity and later evaluated with lower sparsity during inference not only generalize effectively, but also outperform models trained with full attention in motion quality. VSA2 is also flexible: it can replace full attention during the middle of progressive low-to-high resolution pretraining, rebasing early-stage full-attention checkpoints. Experiments show that VSA2 reduces attention computation by half over VSA with lower loss. On 720P videos, it accelerates attention by 8.9x and end-to-end generation by 4.62x compared to the FlashAttention3 baseline, while achieving comparable or better video quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient Architecture, Generative AI, Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QDtFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Video Sparse Attention 2 (VSA2), a trainable sparse attention framework for video DiTs, featuring three key improvements: (1) Fine-Grained Router, which uses smaller pooling regions to compute fine-grained attention scores before aggregating them into block-level scores for more accurate token selection; (2) Per-Sequence TopK, which selects important query–key block pairs globally, allowing adaptive computation allocation across queries under a fixed budget; and (3) Sparse Rebasing with Hard-to-Easy Curriculum, which replaces full attention with sparse attention during high-resolution pretraining and uses higher sparsity during training than inference to improve generation quality. Experiments demonstrate comparable or better video quality than full attention with 90–95% attention sparsity, achieving 8.9× attention speedup and 4.62× end-to-end acceleration at 720P.

Critical strengths

Clear motivation and well-justified design choices. The paper identifies concrete limitations of existing sparse attention methods, particularly the loss of important token interactions caused by coarse pooling and the inefficient allocation of fixed TopK budgets. The proposed designs directly address these limitations, and the ablation results demonstrate that more accurate routing can substantially reduce the required attention budget.

Critical weaknesses
  1. Missing comparisons with recent trainable sparse attention methods. The experimental comparisons primarily focus on VSA and selected NSA variants, while overlooking recent relevant methods such as MoGA (ICLR 2026). In particular, MoGA employs learnable token-level routing to avoid coarse blockwise attention estimation, providing an alternative solution to the routing accuracy problem addressed by VSA2. A comparison or more detailed discussion of their respective accuracy–efficiency trade-offs would help better establish the advantages and limitations of VSA2.
  2. VSA2 largely retains the coarse-to-fine architecture of the original VSA. Its two main architectural modification, finer-grained pooling for routing and global TopK selection, are relatively straightforward extensions. While effective, the paper could better clarify the conceptual novelty of these modifications beyond improving routing accuracy and computation allocation.
  3. The Hard-to-Easy Curriculum is presented as an important finding, but its underlying mechanism remains unclear. The authors attribute the improved motion quality to a regularization effect similar to structured dropout, without providing direct evidence. Moreover, Table 2 shows that motion improvements can come at the expense of prompt following and aesthetics. It remains unclear whether the gains originate from sparse training itself or from the increased attention budget during inference.
  4. The paper motivates VSA2 through the computational challenges of long-video generation and highlights its potential for minute-long videos. However, the main quantitative evaluations are conducted on 10-second videos. Although Figure 11 provides qualitative examples of 30-second generation, there is no systematic evaluation of temporal consistency, motion quality, or computational efficiency at longer durations.
Questions for authors

Please refer to the weakness part.

2 comments

We thank the reviewer for the detailed and constructive review.

W1 (MoGA). Thank you for the pointer; we will add this discussion to the related work. We cannot run additional experiments during the rebuttal period because the original training infrastructure and model checkpoints are no longer accessible to us. MoGA and VSA2 solve the routing problem in different ways:

  • MoGA uses a learned linear router to put each token into one of M groups, with full attention inside each group. The cost is about 1/M of full attention, and it uses a group-balancing loss to keep the router stable.
  • VSA2 keeps hardware-aligned block-sparse attention and makes the router more accurate with pairwise scores at a finer granularity. It adds no loss term and no router parameters; our ablations show a gradient-free router is enough (Fig. 7c, d). VSA2 has lower training loss than full attention at 90–95% sparsity (Fig. 4).

The two papers also test different things. MoGA targets multi-shot long video. We test a drop-in replacement for full attention across pretraining, RL and inference, against a full-attention model trained in the same way.

W2 (novelty). We agree that the two changes are simple to state. However, their effectiveness and efficient implementation at scale are nontrivial. We will make two aspects more visible in the paper:

  • Kernel. A direct implementation of the fine-grained router must materialize the score matrix of size N×N×G×G. Our fused kernel performs GEMM, softmax and G×G pooling in one kernel with two passes and never materializes this matrix (Fig. 5). It gives a clear speedup over the unfused version and keeps the router at 22% of attention time at 220K tokens (Sec. 3.4).
  • Evidence that global topK works. To our knowledge, previous trainable sparse attention methods for video use a per-query budget. We show that a sequence-level budget gives lower loss than per-token topK at the same compute (Fig. 7f), and we use it in all main experiments.

We believe that demonstrating the effectiveness of these methods at scale, together with an efficient kernel implementation, constitutes a meaningful contribution.

W3 (mechanism of Hard-to-Easy Curriculum). We agree that the mechanism is not proven. The paper presents structured dropout only as a hypothesis (Sec. 3.2). On the source of the gain, Table 2 gives some separation for T2V:

  • Sparse training alone does not give it. Exp 1 (train 64, infer 64) has T2V motion −1.34% against full attention.
  • A larger inference budget alone does not give it. Full attention has the largest possible budget, and it is the baseline.
  • The gain appears when the inference topK is higher than the training topK (Exp 2–4: +7.4% to +22.1% T2V motion).

Two limits of this evidence:

  • For I2V, the motion gain is present in all settings, including Exp 1 (+8.72%), so the hard-to-easy effect is specific to T2V in our data.
  • The cost in prompt following and aesthetics is real. Exp 6 shows that a short RL stage at the higher topK recovers most of it (aesthetics 0.00%, following −2.01%) and keeps a smaller motion gain (+7.38%).

W4 (longer videos). We agree that a systematic long-video evaluation would make the paper stronger, and we cannot add one now. Our evaluation is on 10-second videos because the training clips are 5–12 seconds. For longer durations, the paper provides:

  • Fig. 11, which shows 30-second generation. The corresponding video files were not uploaded with the original submission by our mistake; we have now uploaded them as supplementary material.
  • Fig. 8, which measures speed up to 436K tokens, about twice the length of a 720P 10-second video.
  • Eq. 16, which shows that sparsity increases with sequence length for fixed B, R and K.

The paper describes this only as "early evidence". We will make sure the introduction does not overstate it, and we will list the missing systematic long-video evaluation as a limitation.

I appreciate the author's rebuttal, part of my concerns have beed addressed. However, the concerns about the Hard-to-Easy mechanism I think needs more conprehensive and solid experiments to claim its contribution, as the author also says "the hard-to-easy effect is specific to T2V in our data", which is not very convincing. After reading other reviewers' comment, I lean towards a score between 2 and 3, but remain original 2.

Reviewer_Y2oFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Video Sparse Attention 2 (VSA2), a sparse attention mechanism for training and inference in video diffusion models. In some existing coarse-to-fine trainable sparse attention methods, average pooling coupled to the hardware block size makes it difficult for sparse attention selection to capture fine-grained token-level information. Furthermore, fixed-budget per-query Top-K selection cannot flexibly allocate computation according to the difficulty of each query. VSA2 combines fine-grained sparse attention with pooled coarse-grained full attention, achieving a better trade-off between computational budget and information preservation. In addition, VSA2 adopts a sequence-level Top-K selection mechanism that fixes the total selection budget for the entire sequence while allowing different queries to attend to different numbers of KV blocks. This improves the flexibility of attention resource allocation while minimizing additional overhead. VSA2 achieves lower loss than full attention while maintaining 90%–95% sparsity. At 720p resolution, VSA2 achieves an 8.9× speedup in attention computation and a 4.62× speedup in end-to-end generation.

Critical strengths

The flexible allocation of attention across different queries is an interesting design. Fixed per-query Top-K implicitly assumes that different queries should receive the same computational budget, whereas their actual attention requirements may differ substantially. By introducing a sequence-level Top-K budget, attention computation resources can be redistributed across queries, allowing the model to change how computational resources are utilized while maintaining the same total attention computation budget. This idea has potential applicability beyond VSA2 and may inspire further research into computational budget allocation for other sparse Transformers.

Critical weaknesses

Although sequence-level Top-K provides greater flexibility in allocating attention computation resources, it also introduces a new issue. This selection strategy does not necessarily achieve the ideal allocation of computational budget to the queries that genuinely require more context. Global ranking is based on routing scores. A query with a sharp attention distribution may receive more connections, while a query with a flatter distribution may also receive fewer connections; however, the latter does not necessarily require less context. Therefore, this mechanism provides dynamic budget allocation rather than a guarantee of optimal budget allocation. A scoring strategy based on the actual information requirements of each query or its output error might yield better results.

Questions for authors

Have the authors analyzed the distribution of the number of KV blocks actually allocated to different queries, as well as the relationships between these budgets and attention entropy, output error relative to full attention, and generation quality? Furthermore, if end-to-end inference latency is also held constant, can dynamic budget allocation still deliver consistent improvements in generation quality?

1 comment

We thank the reviewer for the careful review and the positive view of per-sequence topK.

Weakness (global topK is not an optimal allocation). We agree. Ranking by routing score gives a dynamic allocation, not a guarantee of the optimal one, and a query with a flat distribution can receive fewer blocks than it needs. In our preliminary study we also tried top-p-based selection (Sec. 3.3) and a dynamic topK variant, and neither gave better results. We kept global topK for two reasons:

  • It is simple and it works (Fig. 7f: lower loss than per-token topK).
  • The total budget is exactly NK blocks per sequence, so the compute cost is easy to control and predict.

Scoring by the real information need of each query or by output error is a good direction, and we will add it to the limitations and future work.

Question 1 (distribution of allocated KV blocks; relation to entropy, output error, quality). We cannot run additional experiments during the rebuttal period because the original training infrastructure and model checkpoints are no longer accessible to us. We agree that analyzing the allocation distribution and its relationship to entropy, output error, and generation quality would help explain why per-sequence topK works. We will explicitly acknowledge this limitation and identify it as future work.

Question 2 (same latency). We do not have a latency-matched comparison and cannot measure one now. What the paper provides is a budget-matched comparison: per-token and per-sequence topK in Fig. 7(f) select the same total number of block pairs (NK), so the fine-branch attention FLOPs are the same and only the allocation changes. At this equal budget, per-sequence topK has lower loss. This does not imply equal end-to-end latency, and there is no human evaluation for this ablation.

Supplementary videos. The video files mentioned in Appendix A were not uploaded with the original submission by our mistake. We have now uploaded them as supplementary material.

Reviewer_huxgOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a novel sparse attention module called VSA2, that is built off of a previous sparse attention module called VSA. Rather than using the attention scores from the coarse attention step for fine-grain routing like in VSA, the proposed VSA2 presents a separate, fine-grained routing step to select a sparse set of block pairs for the BlockSparseAttention module. Additionally, in Section 2.3 the authors propose a specialized training scheme. The authors demonstrate the value of VSA2 in Table 2 and Figure 4 in Section 3.2, and include ablation experiments in Section 3.3.

In the following sections, I will reference the following two related works:

  1. “XAttention: Block Sparse Attention with Antidiagonal Scoring” Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, Song Han Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:69819-69831, 2025.
  2. “Prism: Spectral-Aware Block-Sparse Attention” Xinghao Wang, Pengyu Wang, Xiaoran Liu, Fangxu Liu, Jason Zhu, Kai Song, Xipeng Qiu Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:130606-130622, 2026.
Critical strengths
  1. The paper addresses the important problem related to the computational burden of attention modules.
  2. The proposed VSA2 module is significantly faster than FlashAttention3 (see Figure 8).
Critical weaknesses
  1. I have concerns about the novelty of the method.

    a. In Section 2.2, the authors present their VSA2 method for sparse video attention which leverages pooling for routing a sparse attention module. However, a related works called PRISM [Ref.2] also uses pooling to efficiently compute the routing mask.

    b. In Section 2.3, the authors propose an improved training method for video DiTs. However, the description seems to simply suggest fine-tuning a network instead of training one from scratch. As this is common practice, I do not believe this contribution is novel.

    c. The authors describe the Hard-to-Easy Curriculum as “reducing the sparsity level during inference” (line 291). However, with this description alone, it is difficult to assess the novelty of this particular curriculum learning method.

  2. I have concerns about the significance of the method.

    a. The authors report their main results in Table 2, which is a human evaluation of results. While these results are interesting, it would help to also include standard evaluation benchmarks. In addition, some qualitative examples with the corresponding human evaluation should be included.

    b. The authors compare the wall-clock runtime of VSA2 against FlashAttention3 and report a 8.9x acceleration at a sequence length of 220k. However, I do not see an ablation comparing the quality of each attention type within a video DiT.

    c. The authors contrast their proposed method called VSA2 with a previous method named VSA from “Faster video diffusion with trainable sparse attention” by Zhang et al. However, given that the VSA method is unpublished (except on arXiv) and currently has only 4 citations (10/21/2026), experimentally outperforming VSA is not a sufficiently significant finding. Additionally, the authors do not directly compare against methods like XAttention [Ref.1] or PRISM [Ref.2], yet these methods seem deeply connected to the proposed work.

Questions for authors
  1. How is the proposed sparse attention method different from the method presented in PRISM [Ref.2]?
2 comments

We thank the reviewer for the time and comments. Several points in the review differ from what the paper says, so we clarify them below.

1.a Pooling-based routing and PRISM. Pooling-based routing is not a contribution of this paper. It comes from VSA, as described in Sec. 2.1. Our contributions in Sec. 2.2 are:

  • A fine-grained router whose pooling size R is decoupled from the hardware block size B.
  • Per-sequence topK.
  • A fused kernel that never materializes the fine-grained score matrix.

On PRISM: both papers observe that coarse mean pooling makes block selection less accurate. The solutions and settings are different.

  • PRISM is training-free and inference-only. It calibrates block selection on a pretrained full-attention model.
  • VSA2 is trainable and is used during pretraining and RL, so the model learns under sparsity. It improves selection by computing router scores at a finer granularity and pooling them after the softmax.

XAttention is also inference-only, and we cite it in that category (Sec. 1). We will add PRISM to the related work.

1.b Training recipe. The 480P and 720P stages are pretraining, not fine-tuning: they use the pretraining dataset and the pretraining objective. Progressive low-to-high resolution pretraining is the standard recipe for video DiTs (Sec. 4), and the high-resolution stages take most of the compute. Our finding is that VSA2 can replace full attention in the middle of this pipeline with a seamless transition, and that the final model is comparable to the full-attention one overall and better in motion (Table 2, Fig. 4). We will reword Sec. 2.3 to make this clearer.

1.c Hard-to-Easy Curriculum. We do not claim this as a novel method. We report it as an empirical finding: a model trained with high sparsity and run with lower sparsity at inference has better T2V motion quality than the full-attention baseline (Table 2, Exp 2–4). We think it is useful for the community.

2.a Evaluation and qualitative examples. Our primary metric is flow-matching loss, which is directly aligned with the training objective. We additionally provide human evaluation on 149 prompts (Table 2) and RL reward curves (Fig. 4). We did not use standard automatic video benchmarks for two reasons:

  • Pixel- or feature-distance metrics are not well suited to our controlled pretraining comparison.
  • Existing automatic video benchmarks may have limited sensitivity to the differences between the strong models considered here.

Qualitative examples are in Fig. 3, Fig. 9, Fig. 10 and Fig. 11. Fig. 3 shows samples from the human evaluation of Exp 6. The paper also refers to video files in the supplementary material. By our mistake, these files were not uploaded with the original submission. We have now uploaded them as supplementary material.

2.b Quality comparison against FlashAttention3. Thank you for the question. FlashAttention3 is an exact implementation of full attention, so it is the same baseline as "full attention" in the rest of the paper. The quality comparison of VSA2 against full attention inside a video DiT is our main experiment: Table 2 (7 settings, human evaluation) and Fig. 4 (loss and reward). We will state this equivalence explicitly in Sec. 3.4.

2.c Significance of VSA and baselines. VSA is published in the NeurIPS 2025 main track and has 100 citations on Google Scholar as of Nov 15 2026. Our reference list gives only the arXiv entry, and we will correct it. Our main baseline is full attention trained with the same data, hyperparameters and iterations. VSA and NSA-style designs are compared in the ablations (Sec. 3.3, Fig. 7).

Question (difference from PRISM). Please see 1.a.

We hope these clarifications are helpful, and we ask the reviewer to reconsider the evaluation in light of them. We are happy to answer further questions.

I appreciate the authors feedback. However, the authors start their rebuttal by stating that "several points in the review differ from what the paper says", and I believe this mischaracterizes my review. In fact, nothing my original review contradicts the paper's content, and the authors have not provided evidence to support that my points differ from what the paper says.

Some of my concerns have been resolved.

1a and Q1. This paper's connection to PRISM is resolved.

2b. Clarifying that FlashAttention3 is the implementation of Full Attention resolves my concern about the missing ablation studying.

2c (VSA Baseline). This paper cites "Vsa: Faster video diffusion with trainable sparse attention" which is only published on arVix and still has only 4 citations on Google Scholar. However, the same contribution under the title "Faster video diffusion with trainable sparse attention" is indeed published in the NeurIPS 2025 main track and has 101 citations on Google Scholar. Thus, VSA does serve as a significant baseline, which is a change from my original understanding.

Several of my original concerns remain.

1b. My understanding of fine-tuning is that it involves applying additional training to the parameters of a neural network that have been pre-trained. In this case, a pre-trained DiT network is being fine-tuned in the pre-training stage with the proposed VSA2 module. The authors state this is a core novelty of their paper (line 097). However, given that this approach still appears to simply be fine-tuning a network, I do not believe this contribution is novel.

1c. Line 097 lists "Sparse Rebasing" as a core contribution. Line 086 states: "A crucial component of Sparse Rebasing is the Hard-to-Easy Curriculum". This implies the Hard-to-Easy Curriculum is a core contribution of the paper. In the rebuttal, the authors clarify that they do not believe this core contribution of theirs is novel. However, even more important than this, I believe the "Hard-to-Easy Curriculum" is missing a description altogether.

2a. Reducing only the training loss alone is insufficient evidence that the proposed approach has practical value. Additionally, a human survey is an very interesting approach, but I do not find this convincing without a reference to standardized benchmarks. Furthermore, the results of the human survey are not reproducible, so methods designed after VSA2 do not have baseline results to compare with. Furthermore, I believe the image-to-video generation task (as shown in Figure 3) does have standardized benchmarks. However, if the authors find these benchmarks not suitable, perhaps they can demonstrate the value of the proposed VSA2 module on different application with standardized benchmarks.

2c (Lacking Baselines). Ablation studies in Figure 7 successfully show VSA2 achieves a small training loss compared to VSA. However, the main results in Table 2 and Figure 4 only compare against Full Attention. The authors do not compare against other sparse attention strategies such as XAttention [Ref.1] or PRISM [Ref.2].

Comments on the paper

1 comment

We thank all reviewers for their valuable feedback and constructive comments. We have uploaded a revised manuscript addressing the points raised during the discussion, including clarifications of our contributions, expanded related work, and additional discussion of limitations. All newly added content is highlighted in red for ease of reference. We have also uploaded the supplementary MP4 videos referenced in the paper. We appreciate the reviewers' time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.