acceptodds
Under review as a conference paper at ICLR 2027

Native Token-Level Sparse Attention for Long-Context Video Pretraining

Abstract

Modern video generation models built on bidirectional diffusion transformers (DiT) run quadratic attention over space–time tokens. This O(N²) cost bottlenecks the scaling of video DiT to longer and higher-resolution generation. Sparse attention mitigates the cost by limiting each query to a subset of key–value pairs. Existing sparse attention mechanisms for video DiT mainly apply block-wise selection, which keeps the implementation hardware-friendly and respects the spatio-temporal locality of nearby video tokens. However, block-wise selection is coarse: whenever the relevant context is finer than a block, the attention budget is spent on irrelevant tokens while relevant ones are missed. The resulting gap to full attention is acceptable for accelerating inference, but not for a native sparse attention. It is the attention of the model throughout training, so it pays this gap at every update and has to match full attention. In this paper, we propose VTSA (Video Token-level Sparse Attention), a native token-level sparse attention that is lossless in long-context video pretraining. VTSA selects individual key–value tokens with a small indexer distilled from the model's own attention scores. One selection is shared across the query heads that read one key–value head and across a small tile of neighbouring queries. A custom kernel gathers the selected tokens in the forward pass and scatters their gradients in the backward pass. We compare VTSA with two trained sparse-attention baselines, VSA-style block routing and sliding-window attention, in continued pretraining from dense checkpoints. We use two video DiT backbones with different attention architectures, a 5B multi-head video model (Wan2.2-5B) and a 15B grouped-query audio–video model (daVinci-MagiHuman). VTSA matches full attention while each sparse layer attends to 6–15% of the key–value pairs, at 16k and 32k tokens on the 5B backbone and from 16k to 128k tokens on the 15B backbone. Block routing matches full attention only at two to four times as many tokens. VTSA opens the door to long-context video pretraining with native token-level sparse attention.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, efficient attention, long-context pretraining, diffusion transformer, video generation, attention kernel, native sparse attention, scaling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wnwWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VTSA adapts learned token-level sparse attention to video DiTs with MHA or GQA. A selection is shared across neighbouring queries and heads reading one KV head to fill hardware tiles. A distilled indexer ranks tokens, and custom kernels gather selected KV tokens and scatter gradients. Controlled continued-pretraining experiments on 5B and 15B backbones span 16K–128K tokens. At 6–15% attended fractions, VTSA reports validation losses close to full attention and smaller required budgets than the tested block-routing baseline, with 1.1–2.0× whole-denoiser forward speedups.

Critical strengths
  1. A practical design supported by experiments at substantial scale. The indexer, shared selection, and forward/backward kernels form a coherent approach to trainable token-level sparsity. Evaluation spans MHA and GQA backbones, including a 15B model at 128K context, and reports forward speedups at kernel, attention-layer, and whole-model levels.

  2. Convincing evidence for the benefit of finer selection granularity. Token and block selection share the scorer, token budget, and continued-pretraining conditions, while the exact-score experiment further separates granularity from indexer error. These controls support the finding that token selection can approach dense validation loss with substantially fewer attended keys.

Critical weaknesses
  1. The evidence for “lossless” has a limited scope. The common loss margin is calibrated at 5B/16K, without corresponding uncertainty estimates across the other settings. Generation evaluation covers only three human-focused dimensions at 32K. Their average can conceal individual drops, such as the 15B Identity score decreasing from 85.53 to 79.68 (Table 2). The results support near-dense quality in the evaluated settings, but broader quality preservation remains less established.

  2. The practical evaluation is incomplete. The VSA-style baseline uses VTSA’s scorer and kernel with a fixed 128-token block size, making it a useful granularity control but a limited comparison with existing trainable sparse-attention systems. The reported speedups concern inference forward passes. Backward latency, full training-step throughput, and peak memory are missing. These omissions leave the practical advantage for pretraining less clear than the quality results.

Questions for authors
  1. How sensitive are the reported “lossless” budgets to uncertainty across model sizes and context lengths? Is the 15B Identity drop in Table 2 due to sampling variation or a consistent quality loss?
  2. Does the token-selection advantage persist when the block baseline uses smaller blocks? How does VTSA compare with existing trainable sparse-attention methods at similar quality?
  3. Compared with full attention, how much faster is a complete training step, including backward and indexer training, and what is the peak memory usage?
  4. Could the authors briefly relate shared token-level KV selection across queries to prior work such as VecAttention and FG-Attn, clarifying the contribution in native pretraining?
Reviewer_eMYCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VTSA (Video Token-level Sparse Attention), a fine-grained sparse attention method for long-context video diffusion transformers. It combines a learned token indexer, shared selection across spatiotemporal query tiles and compatible heads, and an efficient GPU kernel to reduce attention computation. Experiments on 5B MHA and 15B GQA models show that VTSA preserves average validation loss within 4*10^{-4} of full attention while attending to only 6%–15% of keys, using 2–4× fewer keys than block routing and achieving 3.2–8.7× attention-kernel speedups. These results demonstrate improved quality–efficiency tradeoffs, although equivalent validation loss does not guarantee identical video generation quality.

Critical strengths
  1. VTSA demonstrates clear advantages of token-level selection over block routing. Across two model backbones and varying context lengths, it achieves near-equivalent validation loss while selecting 2–4× fewer keys. Comparisons using exact attention scores further confirm that fine-grained selection itself contributes to this improvement, beyond indexer accuracy.

  2. VTSA converts sparse attention into practical inference acceleration. By combining shared query selection, a lightweight learned indexer, and a specialized GPU kernel, it efficiently handles irregular token access. The resulting 1.1–2.0× full-model forward speedups demonstrate tangible benefits beyond theoretical computational savings across both MHA and GQA architectures.

Critical weaknesses
  1. The query-sharing strategy lacks direct validation. VTSA shares selected keys across neighboring spatiotemporal queries, but the experiments do not systematically evaluate different tile sizes and shapes. Since existing ablations focus on query summarization and cross-head sharing, controlled comparisons are needed to assess the quality–efficiency trade-off, especially near motion boundaries and scene transitions.

  2. The “lossless” claim is not fully supported by generation-quality results. Near-equivalent average denoising loss does not guarantee consistent generation quality across individual capabilities. For the 15B model, Identity drops from 85.53 to 79.68, while improved Anatomy offsets part of this decline in the aggregate Fidelity score. Without uncertainty estimates and targeted temporal-consistency evaluations, the impact of sparsity on perceptual quality remains unclear.

Questions for authors
  1. Query-sharing trade-off: How sensitive are validation loss and runtime to query-tile size and spatiotemporal shape when evaluated on the same backbone under a fixed key budget? Could comparisons with smaller tiles or per-query selection help isolate the approximation cost of query sharing? Tables 6–7 do not directly address this question. Further evaluation under fast motion and scene transitions would help determine whether fixed local grouping remains reliable when neighboring queries have different attention requirements.

  2. Uncertainty in generation quality: How should the Identity score decrease from 85.53 with full attention to 79.68 with VTSA be interpreted for the 15B model? Could uncertainty estimates or paired evaluations across prompts and sampling seeds establish whether this drop exceeds normal evaluation variability? This would clarify whether the validation-loss equivalence underlying the “lossless” claim also preserves individual generation capabilities, rather than only aggregate quality as reported in Table 2.

Reviewer_GfNKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper draws on ideas from LLM and uses an indexer as a lightweight token-selection mechanism for attention in video generation. The authors emphasize token-level selection on the KV side, in contrast to most existing methods that operate at the block level for QKV, allowing the method to operate at higher sparsity. Under the computational budgets evaluated in the experiments, its loss is close to that of full attention, which the authors describe as “lossless.”

Critical strengths
  • The idea of using an indexer for sparse attention in video generation is straightforward and effective.

  • The experiments cover multiple sequence lengths and pre-trained models.

Critical weaknesses
  • Token-level selection is highlighted as a contribution. However, the method itself is rather straightforward: Q remains block-based, combined with head sharing.

  • The repeated use of “lossless” is confusing, as it could suggest that video generation incurs no loss in quality. The authors define “lossless” as sparse and full-attention models achieving similar validation losses when trained on the same data. However, a sparse model obtained through training need not be a “degraded version” of the full-attention model. I would also recommend visualizing the two models’ validation losses at different denoising timesteps as training progresses, rather than reporting only a final averaged value.

  • The use of “native” in the title could also be clarified. Although the paper specifies that training continues from dense checkpoints, the title may suggest sparse pre-training from scratch. Explicitly referring to long-context continued pre-training would make the scope clearer.

Questions for authors

see weekness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.