- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VTSA adapts learned token-level sparse attention to video DiTs with MHA or GQA. A selection is shared across neighbouring queries and heads reading one KV head to fill hardware tiles. A distilled indexer ranks tokens, and custom kernels gather selected KV tokens and scatter gradients. Controlled continued-pretraining experiments on 5B and 15B backbones span 16K–128K tokens. At 6–15% attended fractions, VTSA reports validation losses close to full attention and smaller required budgets than the tested block-routing baseline, with 1.1–2.0× whole-denoiser forward speedups.
Critical strengths
-
A practical design supported by experiments at substantial scale. The indexer, shared selection, and forward/backward kernels form a coherent approach to trainable token-level sparsity. Evaluation spans MHA and GQA backbones, including a 15B model at 128K context, and reports forward speedups at kernel, attention-layer, and whole-model levels.
-
Convincing evidence for the benefit of finer selection granularity. Token and block selection share the scorer, token budget, and continued-pretraining conditions, while the exact-score experiment further separates granularity from indexer error. These controls support the finding that token selection can approach dense validation loss with substantially fewer attended keys.
Critical weaknesses
-
The evidence for “lossless” has a limited scope. The common loss margin is calibrated at 5B/16K, without corresponding uncertainty estimates across the other settings. Generation evaluation covers only three human-focused dimensions at 32K. Their average can conceal individual drops, such as the 15B Identity score decreasing from 85.53 to 79.68 (Table 2). The results support near-dense quality in the evaluated settings, but broader quality preservation remains less established.
-
The practical evaluation is incomplete. The VSA-style baseline uses VTSA’s scorer and kernel with a fixed 128-token block size, making it a useful granularity control but a limited comparison with existing trainable sparse-attention systems. The reported speedups concern inference forward passes. Backward latency, full training-step throughput, and peak memory are missing. These omissions leave the practical advantage for pretraining less clear than the quality results.
Questions for authors
- How sensitive are the reported “lossless” budgets to uncertainty across model sizes and context lengths? Is the 15B Identity drop in Table 2 due to sampling variation or a consistent quality loss?
- Does the token-selection advantage persist when the block baseline uses smaller blocks? How does VTSA compare with existing trainable sparse-attention methods at similar quality?
- Compared with full attention, how much faster is a complete training step, including backward and indexer training, and what is the peak memory usage?
- Could the authors briefly relate shared token-level KV selection across queries to prior work such as VecAttention and FG-Attn, clarifying the contribution in native pretraining?