- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ESTRA, a training-free sparse attention method for video diffusion transformers. It independently clusters queries and keys, uses query centroids to select individual keys, and introduces Variance-Calibrated Mean compensation for unselected tokens. Experiments on Wan2.1-14B and HunyuanVideo-13B show that ESTRA at 5% attention density improves fidelity to dense generation over the evaluated baselines while reducing latency. At 1% density without dense warmup, it achieves end-to-end speedups of 3.55x and 5.39x, respectively, while retaining higher fidelity than Sol-Attn at its main operating points.
Critical strengths
-
The results at extreme sparsity are promising. ESTRA produces coherent videos at 99% attention sparsity without dense warmup. Maintaining useful generation quality with such a small exact-attention budget is a practical contribution, although the quality loss at 1% density is more noticeable on Wan.
-
Token-level selection makes better use of the attention budget. Using query centroids to score individual keys allows the method to select important tokens across clusters without computing separate selection scores for every query. This addresses the wasted computation that arises when selecting a high-scoring token requires retaining its entire block.
-
The experiments cover several aspects of accuracy and efficiency. The evaluation includes end-to-end latency and fidelity in Table 1, offline attention error, VBench scores, qualitative comparisons, and component ablations in Tables 3 and 5. Testing on both Wan2.1-14B and HunyuanVideo-13B also helps assess whether the improvements extend beyond a single model.
Critical weaknesses
-
I find the treatment of the mean shift and variance in VCM mathematically inconsistent. Section 4 acknowledges that selection shifts the mean of the remaining keys away from the fixed full-cluster center, but omits this first-order mean shift because updating the means would require costly vector operations. The method nevertheless updates the second-order statistics by subtracting selected-token contributions from the full-cluster totals. Since the selected tokens are typically high-score outliers, the resulting second-moment statistic is still measured around a center that no longer matches the remaining keys. I do not see why this selective refinement is justified. If the mean is left unchanged, why not also use the full-cluster variance? In my view, this inconsistency explains the increased error at 10% and 15% density in Table 5. The method refines the variance estimate while leaving the reference mean inaccurate.
-
The cost of irregular token access needs a more detailed analysis. Token-level selection avoids the unnecessary computation associated with selecting entire blocks. However, block-sparse methods such as SVG-EAR and Sol-Attn organize computation around contiguous blocks, whereas selecting individual tokens introduces index preparation and potentially non-contiguous K/V access. Figure 4 provides a runtime breakdown, but it does not separately identify these costs. Query centroids reduce the scoring cost, but this does not by itself establish that gathering selected tokens is inexpensive. At 1% density, these operations could account for a substantial fraction of attention runtime. A separate measurement of index preparation and K/V gathering would help clarify the practical cost of finer selection.
-
The technical novelty appears limited. The clustering, token reordering, and centroid-based routing largely follow SVG2 and SVG-EAR, as the paper explicitly acknowledges. The main additions are token-level selection and VCM, both of which seem incremental within this framework. VCM is essentially a second-order Taylor approximation applied to the compensation term. The ablations do not sufficiently establish its advantage over simpler alternatives, such as using full-cluster variance or standard block-level compensation. More direct comparisons are needed to justify the additional computation and design choices. My concern about the treatment of the first- and second-order terms also limits how much theoretical contribution I see in VCM.
Questions for authors
-
How do the authors justify the mathematical consistency of VCM? Selected tokens are typically high-score outliers, so removing them shifts the mean of the remaining keys. Why is it acceptable to omit this first-order shift while refining the second-order statistics around the original cluster center? If the mean remains an approximation, why not use the full-cluster variance and avoid the additional computation?
-
What is the cost of index preparation and K/V gathering at 1% density? Figure 4 reports the main runtime components. Could the authors further separate threshold estimation, index preparation, gathering, and attention computation? How do the costs of irregular token access scale with sequence length compared with block-level selection?
-
What are the main algorithmic differences from SVG2 and SVG-EAR beyond the selection granularity and VCM correction? Since VCM applies a second-order correction to mean-based compensation, could a simpler correction based on full-cluster variance achieve similar results without subtracting the selected-token statistics?
-
How were the cluster counts chosen? The paper uses 300 query clusters and 1,000 key clusters. How sensitive are quality and latency to these choices, and do the counts need to change for different resolutions or video lengths?
Dear Reviewer 2ex1,
We wanted to briefly check in and see if our response and additional experiments above have addressed your questions. With the discussion deadline approaching, please let us know if anything remains unclear or if you need any further information from us.
Thank you, The Authors