acceptodds
Withdrawn from ICLR 2027

ESTRA: Enabling Extreme Attention Sparsity for Training-Free Video Diffusion Acceleration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video diffusion transformers generate high-quality videos, but attention over long spatiotemporal sequences makes inference computationally expensive. Sparse attention reduces this cost by computing only a subset of query-key interactions exactly, and recent training-free methods further approximate the remaining interactions. These methods, however, degrade quickly at high sparsity, which limits the achievable speedup. We present ESTRA (Extreme Sparsity Through Refined Attention), a training-free acceleration method that preserves generation quality even at extreme sparsity. ESTRA concentrates exact attention computation on a small set of important key-value tokens and approximates the remaining contributions with cluster representatives. To identify these tokens efficiently, it groups similar queries and uses their representatives to score individual keys. A variance-based correction further refines the attention weights of the cluster representatives, improving the accuracy of the approximation. Experiments on Wan2.1-14B and HunyuanVideo-13B show that ESTRA establishes a new quality-efficiency frontier for training-free sparse attention, achieving 3.26× and 4.71× end-to-end speedups at 5% attention density while surpassing Sol-Attn and SVG-EAR in fidelity to dense generation and on most VBench dimensions. Even at 1% density, ESTRA remains more faithful to dense generation than Sol-Attn at 12.5%, while reaching up to 5.39× speedup.

Primary area:
generative models
Keywords:
Video generation, Diffusion transformers, Sparse attention, Training-free acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2ex1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ESTRA, a training-free sparse attention method for video diffusion transformers. It independently clusters queries and keys, uses query centroids to select individual keys, and introduces Variance-Calibrated Mean compensation for unselected tokens. Experiments on Wan2.1-14B and HunyuanVideo-13B show that ESTRA at 5% attention density improves fidelity to dense generation over the evaluated baselines while reducing latency. At 1% density without dense warmup, it achieves end-to-end speedups of 3.55x and 5.39x, respectively, while retaining higher fidelity than Sol-Attn at its main operating points.

Critical strengths
  1. The results at extreme sparsity are promising. ESTRA produces coherent videos at 99% attention sparsity without dense warmup. Maintaining useful generation quality with such a small exact-attention budget is a practical contribution, although the quality loss at 1% density is more noticeable on Wan.

  2. Token-level selection makes better use of the attention budget. Using query centroids to score individual keys allows the method to select important tokens across clusters without computing separate selection scores for every query. This addresses the wasted computation that arises when selecting a high-scoring token requires retaining its entire block.

  3. The experiments cover several aspects of accuracy and efficiency. The evaluation includes end-to-end latency and fidelity in Table 1, offline attention error, VBench scores, qualitative comparisons, and component ablations in Tables 3 and 5. Testing on both Wan2.1-14B and HunyuanVideo-13B also helps assess whether the improvements extend beyond a single model.

Critical weaknesses
  1. I find the treatment of the mean shift and variance in VCM mathematically inconsistent. Section 4 acknowledges that selection shifts the mean of the remaining keys away from the fixed full-cluster center, but omits this first-order mean shift because updating the means would require costly vector operations. The method nevertheless updates the second-order statistics by subtracting selected-token contributions from the full-cluster totals. Since the selected tokens are typically high-score outliers, the resulting second-moment statistic is still measured around a center that no longer matches the remaining keys. I do not see why this selective refinement is justified. If the mean is left unchanged, why not also use the full-cluster variance? In my view, this inconsistency explains the increased error at 10% and 15% density in Table 5. The method refines the variance estimate while leaving the reference mean inaccurate.

  2. The cost of irregular token access needs a more detailed analysis. Token-level selection avoids the unnecessary computation associated with selecting entire blocks. However, block-sparse methods such as SVG-EAR and Sol-Attn organize computation around contiguous blocks, whereas selecting individual tokens introduces index preparation and potentially non-contiguous K/V access. Figure 4 provides a runtime breakdown, but it does not separately identify these costs. Query centroids reduce the scoring cost, but this does not by itself establish that gathering selected tokens is inexpensive. At 1% density, these operations could account for a substantial fraction of attention runtime. A separate measurement of index preparation and K/V gathering would help clarify the practical cost of finer selection.

  3. The technical novelty appears limited. The clustering, token reordering, and centroid-based routing largely follow SVG2 and SVG-EAR, as the paper explicitly acknowledges. The main additions are token-level selection and VCM, both of which seem incremental within this framework. VCM is essentially a second-order Taylor approximation applied to the compensation term. The ablations do not sufficiently establish its advantage over simpler alternatives, such as using full-cluster variance or standard block-level compensation. More direct comparisons are needed to justify the additional computation and design choices. My concern about the treatment of the first- and second-order terms also limits how much theoretical contribution I see in VCM.

Questions for authors
  1. How do the authors justify the mathematical consistency of VCM? Selected tokens are typically high-score outliers, so removing them shifts the mean of the remaining keys. Why is it acceptable to omit this first-order shift while refining the second-order statistics around the original cluster center? If the mean remains an approximation, why not use the full-cluster variance and avoid the additional computation?

  2. What is the cost of index preparation and K/V gathering at 1% density? Figure 4 reports the main runtime components. Could the authors further separate threshold estimation, index preparation, gathering, and attention computation? How do the costs of irregular token access scale with sequence length compared with block-level selection?

  3. What are the main algorithmic differences from SVG2 and SVG-EAR beyond the selection granularity and VCM correction? Since VCM applies a second-order correction to mean-based compensation, could a simpler correction based on full-cluster variance achieve similar results without subtracting the selected-token statistics?

  4. How were the cluster counts chosen? The paper uses 300 query clusters and 1,000 key clusters. How sensitive are quality and latency to these choices, and do the counts need to change for different resolutions or video lengths?

1 comment

Dear Reviewer 2ex1,

We wanted to briefly check in and see if our response and additional experiments above have addressed your questions. With the discussion deadline approaching, please let us know if anything remains unclear or if you need any further information from us.

Thank you, The Authors

Reviewer_TPuhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the quality degradation of training-free sparse attention for video diffusion transformers at very low attention densities. It proposes ESTRA, which builds on the clustered representation of SVG2 and SVG-EAR and modifies both selection and compensation. For selection (Eq. (9)), each query-cluster centroid scores individual keys, and the highest-scoring keys are shared by all queries in that cluster instead of selecting whole key clusters. For compensation (Eqs. (4)–(8)), the remaining keys are represented by full-cluster K/V means, and variance-calibrated mean (VCM) compensation adds a capped logit correction based on the spread of the remaining keys. Experiments on Wan2.1-14B and HunyuanVideo-13B without dense warmup report higher fidelity to dense generation than Sol-Attn and SVG-EAR at 5% density, with end-to-end speedups of 3.26× and 4.71× on an NVIDIA H20, and speedups of up to 5.39× at 1% density.

Critical strengths
  1. The paper is well motivated, and the proposed design follows directly from its analysis. It identifies two sources of error that become critical as the exact-computation budget shrinks: whole-cluster selection spends budget on low-scoring members, and mean-based compensation ignores score variation within each cluster. Token-level selection with scores shared within each query cluster addresses the first at low cost (about 0.4% of dense QK arithmetic on Wan), and VCM refines representative weights using only two scalar statistics per query–key cluster pair. This clear connection between the identified error sources and the proposed components makes the method easy to understand and to adopt as a training-free, drop-in replacement.

  2. The experimental results are consistent across backbones and settings. In Table 1, ESTRA at 5% density achieves better PSNR, SSIM, and LPIPS than Sol-Attn and SVG-EAR on both backbones, even though the baselines use 7.6–12.8% density, and it also achieves larger end-to-end speedups. The advantage is also observed at matched densities (Table 2), in the offline attention-output error analysis (Figure 7), and with a shared dense warmup (Table 4). The ablation in Table 3 further shows that token-level selection accounts for most of the gain (+3.12 dB on Wan and +1.75 dB on HunyuanVideo). These consistent results across multiple settings are the main reason I lean toward acceptance.

Critical weaknesses
  1. The practical benefit of the 1% setting is unclear, and motion is not evaluated. Reducing density from 5% to 1% increases the end-to-end speedup by only about 9% on Wan and 14% on HunyuanVideo while fidelity drops (Table 1), and the increase is even smaller with a shared dense warmup (Table 4). With warmup, the gain over SVG-EAR on Wan is also small (+0.35 dB PSNR, about 1.9% lower latency; Table 4). The VBench evaluation also omits motion dimensions such as dynamic degree and motion smoothness. ESTRA’s subject and background consistency on HunyuanVideo exceed those of dense generation, which may instead reflect reduced motion. Since extreme sparsity is the main claim, the paper should clarify when 1% is preferable to 5%.

  2. The novelty is incremental, and the end-to-end contribution of VCM is not isolated. Clustering (SVG2), query-centroid probing (SVG-EAR), mean-based compensation (SVG-EAR, Sol-Attn), and column-level selection (Chipmunk) have close precedents, making VCM the main new component. VCM drops the mean-shift term in Eq. (5). However, because selection removes the highest-scoring keys, this term is non-positive when averaged over the queries in each query cluster under ideal threshold selection (Eq. (9)), whereas VCM can only add a non-negative correction. This possible overcompensation may explain why VCM increases error at some higher densities with token-level selection in Table 5. Moreover, Table 3 tests VCM only with cluster selection, and Table 5 reports only attention-level error, so the end-to-end benefit of VCM in the final ESTRA pipeline is not shown.

Questions for authors
  1. Could the authors report dynamic degree and motion smoothness for dense generation, ESTRA, and the baselines, especially at 1% density? Results on motion-heavy prompts would show whether the higher consistency scores come with reduced motion.

  2. Could the authors compare the final ESTRA pipeline with VCM, without VCM, and with a tuned constant bias on the representative logits, at 1% and 5% density? The selection rule, budgets, cluster counts, and all other settings should be kept identical, so that the representative-logit bias is the only difference between the variants. Could they also explain why VCM increases error at some higher densities in Table 5?

  3. As a rough consistency check, 0.8 Tsparse+0.2 Tdense0.8\,T_{\text{sparse}} + 0.2\,T_{\text{dense}} computed from Table 1 approximates the reported warmup latencies within about 1% on Wan. For HunyuanVideo, however, all Table 4 latencies are about 500 s lower than this estimate. I recognize that Table 1 reports median latency, Table 4 reports mean latency over 10 prompts, and the warmup uses SVG-EAR's dense attention path. Could the authors provide matched-prompt timing breakdowns for the dense and sparse phases, and explain whether kernel choice, aggregation, or other timing differences account for this offset?

Reviewer_j1H9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The core idea of this paper is to improve approximation accuracy in sparse attention for video generation by introducing within-cluster variance as a bias term. Such methods typically make assumptions about the data distribution. The authors provide some simulations evaluating approximation accuracy to support the idea, along with limited qualitative and quantitative experiments.

Critical strengths
  • For cluster-based sparse attention, incorporating variance is an intuitive and potentially valuable idea.
Critical weaknesses
  • The main concern is the relationship to prior work. The idea of incorporating within-cluster variance is not entirely new. Rectified SpaAttn (Liu et al.) has already proposed essentially the same idea. However, the paper does not cite or discuss this work, nor does it provide a comparison.

  • The paper lacks discussion of the assumptions underlying the method. The method assumes that the covariance of within-cluster key residuals is approximately isotropic. When this assumption does not hold, the method may theoretically perform worse. Indeed, Table 5 in the appendix shows that VCM increases the error at 15% token-selection density.

  • The comparisons are limited. Comparisons with methods such as Rectified SpaAttn and PISA are missing.

Questions for authors

see weakness

1 comment

Dear Reviewer j1H9,

We wanted to briefly check in and see if our response and additional experiments above have addressed your questions. With the discussion deadline approaching, please let us know if anything remains unclear or if you need any further information from us.

Thank you, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Gentle Follow-Up on Author Responses and Additional Experiments

Dear Reviewers,

We would like to thank you again for your time and constructive feedback on our submission. As the discussion period is underway, we wanted to gently follow up on our official responses posted earlier to see if our clarifications and new experiments have addressed your concerns.

In summary, following your suggestions, we have added:

  • Direct end-to-end comparisons with PISA and Rectified SpaAttn on both Wan2.1-14B and HunyuanVideo-13B (for Reviewer j1H9).

  • VBench motion evaluations (Dynamic Degree and Motion Smoothness), end-to-end VCM ablations against fixed logit biases, and warmup backend timing clarifications (for Reviewer TPuh).

  • Mathematical consistency evaluations comparing remaining-key vs. full-cluster variance, fine-grained FlashInfer irregular-access latency breakdowns, and cluster-count sensitivity sweeps (for Reviewer 2ex1).

We remain fully available to run any further checks or clarify any remaining points. We would deeply appreciate your feedback on whether our responses resolve your questions.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.