acceptodds
Under review as a conference paper at ICLR 2027

PARK: Accurate Block Retrieval for Sparse Attention in Video Diffusion Transformers

Abstract

Diffusion Transformers (DiTs) have become a dominant architecture for video generation, but their efficiency is limited by the quadratic complexity of full attention. Sparse attention reduces this cost by retrieving important blocks and computing attention only within them, but inaccurate retrieval can either degrade generation quality or yield unnecessary computation. We identify two retrieval mismatches in methods that retrieve blocks using the averaged representations of query and key blocks: (i) query-side aggregation mismatch, where averaging queries before Softmax fails to preserve their individual attention preferences, and (ii) key-side clustering metric mismatch, where standard Euclidean clustering in the original key space can group keys with dissimilar QK scores under the current query, so their average representation may not accurately represent how the current query scores individual keys. These mismatches can lead to inaccurate block retrieval. To address these mismatches, we propose PARK, a training-free sparse attention method for accurate block retrieval. PARK retains every original query, independently normalizes its attention over key blocks, and then averages these distributions within each query block. It also uses information from the current queries to transform keys before clustering, so that keys receiving similar QK scores are grouped together. A fused GPU kernel further reduces the overhead of block retrieval. Experiments on HunyuanVideo and Wan demonstrate that PARK improves block retrieval accuracy and preserves generation quality while accelerating inference, achieving the best quality-efficiency trade-off among the compared sparse attention methods.

Primary area:
generative models
Keywords:
Video Generation, Sparse Attention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yBbzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PARK, a training-free sparse attention method for video Diffusion Transformers, focused on improving block retrieval. It identifies two mismatches in centroid-based importance estimation: averaging queries before Softmax normalization and clustering keys with Euclidean distance rather than query-dependent score similarity. Per-query Attention-Mass Aggregation (PAMA) retains original queries, normalizes their estimated attention over key blocks, and then averages within each query block. Query-Response Key Clustering (QRKC) derives a metric from the current queries and realizes it through a key transformation followed by Euclidean K-means. A fused PAMA kernel and reuse across denoising steps control overhead. Experiments on four configurations across Wan2.1, Wan2.2, and HunyuanVideo report improved fidelity to dense generation and 1.53×–1.76× end-to-end speedups. The main protocol uses 50 denoising steps, 20% dense warm-up, and periodic clustering/mask refresh.

Critical strengths

1. Well-motivated attention-aware clustering with theoretical justification.

The paper identifies the mismatch between feature similarity and attention similarity, showing that Euclidean key clustering does not necessarily preserve QK-score similarity. QRKC addresses this by deriving a query-dependent metric GQ=Q⊤Q/LG_Q=Q^\top Q/L and transforming key clustering into standard K-means. This provides a principled connection between the approximation error and the proposed clustering objective.

2. Effective correction of query-side aggregation errors.

PAMA addresses the nonlinear approximation error introduced by averaging queries before Softmax. By preserving individual query preferences and aggregating normalized attention weights, it significantly improves block retrieval accuracy. The theoretical analysis and controlled experiments provide convincing support for this design.

3. Efficient implementation with low runtime overhead.

Although PAMA increases block-scoring computation, the fused GPU kernel reduces intermediate memory traffic, while cross-step reuse amortizes retrieval overhead. Experiments demonstrate improved generation fidelity with negligible additional latency, supporting the practicality of the proposed approach.

Critical weaknesses

1. Misalignment between QRKC's optimization objective and attention recall.

QRKC minimizes the mean squared QK-score approximation error (Eq. 10), which does not directly correspond to the Softmax attention mass preserved by Top-p retrieval (Eq. 4). Moreover, its query-induced metric GQ=Q⊤Q/LG_Q=Q^\top Q/L (Eq. 10) weights all queries equally, overlooking variations in attention concentration and block importance. Consequently, minimizing QK-score distortion does not necessarily maximize attention recall.

2. Limited practical contribution of QRKC.

Although QRKC reduces QK-score approximation error by 22.8–36.6% (Table 5), its downstream fidelity improvements are relatively modest. In Table 3, QRKC improves PSNR by only 0.31 dB on HunyuanVideo and 0.10 dB on Wan2.1-14B, compared with PAMA's gains of 1.45 dB and 0.42 dB, respectively. Without statistical significance analysis or matched-fidelity latency comparisons, it remains unclear whether QRKC provides a meaningful additional quality–efficiency benefit.

3. Marginal improvements in perceptual generation quality.

Despite consistently improving PSNR, SSIM, and LPIPS relative to dense attention, PARK shows limited improvements in VBench metrics (Table 2). For example, on Wan2.2-14B, PARK achieves an Aesthetic Quality score of 65.29%, below XAttention's 66.07%. On Wan2.1-1.3B, its Image Quality score of 70.48% is also lower than SpargeAttn's 71.44%. These results suggest that improved attention retrieval fidelity does not necessarily translate into perceptually better video generation.

Questions for authors
  1. Could the authors explain how minimizing QK-score approximation error (Eq. 10) relates to maximizing attention recall (Eq. 4), and whether attention-aware query weighting could improve retrieval accuracy?

  2. Could the authors provide statistical significance analysis and matched-fidelity latency comparisons to demonstrate QRKC's practical benefits beyond PAMA?

  3. Could the authors explain why improved attention retrieval fidelity yields only marginal VBench improvements and sometimes underperforms existing baselines in perceptual quality metrics?

2 comments

We are grateful for your careful review and helpful suggestions, and we appreciate the chance to address these points.

W1 & Q1: Relationship between QRKC’s objective and attention recall.

QRKC is designed to improve attention recall under Top-p retrieval in two ways:

  1. More accurate block-weight estimation. QRKC improves key-centroid estimates of block log-masses by grouping keys with similar QK scores. Applying Softmax to these log-masses gives block attention weights, so more accurate key-centroid estimates of block log-masses help improve attention recall under Top-p retrieval.

  2. More effective block partitioning. QRKC reduces the mixing of important and unimportant interactions within blocks, potentially retaining more attention mass and achieving higher attention recall at a fixed computation budget.

We empirically evaluate these benefits by replacing Euclidean key clustering with QRKC while keeping PAMA and the other settings unchanged. As shown below, QRKC improves both mean and P05 attention recall while reducing attention density on both models.

ModelConfigurationMean Recall (%) ↑P05 Recall (%) ↑Density (%) ↓
Wan2.1-14BEuclidean clustering91.5689.3632.79
Wan2.1-14BQRKC91.9289.9232.22
HunyuanVideoEuclidean clustering90.7087.6226.38
HunyuanVideoQRKC91.2288.9925.51

W2 & Q2: Practical benefits of QRKC.

We evaluate the individual effects of adding PAMA or QRKC to SVG2 and SVG-EAR on Wan2.1-1.3B.
Adding QRKC improves PSNR by 1.04 dB on SVG2 and 0.37 dB on SVG-EAR with nearly unchanged latency, demonstrating its practical benefit independently of PAMA.

configPSNR ↑SSIM ↑LPIPS ↓Latency ↓
SVG 226.210.8880.203435s
SVG 2 + PAMA27.470.9120.166438s
SVG 2 + QRKC27.250.9110.168436s
SVG-EAR26.890.8970.192452s
SVG-EAR + PAMA27.780.9200.158451s
SVG-EAR + QRKC27.260.9090.171454s
PARK28.160.9270.152423s

W3 & Q3: Fidelity versus perceptual quality.

PARK aims to accelerate video generation while preserving the dense model’s output quality. PSNR, SSIM, and LPIPS measure fidelity to dense outputs, while VBench provides a complementary assessment of video quality.
Table 3 in the revised manuscript shows that PARK achieves better fidelity and lower latency than SVG2 and SVOO across all four model configurations, with VBench scores remaining close to those of dense attention. PARK also improves VBench Image Quality over SVG2 by 2.91 and 2.70 percentage points on HunyuanVideo and Wan2.1-1.3B, respectively. These results demonstrate that PARK accelerates inference while preserving both fidelity to dense outputs and perceptual quality.

Thank you for the clarifications and additional experiments. While they partially address my concerns, the connection between QRKC’s objective and attention recall, as well as its consistent incremental benefit beyond PAMA, remains insufficiently established. I therefore maintain my rating.

Reviewer_edzVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PARK, a training-free sparse attention method for efficient video Diffusion Transformers. It identifies two limitations of centroid-based block retrieval: query-side aggregation mismatch and key-side clustering metric mismatch. To address them, PARK introduces Per-query Attention-Mass Aggregation (PAMA) to preserve individual query preferences and Query-Response Key Clustering (QRKC) to improve key clustering based on QK-score similarity. Experiments on HunyuanVideo and Wan demonstrate improved block retrieval accuracy and generation quality preservation, achieving 1.53×–1.76× end-to-end speedups over dense attention.

Critical strengths
  1. The proposed two contributions have sufficient analysis. The paper identifies two key limitations of existing centroid-based block retrieval methods, namely query-side aggregation mismatch and key-side clustering metric mismatch. It provides mathematical analyses of the corresponding approximation errors, offering a well-grounded motivation for the proposed method.

  2. The proposed two modules are straightful. The proposed PAMA and QRKC improve block retrieval accuracy from the perspectives of query aggregation and key clustering, respectively. The overall framework requires no additional training and incorporates a fused GPU kernel to reduce computational and memory overhead, making it practical for deployment.

Critical weaknesses
  1. The conceptual difference with existing methods are limited. PARK largely follows the clustering, token reordering, and block retrieval framework of SVG2 and SVOO, while mainly changes the order of Softmax normalization and query aggregation, while QRKC shares a similar motivation with SVOO’s bidirectional QK co-clustering. The authors should better clarify the methodological novelty beyond these existing approaches.
  2. The actual contribution of each component should be clarified. Although PARK outperforms SVOO, the two methods differ in both clustering and block importance estimation, making it difficult to isolate the benefit of QRKC. Comparisons such as SVG2 + PAMA and SVOO + PAMA would provide stronger evidence. Moreover, Table 3 shows relatively modest gains from QRKC, raising questions about its necessity.
Questions for authors

Please refer to the major weakness part.

3 comments

We are grateful for your careful review and helpful suggestions, and we appreciate the chance to address these points.

W1: Differences from SVG2 and SVOO.

PARK differs from SVG2 and SVOO in both block-importance estimation and clustering.
SVG2 and SVOO estimate block importance using query/key centroids, but averaging queries into centroids before Softmax can distort the attention weights assigned to key blocks, leading to inaccurate block retrieval.
PARK's PAMA retains individual queries, normalizes their attention over key blocks separately, and then averages these attention weights, avoiding query-centroid approximation. QRKC further improves key-centroid approximation accuracy by clustering keys according to their QK-score similarity.
Table 3 in the revised manuscript shows that PARK achieves better PSNR, SSIM, and LPIPS with lower end-to-end latency than SVG2 and SVOO across all four model configurations.

Unlike SVOO’s alternating co-clustering based on opposite-side centroids, QRKC derives its objective from key-centroid approximation error using all original queries. Its linear transformation preserves d=128d=128, whereas SVOO’s affinity dimensions depend on centroid counts (e.g., 256/1,024). Thus, QRKC preserves the dimensionality and complexity of subsequent Euclidean K-means, with a small transformation overhead.

Clustering latency comparison. Both queries and keys have the tensor shapes below, ordered as [batch size, attention heads, sequence length, head dimension]. All settings use 256 query centroids, 1,024 key centroids, and 2 clustering iterations. PARK w/o QRKC uses standard Euclidean key clustering. Lower is better.

Q/K shapePARKPARK w/o QRKCSVOO
[1, 40, 75600, 128]24.51 ms22.59 ms429.54 ms
[1, 24, 115200, 128]22.18 ms20.30 ms396.01 ms

W2: Contribution of QRKC.

We evaluate SVG2 + PAMA, SVG2 + QRKC, and SVOO + PAMA on Wan2.1-1.3B to isolate the contributions of PAMA and QRKC. Adding QRKC alone to SVG2 (SVG2 + QRKC) improves PSNR by 1.04 dB with nearly unchanged latency, demonstrating QRKC’s contribution independently of PAMA. Adding PAMA improves PSNR by 1.26 dB on SVG2 and 0.34 dB on SVOO, also with minimal latency changes. PARK further outperforms SVG2 + PAMA by 0.69 dB, supporting the additional benefit of QRKC.

configPSNR ↑SSIM ↑LPIPS ↓Latency ↓
SVG 226.210.8880.203435s
SVG 2 + PAMA27.470.9120.166438s
SVG 2 + QRKC27.250.9110.168436s
SVOO27.390.9090.170472s
SVOO + PAMA27.730.9150.164474s
PARK28.160.9270.152423s

Thanks for the rebuttal. Most of my concerns have been addressed. Therefore I raise my score to weak accept.

Thank you for your careful review and positive reassessment of our work. We appreciate your constructive suggestions, which helped us clarify the methodological contributions and strengthen the experimental evidence.

Reviewer_yHuaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PARK, a training-free sparse attention method for video diffusion transformers. It addresses two limitations of centroid-based block retrieval through Per-query Attention-Mass Aggregation (PAMA) and Query-Response Key Clustering (QRKC). Experiments across four models demonstrate improved reconstruction fidelity and 1.53× to 1.76× end-to-end inference speedups while largely preserving generation quality.

Critical strengths
  • Well-motivated approach. The paper identifies two sources of block-retrieval approximation error and proposes principled solutions to address them.

  • Consistent quality-preserving acceleration. PARK achieves the best reconstruction fidelity (PSNR, SSIM, and LPIPS) and lowest inference latency among the evaluated methods across four video diffusion models.

  • Thorough validation. The theoretical analysis, approximation-error measurements, attention-recall experiments, and component ablations provide convincing evidence supporting the proposed design choices.

Critical weaknesses
  1. Missing comparisons with recent methods. Recent sparse attention methods, including DFSAttn (ICML 2026), AdaCluster (CVPR 2026), and MOD-DiT (ICML 2026), report higher speedups under their respective evaluation settings. Their omission limits the assessment of PARK's quality-efficiency trade-off, particularly at higher resolutions and longer sequences. Can the authors provide matched comparisons and demonstrate the generalizability of PAMA and QRKC by integrating them into at least one of these methods?

  2. Limited contribution of the theoretical analysis. Section 4 introduces substantial mathematical complexity without clearly establishing its added explanatory value. In particular, the query-side Hessian analysis does not appear to inform the subsequent methodology, and the derived error terms are not empirically validated against actual approximation errors. Although the key-side derivation motivates the proposed clustering metric, the overall theoretical treatment complicates the presentation without a commensurate contribution to understanding the method's effectiveness. Moreover, the direct empirical evidence in Table 4 and Figure 4 is relegated to the appendix, weakening the clarity of the central motivation.

  3. Relation to prior work. Although developed outside the video generation setting, A²ATS (ACL Findings 2025) introduces a similar query-dependent clustering approach.

Minor Comments

  • Equation (2): The formulation masks attention weights after Softmax normalization, which differs from standard masked Softmax attention. Please clarify whether this is intentional or correct the formulation to reflect the implemented sparse attention.

  • Appendix D.1, lines 968 to 969: Remove the redundant sentence.

  • Appendices A and D.2: Correct references to Eq. (7) when referring to the exact block log-mass in Eq. (6).

Questions for authors

see Weaknesses

2 comments

We are grateful for your careful review and helpful suggestions, and we appreciate the chance to address these points.

W1: Comparisons with recent methods

We compare PARK with DFSAttn (ICML 2026) and SVG-EAR (ECCV 2026) on Wan2.1-1.3B. We further integrate PAMA and QRKC into SVG-EAR, which also uses a clustering-based framework.

As shown below, PARK achieves higher PSNR and SSIM, lower LPIPS, and lower latency than DFSAttn and SVG-EAR.
Integrating PAMA or QRKC into SVG-EAR improves PSNR, SSIM, and LPIPS with nearly unchanged latency, supporting the benefits of both modules beyond PARK.

MethodPSNR ↑SSIM ↑LPIPS ↓Latency ↓
DFSAttn23.610.8300.249455s
SVG-EAR26.890.8970.192452s
SVG-EAR + PAMA27.780.9200.158451s
SVG-EAR + QRKC27.260.9090.171454s
PARK28.160.9270.152423s

On HunyuanVideo, PARK achieves 1.70× and 1.77× speedups for longer and higher-resolution videos, respectively, while preserving fidelity to dense outputs.

F ×\times H ×\times WPSNR ↑SSIM ↑LPIPS ↓Speedup ↑
253 ×\times 544 ×\times 96033.200.9470.1511.70 ×\times
81 ×\times 1072 ×\times 192032.040.9500.1861.77 ×\times

W2: Role and empirical validation of the theoretical analysis.

The query-side analysis motivates avoiding query-centroid approximation rather than explicitly computing the Hessian. It shows that the error depends on both query covariance and attention curvature, while Euclidean query clustering does not account for the latter.

We further compare query and key compression using attention-weight TV under the same Euclidean partitions (Table 1 in the revised manuscript).
Full-query, full-key attention serves as the reference.Lower is better.

QueryKeyWan2.1-14BHunyuanVideo
FullFull0.00000.0000
CentroidCentroid0.17740.2092
CentroidFull0.13350.1667
FullCentroid0.09270.1066

Compressing only queries produces larger attention-weight errors than compressing only keys on both models. The analysis and empirical results motivate PAMA, which retains every original query and normalizes before averaging, eliminating query-centroid approximation.


We evaluate the second-order terms against the exact approximation errors across all layers and heads. The higher-order remainders are computed as

RuvQ=ΔuvQ−12⟨Hϕv(q‾u),ΣQ,u⟩F.\mathcal{R}_{uv}^{Q} = \Delta_{uv}^{Q} - \frac{1}{2}\left\langle \mathbf{H}_{\phi_v}(\overline{\mathbf{q}}_u), \boldsymbol{\Sigma}_{Q,u} \right\rangle_F. RivK=ΔivK−ϵivK.\mathcal{R}_{iv}^{K} = \Delta_{iv}^{K} - \epsilon_{iv}^{K}.

To compare their magnitudes, we report

{\operatorname{RMSE}(T_2)+\operatorname{RMSE}(R)} \times 100\%,$$ where $T_2$ and $R$ denote the corresponding second-order term and remainder above. We compute this ratio for each layer–head pair and average across pairs. Under Euclidean clustering, the results are: | Model | Query side | Key side | | :--- | :---: | :---: | | Wan2.1-14B | 75.1% | 85.3% | | HunyuanVideo | 75.0% | 84.7% | The second-order terms dominate the higher-order remainders in magnitude on both the query and key sides. --- Thank you for this helpful suggestion. We have moved the original Table 4 and Figure 4 into the main text, placing the direct empirical evidence alongside the mismatch analysis. ### W3: Relationship to A²ATS. Thank you for pointing out this relevant work. A²ATS and QRKC share a query-second-moment quadratic objective and a Cholesky-based key transformation. We will clarify these differences from A²ATS. **QRKC differs from A²ATS in its motivation and use.** QRKC is derived from our analysis of the error in estimating block attention weights with key centroids. A Taylor expansion of the exact block log-mass identifies the leading approximation error, which motivates QRKC’s clustering objective. In contrast, A²ATS targets token-level attention-score approximation using offline shared codebooks, while QRKC updates its metric and key partitions online using the current queries and keys. ### Minor comments Thank you for pointing out these issues. We corrected Eq. (2) to normalize attention over the retained keys, rather than masking full-attention weights without renormalization: $$A_{ij}^{\mathrm{sp}} =\frac{M_{ij}\exp(Z_{ij})} {\sum_{\ell=1}^{L}M_{i\ell}\exp(Z_{i\ell})}, \qquad \mathbf{O}^{\mathrm{sp}}=\mathbf{A}^{\mathrm{sp}}\mathbf{V}.$$ We also removed the redundant sentence in Appendix D.1 and corrected the equation references in Appendices A and D.2.

Thank you for the detailed rebuttal, which partially addresses my concerns. I would be willing to raise my score if the authors provide comparisons with AdaCluster and MOD-DiT, as previously requested.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.