- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PARK, a training-free sparse attention method for video Diffusion Transformers, focused on improving block retrieval. It identifies two mismatches in centroid-based importance estimation: averaging queries before Softmax normalization and clustering keys with Euclidean distance rather than query-dependent score similarity. Per-query Attention-Mass Aggregation (PAMA) retains original queries, normalizes their estimated attention over key blocks, and then averages within each query block. Query-Response Key Clustering (QRKC) derives a metric from the current queries and realizes it through a key transformation followed by Euclidean K-means. A fused PAMA kernel and reuse across denoising steps control overhead. Experiments on four configurations across Wan2.1, Wan2.2, and HunyuanVideo report improved fidelity to dense generation and 1.53×–1.76× end-to-end speedups. The main protocol uses 50 denoising steps, 20% dense warm-up, and periodic clustering/mask refresh.
Critical strengths
1. Well-motivated attention-aware clustering with theoretical justification.
The paper identifies the mismatch between feature similarity and attention similarity, showing that Euclidean key clustering does not necessarily preserve QK-score similarity. QRKC addresses this by deriving a query-dependent metric and transforming key clustering into standard K-means. This provides a principled connection between the approximation error and the proposed clustering objective.
2. Effective correction of query-side aggregation errors.
PAMA addresses the nonlinear approximation error introduced by averaging queries before Softmax. By preserving individual query preferences and aggregating normalized attention weights, it significantly improves block retrieval accuracy. The theoretical analysis and controlled experiments provide convincing support for this design.
3. Efficient implementation with low runtime overhead.
Although PAMA increases block-scoring computation, the fused GPU kernel reduces intermediate memory traffic, while cross-step reuse amortizes retrieval overhead. Experiments demonstrate improved generation fidelity with negligible additional latency, supporting the practicality of the proposed approach.
Critical weaknesses
1. Misalignment between QRKC's optimization objective and attention recall.
QRKC minimizes the mean squared QK-score approximation error (Eq. 10), which does not directly correspond to the Softmax attention mass preserved by Top-p retrieval (Eq. 4). Moreover, its query-induced metric (Eq. 10) weights all queries equally, overlooking variations in attention concentration and block importance. Consequently, minimizing QK-score distortion does not necessarily maximize attention recall.
2. Limited practical contribution of QRKC.
Although QRKC reduces QK-score approximation error by 22.8–36.6% (Table 5), its downstream fidelity improvements are relatively modest. In Table 3, QRKC improves PSNR by only 0.31 dB on HunyuanVideo and 0.10 dB on Wan2.1-14B, compared with PAMA's gains of 1.45 dB and 0.42 dB, respectively. Without statistical significance analysis or matched-fidelity latency comparisons, it remains unclear whether QRKC provides a meaningful additional quality–efficiency benefit.
3. Marginal improvements in perceptual generation quality.
Despite consistently improving PSNR, SSIM, and LPIPS relative to dense attention, PARK shows limited improvements in VBench metrics (Table 2). For example, on Wan2.2-14B, PARK achieves an Aesthetic Quality score of 65.29%, below XAttention's 66.07%. On Wan2.1-1.3B, its Image Quality score of 70.48% is also lower than SpargeAttn's 71.44%. These results suggest that improved attention retrieval fidelity does not necessarily translate into perceptually better video generation.
Questions for authors
-
Could the authors explain how minimizing QK-score approximation error (Eq. 10) relates to maximizing attention recall (Eq. 4), and whether attention-aware query weighting could improve retrieval accuracy?
-
Could the authors provide statistical significance analysis and matched-fidelity latency comparisons to demonstrate QRKC's practical benefits beyond PAMA?
-
Could the authors explain why improved attention retrieval fidelity yields only marginal VBench improvements and sometimes underperforms existing baselines in perceptual quality metrics?
2 comments
Thank you for the clarifications and additional experiments. While they partially address my concerns, the connection between QRKC’s objective and attention recall, as well as its consistent incremental benefit beyond PAMA, remains insufficiently established. I therefore maintain my rating.
We are grateful for your careful review and helpful suggestions, and we appreciate the chance to address these points.
W1 & Q1: Relationship between QRKC’s objective and attention recall.
QRKC is designed to improve attention recall under Top-p retrieval in two ways:
More accurate block-weight estimation. QRKC improves key-centroid estimates of block log-masses by grouping keys with similar QK scores. Applying Softmax to these log-masses gives block attention weights, so more accurate key-centroid estimates of block log-masses help improve attention recall under Top-p retrieval.
More effective block partitioning. QRKC reduces the mixing of important and unimportant interactions within blocks, potentially retaining more attention mass and achieving higher attention recall at a fixed computation budget.
We empirically evaluate these benefits by replacing Euclidean key clustering with QRKC while keeping PAMA and the other settings unchanged. As shown below, QRKC improves both mean and P05 attention recall while reducing attention density on both models.
W2 & Q2: Practical benefits of QRKC.
We evaluate the individual effects of adding PAMA or QRKC to SVG2 and SVG-EAR on Wan2.1-1.3B.
Adding QRKC improves PSNR by 1.04 dB on SVG2 and 0.37 dB on SVG-EAR with nearly unchanged latency, demonstrating its practical benefit independently of PAMA.
W3 & Q3: Fidelity versus perceptual quality.
PARK aims to accelerate video generation while preserving the dense model’s output quality. PSNR, SSIM, and LPIPS measure fidelity to dense outputs, while VBench provides a complementary assessment of video quality.
Table 3 in the revised manuscript shows that PARK achieves better fidelity and lower latency than SVG2 and SVOO across all four model configurations, with VBench scores remaining close to those of dense attention. PARK also improves VBench Image Quality over SVG2 by 2.91 and 2.70 percentage points on HunyuanVideo and Wan2.1-1.3B, respectively. These results demonstrate that PARK accelerates inference while preserving both fidelity to dense outputs and perceptual quality.