acceptodds
Under review as a conference paper at ICLR 2027

ResiSparse: Residual-Informed Exact-or-Approximate Sparse Attention for Efficient Video Generation

Abstract

Video diffusion transformers repeatedly apply spatiotemporal self-attention, making denoising expensive as video sequences grow. Training-free sparse attention reduces computation, but discarding unselected interactions can make model outputs deviate from their dense-attention counterparts. A recent paradigm computes selected interactions exactly and approximates the rest. Accurate approximation remains essential: replacing individual keys and values with cluster centroids can still distort the output. We introduce ResiSparse, a training-free sparse-attention method guided by a theoretical analysis of centroid-based approximation. This analysis motivates three components: (i) Query-Induced Key Clustering uses query information to reweight key channels before clustering, reducing centroid-induced logit distortion; (ii) Unified Mass–Dispersion Routing combines centroid logits with within-cluster key and value variation to prioritize exact computation using a cost-normalized score; and (iii) Query-Prioritized Taylor Compensation uses a second-order expansion to account for logit variation within approximated blocks, correcting their centroid-based contributions to both the attention numerator and softmax denominator. Across Wan2.1-1.3B, Wan2.1-14B, and HunyuanVideo-13B, ResiSparse outperforms the evaluated training-free sparse-attention methods and achieves a stronger quality–efficiency trade-off, with PSNRs of , , and at , , and denoising speedups, respectively.

Primary area:
generative models
Keywords:
video diffusion models, training-free sparse attention, inference acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FHmROpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents ResiSparse, a training‑free sparse‑attention method for video diffusion transformers. It consists of three cooperating components: QIKC (Query‑Induced Key Clustering), UMDR (Unified Mass–Dispersion Routing), and QPTC (Query‑Prioritized Taylor Compensation). Following the exact‑or‑approximate sparse‑attention paradigm, the method avoids outright discarding unselected token blocks. Instead, it applies second‑order Taylor compensation based on cluster centroids. The goal is to reduce the computational cost of spatiotemporal self‑attention while preserving video generation quality comparable to dense attention. The authors conduct experiments on the Wan2.1 series and HunyuanVideo video models, reporting speedups of 1.59‑2.18× over dense attention with slight degradations in VBench metrics. The authors claim to have theoretically derived the impacts of centroid‑induced logit deviations and intra‑cluster value variation on the attention numerator and softmax denominator, and they use this theory to guide the design of the three modules.

Critical strengths
  1. The topic addresses a well‑known pain point in video generation: spatiotemporal self‑attention constitutes a major computational bottleneck for video diffusion models. The training‑free sparse acceleration scheme requires no fine‑tuning of model weights and carries practical engineering value.
  2. The overall pipeline features clear modular design: QIKC for clustering optimisation, UMDR for error‑aware routing, and QPTC for second‑order compensation. Comprehensive speed‑quality evaluations are performed on popular open‑source video models Wan2.1 and HunyuanVideo, with baselines including SVG‑EAR and other exact‑or‑approximate sparse‑attention counterparts.
Critical weaknesses
  1. Insufficient justification for the valid regime of core theoretical approximations. The second‑order Taylor expansion within QPTC is valid only under δℓ≪1\delta\ell \ll 1. Cases with large intra‑cluster dispersion are not analysed. The compensation term may introduce larger biases at certain denoising steps. Results are supported solely by ablation experiments, lacking theoretical error bounds.
  2. Incomplete manuscript quality and severe figure defects. Many mathematical symbols in the QPTC block diagram of Figure 2 render incorrectly as question‑mark placeholders. The top‑channel selection mechanism drawn in the figure is not reflected in main theoretical formulas. Inconsistencies among theory, schematic diagrams and implementation reveal inadequate manuscript proof‑reading at review submission time.
  3. Baseline comparisons are insufficient. Evaluations focus mainly on SVG‑EAR. More granular error comparisons against other low‑cost approximate‑attention variants and alternative sparse‑attention approaches (e.g., linear‑attention‑style approximations) are missing.
Questions for authors
  1. Please carefully double‑check and fix all formulas and figures in your manuscript.
  2. Could you provide the complete set of VBench metric dimensions and additional baseline results?
6 comments
AuthorsOpenReview ↗

Response to Reviewer FHmR (Part1)

We thank the reviewer for the comments on theoretical validity, presentation, and experimental completeness.

W1: QPTC validity with large logit deviations and across denoising steps.

Reply:

(1) Taylor approximation and error bound. QPTC can improve on the uncorrected centroid estimate without requiring ∣δℓ∣≪1|\delta\ell|\ll1. The original Appendix A.4 retains the exact Taylor remainder in Eqs. (36)–(37), and Eq. (42) separates the remaining error into this remainder and the selected-channel variance-estimation error. Eq. (43) gives the improvement condition 0<σ^ℓ,ij2/2<2(Zij/Z^ij−1)0 < \widehat{\sigma}_{\ell,ij}^{2}/2 < 2(Z_{ij}/\widehat{Z}_{ij}-1), which permits non-negligible higher-order terms. Improvement is conditional on this inequality. We have also made the error bound obtained directly from the same Taylor remainder explicit in Appendix A.4.

(2) Results across denoising steps. We evaluated QPTC on Wan2.1-1.3B using five prompts, five layers, and four denoising steps, yielding 100 sampled locations in total. With clustering and routing fixed, we compute relative L2L_2 error for each query and attention head, then average within each step. Bold percentages indicate reductions in these mean per-query errors:

StepNumeratorDenominatorNormalized output
156.45% (0.04260 →\rightarrow 0.03986)10.88% (0.04484 →\rightarrow 0.03996)8.16% (0.03254 →\rightarrow 0.02989)
256.24% (0.03972 →\rightarrow 0.03724)10.50% (0.04223 →\rightarrow 0.03780)7.95% (0.03060 →\rightarrow 0.02817)
356.18% (0.03596 →\rightarrow 0.03374)10.24% (0.03881 →\rightarrow 0.03484)7.83% (0.02761 →\rightarrow 0.02545)
456.36% (0.03226 →\rightarrow 0.03021)9.84% (0.03556 →\rightarrow 0.03206)7.93% (0.02438 →\rightarrow 0.02244)

(3) Results across logit-variance ranges. The step-wise results above measure the total numerator, denominator, and normalized-output errors. To directly test the approximation underlying QPTC, we compare Z^sjcorr\widehat Z_{sj}^{\mathrm{corr}} with its exact value ZsjZ_{sj} for the key clusters assigned to approximate computation. Here, σℓ,sj2\sigma_{\ell,sj}^{2} is the variance of attention logits across keys in cluster jj for a fixed query ss. The table groups these estimates by σℓ,sj2\sigma_{\ell,sj}^{2} and reports how often QPTC increases the estimation error:

Within-cluster logit variance σℓ,sj2\sigma_{\ell,sj}^{2}Evaluated ZsjZ_{sj} estimates (M)Cases with increased ZsjZ_{sj} estimation error (%)
[0,0.25)[0,0.25)2.040.3411
[0.25,1)[0.25,1)5.240.1393
[1,4)[1,4)14.460.0202
[4,∞)[4,\infty)8.820

Error increases become less frequent at larger logit variances, with none observed for σℓ,sj2≥4\sigma_{\ell,sj}^{2}\ge4. Together with lower mean numerator, denominator, and normalized-output errors at every sampled step, these measurements show that the correction remains useful beyond the small-deviation regime in the evaluated setting.

W2 & Q1: Figure corrections and the top-channel selection mechanism.

Reply:

(1) Figure correction. We apologize for the rendering defects in Figure 2. The figure was created manually and then exported to PDF using WPS. It rendered correctly in WPS, but some symbols and formulas were corrupted in other PDF viewers. Our repeated pre-submission checks did not cover different PDF viewers, so this compatibility issue went unnoticed. We have re-exported the figure, verified the corrected version, and rechecked all figures and formulas in the manuscript.

(2) Top-channel selection in the original submission. Top-channel selection was already specified in the original submission. In the main text, Section 4.3, Eq. (13) computes QPTC's variance estimate over the selected channel set Ω\Omega; Appendix A.4, Eq. (39) defines Ω\Omega through query-energy ranking; and Appendix B and Algorithm 3 use the same selected-channel variances in the implementation. Thus, the theory, diagram, and implementation describe the same mechanism.

AuthorsOpenReview ↗

Response to Reviewer FHmR (Part2)

W3 & Q2: Additional baseline comparisons and per-dimension VBench results.

Reply:

(1) Additional baselines. ResiSparse achieves higher fidelity to dense outputs at lower latency than PISA [1] and Sol-Attn [4] at the evaluated settings. Given the limited rebuttal time and computational resources, the additional baseline comparison covers Wan2.1-1.3B. All methods use the same prompts, seeds, generation settings, and hardware. We set PISA's target density to 0.20 and Sol-Attn's threshold to τ=1.3\tau=1.3 to approximately match ResiSparse's attention density rather than being exactly identical.

MethodDenoising time (s) ↓Density (%)PSNR (dB) ↑SSIM ↑LPIPS ↓VBench (%) ↑
Dense774100.00———76.80
PISA37527.9620.670.75300.237877.50
Sol-Attn36826.8118.760.65350.354974.77
ResiSparse35525.9925.490.84560.113276.35

For training-free acceleration, our objective is to preserve the pretrained model's dense-attention outputs. PISA scores above dense on VBench (77.50% vs. 76.80%) but has lower PSNR and SSIM and higher LPIPS than ResiSparse. A higher VBench score therefore does not imply better preservation of dense outputs. At the evaluated settings, ResiSparse improves PSNR over PISA by 4.82 dB and over Sol-Attn by 6.73 dB, with shorter denoising time. The added comparisons cover training-free exact-or-approximate sparse attention. Linear-attention-style approximations are outside the evaluated baseline set.

(2) Per-dimension VBench results. Following prior sparse-attention studies [1, 2, 5, 6], we evaluated video quality on six VBench dimensions: subject consistency, motion smoothness, dynamic degree, aesthetic quality, imaging quality, and overall consistency. The reported VBench score is their six-dimension aggregate using the official normalization and weighting procedure [3]. Individual scores below are raw scores multiplied by 100.

MethodVBench (%)Subject consistency (%)Motion smoothness (%)Dynamic degree (%)Aesthetic quality (%)Imaging quality (%)Overall consistency (%)
Dense76.8092.6098.5656.9465.3066.5927.31
SVG276.4691.8498.5055.5665.3365.9927.44
SVG-EAR76.3091.9998.5654.1765.2065.8927.34
PISA77.5093.1298.5158.3366.0367.8027.58
Sol-Attn74.7792.9398.7147.2262.6262.5827.06
ResiSparse76.3592.3698.5654.1765.1366.1627.22

Manuscript revisions. We have revised the manuscript and highlighted the changes in blue. We corrected Figure 2, added QPTC validation across denoising steps and logit-variance ranges, and expanded the baseline comparisons and six-dimension VBench reporting.

References

[1] Li et al. PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers. arXiv:2602.01077, 2026.

[2] Li et al. Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation. arXiv:2508.12969, 2025.

[3] Huang et al. VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR 2024.

[4] Li et al. Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification. arXiv:2607.24027, 2026.

[5] Yang et al. Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation. NeurIPS 2025.

[6] Zhou et al. SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing. arXiv:2603.08982, 2026.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer FHmR,

Thank you again for your comments. Our response focuses on the following concerns:

  • QPTC's validity beyond small logit deviations. We clarify the condition under which QPTC reduces approximation error, make the bound derived from the Taylor remainder explicit, and report whether the correction increases errors across sampled denoising steps and logit-variance ranges.
  • Figure defects and consistency with the formulas and implementation. We corrected Figure 2, rechecked the manuscript's formulas and figures, and identified where top-channel selection was already defined in the original equations and algorithm.
  • Missing baselines and VBench details. We added PISA and Sol-Attn comparisons on Wan2.1-1.3B and reported individual scores for all six VBench dimensions evaluated in our experiments.

The corresponding manuscript revisions are highlighted in blue. We would appreciate your feedback on the revised manuscript and our response. Please let us know if any points need further clarification.

Best regards, The authors

Reviewer_HLRyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies approximation error in exact-or-approximate sparse attention for video diffusion models. Building on SVG-EAR's semantic clustering and exact/approximate block computation, ResiSparse introduces query-weighted key clustering (QIKC), mass–dispersion-based routing (UMDR), and a second-order correction to the estimated softmax mass (QPTC). Section 3 decomposes the errors caused by replacing key–value clusters with centroids. Table 1 reports higher dense-reference fidelity and lower denoising latency than SVG-EAR on Wan2.1-1.3B, Wan2.1-14B, and HunyuanVideo-13B. The contribution is best understood as a more accurate and efficient implementation of cluster-based attention compensation within an established sparse-attention framework.

Critical strengths
  1. A well-targeted analysis of approximation error. Proposition 1 (Section 3, Eqs. 5–6) separates the error in the softmax normalization term from the omitted value-weighting contribution. This gives the proposed corrections a concrete mathematical motivation.
  2. Measurable gains over the closest baseline. In Table 1, PSNR improves over SVG-EAR by 0.66/0.86/0.38 dB on Wan-1.3B, Wan-14B, and HunyuanVideo, respectively. Denoising time also decreases from 379/1064/770 s to 355/992/733 s. The gains extend beyond a single model or a proxy metric.
Critical weaknesses
  1. The optimization proxies do not establish control of the final attention error. UMDR bounds the omitted AijA_{ij} term in Eq. (10) and ranks blocks using a cost-normalized form in Eq. (11). That bound is not a bound on normalized attention-output error: denominator errors, interactions across clusters, and deviations between individual queries and their cluster centroid remain outside the routing objective. QPTC corrects the estimated ZijZ_{ij} and its centroid-value contribution in Eq. (13), but does not recover AijA_{ij}; it also drops cross-channel covariance and uses only selected channels. Figure 3(b) primarily measures denominator-contribution error, while Figure 4(a) shows routing coverage close to SVG-EAR. The experiments establish improvements in selected surrogates, not a complete account of why the final token-level attention output improves. Direct measurements of numerator, denominator, and normalized-output error, broken down by layer and denoising step, are needed to substantiate the claimed mechanism.

  2. The end-to-end evidence is insufficient for attributing the reported advantage to the new components. Table 1 is a useful comparison, but there is no complete, matched-setting ablation across all three generators showing SVG-EAR followed by QIKC, UMDR, QPTC, and their combinations, with PSNR, VBench, and latency reported together. This also leaves the effect of the fused implementation difficult to separate from that of the algorithms. The main comparison excludes other exact-or-approximate methods discussed in Related Work, including PISA and Sol-Attn, so broad claims of superiority need to be restricted to the evaluated baselines. The video-quality claims require similar care: VBench AVG on Wan2.1-1.3B is 76.35, below SVG2's 76.46, and the HunyuanVideo value of 77.08 is marginally below SVG-EAR's 77.09. Figure 1 suggests less flickering, but the paper does not measure temporal flicker directly, and the Motion Smoothness differences in Table 4 are small. These omissions matter because the paper's novelty lies in the practical benefit of these additional mechanisms.

Questions for authors
  1. At matched attention density, top-p, and GPU kernels, what are the PSNR, VBench, and latency results for SVG-EAR and for QIKC, UMDR, and QPTC added one at a time, on all three models? How much of the speedup comes from kernel engineering?
  2. For individual query tokens, how much does QPTC reduce errors in the attention numerator, denominator, and final normalized output? When does the uncorrected AijA_{ij} term dominate?
  3. How does the method compare with PISA and other exact-or-approximate baselines at matched accuracy or realized sparsity? Is there a quantitative temporal-flicker metric supporting the interpretation of Fig. 1?
6 comments
AuthorsOpenReview ↗

Response to Reviewer HLRy (Part1)

We thank the reviewer for the detailed analysis and questions on attention-output error.

W1: How do the UMDR and QPTC objectives relate to final attention-output error?

Reply:

(1) Connection to attention-output error. Proposition 1 (Section 3.2, Eqs. (5)–(6)) decomposes the two error sources targeted by UMDR and QPTC, while Figure 4(b) directly measures attention layer output error: MSE against dense attention decreases from 9.93×10−49.93\times10^{-4} for SVG-EAR to 7.79×10−47.79\times10^{-4} for ResiSparse, at a slightly lower realized density. UMDR prioritizes exact computation using a cost-normalized bound on ∥Aij∥\|A_{ij}\| (Eqs. (10)–(11)); QPTC corrects both the estimated denominator contribution and its value-centroid contribution to the numerator (Eq. (13)). Proposition 2 in Appendix A.5 bounds the difference between a query token's output residual and its cluster centroid's residual in terms of the query-cluster radius.

(2) Error measurements by step and layer. We additionally compare ResiSparse with SVG-EAR using five prompts on Wan2.1-1.3B at four denoising steps and five layers (100 locations). For the numerator, denominator, and normalized output, we compute relative L2L_2 error ∥x^−x∥2/∥x∥2\|\widehat x-x\|_2/\|x\|_2 over queries and heads at each location. The average errors decrease by 7.83%, 10.93%, and 18.95%, respectively. Negative values indicate increased error:

Step / layerNumerator error reduction (%)Denominator error reduction (%)Normalized output error reduction (%)
Step 1510.0711.7925.15
Step 256.467.9520.41
Step 357.3011.7917.19
Step 456.8912.5011.60
Layer 018.6468.4426.62
Layer 515.8624.2439.03
Layer 108.9416.04-1.40
Layer 158.0315.37-11.10
Layer 20-1.69-5.517.54

Normalized-output error decreases in all four step averages and three of the five layer averages. Numerator and denominator errors can partially cancel during normalization, so reducing both does not necessarily reduce output error at every layer. Nevertheless, averaged over all sampled locations, ResiSparse reduces normalized-output relative L2L_2 error by 18.95%. Table 1 also shows improved dense-reference fidelity across all three models.

AuthorsOpenReview ↗

Response to Reviewer HLRy (Part2)

W2 & Q1: Component attribution and the contribution of optimized kernels.

Reply:

(1) Component contributions. Given the limited rebuttal time and computational resources, this additional experiment is conducted on Wan2.1-1.3B. All variants are evaluated under matched settings with Top-p=0.85p=0.85 and the same implementations of shared operations; their realized densities differ by at most 0.20 percentage points. Disabling all components gives SVG-EAR, and enabling all components gives full ResiSparse. As noted in Appendix C, our evaluation uses optimized implementations of FlashInfer and K-means. For the requested matched-GPU-kernel comparison, we reran SVG-EAR with the same optimized implementations. This reduced its denoising time from 379 s in Table 1 to 367 s, without changing the SVG-EAR algorithm or attention density.

QIKCUMDRQPTCDenoising time (s) ↓Density (%)PSNR (dB) ↑SSIM ↑LPIPS ↓VBench (%) ↑
———36726.0124.830.83630.124476.30
✓——36826.1925.150.83870.120176.49
✓✓—35425.9925.270.84220.116876.61
✓✓✓35525.9925.490.84560.113276.35

QIKC improves all three dense-reference fidelity metrics with minimal latency overhead, consistent with a better clustering basis. Adding UMDR further improves fidelity while providing most of the runtime reduction through lower-complexity cluster-level routing. QPTC corrects residual approximation errors and yields the best PSNR, SSIM, and LPIPS with little additional cost. Together, the three components improve both fidelity and efficiency: relative to matched-kernel SVG-EAR, full ResiSparse improves PSNR by 0.66 dB and reduces denoising time by 12 s (3.27%).

(2) Kernel contribution. Sections 4.1–4.4 and Appendix B describe the algorithm--system co-design. QIKC reuses query centroids and requires O(Cqd+Nkd)O(C_qd+N_kd) preprocessing; UMDR reduces routing from SVG-EAR's O(CqNkd)O(C_qN_kd) token-level scoring to O(CqCkd)O(C_qC_kd) cluster-level scoring; QPTC uses dsel=32d_{\mathrm{sel}}=32 channels and costs O(Nkdsel+CqCkdsel)O(N_kd_{\mathrm{sel}}+C_qC_kd_{\mathrm{sel}}) instead of forming full covariance matrices. ResiSparse retains SVG2's semantic permutation [3] and SVG-EAR's centroid approximation and online merge [1], while accumulating new statistics during K/V permutation and applying the QPTC correction inside the merge kernel.

We directly compare a PyTorch reference with the fused implementation from Q/K/V input to the final attention output before output projection. The PyTorch reference replaces the fused QIKC transform, cluster-statistics computation, and QPTC-augmented exact/approximate output merge. The fused implementation reduces attention latency at all four tested sequence lengths, with absolute savings increasing from 3.18 to 37.25 ms. These measurements quantify the implementation benefit, while the matched-kernel ablation above shows that ResiSparse remains faster and more faithful than SVG-EAR.

Sequence lengthPyTorch reference (ms)Fused implementation (ms)Latency reduction (ms)Speedup
8,19211.568.383.181.38×
16,38418.9111.527.391.64×
32,76842.5526.6715.881.60×
75,600138.25101.0037.251.37×

(3) VBench results. VBench aggregates the six evaluated dimensions using the official normalization and weighting procedure. VBench evaluates perceptual and semantic video quality rather than agreement with dense-attention output. The objective of training-free sparse attention is to preserve the pretrained model's dense output without retraining; paired PSNR, SSIM, and LPIPS therefore provide more direct measures of this fidelity. A higher VBench score does not necessarily indicate closer reproduction of the dense output. Nevertheless, ResiSparse achieves comparable VBench results: 76.35 vs. 76.30 for SVG-EAR and 76.46 for SVG2 on Wan2.1-1.3B, and 77.08 vs. 77.09 for SVG-EAR on HunyuanVideo-13B. We have therefore revised Section 5 to describe VBench as comparable and to restrict improvement claims to dense-reference fidelity and efficiency.

AuthorsOpenReview ↗

Response to Reviewer HLRy (Part3)

Q2: QPTC's effect on numerator, denominator, and normalized-output errors, and the remaining value-weighting error.

Reply:

(1) QPTC's effect on attention errors. To isolate QPTC, we keep QIKC clustering and UMDR routing fixed and compare QPTC off versus on at the same 100 sampled locations. Following the reviewer's request for individual-query measurements, we compute relative L2L_2 error separately for each query and head before averaging. 90.91% of the individual queries have lower output error.

Mean per-query relative L2 errorQPTC offQPTC onReduction (%)Queries with lower error (%)
Numerator0.0376380.0352616.3193.85
Denominator0.0403610.03616410.40100.00
Normalized output0.0287830.0264877.9890.91

(2) Remaining value-weighting error. Proposition 1 (Section 3.2, Eqs. (5)–(6)) gives Yij=Zijvˉj+AijY_{ij}=Z_{ij}\bar v_j+A_{ij}. QPTC refines the estimate Z^ij\widehat Z_{ij} and its value-centroid contribution to the numerator, while leaving AijA_{ij} uncorrected. Based on Proposition 1 (Section 3.2), we decompose the normalized-output error and compare the two residual norms separately for each query and head. With QPTC enabled, the AA-related residual is larger in 20.96% of queries, most frequently in some layers. Thus, the contribution left uncorrected by QPTC varies across layers.

Q3: Additional baselines and quantitative temporal-flickering results.

Reply:

(1) Additional baselines. At the evaluated settings, ResiSparse is closer to dense outputs and faster than PISA [4] and Sol-Attn [5]. Given the limited rebuttal time and computational resources, the additional baseline comparison is conducted on Wan2.1-1.3B. All methods use the same prompts, seeds, generation protocol, and hardware. We set PISA's target density to 0.20 and Sol-Attn's threshold to τ=1.3\tau=1.3 to approximately match ResiSparse's attention density. Training-free sparse attention aims to reduce computation while preserving the pretrained model's dense-attention outputs. PISA scores above dense on VBench (77.50% vs. 76.80%) despite being 4.82 dB below ResiSparse in PSNR, with lower SSIM and higher LPIPS.

MethodDenoising time (s) ↓Density (%)PSNR (dB) ↑SSIM ↑LPIPS ↓VBench (%) ↑
Dense774100.00———76.80
PISA37527.9620.670.75300.237877.50
Sol-Attn36826.8118.760.65350.354974.77
ResiSparse35525.9925.490.84560.113276.35

(2) Temporal flickering. For temporal flickering, we have evaluated 165 paired videos per model using the official VBench implementation. Scores are multiplied by 100, and higher is better. ResiSparse achieves temporal-flickering scores comparable to dense attention, SVG2, and SVG-EAR across all three models. VBench [2] measures adjacent-frame pixel differences under a protocol with static-scene filtering. It is a useful reference but is not equally sensitive to every artifact: motion can also change pixels, and averaging over a full frame can dilute localized flicker. The paired videos in the supplementary repository's README provide direct visual evidence: in the video pair corresponding to the frames shown in Figure 1, SVG-EAR exhibits visible temporal artifacts that are absent from ResiSparse.

ModelDenseSVG2SVG-EARResiSparse
Wan2.1-1.3B98.1098.0798.1498.13
Wan2.1-14B96.7296.7296.7196.77
HunyuanVideo-13B98.5398.4998.5798.58

Manuscript revisions. We have revised the manuscript and highlighted the changes in blue. We added attention-error and residual analyses, matched-kernel component ablations, PyTorch-versus-fused timings, additional baselines, and temporal-flickering results. We also refined the claims about dense-reference fidelity, efficiency, and VBench quality.

References

[1] Zhou et al. SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing. arXiv:2603.08982, 2026.

[2] Huang et al. VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR 2024.

[3] Yang et al. Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation. NeurIPS 2025.

[4] Li et al. PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers. arXiv:2602.01077, 2026.

[5] Li et al. Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification. arXiv:2607.24027, 2026.

Reviewer_ZJzWOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. ResiSparse is a training-free sparse-attention method for video diffusion transformers. It computes selected blocks exactly and approximates the remaining blocks.
  2. It reduces centroid approximation error through query-informed key clustering, dispersion-based routing, and selected-channel second-order Taylor compensation.
  3. Tests on two Wan2.1 models and HunyuanVideo report improved dense-reference fidelity and 1.59-2.18x denoising speedups over dense attention.
Critical strengths
  1. The error decomposition separates attention-mass error from value-weighting error. It gives a clear reason for improving clustering, routing, and compensation (Section 3.2).
  2. The method makes concrete improvements within the existing framework. Table 1 reports better PSNR, SSIM, and LPIPS and lower latency than SVG-EAR on all three models.
Critical weaknesses
  1. The novelty is mainly a refinement of an existing framework. The paper should explain more directly how QPTC's second-order mass correction differs from PISA's first-order compensation and why this change matters. PISA is discussed but not evaluated.
  2. The paper lacks final-video ablations of QIKC, UMDR, and QPTC. Figures 3-4 provide useful local analyses, but use 50 clustering iterations, whereas generation mainly uses two warm-start iterations (Appendix C). Each component's contribution under the actual settings remains unclear.
Questions for authors
  1. What benefit does QPTC offer over PISA's first-order compensation? A direct comparison would help establish the methodological contribution.
  2. How do QIKC, UMDR, and QPTC individually and jointly affect final-video fidelity and denoising time at a controlled attention budget?
  3. Do the mechanism-level gains in Figures 3-4 remain under the two-iteration warm-start clustering used during generation?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ZJzW

We thank the reviewer for recognizing the theoretical motivation and the consistent improvements.

W1 & Q1: Novelty and the benefit of QPTC over PISA.

Reply:

(1) Theoretical contribution. Our contribution is a theoretical framework that decomposes errors in centroid-based attention compensation and guides three complementary designs for reducing them. Proposition 1 in Section 3.2 separates errors in the estimated denominator contributions from the omitted within-cluster value-weighting contribution. This decomposition motivates our three complementary components and provides a basis for analyzing further improvements to exact-or-approximate sparse attention in the future.

(2) Comparison with PISA. For PISA [1], the first-order denominator term vanishes because centered key deviations sum to zero; its compensation acts on value weighting in the numerator. QPTC adds a complementary second-order correction to both the numerator and denominator, estimated from selected-channel logit variance. We evaluated PISA vs. PISA + QPTC on Wan2.1-1.3B. Adding QPTC improves all three similarity metrics with a 0.53% increase in denoising time, while VBench remains essentially unchanged. These results show the additional benefit of QPTC alongside PISA's first-order compensation.

MethodDenoising time (s) ↓Density (%)PSNR (dB) ↑SSIM ↑LPIPS ↓VBench (%) ↑
Dense774100.00———76.80
PISA37527.9620.670.75300.237877.50
PISA + QPTC37727.9620.890.76870.228577.49

W2 & Q2: Component contribution ablations in video generation.

Reply:

Component contributions. Progressively adding QIKC, UMDR, and QPTC improves PSNR, SSIM, and LPIPS, with full ResiSparse achieving the best fidelity among these variants. Given the limited rebuttal time and computational resources, this additional ablation is conducted on Wan2.1-1.3B. All variants are evaluated under matched settings with the same kernels for shared operations, and their realized densities differ by at most 0.20 percentage points. Disabling all components gives SVG-EAR, and enabling all components gives full ResiSparse. For this kernel-matched comparison, we reran SVG-EAR with the optimized FlashInfer attention backend and K-means implementation used by ResiSparse; this reduced its denoising time from 379 s in Table 1 to 367 s without changing its algorithm or density.

QIKCUMDRQPTCDenoising time (s) ↓Density (%)PSNR (dB) ↑SSIM ↑LPIPS ↓VBench (%) ↑
———36726.0124.830.83630.124476.30
✓——36826.1925.150.83870.120176.49
✓✓—35425.9925.270.84220.116876.61
✓✓✓35525.9925.490.84560.113276.35

QIKC improves all three dense-reference fidelity metrics with minimal latency overhead, consistent with a better clustering basis. Adding UMDR further improves fidelity while providing most of the runtime reduction through lower-complexity cluster-level routing. QPTC corrects residual approximation errors and yields the best PSNR, SSIM, and LPIPS with little additional cost. Together, the three components improve both fidelity and efficiency: relative to matched-kernel SVG-EAR, full ResiSparse improves PSNR by 0.66 dB and reduces denoising time by 12 s (3.27%).

Q3: Effectiveness under two-iteration warm-start clustering.

Reply: The mechanism-level gains persist under two-iteration warm-start clustering. We evaluate all three components on Wan2.1-1.3B using five prompts, four denoising steps, and five layers:

  • QIKC: Relative to raw K-means, QIKC reduces relative logit MSE from 1 to 0.85182 (14.82% lower), with improvements at 99/100 sampled locations.

  • UMDR: At 20% candidate token-pair budget, UMDR captures 82.09% of the candidate blocks' total ∥Aij∥\|A_{ij}\|, compared with 83.61% for SVG-EAR and 79.72% for Top-pp, at approximately the same realized density. UMDR therefore approaches SVG-EAR's routing coverage while reducing complexity from O(CqNkd)O(C_qN_kd) to O(CqCkd)O(C_qC_kd).

  • QPTC: With clustering and routing fixed, QPTC reduces the NRMSE of the approximated key clusters' denominator contributions from 0.87024 to 0.82230 (5.51% lower).

Manuscript revisions. We have revised the manuscript and highlighted the changes in blue. The revisions clarify the distinction between PISA and QPTC and add their direct comparison, component ablations, and two-iteration warm-start analyses.

References

[1] Li et al. PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers. arXiv:2602.01077, 2026.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer ZJzW,

Thank you again for your thoughtful comments. Our response focuses on the following concerns:

  • Novelty and QPTC's benefit over PISA. We clarify how the error decomposition guides the three components, explain the difference between QPTC's second-order correction and PISA's first-order compensation, and compare PISA with and without QPTC.
  • Component contributions to final-video quality and runtime. We added ablations on Wan2.1-1.3B that progressively introduce QIKC, UMDR, and QPTC under matched settings and report video fidelity and denoising time.
  • Effectiveness with two clustering iterations. We repeated the mechanism analyses using the two-iteration warm-start clustering employed during generation.

The corresponding manuscript revisions are highlighted in blue. We would appreciate your thoughts on whether these clarifications and results address your concerns. Please let us know if any points need further clarification.

Best regards, The authors

AuthorsOpenReview ↗

Looking Forward to Your Feedback

Dear Reviewer,

As the discussion deadline approaches, we would appreciate your feedback on our responses and revisions so that we can address any remaining concerns before the deadline.

Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_ZJzWOpenReview ↗

Reply to the authors' rebuttal

Thank you for your response and the new experiments. They help me better understand the method. However, I still have three concerns:

  1. Novelty: I agree with Reviewer FHmR that the method is mainly an improvement over existing work. The three components improve clustering, routing, and compensation, but the overall framework is very similar to SVG-EAR.

  2. Experiments: Adding QPTC to PISA improves PSNR by only 0.22 dB, while the denoising time increases from 375 to 377 seconds. The new experiments are only on Wan2.1-1.3B.

  3. Presentation: Some symbols in Figure 2 were not shown correctly in the original paper. These basic mistakes raise concerns about the quality of the work.

For these reasons, I have changed my rating from 3 to 2.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Responses to Reviewers

We summarize our responses to the three reviewers' concerns below.

  • ZJzW: novelty, component contributions, and two-iteration warm-start clustering. Adding QPTC to PISA improves all three dense-reference fidelity metrics at unchanged density, supporting its complementary role alongside first-order compensation. We added ablation experiments to analyze the contributions of the three components. Additional analyses confirm that the mechanism-level gains persist under the two-iteration warm-start clustering used during generation.

  • HLRy: attention-output errors, component contributions, and kernel contributions. The step- and layer-wise analysis shows 18.95% lower normalized-output relative L2 error than SVG-EAR, averaged over 100 sampled locations. With clustering and routing fixed, QPTC reduces mean per-query relative L2 output error by 7.98%. After matching shared kernels, ResiSparse improves PSNR by 0.66 dB and reduces denoising time from 367 to 355 s. PyTorch-versus-fused timings quantify implementation gains. We also added PISA/Sol-Attn comparisons and quantitative temporal-flickering results, with comparable flickering scores across methods on all three models.

  • FHmR: theoretical validity, presentation, and evaluation. We clarified the original conditional error-reduction analysis, made its Taylor-remainder bound explicit, and tested QPTC across sampled denoising steps and logit-variance ranges. We corrected Figure 2's rendering and identified the channel-selection definitions already present in the original formulas and Algorithm 3. The added PISA/Sol-Attn comparisons show better dense-reference fidelity at lower latency and similar realized densities; all six evaluated VBench dimensions are reported.

AuthorsOpenReview ↗

Main Contributions of ResiSparse

We thank the reviewers and chairs for their time and consideration. Below are the main contributions of ResiSparse.

  • Theoretical framework. Proposition 1 separates errors in estimated denominator contributions from omitted within-cluster value weighting. This decomposition explains the errors introduced by centroid-based compensation and provides a framework for improving exact-or-approximate sparse attention.
  • Three complementary algorithms. Guided by this analysis, QIKC improves key clustering using query energy; UMDR prioritizes exact computation through a cost-normalized residual bound while reducing routing complexity; QPTC corrects estimated numerator and denominator contributions using selected-channel second-order statistics.
  • Quality and efficiency. Across all three models, ResiSparse improves PSNR, SSIM, and LPIPS and reduces denoising time relative to SVG-EAR, with comparable VBench scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.