- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Rectified SpaAttn is a training-free method for accelerating DiT-based visual generation. It addresses two effects of sparsification: amplified attention weights on retained interactions and lost contributions from skipped interactions. IPAR constructs a pooled full-attention approximation for weight calibration, while GAPR selectively compensates skipped blocks using an estimated gain–error criterion. Experiments include multiple generative models, ablations, and human preference evaluation, reporting up to 3.33× and 2.08× end-to-end speedups over FlashAttention-2 on HunyuanVideo and Wan2.1-I2V, respectively.
Critical strengths
-
Practical acceleration without retraining. The method supports efficient block-sparse computation and integration with caching and distilled models. The reported results demonstrate useful quality–latency trade-offs across video-generation workloads.
-
Informative ablations. Starting from a Jenga-based sparse baseline [1], Table 4 shows that naive pooling correction can worsen quality, whereas IPAR and GAPR improve it with approximately 1.4% additional latency. This supports the practical importance of an accurate compensation reference.
[1] Zhang et al. Training-Free Efficient Video Generation via Dynamic Token Carving. NeurIPS, 2025.
Critical weaknesses
-
Novelty is insufficiently established against closely related work. Re-ttention [1] already identifies sparse-softmax normalization distortion, rescales retained weights using the sparse/full denominator ratio, and recovers omitted contributions through residual caching. Thus, both the central diagnosis and the “rescale retained interactions + recover skipped contributions” structure have precedent. Rectified SpaAttn instead estimates corrections from current-step pooled interactions, which is a meaningful implementation distinction, but Re-ttention is neither cited nor evaluated. SVG-EAR [2] also provides training-free centroid compensation and error-aware routing. Together, these works require a clearer account of the proposed method’s incremental contribution and matched-budget comparisons.
-
GAPR’s mathematical justification contains substantive problems. Equation (13) equates attention-weight error with a softmax of logit differences, which generally does not hold: identical logits yield zero error but a nonzero softmax of their difference. Moreover, under the mean-pooling definitions in Equation (14), signed logit residuals sum to zero over a complete visual query–key block. The absolute value of their sum in Equation (16) therefore cannot measure within-block approximation error, making the gate appear degenerate as written. These issues undermine the claimed necessary-and-sufficient guarantee and raise questions about consistency between the equations and implementation.
[1] Chen et al. Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshape. NeurIPS, 2025.
[2] Zhou et al. SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing. arXiv:2603.08982, 2026.
Questions for authors
-
What is the precise mathematical contribution beyond Re-ttention and SVG-EAR? Can matched quality–latency comparisons, including dense refresh costs, establish the benefit of current-step pooling over cached statistics?
-
What error estimator does the implementation actually use? How does it avoid signed-residual cancellation in Equation (16), and does it match the reported experiments?
-
Can the authors provide a valid derivation of Equation (13), specify the objective and assumptions behind the claimed guarantee, and show whether GAPR outperforms unconditional compensation with the sparse mask fixed?
6 comments
Response to Reviewer ZV4u (2/3)
W2: GAPR’s mathematical justification contains substantive problems. Equation (13) equates attention-weight error with a softmax of logit differences, which generally does not hold. The absolute value of their sum in Equation (16) therefore cannot measure within-block approximation error, making the gate appear degenerate as written.
Thank you for your valuable and constructive feedback.
As you correctly pointed out, the attention-weight error is not equivalent to the Softmax of logit differences. In the original Eq. (13), the final equality was intended to express that we use pre-Softmax logit discrepancies as a surrogate for attention-weight errors, rather than implying strict mathematical equivalence. Regarding Eq. (16), our intended formulation assumes that the logit residuals and pooled logits are evaluated in terms of their element-wise absolute values before summation. Specifically, was intended to represent the absolute magnitude of the logit residual induced by Q/K pooling in Eq. (14). The outer absolute-value notation in the original Eq. (16) was introduced to maintain notational consistency with the ideal decision criterion .
We apologize for these inaccurate and unclear formulations. Following your comments, we have corrected Eq. (13) and (16) in the revised manuscript and further clarified the design objective and implementation of GAPR. Specifically, for a non-critical block, the restored attention mass and the pooling error are defined as:
$G_{n,m}=\sum_{i\in B_n^q}\sum_{j\in B_m^k}a_{i,j}^{\mathrm{pool}},\qquad E_{n,m}=\sum_{i\in B_n^q}\sum_{j\in B_m^k}|a_{i,j}-a_{i,j}^{\mathrm{pool}}|$ .
GAPR targets the comparison . Evaluating this criterion directly requires dense token interactions and normalized attention weights, undermining computational efficiency. For a fixed query, softmax preserves the ordering of scores and attention weights: . Furthermore, the relationship between attention scores and relative attention weights can be expressed as: . Accordingly, score perturbations reflect changes in relative attention preferences. Prior works, such as Top-k Attention and QUEST, have also leveraged score estimates to screen attention interactions. These properties and precedents motivate our surrogate design. Therefore, we use scores and their deviations in the score domain as surrogates for this criterion:
.
Our implementation is consistent with the corrected Eq. (16), where the absolute values are computed before summation. Please refer to lines 27, 33, and 38 of rectified_spaattn/gapr_mask.py in our supplementary code.
We acknowledge that the proposed surrogate is empirically motivated. Nevertheless, the statistical evaluations in Appendix J demonstrate high rectification decision agreement with the original criterion across models, denoising steps, and attention layers. Appendix J also provides visualizations of the rectification masks. These results empirically support the surrogate's decision consistency.
We sincerely thank you again for your valuable feedback.
Q1: What is the precise mathematical contribution beyond Re-ttention and SVG-EAR? Can matched quality–latency comparisons, including dense refresh costs, establish the benefit of current-step pooling over cached statistics?
Thank you for your valuable comments.
Compared with Re-ttention and SVG-EAR, our contribution lies in mathematically characterizing the inherent systematic biases introduced by attention sparsification. Building upon this insight, we develop a rectification framework based on current-step Q/K interactions, with dedicated correction mechanisms for both retained and discarded attention.
To demonstrate the advantages of current-step pooling in Rectified SpaAttn over the cached statistics used in Re-tention, we conduct both quality-matched and latency-matched comparisons. As shown in Table, current-step pooling achieves lower inference latency at comparable generation quality and higher generation quality at comparable inference latency.
| Method | ImageQual ↑ | SubConsist ↑ | Latency ↓ | Speedup ↑ |
|---|---|---|---|---|
| CogVideoX-2B | 65.72 | 94.97 | 54.43 s | 1.00× |
| Re-ttention | 64.87 | 94.80 | 29.78 s | 1.83× |
| Rectified SpaAttn | 65.38 | 94.90 | 29.10 s | 1.87× |
| Rectified SpaAttn | 64.90 | 94.77 | 25.22 s | 2.16× |
Response to Reviewer ZV4u (3/3)
Q2: What error estimator does the implementation actually use? How does it avoid signed-residual cancellation in Equation (16), and does it match the reported experiments?
Thank you for your valuable comments.
As shown in the revised Eq. (16): , in practice, we first take the element-wise absolute values of the logit residuals before summation, without the issue of signed-residual cancellation. We apologize for the lack of clarity in our previous formulation. All experiments reported in this paper were conducted using an implementation consistent with the revised Eq. (16).
Q3: Can the authors provide a valid derivation of Equation (13), specify the objective and assumptions behind the claimed guarantee, and show whether GAPR outperforms unconditional compensation with the sparse mask fixed?
Thank you for your valuable and constructive comments.
Following your feedback, we have revised Eq. (13) by removing the final equality and explicitly clarifying that the score-domain criterion serves as a surrogate for the ideal probability-domain criterion. Statistical experiments across different models, denoising steps, and attention layers demonstrate high consistency between the surrogate and the ideal criterion in rectification decisions, providing empirical support for its effectiveness.
The paper provided ablation studies in Table 4, Fig. 7, and Fig. 11, demonstrating the advantages of GAPR over no compensation in terms of both generation quality and attention approximation error.
To further demonstrate the advantages of GAPR over unconditional compensation, following your constructive suggestion, we conduct additional comparisons with random compensation. Specifically, random compensation randomly selects the same number of attention blocks as GAPR for rectification, ensuring an identical rectification budget. Consistent with Fig. 11, we evaluate the L1 error between sparse and full attention outputs.
The table below presents the results on Wan2.1-T2V, where the baseline refers to Jenga without attention rectification. GAPR consistently outperforms random compensation across all sparsity levels, whereas random compensation can even introduce larger attention errors than the baseline at high sparsity ratio. This is because random compensation may introduce errors that outweigh its compensation benefits.
| Method / Sparsity | 60% | 65% | 70% | 75% | 80% | 85% | 90% |
|---|---|---|---|---|---|---|---|
| Baseline | 322.05 | 370.14 | 443.77 | 519.87 | 688.31 | 910.76 | 1185.73 |
| Random Compensation | 311.28 | 358.47 | 436.58 | 510.29 | 695.32 | 943.81 | 1294.26 |
| GAPR | 272.56 | 323.42 | 368.82 | 423.96 | 516.36 | 607.71 | 754.73 |
Response to Reviewer ZV4u (1/3)
Many thanks for the reviewer's valuable comments, which help us a lot to improve our work. We address your concerns as follows.
Thank you for your important and constructive feedback.
We acknowledge that Re-ttention has investigated the normalization bias in sparse Softmax, and proposed rescaling retained attention weights and caching residuals to recover discarded contributions. Although both methods aim to improve sparse attention approximation, we would like to emphasize that Rectified SpaAttn differs substantially from Re-ttention in both its core design philosophy and implementation mechanism.
Core Design Philosophy. Re-ttention primarily exploits temporal redundancy across adjacent denoising steps, caching and reusing historical normalization ratios and residuals to approximate the current attention output. In contrast, Rectified SpaAttn starts from a theoretical analysis of sparse attention errors, revealing the systematic biases introduced by sparsification in both retained and discarded attention contributions. Motivated by this insight, It constructs an implicit full-attention reference from pooled Q/K interactions to jointly guide the rectification of these biases. Thus, while sharing a similar optimization objective, the two methods follow distinct design principles and underlying motivations.
Implementation Mechanism. Re-ttention relies on periodic full-attention computation to cache normalization ratios and residuals for reuse in subsequent denoising steps. In contrast, Rectified SpaAttn requires neither historical caching nor periodic full-attention computation. Instead, it constructs a lightweight global attention estimate directly from pooled Q/K interactions at the current step, which is then used to rectify both retained and discarded attention contributions. The key distinction lies in their information sources and rectification mechanisms: Re-ttention exploits temporal information across denoising steps, whereas Rectified SpaAttn performs current-step bias rectification using Q/K interactions.
Following your suggestion, we have added Re-ttention in the Related Work section (Lines 172–174 of the revised manuscript) and introduced Appendix G to further clarify these distinctions in both design philosophy and implementation. Regarding SVG-EAR, we have already provided a detailed comparison in Appendix E of the original manuscript (Appendix F in the revised version). Please refer to Appendices F and G of the revised manuscript for further details.
We have conducted additional comparisons under comparable computational budgets, as summarized in the table below. For Re-ttention, we adopt CogVideoX-2B with its default generation settings (720 × 480 resolution, 48 frames). The comparison with SVG-EAR was already included in the main experiments and is reproduced here for convenience. We sincerely thank you again for your valuable feedback.