acceptodds
Under review as a conference paper at ICLR 2027

Rectified SpaAttn: Revisiting Attention Sparsity for Efficient Video Generation

Abstract

Diffusion Transformers dominate video generation, but the quadratic complexity of attention computation introduces substantial latency, hindering real-world applications. Attention sparsity reduces computational costs by preserving interactions with critical tokens while discarding those with non-critical tokens. Existing methods primarily focus on token reordering or importance metric design to identify critical tokens, but often suffer performance degradation at high sparsity. In this paper, we revisit attention sparsity and theoretically show that existing sparsity paradigms induce systematic biases in attention allocation: (1) excessive focus on critical tokens amplifies their attention weights; (2) complete neglect of non-critical tokens causes the loss of relevant attention weights. To address these issues, we propose Rectified SpaAttn, which rectifies sparse attention allocation with implicit full attention reference, thereby enhancing the alignment between sparse and full attention maps. Specifically, we implicitly capture the full-attention distribution of critical and non-critical tokens by pooled query-key interactions, which is then used to rectify sparse attention allocation: (1) for critical tokens, we show that their bias is proportional to the sparse attention weights, with the ratio governed by the amplified weights. Accordingly, we propose Isolated-Pooling Attention Reallocation, which calculates accurate rectification factors by reallocating multimodal pooled weights. (2) for non-critical tokens, recovering attention weights from the pooled query-key yields attention gains but also introduces pooling errors. Therefore, we propose Gain-Aware Pooling Rectification, which ensures that the rectified gain consistently surpasses the induced error. Moreover, we customize and integrate the Rectified SpaAttn kernel using Triton, achieving up to 3.33× and 2.08× speedups on HunyuanVideo and Wan 2.1, respectively, while maintaining generation quality even at high sparsity.

Primary area:
optimization
Keywords:
Attention Sparsity, Video Generation, Diffusion Transformers, Efficient AI, TriTon

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZV4uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Rectified SpaAttn is a training-free method for accelerating DiT-based visual generation. It addresses two effects of sparsification: amplified attention weights on retained interactions and lost contributions from skipped interactions. IPAR constructs a pooled full-attention approximation for weight calibration, while GAPR selectively compensates skipped blocks using an estimated gain–error criterion. Experiments include multiple generative models, ablations, and human preference evaluation, reporting up to 3.33× and 2.08× end-to-end speedups over FlashAttention-2 on HunyuanVideo and Wan2.1-I2V, respectively.

Critical strengths
  1. Practical acceleration without retraining. The method supports efficient block-sparse computation and integration with caching and distilled models. The reported results demonstrate useful quality–latency trade-offs across video-generation workloads.

  2. Informative ablations. Starting from a Jenga-based sparse baseline [1], Table 4 shows that naive pooling correction can worsen quality, whereas IPAR and GAPR improve it with approximately 1.4% additional latency. This supports the practical importance of an accurate compensation reference.

[1] Zhang et al. Training-Free Efficient Video Generation via Dynamic Token Carving. NeurIPS, 2025.

Critical weaknesses
  1. Novelty is insufficiently established against closely related work. Re-ttention [1] already identifies sparse-softmax normalization distortion, rescales retained weights using the sparse/full denominator ratio, and recovers omitted contributions through residual caching. Thus, both the central diagnosis and the “rescale retained interactions + recover skipped contributions” structure have precedent. Rectified SpaAttn instead estimates corrections from current-step pooled interactions, which is a meaningful implementation distinction, but Re-ttention is neither cited nor evaluated. SVG-EAR [2] also provides training-free centroid compensation and error-aware routing. Together, these works require a clearer account of the proposed method’s incremental contribution and matched-budget comparisons.

  2. GAPR’s mathematical justification contains substantive problems. Equation (13) equates attention-weight error with a softmax of logit differences, which generally does not hold: identical logits yield zero error but a nonzero softmax of their difference. Moreover, under the mean-pooling definitions in Equation (14), signed logit residuals sum to zero over a complete visual query–key block. The absolute value of their sum in Equation (16) therefore cannot measure within-block approximation error, making the gate appear degenerate as written. These issues undermine the claimed necessary-and-sufficient guarantee and raise questions about consistency between the equations and implementation.

[1] Chen et al. Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshape. NeurIPS, 2025.

[2] Zhou et al. SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing. arXiv:2603.08982, 2026.

Questions for authors
  1. What is the precise mathematical contribution beyond Re-ttention and SVG-EAR? Can matched quality–latency comparisons, including dense refresh costs, establish the benefit of current-step pooling over cached statistics?

  2. What error estimator does the implementation actually use? How does it avoid signed-residual cancellation in Equation (16), and does it match the reported experiments?

  3. Can the authors provide a valid derivation of Equation (13), specify the objective and assumptions behind the claimed guarantee, and show whether GAPR outperforms unconditional compensation with the sparse mask fixed?

6 comments
AuthorsOpenReview ↗

Response to Reviewer ZV4u (1/3)

Many thanks for the reviewer's valuable comments, which help us a lot to improve our work. We address your concerns as follows.


W1: Novelty is insufficiently established against closely related work. Re-ttention already identifies sparse-softmax normalization distortion, rescales retained weights using the sparse/full denominator ratio, and recovers omitted contributions through residual caching. SVG-EAR also provides training-free centroid compensation and error-aware routing. Together, these works require a clearer account of the proposed method’s incremental contribution and matched-budget comparisons.

Thank you for your important and constructive feedback.

We acknowledge that Re-ttention has investigated the normalization bias in sparse Softmax, and proposed rescaling retained attention weights and caching residuals to recover discarded contributions. Although both methods aim to improve sparse attention approximation, we would like to emphasize that Rectified SpaAttn differs substantially from Re-ttention in both its core design philosophy and implementation mechanism.

  • Core Design Philosophy. Re-ttention primarily exploits temporal redundancy across adjacent denoising steps, caching and reusing historical normalization ratios and residuals to approximate the current attention output. In contrast, Rectified SpaAttn starts from a theoretical analysis of sparse attention errors, revealing the systematic biases introduced by sparsification in both retained and discarded attention contributions. Motivated by this insight, It constructs an implicit full-attention reference from pooled Q/K interactions to jointly guide the rectification of these biases. Thus, while sharing a similar optimization objective, the two methods follow distinct design principles and underlying motivations.

  • Implementation Mechanism. Re-ttention relies on periodic full-attention computation to cache normalization ratios and residuals for reuse in subsequent denoising steps. In contrast, Rectified SpaAttn requires neither historical caching nor periodic full-attention computation. Instead, it constructs a lightweight global attention estimate directly from pooled Q/K interactions at the current step, which is then used to rectify both retained and discarded attention contributions. The key distinction lies in their information sources and rectification mechanisms: Re-ttention exploits temporal information across denoising steps, whereas Rectified SpaAttn performs current-step bias rectification using Q/K interactions.

Following your suggestion, we have added Re-ttention in the Related Work section (Lines 172–174 of the revised manuscript) and introduced Appendix G to further clarify these distinctions in both design philosophy and implementation. Regarding SVG-EAR, we have already provided a detailed comparison in Appendix E of the original manuscript (Appendix F in the revised version). Please refer to Appendices F and G of the revised manuscript for further details.

We have conducted additional comparisons under comparable computational budgets, as summarized in the table below. For Re-ttention, we adopt CogVideoX-2B with its default generation settings (720 × 480 resolution, 48 frames). The comparison with SVG-EAR was already included in the main experiments and is reproduced here for convenience. We sincerely thank you again for your valuable feedback.

MethodImageQual ↑SubConsist ↑Latency ↓Speedup ↑
CogVideoX-2B65.7294.9754.43 s1.00×
Re-ttention64.8794.8029.78 s1.83×
Rectified SpaAttn65.3894.9029.10 s1.87×
MethodImageQual ↑SubConsist ↑Latency ↓Speedup ↑
HunyuanVideo-13B67.3196.332425 s1.00×
SVG-EAR65.8896.77984 s2.46×
Rectified SpaAttn65.9996.82970 s2.50×
AuthorsOpenReview ↗

Response to Reviewer ZV4u (2/3)

W2: GAPR’s mathematical justification contains substantive problems. Equation (13) equates attention-weight error with a softmax of logit differences, which generally does not hold. The absolute value of their sum in Equation (16) therefore cannot measure within-block approximation error, making the gate appear degenerate as written.

Thank you for your valuable and constructive feedback.

As you correctly pointed out, the attention-weight error is not equivalent to the Softmax of logit differences. In the original Eq. (13), the final equality was intended to express that we use pre-Softmax logit discrepancies as a surrogate for attention-weight errors, rather than implying strict mathematical equivalence. Regarding Eq. (16), our intended formulation assumes that the logit residuals and pooled logits are evaluated in terms of their element-wise absolute values before summation. Specifically, Δsi,j\Delta s_{i,j} was intended to represent the absolute magnitude of the logit residual induced by Q/K pooling in Eq. (14). The outer absolute-value notation in the original Eq. (16) was introduced to maintain notational consistency with the ideal decision criterion ∣G∣>∣E∣|G|>|E|.

We apologize for these inaccurate and unclear formulations. Following your comments, we have corrected Eq. (13) and (16) in the revised manuscript and further clarified the design objective and implementation of GAPR. Specifically, for a non-critical block, the restored attention mass and the pooling error are defined as:

$G_{n,m}=\sum_{i\in B_n^q}\sum_{j\in B_m^k}a_{i,j}^{\mathrm{pool}},\qquad E_{n,m}=\sum_{i\in B_n^q}\sum_{j\in B_m^k}|a_{i,j}-a_{i,j}^{\mathrm{pool}}|$ .

GAPR targets the comparison Gn,m>En,mG_{n,m}>E_{n,m}. Evaluating this criterion directly requires dense token interactions and normalized attention weights, undermining computational efficiency. For a fixed query, softmax preserves the ordering of scores and attention weights: ai,j>ai,k⇔si,j>si,ka_{i,j}>a_{i,k}\Leftrightarrow s_{i,j}>s_{i,k}. Furthermore, the relationship between attention scores and relative attention weights can be expressed as: log⁡ai,jai,k=si,j−si,k\log\frac{a_{i,j}}{a_{i,k}}=s_{i,j}-s_{i,k}. Accordingly, score perturbations reflect changes in relative attention preferences. Prior works, such as Top-k Attention and QUEST, have also leveraged score estimates to screen attention interactions. These properties and precedents motivate our surrogate design. Therefore, we use scores and their deviations in the score domain as surrogates for this criterion:

∑i∈Bnq∑j∈Bmk∣si,jpool∣>∑i∈Bnq∑j∈Bmk∣Δsi,j∣\sum_{i\in B_n^q}\sum_{j\in B_m^k}|s_{i,j}^{\mathrm{pool}}|>\sum_{i\in B_n^q}\sum_{j\in B_m^k}|\Delta s_{i,j}| .

Our implementation is consistent with the corrected Eq. (16), where the absolute values are computed before summation. Please refer to lines 27, 33, and 38 of rectified_spaattn/gapr_mask.py in our supplementary code. We acknowledge that the proposed surrogate is empirically motivated. Nevertheless, the statistical evaluations in Appendix J demonstrate high rectification decision agreement with the original criterion across models, denoising steps, and attention layers. Appendix J also provides visualizations of the rectification masks. These results empirically support the surrogate's decision consistency. We sincerely thank you again for your valuable feedback.

Q1: What is the precise mathematical contribution beyond Re-ttention and SVG-EAR? Can matched quality–latency comparisons, including dense refresh costs, establish the benefit of current-step pooling over cached statistics?

Thank you for your valuable comments.

Compared with Re-ttention and SVG-EAR, our contribution lies in mathematically characterizing the inherent systematic biases introduced by attention sparsification. Building upon this insight, we develop a rectification framework based on current-step Q/K interactions, with dedicated correction mechanisms for both retained and discarded attention.

To demonstrate the advantages of current-step pooling in Rectified SpaAttn over the cached statistics used in Re-tention, we conduct both quality-matched and latency-matched comparisons. As shown in Table, current-step pooling achieves lower inference latency at comparable generation quality and higher generation quality at comparable inference latency.

MethodImageQual ↑SubConsist ↑Latency ↓Speedup ↑
CogVideoX-2B65.7294.9754.43 s1.00×
Re-ttention64.8794.8029.78 s1.83×
Rectified SpaAttn65.3894.9029.10 s1.87×
Rectified SpaAttn64.9094.7725.22 s2.16×
AuthorsOpenReview ↗

Response to Reviewer ZV4u (3/3)

Q2: What error estimator does the implementation actually use? How does it avoid signed-residual cancellation in Equation (16), and does it match the reported experiments?

Thank you for your valuable comments.

As shown in the revised Eq. (16): ∑i∈Bnq∑j∈Bmk∣si,jpool∣>∑i∈Bnq∑j∈Bmk∣Δsi,j∣\sum_{i\in B_n^q}\sum_{j\in B_m^k}|s_{i,j}^{\mathrm{pool}}|>\sum_{i\in B_n^q}\sum_{j\in B_m^k}|\Delta s_{i,j}|, in practice, we first take the element-wise absolute values of the logit residuals before summation, without the issue of signed-residual cancellation. We apologize for the lack of clarity in our previous formulation. All experiments reported in this paper were conducted using an implementation consistent with the revised Eq. (16).

Q3: Can the authors provide a valid derivation of Equation (13), specify the objective and assumptions behind the claimed guarantee, and show whether GAPR outperforms unconditional compensation with the sparse mask fixed?

Thank you for your valuable and constructive comments.

Following your feedback, we have revised Eq. (13) by removing the final equality and explicitly clarifying that the score-domain criterion serves as a surrogate for the ideal probability-domain criterion. Statistical experiments across different models, denoising steps, and attention layers demonstrate high consistency between the surrogate and the ideal criterion in rectification decisions, providing empirical support for its effectiveness.

The paper provided ablation studies in Table 4, Fig. 7, and Fig. 11, demonstrating the advantages of GAPR over no compensation in terms of both generation quality and attention approximation error.

To further demonstrate the advantages of GAPR over unconditional compensation, following your constructive suggestion, we conduct additional comparisons with random compensation. Specifically, random compensation randomly selects the same number of attention blocks as GAPR for rectification, ensuring an identical rectification budget. Consistent with Fig. 11, we evaluate the L1 error between sparse and full attention outputs.

The table below presents the results on Wan2.1-T2V, where the baseline refers to Jenga without attention rectification. GAPR consistently outperforms random compensation across all sparsity levels, whereas random compensation can even introduce larger attention errors than the baseline at high sparsity ratio. This is because random compensation may introduce errors that outweigh its compensation benefits.

Method / Sparsity60%65%70%75%80%85%90%
Baseline322.05370.14443.77519.87688.31910.761185.73
Random Compensation311.28358.47436.58510.29695.32943.811294.26
GAPR272.56323.42368.82423.96516.36607.71754.73
Reviewer_zvUYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3
Read review
Paper summary

This work argues that existing sparse attention paradigms lead to systematic biases in attention allocation and proposes Rectified SpaAttn to correct this by utilizing an implicit full-attention reference. The effectiveness of the method was validated across multiple foundation models.

Critical strengths
  1. The experiments in this paper are comprehensive, involving quantitative testing on HunyuanVideo, Wan2.1, and Flux.
  2. The motivation of this paper is clear. This paper rectifies sparse attention allocation with implicit full attention reference,
Critical weaknesses
  1. This paper contains a large number of equations, yet some parts are difficult to understand or ambiguous. There may be issues with the derivation of certain equations and the validity of the equalities presented. For instance, regarding the final equality sign in line L353 of Equation (13), the difference between two softmax outputs is not equivalent to applying the softmax function to the difference between the two sets of inputs.
  2. Some of the methodological descriptions in the paper lack precision, and symbol definitions are unclear; this risks overstating the method's contributions and making it harder to understand. For instance, the statement in lines 363–364 that the "design corresponds to a necessary and sufficient condition" is problematic, as the condition G>EG>E is not equivalent to a reduction in attention weight error.
Questions for authors
  1. Clarify the validity of the final equality in Equation 13.
  2. Clarify the rationale behind the sufficient and necessary conditions in lines 363–364.
1 comment
AuthorsOpenReview ↗

Response to Reviewer zvUY

Many thanks for the reviewer's valuable comments, which help us a lot to improve our work. We address your concerns as follows.


W1 & Q1: This paper contains a large number of equations, yet some parts are difficult to understand or ambiguous. There may be issues with the derivation of certain equations and the validity of the equalities presented. Clarify the validity of the final equality in Equation 13.

We sincerely thank you for your constructive and valuable feedback.

In the original Eq. (13), the final equality was intended to express our use of pre-softmax score deviations as a surrogate for attention-weight deviations, rather than implying strict mathematical equivalence. We apologize for this misleading formulation. Following your feedback, we have corrected Eq. (13) in the revised manuscript by removing the final equality and explicitly clarifying that the score-domain criterion serves as a surrogate for the probability-domain criterion. Apart from Eq. (13), we have also carefully verified the correctness of the remaining equations.

As you noted, our method involves numerous equations with superscripts and subscripts to distinguish different quantities and comprehensively describe the methodology. To improve clarity and readability, following your constructive suggestion, we have added a notation table in Appendix A of the revised manuscript for your reference.

W2 & Q2: Some of the methodological descriptions in the paper lack precision, and symbol definitions are unclear; this risks overstating the method's contributions and making it harder to understand. Clarify the rationale behind the sufficient and necessary conditions in lines 363–364.

Thank you for your valuable feedback.

Following your feedback, we have revised this section to clarify the design objective and implementation of GAPR. Specifically, we have removed the original statement that the "design corresponds to a necessary and sufficient condition" to avoid overstating the theoretical guarantees.

The objective of GAPR is to rectify a discarded attention block when the expected gain in attention weights exceeds the approximation error introduced by pooling. To achieve this efficiently, we introduce a lightweight score-domain surrogate for the ideal probability-domain criterion, motivated by the order-preserving property of Softmax and established score-based attention screening practices in prior work. Furthermore, our experiments across different models, denoising timesteps, and attention layers demonstrate the effectiveness and generalizability of this surrogate.

We would like to further emphasize that the key contribution of this work lies in theoretically identifying the fundamental source of sparse attention errors: sparsification introduces systematic biases in both retained and discarded attention contributions. Building upon this insight, we propose a bias rectification approach that leverages an implicit full-attention reference to compensate for these errors. Unlike prior methods that primarily focus on optimizing sparsity masks to minimize discarded attention information, our approach seeks to recover lost attention information through bias rectification. Although the proposed rectification cannot completely eliminate sparsification-induced biases, our experimental results demonstrate that it effectively mitigates these biases and substantially improves generation quality. We sincerely thank you again for your valuable feedback.

Reviewer_gwVBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Rectified SpaAttn, a method for accelerating video generation via sparse attention that requires no additional training. Addressing the issue where sparsification not only discards information from skipped tokens but also inflates the weights of retained tokens due to softmax re-normalization, this approach constructs a low-cost, approximate reference of the full attention mechanism through the pooling of query–key interactions. It employs IPAR (Interaction-Pooling-Aware Rectification) to apply distinct pooling granularities for text and video tokens, thereby correcting the weights of the retained components. Furthermore, it utilizes GAPR (Gain-Aware Pooling-Error Rectification) to estimate the trade-off between the recovered contribution and the pooling-induced error, selectively restoring the influence of the skipped interactions.

Critical strengths
  1. The motivation to correct probability distortions caused by softmax renormalization during attention sparsification is practically important.

  2. The paper is clearly written and easy to follow.

Critical weaknesses
  1. The paper primarily evaluates video quality using Vision Reward and selected VBench metrics. However, comparable perceptual quality does not necessarily imply fidelity to the outputs generated with full attention. Since attention sparsification methods generally aim to reduce sparsification-induced errors and preserve consistency with full attention, I recommend additionally reporting PSNR, SSIM, and LPIPS under identical prompts, random seeds, and sampling settings.

  2. Equation (13) on page 7 converts ∣aij−aijpool∣|a_{ij}-a^{\mathrm{pool}}_{ij}| into softmax⁡(Δsij)\operatorname{softmax}(\Delta s_{ij}), justified by the approximate equality of the softmax normalization denominators for the true and pooled attention. However, due to the nonlinearity of softmax, the following quantities are generally not equal:

    ∣softmax⁡(s)−softmax⁡(spool)∣≠softmax⁡(s−spool).\left|\operatorname{softmax}(s)-\operatorname{softmax}(s^{\mathrm{pool}})\right| \neq \operatorname{softmax}(s-s^{\mathrm{pool}}).

    For example, when s=spools=s^{\mathrm{pool}}, the actual probability error is zero, whereas softmax⁡(0)\operatorname{softmax}(0) is a nonzero uniform distribution. This issue persists even when the two normalization denominators are exactly equal. Therefore, the discussion of approximate denominator equivalence in Appendix A is insufficient to justify Equation (13).

  3. The related-work discussion of caching-based acceleration is incomplete and omits several relevant methods, such as:

[1] MixCache: Mixture-of-Cache for Video Diffusion Transformer Acceleration

[2] EchoCache: Energy-Guided Cross-Modal Caching for Efficient Audio-Driven Video Generation

[3] Magcache: Fast video generation with magnitude-aware cache

[4] Omnicache: A trajectory-oriented global perspective on training-free cache reuse for diffusion transformer models

[5] Ab-cache: Training-free acceleration of diffusion models via adams-bashforth cached feature reuse

Questions for authors

See Critical Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer gwVB

Many thanks for the reviewer's valuable comments, which help us a lot to improve our work. We address your concerns as follows.


W1: The paper primarily evaluates video quality using Vision Reward and selected VBench metrics. I recommend additionally reporting PSNR, SSIM, and LPIPS under identical prompts, random seeds, and sampling settings.

Thank you for your valuable and constructive comments.

Following your suggestion, we further evaluate PSNR, SSIM, and LPIPS on Wan2.1 and compare our method with SVG2, the state-of-the-art sparse attention method reporting the best performance on this model. As shown in the table below, our method achieves higher generation consistency under comparable computational costs. Furthermore, the image generation experiments in Table 8 of the original manuscript (Table 9 in the revised version) provide a comprehensive evaluation in terms of PSNR, SSIM, and LPIPS, further demonstrating the higher generation consistency of our method.

ModelMethodPSNR↑SSIM↑LPIPS↓Speedup↑
Wan2.1-T2VSVG223.6820.7890.1961.89
Ours24.0300.8220.1381.92
Wan2.1-I2VSVG224.5100.8120.1791.84
Ours24.8600.8260.1581.89

W2: Equation (13) on page 7 converts ∣ai,j−ai,jpool∣| a_{i,j} - a_{i,j}^{pool} | into softmax(Δsi,j)\text{softmax}(\Delta s_{i,j}). However, due to the nonlinearity of softmax, the ∣ai,j−ai,jpool∣=softmax(Δsi,j)| a_{i,j} - a_{i,j}^{pool} |=\text{softmax}(\Delta s_{i,j}) in Equation (13) is not justified.

Thank you for your valuable and constructive suggestion.

As you correctly noted, the nonlinearity of softmax means that ∣ai,j−ai,jpool∣=softmax(Δsi,j)|a_{i,j}-a_{i,j}^{\mathrm{pool}}|=\mathrm{softmax}(\Delta s_{i,j}) does not hold. In Eq.~(13), the equality was intended to express our use of score deviations as a surrogate for attention-weight deviations, rather than implying strict mathematical equivalence. We thank the reviewer for identifying this issue and apologize for the incorrect equality in Eq. (13). Specifically, for a non-critical block, the restored attention mass and the pooling error are defined as:

Gn,m=∑i∈Bnq∑j∈Bmkai,jpool,En,m=∑i∈Bnq∑j∈Bmk∣ai,j−ai,jpool∣G_{n,m}=\sum_{i\in B_n^q}\sum_{j\in B_m^k}a_{i,j}^{\mathrm{pool}},\qquad E_{n,m}=\sum_{i\in B_n^q}\sum_{j\in B_m^k}|a_{i,j}-a_{i,j}^{\mathrm{pool}}| .

GAPR targets the comparison Gn,m>En,mG_{n,m}>E_{n,m}. Evaluating this criterion directly requires dense token interactions and normalized attention weights, undermining computational efficiency. For a fixed query, softmax preserves the ordering of scores and attention weights: ai,j>ai,k⇔si,j>si,ka_{i,j}>a_{i,k}\Leftrightarrow s_{i,j}>s_{i,k}. Furthermore, the relationship between attention scores and relative attention weights can be expressed as: log⁡ai,jai,k=si,j−si,k\log\frac{a_{i,j}}{a_{i,k}}=s_{i,j}-s_{i,k}. Accordingly, score perturbations reflect changes in relative attention preferences. Prior works, such as Top-k Attention and QUEST, have also leveraged score estimates to screen attention interactions. These properties and precedents motivate our surrogate design. Therefore, we use scores and their deviations in the score domain as surrogates for this criterion:

∑i∈Bnq∑j∈Bmk∣si,jpool∣>∑i∈Bnq∑j∈Bmk∣Δsi,j∣\sum_{i\in B_n^q}\sum_{j\in B_m^k}|s_{i,j}^{\mathrm{pool}}|>\sum_{i\in B_n^q}\sum_{j\in B_m^k}|\Delta s_{i,j}| .

We acknowledge that the proposed surrogate is empirically motivated. Nevertheless, the statistical evaluations in Appendix J demonstrate high rectification decision agreement with the original criterion across models, denoising steps, and attention layers. Appendix J also provides visualizations of the rectification masks. These results empirically support the surrogate's decision consistency, while the ablation studies in Tab. 4 and Fig. 7 of paper further demonstrate GAPR's benefits for generation quality.

Following your feedback, we have revised Eq. (13) by removing the final equality and clarified the corresponding descriptions. Please refer to Sec. 4.2 of the revised manuscript for details. We sincerely appreciate your valuable comments.

W3: The related-work discussion of caching-based acceleration is incomplete and omits several relevant methods.

Thank you for your valuable suggestion.

Following your suggestion, we have expanded the discussion of cache-based acceleration methods in the Related Work section of the revised manuscript, including MixCache, EchoCache, MagCache, OmniCache, and AB-Cache, to provide a more comprehensive overview of related research.

AuthorsOpenReview ↗

Follow-up to Reviewer gwVB

Dear Reviewer gwVB,

As the discussion period is approaching its end, we would like to politely ask whether you have any remaining questions or concerns. We remain available for further discussion until the very end of the discussion period and would be happy to provide any additional clarifications.

Thank you again for your valuable time, effort, and constructive feedback.

Comments on the paper

2 comments
Area_Chair_YBe5OpenReview ↗

Discussion Reminder

Dear Reviewers,

I hope this message finds you well. The rebuttal is now available in the discussion forum. I would like to kindly remind you to check the rebuttal and leave comments for the authors, which would be very helpful.

Thank you very much for your time and effort. Please let me know if you have any questions.

Best

AuthorsOpenReview ↗

Rebuttal Summary of Submission 4

Dear Area Chair,

Thank you for your time and consideration. We sincerely thank all reviewers for their constructive suggestions and valuable insights. Below is a summary of the discussions and improvements during the rebuttal phase.

First, we appreciate the reviewers' recognition of Rectified SpaAttn's motivation, technical contributions, and practical effectiveness. Reviewer gwVB highlighted that correcting attention distortions caused by sparse Softmax is "practically important." Reviewer zvUY acknowledged the clear motivation and comprehensive evaluations across HunyuanVideo, Wan2.1, and Flux. Reviewer ZV4u recognized the practical acceleration benefits and informative ablations, particularly the effectiveness of IPAR and GAPR in improving sparse attention approximation with minimal computational overhead.

Second, the reviewers raised valuable concerns primarily regarding the mathematical formulation of GAPR (Eq. 13) and the effectiveness of its empirically motivated design. In response, we carefully revised the manuscript, clarified the theoretical scope of GAPR, and conducted additional experiments to address these concerns.

1. Mathematical Formulation and Theoretical Clarification

  • Correction of Eq. (13) (gwVB W2; zvUY W1; ZV4u W2). We corrected the unclear expression in Eq. (13) by removing the incorrect equality and explicitly clarified that GAPR employs a score-domain surrogate rather than a mathematically equivalent probability-domain criterion.
  • Clarification of the theoretical guarantee (zvUY W2; ZV4u Q3 and follow-up). We clarified that the original "necessary-and-sufficient" statement applies only to the positivity of the idealized net-gain metric, not to the practical score-domain criterion.
  • Implementation consistency (ZV4u W2, Q2). We clarified that the absolute values of logit residuals are computed element-wise before summation, avoiding signed-residual cancellation. We also verified that the corrected formulation matches the implementation used in all reported experiments.
  • Improved mathematical presentation (zvUY W1). We carefully checked the remaining equations, clarified the underlying assumptions and notation, and added a notation table in Appendix A to improve readability.

2. Empirical Justification and Effectiveness of GAPR

  • Empirical validation of the gating criterion (gwVB W2; ZV4u W2/Q3). We provided additional statistical evaluations in Appendix J, showing high agreement between the practical surrogate and the ideal probability-domain criterion across different models, denoising timesteps, and attention layers.
  • Additional ablation evidence for GAPR (ZV4u Q3 and follow-up). Following the reviewer's request, we conducted fixed-mask comparisons against unconditional compensation and random compensation. Together with the original ablations in Table 4 and Figures 7 and 11, these results demonstrate the effectiveness of gain-aware selective compensation over unconditional or random compensation.

Beyond the two central concerns, We clarified the distinctions from Re-ttention and SVG-EAR, emphasizing our theoretical characterization. We added comparisons under comparable quality and latency budgets and expanded the discussion in the Related Work section and Appendices F and G.

We appreciate the reviewers' continued engagement during the discussion. Reviewer zvUY acknowledged our careful responses and increased both the presentation and soundness scores. Although the reviewer maintained the original overall rating, no specific unresolved technical concerns were identified in the follow-up discussion. Reviewer ZV4u further requested a fixed-mask comparison, which we subsequently provided with detailed experimental results. We hope that our additional response has adequately addressed the reviewer's remaining concerns.

We believe most of concerns raised during the review have been addressed through expression corrections, implementation verification, and targeted experimental evidence. In particular, the revised formulation no longer relies on the disputed equality in Eq. (13), while the additional experiments directly validate the effectiveness of GAPR's selective compensation. Together with the comprehensive evaluations and ablations in the original submission, these results provide a solid foundation for the paper's technical contributions and empirical conclusions.

Finally, we sincerely thank all reviewers for their valuable feedback, time, and effort, and especially thank the Area Chairs for their thoughtful consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.