Rectified SpaAttn: Revisiting Attention Sparsity for Efficient Video Generation

Diffusion Transformers dominate video generation, but the quadratic complexity of attention computation introduces substantial latency, hindering real-world applications. Attention sparsity reduces computational costs by preserving interactions with critical tokens while discarding those with non-critical tokens. Existing methods primarily focus on token reordering or importance metric design to identify critical tokens, but often suffer performance degradation at high sparsity. In this paper, we revisit attention sparsity and theoretically show that existing sparsity paradigms induce systematic biases in attention allocation: (1) excessive focus on critical tokens amplifies their attention weights; (2) complete neglect of non-critical tokens causes the loss of relevant attention weights. To address these issues, we propose Rectified SpaAttn, which rectifies sparse attention allocation with implicit full attention reference, thereby enhancing the alignment between sparse and full attention maps. Specifically, we implicitly capture the full-attention distribution of critical and non-critical tokens by pooled query-key interactions, which is then used to rectify sparse attention allocation: (1) for critical tokens, we show that their bias is proportional to the sparse attention weights, with the ratio governed by the amplified weights. Accordingly, we propose Isolated-Pooling Attention Reallocation, which calculates accurate rectification factors by reallocating multimodal pooled weights. (2) for non-critical tokens, recovering attention weights from the pooled query-key yields attention gains but also introduces pooling errors. Therefore, we propose Gain-Aware Pooling Rectification, which ensures that the rectified gain consistently surpasses the induced error. Moreover, we customize and integrate the Rectified SpaAttn kernel using Triton, achieving up to 3.33× and 2.08× speedups on HunyuanVideo and Wan 2.1, respectively, while maintaining generation quality even at high sparsity.