Residual Forcing: Structure-Aware Sparse Attention for Autoregressive Video Diffusion

Autoregressive video diffusion enables streaming long-video generation, but self-attention over cached visual history remains a major inference bottleneck. Training-free block-sparse methods can flexibly select key blocks for each query block, yet routing based on inexpensive mean-pooled scores can miss concentrated interactions amplified by softmax. These misses degrade generation quality, while compensating with more key blocks erodes the efficiency gains; independently changing the selected context across chunks can also introduce periodic visual discontinuities. In this paper, we show that many of the missed high-response interactions follow identifiable attention structures that can be protected at low cost. We therefore introduce $\text{Residual Forcing}$, a training-free framework that separates structure preservation from residual attention selection. It protects spatially aligned key blocks, their axial halo tokens, and shared salient keys, then routes only the residual key space using reusable head-level priors and block-adaptive budgets. To improve temporal continuity, it shares candidate-frame choices across chunk boundaries without increasing the frame budget. Fused routing and dynamic sparse-attention kernels efficiently execute the resulting heterogeneous routes. On LongLive, Residual Forcing achieves a $4.12\times$ self-attention speedup in fixed-cache profiling and a $1.25\times$ end-to-end speedup for 30-second generation, while maintaining competitive video quality and mitigating chunk-boundary discontinuities.