Eliminating Context and Timestep Redundancy from Autoregressive Video Diffusion

Few-step autoregressive video diffusion enables streaming and interactive generation, but commonly retains full temporal attention in every layer and a fixed denoising budget for every latent block. Our analysis reveals a consistent layer-wise pattern of temporal-context redundancy: similar subsets of Diffusion Transformer layers assign little attention to historical context across the evaluated training stages and datasets, as well as samples and timesteps within distribution matching distillation (DMD). In contrast, latent frames within a video vary in denoising difficulty and require different computational budgets. To address these two forms of redundancy, we introduce Static Pruning and Adaptive Denoising (SPAD), which reduces computation within each forward pass by removing historical cross-block attention from weak-temporal layers and adapts the number of denoising passes to content difficulty. To avoid online attention ranking, we identify weak-temporal layers from offline profiles and remove redundant historical-context computation using a fixed pruning pattern throughout training and inference. To allocate denoising steps on demand, a lightweight confidence head estimates proximity to the clean endpoint from the current trajectory state. We further introduce probabilistic timestep subsampling during DMD, exposing the generator to variable-span transitions and the states they induce to better accommodate inference-time skipping. Evaluations demonstrate that SPAD accelerates both training and inference while alleviating long-horizon degradation, supporting efficient autoregressive video generation without sacrificing visual quality.